Episode 53: What Should I Measure? with Dr. Haywan Chiu

Alexandra

Hello and welcome to Heart, Soul and Data, where we explore the human side of analytics to help amplify the impact of those out to change the world. With me, Alexandra Mannerings. Today is a very exciting episode. We are trying something new for the first time and I have a very willing guinea pig to help me out with this.

Dr. Haywan Chu is going to join me and we are going to do a live coaching session together. He has a question about what he should measure in his business. So, Dr. Chu, I’m going to hand it over to you, and I’d love to hear a little bit more about your business and then where your question comes from.

Dr. Haywan Chiu

Hey, Alexandra. Thanks for having me on here. So my name’s Hagan. I’m a podiatrist with a special teeth in diabetic foot and ankle infections, and I’m salvage. I’ve been in private practice for three and overall in practice for six in Albuquerque, New Mexico. So just the pandemic, students and residents, we’re not allowed to scrub cases with us. And we are very involved in surgical education for these trainees.

So when they were not allowed to scrub, in some cases, we felt that there was going to be a detriment to the next generation of physicians in general. So we were trying to figure out what can we do in our part to help with this, our local group of trainees. And we saw it well, I’m not super busy.

My private practice is still building. I have time. So I bought a camera, figured out how to do some filming. I started filming my surgery and my partner surgeries. We figured out how to do some video editing and voiceovers and recording. So after a ton of trial and error, we were able to come up with a whole comprehensive surgery video library of our own surgeries with voiceovers where we give insight into each procedural step that we do.

So we were able to pause a certain time and dynamic maneuvers and kind of go into detail about how we do it, when in actuality it takes like one or 2 seconds to do like there’s a 5 minutes of thought behind it that we were are now able to go into. And this is a kind of education that has not been done before for our field where, you know, there are cadaver videos and single person narration where kind of monotonous and this is a step by step, but we know that no one’s ever really kind of went further and deeper on actual patients with a real pathology.

So we named this video library, the foot and Ankle Surgery Academy, and put a dot com to it, put it up on to a website and started charging a subscription fee for it. We’re now at over 400 members multiple residency programs have subscribed with their CME funds and we’re slowly growing our subscription membership and our video library. Now with over 500 over 100 hours of content, over 50 videos, 50 surgeries.

And so this brings me to kind of like was like, what’s our next step? You know, our mission is to offer and doctors and podiatrists in general, specifically when it comes to surgical technique. And that’s only one aspect of being a doctor where there’s it’s a multifactorial thing to be good at surgery, good at being a good doctor in general.

But if we can impact one of those small factors with our video or whatever our mission is, I think that’s awesome thing.

Alexandra

And I know that one of the things that brought you in here was this idea of saying, Well, how do we know if what we’re doing is effective and how do we know whether what do we need to be measuring? What do we need to be monitoring to understand what we might need to improve or what we might need to change?

Haywan

Yeah, and we’ve had a lot of amazing feedback. A lot of it is, Oh, I wish I had this years ago or I use this before every case. Some people will listen to their watch our video the night before, and when they’re driving to their case, they’ll listen to it again and kind of like mentally imagine the steps.

So it’s had some impact. What that impact is, I don’t know. And the next step is when it comes to surgical education, like, people need to see the surgery and people need to practice it. And we’re you know, our goal is to just help people with their surgical proficiency in general. And videos are a part of that component.

My question that I had for my kind of company moving forward, and just to give context, we’re not nearly generating enough profit to be able to outsource this kind of work to professionals. And if we look at our total collections, it’s only half of what we would have been able to afford to hire somebody. So we would still be in the hole if we had done this professionally.

So all the editing and recording stuff is done by mostly me and my time is limited. This isn’t something we’re thinking about doing as, you know, our full time job. We’re not looking at quitting our day job to do. This is just kind of like a hobby that turned into more of a business now that we’re actually generate some good income from it.

But again, not enough to be able to just hand off all the work to a professional to start filming and recording a video and if we had to do it, like because I know like what we’re supposed to look for, I can also put in photos of a journal article and that’s relevant to the topic. And then people don’t need to like pause and look it up and stuff like that, which is something that’s never been done in other surgery videos.

And I think that professional touch from someone who really understands what we’re supposed to know for the procedure is something that is helpful, is hard to teach, and I know it pigeonholes me into a situation where I may not be able to completely let go of this task, but obviously I’m going to try and delegate some of this and train somebody to do part of this work.

But I will have a hand in something. So that’s kind of a bottleneck in our production, is finding the surgeries, being able to coordinate our schedule so that I can film it. We can try elsewhere some of this filming, but it’s expensive or requires a lot of training. I mean, there’s a reason this has never been done before in our lives, but you know, it’s a blessing and a curse in a way.

So basically that that was just to get context on the limitation of how our production can be moving forward. So if I want hands on creation of Dry labs, which is more of a hands on experience thing, which is another component, I feel there’s a portion to surgical prep that will take time away from what were our strength is in the first place.

We just creation of these videos. So I think it’s important to make a well informed decision on where to spend my time so that I don’t lose on opportunity cost of doing something that was doomed to fail to begin with. So I am hoping that I can measure something that can tell me if I should continue doing videos or if I should spend some time to branch out into these hands on lab kind of thing.

Alexandra

I love it. There’s so much great information here and you’ve done so much of the important groundwork. I’d like to just review some of the ground work pieces, one to make sure I understand them, but also because I think it’s a great illustration for a lot of people who might be listening of how you think about some of these decisions that we come up against in the programs and the activities that we do.

For example, the fact that you recognize that your mission is is to improve surgical outcomes, right? By improving physician performance that you want the best possible physician performance in the surgery or surgery performance so that people have the best possible outcomes from those surgeries. But you also recognize that the surgical proficiency is just one piece of a good surgical outcome.

And I think a lot of organizations that have a similar mission of yours of creating, you know, better outcomes in the world are in a similar position where their activities are not solely responsible for the outcomes, and in many cases, you know, we’re part of very complex systems. So the fact that you’re aware of that and we’ll use that later on, you know, in how we talk about what we could measure and how we understand our own impact, look, I think that’s a really great thing they already are aware of.

And then I also appreciated that you brought up the fact that it’s you have a mission, but then you also have certain level of financial obligations, right? Whether it’s a particular goal, right. That you want to make this a self sufficient as possible or also that you have to decide where to spend your resources. And so you have to make a decision in terms of both your time and your money and where those resources go that will most impact your mission.

And I think, again, this duality where we have obligations to our mission, but we have also obligations to the survival of our organization and the effectiveness of our organization within its own operations. And sometimes it’s difficult to keep those two things in mind because they’re oftentimes very different in our what we measure. And they may be thought about by different people.

I think for you, obviously, you have to keep them both in mind because you’re responsible for almost all of this. But for many of us, you might have fundraisers or financial people who are managing the finance side and then you might have programs or production or operations dealing with the other side. But really they feed each other, right?

Because if your organization doesn’t survive, there’s no mission that you will be able to accomplish. So I think you’ve laid out that very complex context that we have to really think about here when we’re trying to figure out, well, what information do I need to be able to make the most impactful decisions. And so then when I start to think about what information and what metrics we might want to put in place to be able to answer your questions of where do you spend your time, where do you spend your money, and how do you know the impact of what you’re doing?

Because that might that’s going to influence where you spend your time and where you spend your money and your resources. And so we can start with that first piece of how do we know if we’re supporting the piece that we can impact with surgical proficiency? And I think you brought up that you’re getting feedback from students, which is a really great qualitative source of data and I don’t think should be dismissed.

So you certainly want to keep track of what they’re saying, What your students are saying is helpful, right, Whether they’re residents or whether they’re practicing physicians right now, that the fact that they’re giving you that feedback is a really great qualitative source to understand the impact you’re having directly on the people who are watching your videos. But then we might need to take it one step further and say, all right, well, how do we know that that doesn’t just make them feel better, but it actually results in better surgical proficiency?

And this is where things get more complicated, because I think that one thing you could consider would be to do some sort of small equivalency study. And what I mean by this would be that you actually enroll a certain proportion of your students who watch videos and then find a certain proportion of people who are not part of your program.

And you could see whether there are measurable outcomes, right? So for example, you could pick one adverse event. So something bad that happens in surgery that can be highly attributable to surgeon proficiency. And this is really important. If you picked something like that, you need to pick something that is that surgeons impact very highly. And so you would have to use your medical skill to identify what that could be.

But if you identified a particular outcome that you want to avoid, that can be avoided by good surgical training, you could then count up, Well, how often do these things happen? Like you’d probably have to do self-reporting, right? Where physicians could self-report that I understand that would be very difficult to do. Right. There’s a lot involved in being able to do that study.

So but I’m curious what comes to mind when I even just bring that up for you.

Haywan

Yeah, a few things. It depends on like obviously a student who’s still who doesn’t have their podiatry degree that would not be involved in anything other than retraction. But there are times where even a student who is doing retraction, I can tell where they, you know, that they’re well prepared. They know what’s the next step is things like that.

And those are that being proactive and not reactive. And that’s something that is rare in a student. But if they have it, then I kind of know that, oh, they’re going to they’re going to be fine when they’re graduate. When it comes to a resident who is in training, they’re under direct supervision by the doctor and they’re involvement is also limited to maybe some of the more basic stuff, like skin incision dissection to the irrelevant anatomy when it gets kind of hairy, where, you know, if they mess up, then it can be really bad.

It would be like making bone cuts or, you know, drilling holes into the bone. That kind of stuff is often given to the junior resident to do until they have mastered the previous steps. So there, you know, we do take a lot of care to avoid having adverse outcomes. And so I feel that if we were to measure adverse outcomes related to a resident’s being involved in surgery, we’re not going to find it.

And it’s even more indicative of how attentive the attending is and how careful they are in allowing what the resident should or should or should not be able to do. And then when it comes to the attending practicing, I just see a whole host of ethical barriers related to them. Self-tracking in this and reporting it to this or I’m not even their medical board is just like I’m just guy making videos and I got the exam.

Alexandra

Results.

Haywan

To me yet. So yeah my brain is buzzing towards more. Yeah I’m not sure.

Alexandra

Yeah no and I did this kind of on purpose because I think that in a perfect world when we talk about being able to prove effectiveness, the things we want to do, what I described and the reason I wanted to ask you that is because very rarely is that going to be possible. Right? There’s going to be all sorts of issues.

One, as you brought up, different groups are going to respond different ways, right? So you have different groups of students, right? They might be residents, they might be attendees, they might still be in medical school. And so you bring up a great point in that that when we talk about doing a controlled experiment is what I described that you really need to be able to manage as much of the variation as possible.

And that’s oftentimes very difficult in real life. So that’s the first problem is, well, how do we manage all these different people with different that are interacting with your program or your intervention in different ways and they come from different places, right? That’s a student versus an attendee. But then secondly, like you said, there’s ethical issues with collecting some of this data.

There would be potentially concerns that if you like, you said, you’re not going to an attendee is not going to allow a resident to make a mistake, they’re going to intervene and they should intervene. And so again, there’s all sorts of complicated issues when we talk about actually running experiments in real life because, oh wait, we’re talking about real people.

And so to some extent I kind of wanted to have that discussion of saying, look, this would be the perfect way of doing it. Now we see all those barriers to the perfect scientific, and I’m putting that in air quotes, the perfect scientific way of doing it. So what could we do that would be a step down from that potentially in terms of the quality, like what’s the term for it?

But the idea that the purest source of information would be to do a controlled experiment, but there’s a lot of barriers to that. So what would be something where we might give just a little bit up in the quality of that information, but we’re going to then not have to deal with all of the things that are going to make that impossible.

And so the next thing that I thought about is what we call a proxy variable, right? What would be something that would estimate a student’s future surgical success but is much easier to measure and doesn’t have the same ethical problems or the same limitations as that controlled experiment. And so you mention one thing that I thought was interesting, which is the proactive ness of a student.               

The idea that a proactive student is going to be much more effective as an independent practitioner than a student who is really reliant on somebody else. And so I was curious, you know, how when you say, Oh, when I see a student, it’s proactive, how do you know if a student is proactive or not?

Haywan

Actually, I just thought of something that came to mind. Oh, perfect. When I was a senior resident, there was a student who was in one of my cases where, as I was dissecting, usually what students do is they’ll hold the retraction in their hand. They know they’re supposed to retract at some point. They just they’re afraid to put it in there.

So this particular student, she had the retractor in her hand, and I put the retraction where I needed it. And as I’m working, as I’m working, we’re not usually moving in one place when we’re done with our dissection work in one part of the surgical field, we start moving it down the incision. Usually, as I’m moving down the incision, the students are retracting the same spot, but they’re just kind of staring at me, doing my work, struggling as I’m like, okay, so I usually have to grab the retractor, move it down the field to where I need to see, what I need to see.

But this one particular student was automatically following me as I was doing my work and I was like, Wait a minute, this never happens. And she ended up matching into one of the top residencies in the country for podiatry. And now she’s an instructor at that program and has a okay, well, that’s not a surprise that I knew she was going to do fine.

So may be like I can have a list of somewhat objective maneuvers that students would do if they had a higher level of understanding, such as following along automatically with the retraction, adjusting the light when appropriate. Because sometimes the aura is right above the student, but somehow the foot move, the light drifts and we’re operating in the dark.

I can’t see shit. Sorry. It’s like frustrating. And I come. I just. I just, like, I can’t see anything. You can’t see anything? Like, just move it. Yeah, we’re allowed to touch it. Obviously, there are situations where the student is touching the light every, like one, every minute. It’s okay. You’re touching it too much, but maybe appropriate adjustment of lighting and that can kind of like come up with a list of proficient assistant maneuvers, I guess, or some, you know, something along those lines that that we can have the supervisor do a little survey and write down.

Yeah, that’s just one idea.

Alexandra

Yeah, I think that’s a great place to start because the idea would be that and I like what you said, that they’re semi objective, right? You’re still going to have to have an observer say like are they actually tracking with the surgeon. Right. And they’re adjusting and moving along with the surgeon without being asked to. There’s some subjectivity to that.

But you’ve created an action that is easy to define and shouldn’t be too difficult to identify whether it happened or not. And what you’ve done is you’ve done a really good job of saying well, and that action represents a lot of complex stuff behind the scenes, right? It reflects the student understanding what will come next in the surgery.

It reflects the student being confident enough in his or her own abilities to be able to then take action rather than wait and be told it represents a certain amount of assertiveness and willing to control the situation, which are all characteristics that in your expert opinion, go along with being a proficient surgeon. And now, of course, there’s always then flip.

Now the flip side of like, well, you know, are there issues that you may be picking up that may confound this? Like, well, students who’ve been in the program longer are more likely to do that than students at the very beginning or students that feel more comfortable with you or the attending surgery or surgeon or whoever it is might be also more willing to do that.

Or like there may be other dynamics that could get picked up. But that’s always the case when we’re trying to measure complex things. So we want to just be aware of where we might be measuring something other than what we intend whenever we use a proxy variable. But I think that’s a great start. And then the idea could be that you could do that, like whether, you know, you mentioned you’re really active with your own students and in training some of these students or you could enroll some of your fellow physicians to also be part of this and then you could just offer you half of them access to the videos and half not, and

Then track over several months whether you see a difference in their ability to do this. Or better yet, you could do a same subject study where you have your student watch a video for a particular procedure and see how they perform in that procedure and then not watch a video for another procedure or a procedure that you don’t have any videos for.

And you could see whether there’s a within variation of the same person, you know, in terms of their ability to be proactive based on the knowledge and experience gained from the video. Because that’s really what we’re trying to see, right, is do your videos help promote that confidence and understanding? And then the idea would be that once you’ve done that, if you’re able to show that your videos have that impact, you don’t have to keep doing that study.

Right? The idea is that then you’ve demonstrated the effectiveness of the videos. And so by tracking how many people watch videos and how many procedures, you could make a generalization about the overall impact of your program. You know, so if you know that students that watch your videos are 50% more proficient and confident than those who don’t, based on these proficiency behaviors, then you could say that we can generalize that to all of our students and make an estimate.

That’s the overall impact of our program. Has all that that.

Haywan

I’m say, may I run a randomized trial? Now?

Alexandra

It’s not a fully randomized trial because we’re cutting a lot of corners here.

Haywan

Of just, oh, got all the variables I’m going to have to.

Alexandra

Control. Right.

Haywan

thinking like each group has its own kind of set of nuances where if I were to do a survey, it has to be like very thoughtful and it’s like, what would be the best group of students or residents? And, you know, these videos I feel, are most beneficial for residents because they’re the ones who actually are doing parts of the procedure and seeing the relevant enemy and surgical maneuvers are most helpful for someone in that at that level of their career.

Alexandra

Yeah. And so then so like we’ve sort of talk we’re moving our way down the levels of like scientific complexity, right of it. So we talked about like fully watching actual outcomes. Then we said, well, what if we could look at measuring proficiency and using some actions that could imply proficiency? The next step down could simply be like a pre post survey that you ask of the people who are participating with your videos where before they watch it, you’re asking a few key questions about knowledge for a particular procedure.

And then after they watch the video, you ask them some post questions that reflect Did they learn from participating in your video, really critical elements that you, from your experience, would say these are the important things they should have gotten out of that video, Right. So it doesn’t require any intervention within the surgery itself. But just we’re measuring advancement of knowledge.

And then you could also have, especially for your residents, you could ask a certain subset of them to do some pre post surgery surveys. So just say, hey, we’re interested in some of you who might have been doing this surgery and you could pick just one or you know that you’re coming up in your residency. I don’t know if residencies if the surgery sort of what you learn goes in a particular order.

So if you sort of knew where they were in their surgery, you’re in their residency, you could, you know, think that they haven’t done a particular procedure. But you could also ask them kind of before and after of before you’ve done this procedure, how effective do you think you’ll be? And then after you’ve done it, how effective do you think you were and see whether or not you can ask a few questions that might have them reflect on where the video helped them or not in a specific area?

And then again, that’s even simpler than watching what’s happening in surgeries. It’s weaker evidence because it’s completely self-reported, but it still might give a pretty good sense of are they finding value in this? And you could test knowledge gained right from having watched a video about a particular surgery.

Haywan

Yeah, I’m just thinking like this is kind of like a qualitative study.

Alexandra

Where it would be very it would be very qualitative because it’s completely self-reported, But you can still structure it in a way that is giving you repeated information. So I think a lot of people think about qualitative that it’s not necessarily something that can be, I don’t know, use with any statistical authority, but it can be. I mean, if you do a well-structured pre-post test that is geared towards measuring knowledge gained in some really important areas like you said, for you where you’re like from this video, they need to learn this thing.

That’s what’s going to really make them better at this procedure. You could come up with a couple of questions to ask before and after the video and you would see whether they learned that with the idea that if they learned it, they’re going to use it when they go into that surgery. Does that make sense?

Haywan

It does. I’m just trying to think so. I guess one example would be, let’s say a correction of a bunion like that. Yeah, like the dissection is important. So some bunions we correct by cutting the bone, moving the bunion bump over fixing it with a screw or two. And I think the most taken for granted and somewhat hard part is the moving the bunion bone over part where most people and initially when I was doing my first few bones, I was struggling with this is being able to move the bump over enough and usually that involves dissection enough dissection to free up all the soft tissue attachments or stretching out the soft tissue on the other side that’s tight so that when I’m finally ready to move it over, it’s a real simple maneuver. And I’ve taken from it, like we all talk about that actual part during surgery. Oftentimes people just to do it and move it and then they think about it or or think to talk about it or exactly to get that.

That’s something that I struggled with initially. And I figured out like, okay, this is what I need doing in the future. And just trying to think like usually that’s a part that the attending does. The resident Sometimes they’re doing like, how would I design some pre post measurement of that?

Haywan

Well, in some ways it prefers because we’ve sort of moved down to like, okay, we’re going to just survey or do a quiz of students. And so what you could do is prior to them ever watching this video, you could have a couple of questions of like which factor would be most likely to limit the success of this surgery.

Alexandra

And you could give them four options, three of which are really would have little to do with the outcome. And the fourth that could have to do with actually like what you were describing with your ability to actually correctly relocate the bunion. And then after they watch the video, you asking the same question again and you see if their answer change, because the idea would be that if what they know has changed after watching your video, you’re impacting the knowledge they have.

Alexandra

Right.

Haywan

Okay.

Alexanrda

And so so I’m saying is we’re taking we’re stepping down in terms of the intensity of what we’re measuring. And so you’re getting farther and farther away from the actual outcome you’re trying to measure, but you’re trying to find a way of estimating that in a simpler and simpler way. Does that make sense?

Haywan

Yeah. In my brain is taking me towards a question of like the video. I want the video to help people be able to do more understanding more as they’re actually doing the surgery. Mm hmm. So would it make sense to ask these questions after watching the video or after actually doing the procedure and watching the video?

Alexandra

You could do both, right? So you could say what was the impact just on the video by itself? And what’s the impact of the video combined with the surgery? But you get the point, which is obviously the thing we actually want to effect is surgery, not the just I can think the thing, but whether what is it that to to know and not to do is not to know.

Right. You need to have the doing part to show that you actually really got that knowledge. And so again, one step even farther down, which would be completely qualitative and not structured, would be to ask people routinely after they perform surgeries and you could maybe have it where, you know, members in your community receive this survey every so often to say what was one thing that you can actually specifically pinpoint you did differently in that surgery because of this video?

Right? Like, can you tie one procedure that you did one way you moved your hand, one intervention that you did? Can you pinpoint something that you did differently because you watched the video that you wouldn’t have done or that you didn’t do in the last time you did the surgery or that you wouldn’t have done when you thought about it like I did that because you told me that.

And then you can start to collect information and you could give them the option of I did nothing different, right? So you could start with just a simple what proportion of people who respond to this tell me that they did something different versus not because of it? And then if you can start to identify, oh, there’s particular parts where, look, they say this made them do something different that lets you know that actually something went different.

You have a problem then with non-response. Well, you’re going to get non-response bias where maybe only people who actually felt impacted by your videos respond and those who didn’t don’t respond. So you’ll have to deal with that as well. But it is a way of again, saying like what actually is the impact here, that maybe that’s.

Haywan

Something that makes sense. I’m just trying to think like if I was actually going to execute it leg, would I have to incentivize them? Like, a lot of people don’t respond to my marketing emails, but yeah, you know, that’s normal. I’m just like.

Alexandra

Service take people’s time. So absolutely, you can always offer something in exchange, like you could offer a free month in exchange for agreeing to complete four surveys after four surgeries or something like that. And I think that that’s a completely acceptable practice in any kind of survey development, which is compensate people for their time and their willingness to give you information.

Haywan

Yeah, it makes sense.

Alexandra

Yeah. There is then the flip side, which is and we haven’t really talked as much about this, and I know we’re running out of time, which is it’s not just about measuring the impact and effectiveness of your videos. There’s also the piece that you want to make sure that you reach a certain financial goal, right? And so you’re trying to decide like, do we move into dry labs?

Do we continue with the videos? And that’s going to be a similar setup. But with completely different goals, which is you want to see what drives student engagement and willingness to pay for the service, right? So you may find that certain techniques or certain approaches or certain surgeries or certain, you know, different elements of what you do in your business are more likely to have students stay in your community or students buy into your community for the first time.

And so you may want to do something sort of similar where you look at, All right, if this is the financial goal that I want to get, where do I see like the highest levels of student engagement? Which videos do I see the students watching the most? And I’m going to do more in that area because I’m assuming your platform will allow you to track where students are watching videos or which videos have a high dropout rate, like where people not finishing a video.

Right. And are there things that I can say? All right. There’s something wrong with this particular video or something that we could improve in this particular video to drive engagement, because the more engagement you have, the more likely people are to want to keep participating in your program. It’s easier to experiment with your videos than it is on people getting surgery.

So you could try a few things in terms of looking at your videos and your engagement and in how long people want to stay in your membership and you can actually do some of your own little experiments to see which of these drive engagement the most. And you can do the same thing with your marketing. You said people don’t really respond to your marketing emails.

That would be a place to start. Some also little small experiments where you have this outcome, which is did they join my membership? That’s the outcome you want. They’re right. And so you can look at, well, if I send email a voice, email B do I get a difference in response? And you can. That’s very easy to do.

It’s very difficult in the medical field, very easy to do with emails. And so you can do some much more simple experiments in your marketing and in your engagement at ways you can’t do with your outcomes that you really want to have in the medical space, if that makes sense.

Haywan

Yeah, that makes sense. And I’ll have to sit down and really think about this.

Alexandra

Well, and like I said, we sort of ran out of time because I got really interested in your, in your medical measurements. So maybe we could do around to where we talk about how to actually look at what you measure for fighting and outreach and engagement.

Haywan

Yeah, for sure. It would be fun, but I don’t know.

Alexandra

So we talked about a lot of things and a lot of limitations and challenges on things. Do you have something that you feel you could consider putting in place for looking at the outcome of your program? Or where might questions still remain on that?

Haywan

Yeah, it’s just like training to be proficient in a surgery. It depends on your surgical volume, your training, mentorship. You know, there are better mentors and others who are more engaging and will let you do more than others. And then there’s the ability for the resident to do independent self study. And of those three things, I can only impact one of those to provide a better resource for a self study.

And one one thing I thought about measuring was maybe how long it takes for them to study, to feel prepared. And if our videos can reduce the time it takes, obviously some don’t even study some studies more than others. There’s so many variables that even measure that things. But everything I’m thinking of is the feel. It feels like, Yeah.

Alexandra

No. Well, and again, we talked about this at the very beginning, which is you can’t control all of it. And most of us with any intervention we’re doing, whether you’re talking about social services, whether you’re talking about health care, whether you’re talking about improvement in the arts right there, you really only control one little sliver of a very complex system.

And so that’s why, again, as we sort of went through the levels of complexity of what we measure and we got simpler, you can home in more and more on just looking at do I drive change in the piece that I control, right? And that’s where I said when I got to the point of, well, could you measure a piece of knowledge that you want one video to impart and just look at whether your video actually imparts that piece of knowledge?

I like your idea of saying, All right, well, do we does it reduce study time? Because that’s something that you could use to drive engagement. That’s something that is advantageous to the students themselves, right? To say, yes, we reduce study time. It’s okay if it doesn’t work for everybody, right? He said, Well, what if they don’t even study at all?

Well, fine, they don’t. And you’re not going to impact those students. But if you can show that even of 30% of your students, you reduce study time by 50% or whatever it is, that’s a meaningful outcome for students who might want to join. And so, again, we a lot of what we talked about was looking at surgical outcomes like outcomes for the patient, but the patient’s not paying for this service, right?

Your students are paying for this service. And so if you could also document, direct benefit to your patients or to your participants, not just to the patients, that could also be probably an easier to measure outcome, but also one that would support your other goals in terms of making it self sufficient to show that there were direct outcomes to your students, right?

That you clearly are imparting more knowledge and the study time one I think could be helpful because if you are more prepared, you obviously you know more right like you already learned that piece and so you are also capturing just gain of knowledge, which is the piece that you can control. Right? You’re controlling that gain of knowledge piece.

Haywan

I can consider figuring out a way to measure time saved to feel prepared for a case for a students and residents. And I can come up with a couple key procedural steps that are important for two of the most common procedures we do and measure A So whenever we go into a surgery, we call it scrubbing in. So it may be like a pre survey will be two groups of those are studied normally versus those that study normal you plus or video post and see much more insightful.

One group is compared to the other. Yeah, yeah. I think that makes sense. I think that’s doable.

Alexandra

Yeah, exactly. And that’s what we said is we had to kind of find what’s going to give you enough information, but be doable. And that is the balance of collecting data. And I think the piece here is that you can do smaller groups, more intensive, right? So being to actually look at students coming into the surgeries, asking them some questions before and after the surgeries will give you deeper, richer, more information than just like an embedded survey in your learning platform.

But you can reach a lot more people with an embedded survey right before and after a video that they watch. And so you could do both. You could do everyone who watches. Like you said, the two most common procedures, they get asked questions before and after those two videos. Then for 15 of your students, you’re going to do a much more in-depth review of their knowledge and see what differences you find in the more in-depth one.

And it gives you sort of a couple of points to triangulate on potential impact.

Haywan

Yes. Now.

Alexandra

Good. Well, that’s better than just be like I still feel like I have no idea whatsoever. At least we’ve got lots of ideas for you.

Haywan

Yeah, I’m like, Oh, should I? I mean, should I send them a prepaid envelope? And they can still and draw in what they are supposed to do, get and ship it to me.

Alexandra

And that’s true. You opened up a whole other question of how do you gather some of that information from. Yeah, a whole other set as well. Yeah.

Haywan

Yeah.

Alexandra

Awesome. Well, I really enjoyed this. I hope it was at least a little helpful and useful for you.

Haywan

Yeah. Yeah, it was very helpful. Thank you.

Alexandra

Excellent. And I would love, like I said, we can have a follow up and talk either about the financial obligations or also just, you know, if in six months we could check back in and see if you’ve tried any of these and what you learned or what challenges you ran into or where you may have had to pivot because of real life.

Haywan

Yeah, that sounds good. We’ll see what happens.

Alexandra

Well, thank you so much for your time today. I really, really appreciate it.

Haywan

Yeah. Thank you. Thanks. Your time, too, is helpful.

Alexandra

You have been listening to Heart, Soul and Data. This podcast is brought to you by Merakinos. Is an analytics education, consulting and data services company devoted to helping nonprofits and social enterprises amplify their impact and drive through data. You can learn more at Merakinos.com