The promise of artificial intelligence (AI) and large language models (LLMs) hold a massive amount of potential within healthcare workflows. Some of the most enticing benefits include ambient listening, note taking, and clinical decision support. While these are exciting goals to strive for, success relies on implementation, buy-in from clinical and administrative staff, and continual performance improvement throughout the process. Only then can the true power of AI be realized within a health system.
On today’s episode of In Network's Designing for Health podcast, Nordic Chief Medical Officer Craig Joseph, MD, talks with Scott MacDonald, MD, Chief Medical Information Officer at the University of California Davis Health. They discuss his background and how he became a CMIO, how he’s changing professional development for physicians, and the integration of large language models into workflows. They also talk about how to leverage AI’s ambient listening capabilities, reducing note bloat, and why tech adoption can sometimes be uneven across medical specialties.
Listen here:
In Network's The Consulting Show podcast feature is available on all major podcasting platforms, including Apple Podcasts, Amazon Music, iHeart, Pandora, Spotify, and more. Search for 'In Network' and subscribe for updates on future episodes. Like what you hear? Leave a 5-star rating and write a review to help others find the podcast.
READ THE TRANSCRIPT
Show Notes:
[00:00] Intros
[01:29] Dr. MacDonald’s background
[03:57] The PEP squad
[12:42] The results of the PEP squad
[16:51] The AIScribe
[27:24] Leveraging AI to improve doctor’s notes
[34:14] Epic’s new message prompt tool
[39:13] Dr. MacDonald’s favorite well-designed thing
[42:45] Outros
Transcript:
Dr. Craig Joseph: Dr. Scott McDonald. Welcome to the pod. How are you, sir?
Dr. Scott MacDonald: I'm doing just fine. Thanks for having me. Looking forward to this discussion.
Dr. Craig Joseph: Yeah, it's going to be fun. Where do we find you today? Where are you?
Dr. Scott MacDonald: I'm at the University of California Davis health system in Sacramento, California.
Dr. Craig Joseph: Yeah, so is Davis in Sacramento? Those are two cities that don't overlap. Is that correct?
Dr. Scott MacDonald: Yeah, they're about 12 miles apart across a big flood project. Davis is a little small college town. But Sacramento is where most of the population is. So, there's a lot more medical pathology here. So that's why we are able to do our medical school.
Dr. Craig Joseph: It's kind of false advertising, though. I think everyone would agree.
Dr. Scott MacDonald: I would agree.
Dr. Craig Joseph: All right.
Dr. Scott MacDonald: We sometimes will call ourselves the East Campus and they're the West campus.
Dr. Craig Joseph: So, we've established that UC Davis is not always in Davis, I think that people are interested in that. They might also be interested in hearing about what you do at UC Davis. So, you're the chief medical information officer. How does one become the chief medical information officer at UC Davis, located in Sacramento, California?
Dr. Scott MacDonald: Well, first of all, you could become a doctor. It doesn't matter what kind of specialty you're in. There's lots of people that are, CMIOs who were emergency physicians, internal medicine physicians like myself, you name it. So, it helps to become board certified in clinical informatics, which is another topic we could talk about if you haven't already, on this podcast. So, it's like having a second medical specialty that you do in concert with your primary one. And it certainly helps to have some, interest or an enthusiasm for using health information technology to improve health and healthcare and you don't have to be a programmer for that. But you have to have an interest in using it tools to make things better. So, you've got those things, and it helps to have some leadership skills, some change management skills. Those play a big role in my job and if you got those, then you can get a job, maybe as a medical director in informatics first or work as a physician builder or just an informaticist who works on projects. Then you can rise up through the ranks like I did and have some more leadership responsibility for informaticists across an organization.
Dr. Craig Joseph: One of the things you didn't mention that I think you must be good at is getting yelled at by your colleagues. Is that not a key responsibility?
Dr. Scott MacDonald: Well, I take a different approach. I'm able to sell things upfront and phrase them in a way so I don't get yelled at very often.
Dr. Craig Joseph: Wow.
Dr. Scott MacDonald: That's part of change management, right? Building a reputation that you can actually solve problems for people. So, people come to you and ask for help rather than yell at you for what somebody did.
Dr. Craig Joseph: Well, clearly, we've established` multiple things. One of the new ones is that you are more effective at change management than I have been in my past, 15 years. But let's move on to one of the ways that you help your colleagues, and this is what actually drew my attention. You and I have known each other for a long time. But I read an article that you were an author on, about the PEP squad. And, when I first saw that, I thought, wow. That's cheerleading and then something along those lines, but, you don't typically see cheerleading, articles in the scholarly journals. So, what is the PEP squad at UC Davis?
Dr. Scott MacDonald: Yeah. No pom poms or flips on our team. It's what a lot of places may call sprint approaches, but this is a team. PEP stands for practice experience program. So really, we're all about improving the practice experience for physicians at Davis and it has a couple different facets. One is about training. We know from a lot of data in the literature, and in my practical experience, that good training makes a huge difference in what people think about their EHR and that also has a big impact on their burnout and their professional happiness. If you look at some of the KLAS data that supports that. So, we recognized ten years ago or so, well, now 7 or 8 years ago that we really needed to make an effort to really improve the training we were giving. Mostly our physicians is our main area focus, but we also help out other providers. But physicians are our main cohort that we train. We also recognize that everybody is a little different. People, they did pay attention to this part of class, but they missed that other part of class. So, it's not a one size fits all training program. We have really made an effort to tailor our training to everybody's individual needs and we do that by having really a one-on-one or at the elbow model for training. We do two main things that drive the individualized curriculum for each physician, and that's a survey we give just to kind of get a feel for what they're bothered by, and it may not be something that they're particularly bad at based on data, but it's something that really sticks in their craw and makes their professional life quality poor. So, let's try to address a few of the issues that they request us to help with. We also look a lot of data, and we use the Epic signal data extensively to identify in what areas are they an outlier compared to their peers? So, we'll look and see if they're spending hours after work, the pajama time, if you know that term. If they're writing their notes or if they're doing their in-basket work or whatever it might be. So, we will prepare some lessons before our trainers meet with them to really address those specific areas that we know from the data that they're struggling with. So that's our main approach. We have historically done about a four-hour training session, once a year for each physician and we don't block down their schedule for the entire half day, but we have them have a few patients there so that the trainers can observe them interacting with patients. That's another opportunity where they can notice some inefficiencies in how they're both using the EHR as well as interacting with the patient with that third party in the room of the EHR.
Dr. Craig Joseph: All right so there's a lot to unpack there. Let me start with just a basic clarification. So, we're talking about ongoing training, we're not talking about a new physician to your practice. So, that's correct, right?
Dr. Scott MacDonald: We do both actually. So, the bulk of our work really is our larger population of existing physicians, but we also have a different training group here that does the bulk of the onboarding training for new physicians, both attendings and residents. So, they do the initial day or two training. But then we have a plan where we follow up at 30, 60 and 90 days where we do some additional pre-planned time with them to make sure that they're getting up to speed as they start with us. Then we have that annual training with them thereafter.
Dr. Craig Joseph: Okay and for about how long have you been doing that annual four hours of training with every physician?
Dr. Scott MacDonald: Yeah. I think we started it 6 or 7 years ago, it's in that realm.
Dr. Craig Joseph: It must be a very small percentage of healthcare systems that dedicate that much time in an ongoing way. The majority of places I've worked with, there's zero hours of ongoing physician time. Zero. So, four hours for five, six, seven years. Is this mandatory?
Dr. Scott MacDonald: No, it's highly encouraged, and a few physicians don't take advantage of it. That frees up some of the time. We have a team of about 12, including trainers and builders and project managers that do the work. The harder part really is the resident training, because most of them are doing predominantly inpatient work through most of their training, and their amputee experience is more limited. So, trying to get on, a schedule to protect time to do that for residencies is difficult. So, for residents, we tend to do more group-based trainings. Meeting, doing a presentation at the didactic sessions that they have as trainees. So, we're still kind of exploring different options to get in front of residents and get them trained up. So, we haven’t cracked that nut perfectly.
Dr. Craig Joseph: Yeah, so that's great. So, if I'm an attending, I'm going to get an invitation about once a year and it's going to be a morning session or an afternoon session, you're going to ask me, what are things that I find today that I think I need help with. And then you're going to look at all the data points that you get from the EHR vendor about how I'm doing, which might actually contradict everything that I've just told you. I think I'm struggling in documentation, and you might come to the conclusion that I'm really good at that, but I'm really bad at ordering or something along the lines. And then you’ve got to figure out, okay, how are you going to hit both of those things in four hours.
Dr. Scott MacDonald: We've evolved that a little bit. We're doing some shorter sessions now. We also have a trainer sort of assigned to each clinic. So, everybody in the clinic has a person that they can go to if they have a problem, who can come out and work with them elbow to elbow or just send them some tips. Depends on how complicated it is. So, the protected time is still there, but it's smaller and kind of spread out. Scheduling people every year for four hours turned out to be very difficult. So, we've kind of evolved the model a little bit. We still have elbow to elbow training, but it's a little less.
Dr. Craig Joseph: And are they still observing the docs with 1 or 2 patients?
Dr. Scott MacDonald: They’ll offer. It’s not mandatory. But they suggest it often. Let me watch you work with the patient.
Dr. Craig Joseph: I think it would be scary. Some of the things that you would find that people are doing that they think is the way it was, the way I was trained, I'm sure of it. And you're like, that was really an inefficient workflow that you just selected for yourself.
Dr. Scott MacDonald: Yeah, I used to do it when we first started, in my early days when I had less responsibility, I would actually do some of that training for new physicians as they came on board. But I saw a lot of weird stuff, and I was able to disavow them of those workflows. So, it was rewarding.
Dr. Craig Joseph: Often just having physicians in the same clinic share workflows can be fairly clarifying. So even two folks who neither of them may be a power user, but I've seen Doctor A go, ‘well, this is how I do a referral for cardiology’ and Doctor B is like, ‘wait, how did you do that?’ And there's nothing magical there, just that, ‘Oh, well I picked up on this thing about referrals, but when you did your version, you have something that I didn't know about’ and just sharing, even in an unorganized way is helpful. So, what have been the results of the PEP squad?
Dr. Scott MacDonald: Part of the PEP squad is also system enhancements too. So, we actually have depending on how well we're staffed, a couple Epic certified builders on our team too. So that's a bit more now on a sprint basis. So, we'll have set aside weeks where the builders are going to be working with a given specialty or clinic. We have sort of a menu of items that we can build that have a minimal amount of governance around them, mostly things that are going to have impacts on the rest of the health system. So we can build new order sets and review tools and documentation templates and those sorts of things for a given specialty. So, we have that sort of time set aside with the given specialty. On outcomes, it's hard to get a lot of quantitative outcomes because there's so many other factors happening in the world of medicine in the hours these days. So, we haven't been able to show clear changes to, for example, in levels or signal data about time in the charts, because that has so much more to do with the workload that's coming in and other factors like a global pandemic that was big confounder for all of our data over the past few years. But certainly, the soft results, the testimonials that we've got from our physicians are fantastic and everybody really has found value in spending time with the trainers and getting new tools in the system for them. And they look forward to interacting with the trainers when they come out periodically to their clinics.
Dr. Craig Joseph: Yeah. Well, I think the key metric might be what percentage of people actually agree to this, right? This is not a required program and so if I'm giving you my time, which is generally the thing I have the least to give, then then clearly, I'm finding benefit from it.
Dr. Scott MacDonald: Yeah. I don't have that statistic on my finger, on the tip of my tongue, but we've got it somewhere. We've got a lot of data.
Dr. Craig Joseph: Well, and it's great that the health system is seeing this as something that's really important and, putting the money into it. I'm sure it's not inexpensive to have 10 or 12 resources dedicated to simply making sure the physicians are as up to date as they can be on the on the latest and greatest uses of the software. Or, not up to date, not the latest and greatest of the basics, that they were supposed to have learned the first time, that maybe they didn't hear that part of the class or missed that part of the class.
Dr. Scott MacDonald: Yeah, that's a really important point. When you're coming in to learn a new EHR, it's just a firehose of information. People complain about how EHRs aren’t simple like my iPhone, but healthcare is just a lot more complicated than a weather app. Well, weather is complicated, but looking at the weather is not. So, there's no way anybody can really learn everything they need to learn during a couple days training when they join it. So, it's really important to have them be effective physicians to know how to do stuff. We hope that it'll pay off in terms of decreased physician turnover because we know it takes half a million dollars to replace a physician who leaves, give or take. So, we're hoping it's in some way affecting those sorts of numbers. And having an ROI just hard to prove to the level that will satisfy a CFO.
Dr. Craig Joseph: Yeah. Well, you're doing it. So, congratulations so far. And if you want us to talk, myself and all of our listeners, we can talk to your CFO, if you think that will be helpful, I suspect it won't be. But, on behalf of all the listeners, we're happy to do that. Talking about other things that will make your CFO maybe happy, an AI Scribe. See, I just transitioned us to an AI Scribe. So, you have, it sounds like, gone all in on one particular vendor. Tell us what that AI Scribe is or what is an AI Scribe, first of all, and who's the vendor and how's all that working out?
Dr. Scott MacDonald: Yeah, so AI Scribe is one of the names. You may also hear ambient listening is another term of art. But my take is that ambient listening doesn't mean much. It just sounds like somebody is eavesdropping, but this is actually doing something for you. It's using AI, obviously, but it's a scribe. So, it provides a lot of the functionality that a human scribe might provide a physician. And in a little more technical detail, what it does is its sort of like Dragon or one of the medical transcription voice recognition software’s that can record your voice, that you can then use to build your progress note or H&P, depending on your practice area. But in this case, the system generates the transcript of the entire visit. So, it's not just the physician dictating what happened, but it actually is capturing the patient's voice. Not only the patient's voice, but their families. So, the parent’s voice of this pediatric visit or, when I see a demented geriatric patient, that's the patient's children that are giving me the history and all that's captured by the transcription part of the AI Scribe software. Then the next phase of the software is it takes that using takes that transcript using a large language model to convert that transcript into a progress note. So the few minutes I would spend for every patient before writing a note, now, the system will generate a note for me that I just edit, make a few corrections and changes. So, it saves me quite a bit of time, a couple minutes on average across the board and then, I can just edit that note, sign it, and I can go. We're using a vendor are called Abridge, which is one of the kind of two that were initially out of the gate that were more integrated with Epic. Nuance, which is the Dragon company owned by Microsoft now was one of the other vendors. But we went with Abridge. We did a pilot with them this summer, with 30 physicians, and we found pretty good results. We measured a lot of things about time in the system, the quality of the notes as rated by the physicians, what patients thought about it if it improved their ability to engage with each other during the visits by taking the third entity out of the exam room, and generally got good results on all these. So, for now, we're going to go ahead and roll out to our physician or attending physicians for now, anybody who wants it, it's not mandatory to use it. We find that it's a lot better in certain specialties than others. So, as long as our budget lasts, we’ll be able to provide this to as many physicians as we can get it in the hands of.
Dr. Craig Joseph: So you said that you're using it, and we're talking outpatient, I'm assuming we're talking in the clinics.
Dr. Scott MacDonald: Yeah, primarily it's Abridge, so that they're good for quite a few different specialties in the ambulatory space, and then they're about to release official support for the emergency department. But they're working on inpatient as well. Workflows are a bit different there, especially in an academic health system where there's a lot of members of the team gathering information from different places. It's not all verbal information from the patient. So, it may work. It may still end up working better in arbitrary, because so much of that visit is verbal in that exam room where you can actually capture the words, it kind of leads to one of the awkward pieces of the AI Scribe that's vendor independent, which is that it doesn't know what you know. When a patient says it hurts right here, it doesn't know that. Or when I hear a murmur, it doesn't know that. So, it has to change the dynamic with the patient quite a bit. Especially to hear during the exam to say, okay, I'm going to be talking to the computer now. You don't need to answer me or ask questions when I talk about right upper quadrant passes.
Dr. Craig Joseph: Fair enough. Have you gotten used to that being able so you have to kind of essentially have to dictate your physical exam, either as you're doing it or afterwards. And how have patients reacted? Obviously, you're telling them that you're going to be recording them and why and how it benefits them, and how do they react?
Dr. Scott MacDonald: Generally, pretty good. California's a two-party consent state for recording. So, we have to get verbal consent and document that in the record each time. But we have some fliers up on the wall so people can read about it. So that way we don't have to spend a whole lot of time describing the system in detail and wasting precious clinical time. But most people are agreeable to it. I tend to read the body language, and if somebody looks at me a little askance or seems uncomfortable that I won't use it. But majority of people think it's pretty cool and based on our patient survey data, a lot of patients feel like it's really improved the engagement. And in the interaction with the physician, just because I don't even log into the computer oftentimes anymore, I'm just sitting straight in front of them, looking them in the eye because I know something else is taking care of my documentation. So, I don't need to worry about that.
Dr. Craig Joseph: Awesome, and is this on your phone? That's doing the recording? Where's the recording being done? Is there a device in all the rooms?
Dr. Scott MacDonald: No, I'm just using my phone. I use the mobile app that our EHR vendor puts out. So, it's embedded right in there. So, I don't even need a separate app for it.
Dr. Craig Joseph: Okay, and it's made you faster, it sounds like the note comes out fairly complete from your perspective.
Dr. Scott MacDonald: Yep. Takes about a minute or so for the note to be generated. Minute and a half maybe. And it's usually pretty accurate. We came up with a score, a note quality scoring mechanism, and it was 3.9 on a 4.0 scale, basically on a sample of 300 notes that we gave, most of the kind of errors are fairly small and inconsequential. That might even really be interpreted more of a matter of style rather than medical accuracy. There's the occasional mistake where it thought it was a different drug name that it interpreted wrong. So, I have to make sure those are all correct. But it gets 90-95 plus percent of those correct when I'm talking about obscure drug names.
Dr. Craig Joseph: Okay. Fair enough. So, you're expecting to make this available to all your physicians and some of them will use it and some will think that they're doing fine it sounds like.
Dr. Scott MacDonald: Yeah, specialties who have a lot more structured documentation, like ophthalmology is one extreme example where a lot of their notes is all just intraocular pressures and refraction indexes and things like that that are just not spoken. So, it doesn't make sense to use AI Scribe for a lot of ophthalmology. If it's an area or patient complaint that's a lot more functionally based, like how is your vision impairment affecting you? Well, that might be better to include AI Scribe for. Certainly, primary care where we're taking the initial history and exam and trying to figure out what's going on with the patient. AI Scribe works really well. Places in the middle might be, orthopedics. We're going to test that out. I think they're starting some of them this week. And so a lot of their database is based upon their imaging in history, but less about a detailed history of present illness from the patient like an internist would use. So, it's interesting to see how the different specialties work. Dermatology is very photograph based. There's not much history there so AI Scribe is not terribly useful, sometimes useful, but not terribly for them.
Dr. Craig Joseph: Yeah. No, that all resonates. I was going to start making jokes about ophthalmologist notes, because ophthalmologist notes don't make any sense to any physicians besides ophthalmologist. I'm not even sure it makes sense to them.
Dr. Scott MacDonald: Yeah, I use a large language model to help interpret ophthalmology notes.
Dr. Craig Joseph: Yeah, that actually resonates with me. You were talking about dermatologists, I remember, I think I was a resident just doing a month of dermatology, pediatric dermatology and I went in to see a patient and came out to discuss it with the attending before the attending followed me in. And I said, it looks like this, but it can't really be that because the history’s all wrong and he looked at me, goes history? What do I care about the history? And he walked in and said, it's exactly what you thought it was at first and the fact that the history of the way the rash presented, it's irrelevant. It's what do you see? Yeah, that resonates a lot.
Dr. Scott MacDonald: Yeah. It's still psoriasis no matter what.
Dr. Craig Joseph: Yeah. It's not going to change if it's itchy or not. I can't convince him that it's not that, so one of the things that you talked about was note quality and you had mentioned, when we were preparing for this, that you have kind of a program called the note matrix program where you kind of go over dos and don’ts for documentation. Is that still necessary if ambient listening and AI scribing becomes Uber successful or is that something that the old folks like myself who put notes together manually, care about.
Dr. Scott MacDonald: Yeah, no, it's still relevant. So that note matrix program, it's one of the projects of the PEP squad, actually. And so, we put a lot of work into a really multidisciplinary group to figure out what should be in a note, what shouldn't be in a note across the whole enterprise. So, this is discharge summaries, missing admission reps, progress notes, discharge summaries in the clinic, in the hospital. Because we all know there's a huge problem with note bloat over the past decade or two. It's so easy to add a lot of text to a note or copy forward from previous notes, and it leads to not only potential inaccuracies in the notes, but they're really hard to read and they're time consuming for the audience to read. And so, we wanted to try to get those tightened up so that the notes can serve the many purposes that notes have to serve. Which is briefly, is documenting what you did and what's going on, communicating to your colleagues on the care team. But to be honest, it's a lot of other things. We kind of recognize those and brought representatives to represent those points of view. So, there's the compliance piece. What do we need to document to comply with federal and state law? What do we need to get in there to capture quality indicators for quality reporting improvement efforts? What needs to be in there for coding so that we can build and get paid for the work we're doing? What do the patients need to see? Because now, since the 21st Century Cures Act, our patients are another audience of our notes. So, we're trying to get all of these constituencies of the note once we got all their voices in in our matrix. So, for each progress note type, we've got one column of all the different things that historically have been in notes. And then we have commentary to say this has to be in there, or this can be in there, or this shouldn't be in there. And then what we've done to support that as we build tools in our EHR to help make it easier to do that, not just templates, which are hard to get people to use a system generated entire note template, but rather we put in the modules from the template. So, we've got a really good way to record or to summarize social medical history, that sort of thing. So, we've created smart links and a lot of them are using some of the hover bubble smart links. Now that you can cursor over it, it'll show you a big block of text, but it's not text that really needs to be in the file. No other people a year from now isn’t going to want to read that. So, we've done what we can to minimize the amount of actual text that's in the note, while still making the function of chart review, which is actually also a function of a note, is this is also kind of where you review information about the patient by bringing it into your note. So, we're trying to walk a fine line between supporting that note review function, along with all the other, people who consume the content of it. So now it's another big change management process. So now we're going to be leveraging our PEP squad again as we work with physicians, either as they're getting onboarded or are working with them throughout the years to say, hey, we got these new things that we'd like, let's work with you and tighten up your note template you use every day. It'll make them easier for you to write and easier for you to for other people to read. Now going back to the AI Scribe piece of that, the AI Scribe generates sort of discrete blocks of text. It doesn't generate the whole note per se, so it gives you a chunk that's the HPI, and it gives you a chunk that it's the physical exam and the chunk that's the assessment and plan. So, most of us then will use those chunks in our kind of preexisting note templates. So, our preexisting note templates may have other things in there. Some people put those disclaimers about inaccuracies, or they put notes to their patients who are reading the notes to help them interpret it. Or they may bring in smart links to have some data that automatically comes in for them to review and is referred to. So, all that stuff still works using AI Scribe. It's just those chunks where you used to have to either type or dictate an HPI. Now that section is being populated by AI, and those are the most time-consuming chunks to generate.
Dr. Craig Joseph: Yeah. What I love about this, besides it being incorporated into the PEP squad is that you're kind of following some of the principles of human-centered design, right. As you said it yourself, make it easy. You want to make it easy to do the right thing. So oftentimes what groups do, in my experience, is they they'll write a document. Hey, this is what should be in this note, this is what should maybe be in this note, this is what should never be in this note. And then they email that document to you, and that's fabulous. But you're not making it easy to do the right thing because I read it, but then I forgot it the next day when I need it. And when do I need it? Well, I need it right when I'm writing the note. And so, kind of making that easy for me to get and kind of incorporating it, so very little work to do. That's the way to go. So that sounds good. I think it's smart that you're not kind of putting your hands up and going well the AI, the LLM is going to write the note for us, so it doesn't matter anyway.
Dr. Scott MacDonald: Yeah. Because a lot of the problem is getting rid of stuff. And in fact, the, the AI Scribe is a little bit verbose for my taste, I tend to write fairly curt notes. So, my notes actually got a little longer, but I'm still spending less time doing them, and I have a lot less mental exertion at the end of the day. So, it's still a win.
Dr. Craig Joseph: That's great. So, let's start to kind of close up. I just wanted to kind of talk about art. And I don't mean art, the stuff that we draw and sculpt, but art, which is an Epic product or the name of an AI that kind of helps physicians start the response to a message. And so, you guys have been using it. What do you think? I understand it's kind of early days, but does it actually help you?
Dr. Scott MacDonald: Occasionally. Yeah, I just thought of a cheesy joke. So, it's the art formerly known as draft in basket responses. A play on our purple friend in Minnesota. So, what it does is basically on the first incoming mychart patient portal message from a patient. We can generate a prompt that goes through Epic's pipeline. That sends that message from the patient with what other information we choose to send along with it to OpenAI is what actually does it. And that prompt tells the AI to generate a response, the kind of response that we want. So, it works. It's somewhat helpless sometimes and somewhat helpful. I would say more than anything, it provides sort of a draft that you can get started with, and it provides some of them the social niceties and then maybe a sentence or two that are useful, but it doesn't make medical decisions. You still have to look at all the data that's appropriate for the question. Make a decision. You still may need to place orders, fill in specific details, but it can provide sort of a skeleton or a framework for you to generate your response to the patient. And because, the LLM can be a little bit more empathetic than a human can be because it has the time, those extra microseconds, to generate the: Hi. How were you doing type stuff? I kind of appreciate that, that stuff that, I could type out, but it's going to take me a couple seconds to type all that stuff out. So, if the lab does it for me, great.
Dr. Craig Joseph: Yeah, when I was a medical student, we didn't have LLMs back then. Dr. McDonald, let me assure you. I was seeing a dermatologist, and he walked in, and I hadn't seen him in a long time, and he asked me how our vacation had been a year prior. And I was like, oh, my gosh, I can't believe you remembered that, and then I saw on his note that his medical assistant, who was scribing for him, had written down, that I was going on vacation. So, he had just read the note from a year prior, and I was like, it's kind of cheating, but it's also kind of brilliant. And so, yeah, if you have a computer kind of move in that direction. Now, to be clear, just in case anyone's listening, this is a private LLM right? So, you're not actually sending patient data that's available to OpenAI. It's all kind of behind a firewall, as I understand it.
Dr. Scott MacDonald: No, we send what we want to send that the LLM doesn't have access to the chart per se, but if we want to send a medication list for the patient along with a query that's about a medication issue or a refill, then we can send the med list with that.
Dr. Craig Joseph: Okay. Yeah. Because there's no there's no private information there.
Dr. Scott MacDonald: But the system also categorizes the incoming messages in four different types. So, we can send an LLM prompt for each of those different types. So, we can fine tune our prompts a little bit to get the get appropriate responses. So that's something we're hopefully going to get to more this year is working on making those prompts a little bit more effective. There's an Epic tool called the Prompt Engineering Testbed that we're hoping to get some of our informatics physicians involved in and learn about how to do prompt engineering more effectively through the output of this.
Dr. Craig Joseph: Yeah. Prompt engineering is the new term for programing, as someone said.
Dr. Scott MacDonald: Like natural language programing, because you don't have to follow any particular syntax. You just have to tell it what you want and then like your puppy, it tries to please you.
Dr. Craig Joseph: Haha. I have never heard an LLM referred to as a puppy, but you're not far off there with that metaphor, I love it. Awesome. Well, this has been a great conversation. As you know, we love to end the conversations that we have with a question about design. Is there something in your life or are there some things in your life that are so well-designed, they bring you joy when you use them? And so, Dr. McDonald, is there something in your life that's, pretty awesome, and it's because of the design?
Dr. Scott MacDonald: Yeah, I think so. And it's my own thing that I designed, but I use it every day, and it's a combination of my home automation system and 3D printing. So, I'm kind of a coffee snob, and I roast my own coffee beans, and I brew a pour over every morning with just the right temperature. So, my electric kettle is really good at maintaining a good temperature, but it doesn't have a remote turning on functionality. So, I was able to find, it's basically called a finger bottle. So, it's a little home automation device that has a little finger that comes out and just presses down. And so it can press the on button for my coffee, my hot water kettle. And so, in order to get the finger to hit the button in the right place, I 3D printed a little, piece of plastic that clips on to the hot water kettle just so that finger is red in the right place, at the right height, and centralized over the button to, to turn it on. Now, if I still have to press a button on my phone to get that to happen. I have to spend five calories to do that. So, we don't have that. So, then I built a home automation connected to my wake-up alarm on my phone so whenever my alarm goes off, it automatically turns on some lights. It tells the finger bot to turn on the hot water so that by the time I wake up a little bit and go downstairs, the water's hot and ready for my pour.
Dr. Craig Joseph: That's impressive and the fact that the plastic doesn't melt, so it must not get too hot on the hot water.
Dr. Scott MacDonald: Yeah, it's far enough away. And if it melted, I could just use a different type of filament that's a little more heat tolerant. But the one I use works fine.
Dr. Craig Joseph: All right. And so, then you have coffee. You’re pouring it over at the exact right temperature because it maintains that.
Dr. Scott MacDonald: Yeah. So freshly ground beans that were roasted within the past few days and then I just pour. It’s basically chemistry, I was a biochemistry major. So, it's just basically an extraction. You're just trying to extract as much of the volatile flavor compounds and caffeine from the ground. So, you got to have the right surface area to volume ratio of the grounds, the right temperature of your solvent water and the right dwell time and all that stuff is what makes a really good cup of coffee.
Dr. Craig Joseph: So, what time should we be at your house? To try some of this coffee.
Dr. Scott MacDonald: About 6:30
Dr. Craig Joseph: All right. I noticed you're not giving us your address, but we'll work on that part, but wow. That's cool. So, your favorite design is your own design, and I think that's probably true for many of us.
Dr. Scott MacDonald: Yeah, it combines a lot of just the things I'm interested in 3D printing, CAD, home automation, computers, coffee. It's all wrapped up in this one little package.
Dr. Craig Joseph: It's all right there. That's awesome. Well, it has been a pleasure talking with you about all the cool, fun things you're doing at UC Davis, by the way, which is found and located in Sacramento. Appreciate the time and thanks for all the work that you're doing. And I think a lot of it you're moving away some problems for the rest of us, kind of proving that, hey, some of these things do make sense financially and helping physicians and clinicians in general, become less burdened and get back that joy of medicine and that ability to actually do what we all went to school to do. So, on behalf of everyone, I say thank you and we appreciate it.
Dr. Scott MacDonald: You're very welcome.