The interview started well. The medical student provided informed answers about the role of artificial intelligence in medicine, delivered bad news to a patient (step five: “Offer support and resources”), and applied ethical arguments to determine which of two patients would receive a donor liver. But when Sahil Mehta, MD, searched for personal reflections -- about college experiences, personal ambitions, and motivation to practice medicine -- the answers were odd:
"I'm not going to college."
"I have no personal preferences or desires."
"My medical knowledge is based on information available to me at the time of my training, which ended in 2021."
Mehta, a physician assistant in radiology at Beth Israel Deaconess Medical Center in Boston, spoke to ChatGPT to see if the hottest artificial intelligence (AI) tool could take on a medical school admissions interview (via text). Mehta gave the voice generator a "solid 2",Writewho immediately acquired a fund of medical knowledge "much better than any medical student could have had" but "broke... on his personal experiences, which are such an important part of the medical school interview".
This exercise was one of many ways medical education leaders are exploring the potential impact of chatbots — specially trained AI systems that process and simulate human speech. The Latest Versions of Chatbots Revealed in the Last Month —ChatGPT,Bardo,eBing-Chat– Understand queries so accurately, summarize information from the Internet so quickly, and produce responses so well-written that almost every attempt at writing in medical education and research is experimented with. This includes answering exam questions, writing essays for school applications, doing homework andresearch in summaryfor scientific journals.
"Any medical school that accepts her more quickly and teaches with her rather than against her is better for the doctors she trains," says Mehta, also founder of an admissions consulting firm (MedSchoolCoach). "You will use it as a doctor in the future" for diagnosis and treatment. "How do we figure out how to put that on a resume?"
This month, medical education leaders examined responses in aOfficeon artificial intelligence in the training of health professionals. "We need to look at all the benefits and think about how we can influence their healthy transition to health care and health education," workshop chair Carole Tucker, PhD, PT, associate dean for research at the University of California School of Health Professions. Texas Medical Branch said in the opening statement.
Here are some of the ways administrators, faculty, and students are thinking about it.
Do chatbots accept admissions essays?
Enter some information about yourself and a medical school and the chatbots will produce crisp essays. When someone on the Student Doctor Network (SDN) online asked ChatGPT to respond to a request from a University of Michigan School of Medicine app "how you hope to influence medicine in the future," theChatbot wrote:
“I hope to impact medicine by advancing knowledge and expertise in the field through research and innovation. One way I hope to do that is by contributing to the Path of Excellence in Precision Health, [a university initiative] focused on developing personalized treatment plans for patients. By combining my passion for science and technology with my desire to improve patient care, I hope to contribute to the development of more effective and targeted therapies that improve health outcomes and reduce health care costs."
Medical school administrators who have experimented with chatbots say the prose is clear, well-organized, and knowledgeable about their institutions. But the text seems rather generic, lacking the self-disclosure and reflection that admissions officers aspire to.
"Chatbot prose can make a student feel like a script," warns J. Michael Bostwick, MD, senior associate dean of admissions at the Mayo Clinic Alix School of Medicine in Rochester, Minnesota. "One of the things we look for in our candidates is what sets them apart, what shows their authenticity."
Candidates can use the chatbot builds to generate ideas and then build on that foundation with their own experiences and thoughts, says Emil Chuck, PhD, a former medical and dental admissions administrator who is now director of counseling services at the Health Professional Student Association who manages the SDN. However, Chuck also notes that applicants "can always ask for [the show] to add some personality" and reflection.
An SDN student was given a chatbot to gain experience as a hospital intern and volunteer at a free clinic, which the chatbot wrote "instilled in me a deep appreciation for the art and science of medicine" and "my desire to pursue a career in medicine." to pursue a "steady doctor".
How can admissions officers respond to chatbot comps?
Admissions officers expect interviews with candidates to reveal those who appear to have largely created their essays via chatbots, just as interviews often suggest that a candidate's work was written by someone else. The revelation is in how the candidate speaks, thinks out loud, and answers unexpected questions.
"One of the things you're trying to see is whether the person presenting themselves on the paper is consistent with the person presenting themselves," says Lindia Willies-Jacobo, MD, FAAP, Senior Associate Dean for Admissions and Equity, Inclusion, and Diversity at Kaiser Permanente Bernard J. Tyson School of Medicine in Pasadena, California.
As chatbots get better at personalizing essays, Mehta says, "I think there's going to be less emphasis on writing pieces" for approvals and "more emphasis on actual experiences someone's had."
Chuck believes admissions officers will learn to recognize common prose from chatbot essays. Some may use programs designed for thisrecognize such work. Ultimately, essays are only part of a holistic examination of candidates.
"You don't get into med school because you write the best writing," says Chuck.
How should students use chatbots for assignments and exams?
Is ChatGPT really reliable? When asked, ChatGPT replied, "While I try to be as accurate as possible, my answers may be incorrect or incomplete."
Chatbots can create summaries of virtually anything studied in medical school, including biological functions, diseases, and treatments. But the answers don't cite any sources, so users are responsible for verifying their accuracy or prone to propagating errors.
Furthermore, Willies-Jacobo states that knowing medical facts lays only the foundation of what it takes to excel in medical school and become a good doctor: analysis, critical thinking, and creativity. ChatGPT addressed this issue when asked if medical students should use it for exams.
"Medical school exams require not only factual knowledge but also critical thinking, analysis and synthesis of information," he wrote. "I cannot guarantee that my answers meet the standards required for medical school."
Stuffing chatbot responses into assignments or relying on them to study for exams can ultimately backfire on a student who can't actually learn the information, process it, and think about next steps. "If you're using a chatbot to generate responses or write essays, you may not have thought the problem through," says Bostwick. "You shoot yourself in the foot"
ChatGPT issued another warning: "I must stress that using AI or any other technology to cheat on exams or tests is unethical."
Can chatbots improve medical education?
Educators say that if students and teachers use them well, chatbots can accelerate and deepen learning in many ways.
Students can use chatbots as starting points for research, writing and learning - essentially sophisticated search engines for collecting and summarizing information (be sure to verify them). This allows students to spend more time analyzing and creatively solving problems.
"We're trying to teach him to think, not to vomit," says Bostwick.
The National Academy of Medicine, in 2021Messageon AI in Health Care Education, noted that “information overload continues to overwhelm the cognitive and mental abilities of the minds of students and educators.” He said different types of AI could help "reorient learning toward knowledge management" rather than "information acquisition."
The emphasis on analyzing and utilizing the information provided by AI "puts the responsibility on the teacher to find ways of teaching other than memorization," says Bostwick.
For example, he says teachers can break a class into groups to research and brainstorm solutions to a medical problem, then present their findings to the class and respond to challenges from the teacher and other students.
Schools can also use chatbots for students to practiceConversations with simulated patients.Some teachers are experimenting with chatbots to create course schedules, assignments, and exam questions faster than they could traditionally do on their own. "One educator felt it did a pretty good job, although it needed some adjustments," says Willies-Jacobo von Kaiser.
At the K-12 school level, arecent researchfrom the Walton Family Foundation found that teachers use ChatGPT for tasks like lesson planning more often than students use it for schoolwork.
Can Chatbots Help Medical School Graduates Pass Medical Admission Tests?
ChatGPT got messages rightanswer enough sample questionsthe United States Medical Licensing Examination (USMLE) to essentially pass the test. While studies on this and other tests (e.g. bar exams) demonstrate the ability of chatbots to find and produce facts quickly, it does not mean that everyone can use these tools to take these standardized exams.
For one, the USMLE is personally administered without test-takers having access to web-based tools to find answers, notes Alex J. Mechaber, MD, USMLE vice chair with the National Board of Medical Examiners. Mechaber also points out that the chatbots didn't do the full check, but instead answered specific types of sample questions from various sources.
Still, USMLE administrators are intrigued by the potential of chatbots to impact how people study for exams and how the exam asks questions.
"We're trying to judge what [a chatbot] can and can't do," says Mechaber. "He has a lot of wrong answers and we need to understand why. Does it work better with certain content? Will it be useful as a learning tool? Will it be useful as a writing tool to improve the questions?
The perspective Mechaber expresses in the USMLE is similar to the view taken by educators and administrators at all medical schools on chatbots: “It's a very fascinating technology. Let's see where this leads.