The Journal of Graduate Medical Education (JGME) once again offered two of its most popular and engaging interactive workshops at the ACGME Annual Educational Conference to help attendees take a scholarly approach to program evaluation and to work together to help each other get over barriers to writing. JGME editors and staff members also returned to the ACGME Hub in the Exhibit Hall to meet members of the graduate medical education community and answer questions.
Count It Twice: Conducting a Scholarly Program Evaluation and Achieving Scholarship
Programs make evaluation decisions on a daily basis, asking questions like, “Should we retain component X or Y in this curriculum?” “Did this innovation address the challenges from our old approach?” or “What should we report to accrediting bodies or funders?” As the answers to these questions are reported, however, faculty members, hospital leaders, and others may challenge the evaluation data. Taking a scholarly approach to program evaluation can help to effectively address skeptics, while also turning evaluation into scholarship.
This interactive JGME session, led by Editor-in-Chief Gail Sullivan, MD, MPH; Executive Editor Nicole Deiorio, MD; and Deputy Editors Deborah Simpson, PhD and Lalena Yarris, MD, MCR, aimed to help participants (1) distinguish educational program evaluation from research; (2) design an evaluation with a scholarly approach using the American Evaluation Association’s standards; and (3) explain the central roles played by key interest holders.
The first part of the session introduced concepts and criteria for educational scholarship, along with an overview of systematic evaluation—what it is, what it isn’t, and how to strengthen it. Participants were then led through the steps of program evaluation, from writing specific questions informed by the evaluation model, to identifying evidence that would matter to interest holders, to collecting that evidence and implementing an actionable plan.
Participants received an Evaluation Blueprint Form and broke into small groups to ask a common GME evaluation question and brainstorm about who and what roles would be affected by the answer (i.e., “Who else cares about this?”). Each group was assigned an interest-holder role (e.g., program director, faculty member, resident, clinical leader) and was asked to address what information/evidence would inform their decision or change their perspective on what is or is not working.
The session concluded with a discussion on how to turn evaluation data into scholarship. Participants were given a Publication Checklist with questions that could be used to help organize data into background and objectives, methods, results, and conclusions. With more than 100 participants, this session was one of JGME’s most engaging and popular offerings.
Less Angst and More Fun: Writing Strategies and Writing Groups for Scholarship
“Why is writing so hard?” This was the question JGME’s second session aimed to answer, first by uncovering the barriers many people encounter when writing, and then presenting strategies and solutions, including how to use groups to facilitate time to write and create accountability. This hands-on, practical session, also led by Drs. Sullivan, Deiorio, Simpson, and Yarris, immediately placed participants in small groups to ask what keeps them from writing, and to brainstorm solutions.
As expected, the most common barrier was “I don’t have time,” followed by some less obvious obstacles, such as fear of criticism, perfectionism, and self-doubt about having anything worthwhile to say. The presenters discussed several strategies, including reducing “time bandits” like social media and emails, as well as setting aside small chunks of time daily to write rather than feeling like an entire day is necessary to “binge write.” While some participants reported better results writing for long periods (and for some that’s true), others were encouraged by the evidence that writing a little at a time can increase productivity. Different strategies are effective for different people, so trying new ways of working may bring results.
As for expecting perfection, both presenters and participants offered advice, with one attendee saying that all writers should expect—and even welcome—a “bad” first draft. The most successful writers are those who have higher confidence in their writing ability, regardless of quality. They write more and more often, with increased opportunities to edit and improve later.
On this note, participants were asked again to leave their comfort zones and “free write” on any topic they chose, the only rule being to write for a set length of time without stopping and not going back to edit.
“Even if you just write ‘I can’t think of anything’ over and over, just do it until something comes,” Dr. Simpson advised.
Afterward, participants shared with each other and gave feedback on what ideas and storylines could be developed from the exercise.
The session ended with a discussion about writing accountability groups (WAGs). These groups can increase one’s productivity by encouraging writing regularly, in short intervals, and on schedule, but they can also have process goals like building a community. Examples of how a WAG can be structured were offered, along with examples of established groups, such as the non-profit, Shut Up & Write.
This intimate, lively session provided several insights, and even some laughs for attendees who left feeling more confident to apply what they learned.
Journal Notes blogger Kevin Gladish is a staff editor for the Journal of Graduate Medical Education. He’s been at the ACGME since 2016, and is also a performer, writer, and storyteller.