Emerging Trends in Competency-based Assessment

Lecture by: Eric Holmboe, MD

Dr. Eric Holmboe, a board certified internist, is Senior Vice President for Milestones Development and Evaluation of the Accreditation Council for Graduate Medical Education. Prior to joining the ACGME in January 2014, he served as the Chief Medical Officer and Senior Vice President of the American Board of Internal Medicine and the ABIM Foundation. He is also Professor Adjunct of Medicine at Yale University, and Adjunct Professor at the Uniformed Services University of the Health Sciences.

(UW NetID Required)

SPONSORED BY THE OFFICE OF GME

Sponsored by the Office of GME

Following Dr. Holmboe’s lecture, a small group of participants had a follow up discussion with Dr. Holmboe on topics related to competency-based assessment. Below is a summary of selected questions and answers:

Q: What are your suggested strategies for faculty training and buy in during the milestone implementation processes?

Help faculty create a shared mental model by providing training and creating opportunities for dialogues. Adopt and practice the principle of “Co-Production, Co-Creation”. This applies to residents as well who will need training in milestone-based assessment. The article on rater cognition is an excellent resource to consult.
At the more fundamental level, remember that adopting this new assessment framework poses a real identity threat to established long-time faculty who view the new mandate as discounting of their contribution to advancing educational excellence. Remember that for many, change triggers loss and grief. Two excellent books to consult include: Transitions: Making sense of life’s changes (William Bridges) and Leadership on the Line (Ronald Heifetz, Marty Linsky). Apply communication strategies from breaking bad news skills and motivational interviewing and be empathetic to faculty who are resistant to change.

Q: How can ACGME help us access national best practices in milestone implementation in a timely and efficient manner?

Some of the best practice resources are being posted on the ACGME Web site, which is being significantly revamped. The site will include updated assessment resources. Also a distance learning specialist is being hired to help with resource dissemination.

Q: What happened to the ACGME portfolio project?

It tanked due to the complexity of the IT issues - competing commercial programs, and the necessity for a portfolio to be maintained over very long periods of time. He supported the idea of portfolios as an assessment and a showpiece strategy for learners, particularly when the learner must take ownership.

Q: How do we choose public health based outcomes and judge quality of care?

This was often a moving target, and where the ACGME needed input from the specialties/subspecialties.

Q: Accreditation bodies are scary things. How do you (ACGME) reconcile your developmental and regulatory roles?

Regulatory role is important as there are some really unbelievable bad things still going on, and we have to protect residents and their education.  Trying to be more open minded, but still a lot of people with an old-style mindset – especially in the boards. Improving access to info such as tools and best practices, redesigning web-site, etc.  It’s all about the balance required in self-regulation of professions. Please let me know if you encounter any difficulties with ACGME staff or process.

Q: How to use EPAs in conjunction with the milestones to engage faculty.  

a) Medical educators have a hard time distinguishing competencies, sub competencies and milestones and instead think along EPA terms.
b) However, thinking along the lines of EPAs is necessary but not sufficient because faculty need to "diagnose" and so need to have access to mapped competencies, etc.
c) Competency based assessment should be viewed as an ongoing faculty development effort this is circularly related to providing trainee assessment.
d) Also the definition and use of EPAs is evolving. 

Q: I am in a group developing EPA's for Emergency Medicine. One concern that's been expressed is, if an evaluator is "signing off" on an EPA, how do we ensure that the evaluator is considering all the various elements/competencies underlying the EPA? That is, if they say "yup,  this resident can manage a patient with multiple chronic diseases" how do you ensure the evaluator is not just thinking about Medical Knowledge or patient care but is also considering communication, professionalism, appropriate systems use, etc.

The “nesting" feature was useful in that some faculty will only be able to assess a small piece of a bigger EPA, mainly due to the way we currently structure rotations. So while an evaluator might not be able to say "manages a patient with multiple chronic diseases" they might be able to say "manages the features of other underlying chronic disease during an acute exacerbation of a medical problem" (i.e. canstay on top of a patient's diabetes while they are hospitalized for a COPD exacerbation). The CCC/PD can look at the bigger, more global picture while an "average" faculty member might assess the smaller pieces nested within.