How Are USMLE Questions Written? 9 Open Secrets for Impressive Boards Scores (2024)

The USMLEs are the most important exams you’ll take in your medical career. Want to impress residency program directors and get an interview at your top choice? Step 1 is the most cited factor in giving interviews. Need to make up for a low Step 1 score? Then Step 2CK is your last best chance.

If that stresses you out and creates sleepless nights, you’re not alone.

But what if you knew the best method to prepare for Step 1, your Shelf exams, or any other USMLE? Even better, what if that advice came from the people writing the tests?

Not some anonymous Redditor telling you to “memorize First Aid and do UWorld three times.” Or the person who secretly failed Step 1 but is telling everyone “UFAP + Sketchy.” Or the Anki user who can’t keep up with his daily reviews but swears by Zanki/Brosencephalon.

The. National. Board. Of. Medical. Examiners.

Table of Contents

The NBME Publishes the Rules for USMLE Question-Writing

You’re in luck. The National Board of Medical Examiners (AKA the NBME) writes the USMLEs, Shelf Exams, and some specialty boards. Unknown to many, the NBME also publishes the instructions they give to question writers. In it, they detail how they write each question. They also tell you exactly what they are looking for students to do – and to not do.

In short, the NBME explains what and how you should be studying.

Does reading through an academic document on an arcane topic like MCQ-writing make you sleepy? Then you’re in luck! I’ve combed through the 100+ page document to find the nine most important points for you, dear reader.

The surprising result? Impressive USMLE scores are more straightforward to achieve than you think.

The less surprising result? If you don’t know what you’re doing, your score will suffer.

Here are the nine most essential takeaways from the NBME question writer’s guide.

1. Questions Test the Application of Knowledge, NOT Recall of Facts

“Each item should assess application of knowledge, not recall of an isolated fact.”

– “Chapter 5: Basic Rules for Writing One-Best-Answer Items.” Constructing Written Test Questions For the Basic and Clinical Sciences.

The NBME makes one thing clear, over and over. They are in the business of testing the application of knowledge.

They don’t care if you can memorize a ton of facts. (And design the questions to weed out people who memorize – more on this later).

Their goal is to assess “higher level[s] within Bloom’s taxonomy, such as ‘application of knowledge.'”

The NBME’s Challenge: Master the Material

Here is a sampling of other direct quotes from the NBME on the importance of testing application:

• “Questions should NOT focus on the direct assessment of isolated facts.”
• “Avoid asking about the leading cause of death in some subpopulation; instead, focus on the application of this knowledge.”
• “The NBME decided that it was not important to assess whether students know definitions; instead, the goal became assessment of whether or not test-takers could apply ethical principles on their decisions related to patient care.”

I could go on.

How many people do you know who memorize lists of facts? Or speed through First Aid trying to cover it multiple times. Or cram minutiae into their head using Zanki or Brosencephalon?

I know a ton, and I’m sure you do, too.

Cramming facts is the equivalent of memorizing recipes for a cake-baking contest. You may impress your friends by listing the 6 heart failure symptoms from First Aid. Memorizing facts may even help you crush your med school exams. But it won’t help when Step 1 asks to differentiate between viral myocarditis and ischemic cardiomyopathy.

The USMLEs won’t ask you to regurgitate facts.

The NBME is very clear: focus on mastery of material, not memorization. It’s much better to master half of First Aid and know how to use it than to “cover” it twice without understanding.

2. Apply Important Concepts; Don’t Chase the “Zebras”

“Each item should focus on an important concept or testing point.”

– “Chapter 5: Basic Rules for Writing One-Best-Answer Items.” Constructing Written Test Questions For the Basic and Clinical Sciences.

Another point of emphasis of the NBME is that they should test essential concepts, NOT zebras. “Focus items on common or potentially catastrophic problems; avoid ‘zebras’ and esoterica.”

What about those hysterical message board lurkers? The ones screaming their test was “nothing like UWorld”? The ones telling you that their Step just tested minutiae and tiny details. Or who say that they’ve done UFAP+Sketchy four times? Or who’ve memorized all 20K+ cards of Zanki?

The approach by a large number of med students is to try and learn every possible fact or disease. They’re the ones asking, “does this resource cover EVERYTHING I need to know?” The more “comprehensive,” the better.

They focus so much on memorizing esoterica that they forget the key concepts. Would it make sense for the NBME to test rare congenital conditions as often as heart failure? Absolutely not.

Which leads us to our next point.

3. The Number of Questions is Proportional to the Topic’s Importance

This may seem obvious, but more important topics have more questions than less important topics. “Important topics should be weighted more heavily than less important topics. The testing time devoted to each topic should reflect the relative importance of the topic.”

In other words, expect more questions about heart failure, asthma, or kidney failure. There are comparatively fewer questions on rare genetic disorders (although they do exist).

Whether you’re starting out, or taking your exam next week, the message is clear. Your time and effort should be proportional to the clinical importance of the topic.

4. Clinical Vignettes Weed Out Memorizers

The most important takeaway from the question writers guide? The NBME wants you to apply important concepts.

Vignettes force you to apply key concepts. As a side benefit, they also weed out memorizers.

Here is the NBME on why they love vignettes:

“Questions with a clinical vignette as part of the item stem have several benefits. First, the authenticity of the examination is greatly enhanced by using questions that require test-takers to “solve” clinical problems. Second, the questions are more likely to focus on important information, rather than trivia. Third, these questions help to identify those test-takers who have memorized a substantial body of factual information but are unable to use that information effectively in clinical situations.”

Do vignettes force you to apply knowledge to solve a problem? Check.

Do they focus on critical concepts and not esoterica? Check.

Do vignettes weed out memorizers? Checkmate.

5. There’s a Single Question Format: “Single Best Answer, Closed Responses”

At Stanford, one of our hardest exams was anatomy. Why was it so hard?

The format of the test made it brutal. Many questions gave you five statements and asked which was false.

Example of a question never to be seen on the USMLE:

Determine which of the following statements is false.

1. The sternum is anterior to the vertebral bodies.
2. A shoulder is proximal to the corresponding humerus.
3. One’s umbilicus is inferior to the mentum.
4. The first portion of the small intestine is the duodenum.
5. The right vagus nerve is lateral to the external carotid artery.

The content of our Stanford anatomy exams wasn’t abnormally challenging. But the test FORMAT made it (unnecessarily) difficult.

USMLE questions are hard, but not because the format is unfair

Stanford isn’t the only med school whose complicated formats make tests harder than necessary.

Have you ever had a true-false question that didn’t make sense, even after reading it five times? Or a problem only a mind-reader could get? (And even then, it still would have been a struggle.)

The great news is that the USMLEs have none of these such questions. They all have a single question format, called, “single best answer, closed response.”

“Single best answer” refers to the fact that there is only ONE best answer.

“Closed response” refers to the fact that the question is a complete sentence, not a fragment.

What does this mean for you?

First, med school exams are often not a good predictor of USMLE performance. This is particularly true if they 1) test minutiae, 2) require lots of rote memorization, and 3) use ambiguous test formats. So if you’re a preclinical student, don’t obsess about your preclinical grades. Not only do residency program directors not seem to care about preclinical grades. (Most schools are pass-fail anyway). Preclinical classes also aren’t often good indicators of how well you can apply critical concepts.

Second, you don’t have to prepare for many different test formats. Rejoice! (And put your energy towards mastery/application of important topics).

6. The “Cover-the-Options” Rule

This may seem like a shock since most medical school exams don’t follow this format. But the NBME has a rule called the “Cover-the-options” rule:

”If a lead-in is properly focused, a test-taker should be able to read the stem and lead-in, cover the options, and guess what the right answer is without seeing the option set.”

Translation: You don’t need to look at the answer choices to answer a question.

Let’s look at an example that doesn’t fit the “Cover-the-options” rule, courtesy of the NBME.

“Which of the following is true about pseudogout?

1. It is clearly hereditary in most cases
2. It seldom associated with acute pain in a joint
3. It may be associated with a finding of chondrocalcinosis
4. It occurs frequently in women
5. It responds well to treatment with allopurinol”

Try it yourself. Read the question and cover the answer choices.

The result? For this question, you’d have no idea what the answer was. Hence, this sort of item would never show up on a USMLE/Shelf Exam.

“Single Best Answer” Format

Additionally, note how this question doesn’t fit the “single best answer” format. Why? Because it’s vague. Particularly because the options aren’t listed on a “single dimension.” (What is more correct – the incidence in women, or its association with chondrocalcinosis?).

So, how can you use the “Cover-the-options” rule?

Ever done a Qbank question and wondered if the item was poorly written? This is a great test you can use to identify poorly written questions.

Also, when you’re doing questions, practice covering the answers. Not only will you talk yourself out of the right answer less. It’s also an excellent way to test yourself. If you can read the stem, then answer the question without knowing the answers, your scores will climb.

7. No One’s Trying to Trick You

Ever heard that the USMLEs try to “distract” you? That they’re trying to “trick you”?

It’s not true.

Vignettes may be challenging (see: weeding out memorizers, above). Students may not know how to interpret clinical scenarios. However, that doesn’t mean that the NBME is trying to mislead you.

The NBME creates vignettes that test your ability to apply important concepts.

Their goal is “avoiding…‘red herrings’ (information designed to mislead the test-taker).” They instruct test-writers: “patients in vignettes should tell the truth.”

They even caution against using real patient cases. Why? “Real patients are complicated, and the elements that are complicated are not always those that are important for assessment.”

What does this mean for you?

Virtually every sentence for Step 1 has a purpose. (You may not KNOW the purpose, but that’s a different story). For Step 2, the NBME claims (and experience suggests) that there may be unnecessary information.

However, they’re not trying to mislead you intentionally.

8. Vignettes Follow the Typical H&P Format

If you’ve done a lot of Qbank questions, you may have already realized this. Every vignette follows the same chronologic format as a typical history and physical.

In other words, the chief complaint is never after the HPI, which is always before the PMH. They may omit things, but they won’t give you lab results in the first sentence unless it were the chief complaint. Nor will they give you the physical exam before they tell you the PMH.

9. The NBME Knows If You’ve Shared Answers

The NBME details the statistical tests to evaluate question validity. Here is a non-exhaustive list of some of the most important things they look for:

The number of students who get a question correct

How well an item discriminates between high-scorers and low-scorers

The change in people getting an item correct over time

Have you ever wondered how the NBME prevents cheating? For one, they have strict test-taking guidelines for everyone who takes the test. (Fingerprinting, turning out pockets, etc.)

Second, they look at the CHANGE in percentage correct over time. In other words, what if 30% of test-takers were getting an item one month, then 70% start getting it right the next month? The NBME will know something changed. Here are their words:

“A big change in the P-value or discrimination for an item over time…could indicate that the item has become “exposed” (known beforehand to test-takers).”

Translation: how would the NBME know if students had divulged test items? More otherwise low-scorers would start getting those items correct.

For such a high-stakes exam, people will always try and get ahead (read: cheat). First Aid started at Yale School of Medicine. The rumor is that the authors asked students what had been on their test, and published the answers. I’d also be shocked if QBank writers didn’t “accidentally” find out what kinds of questions were on the exam.

The NBME’s statistical analysis suggests they have a simple and effective counter-measure against cheating.

Concluding Thoughts

What is my top take-away from the 100+ pages of NBME question writing instructions?

Apply important concepts. Don’t memorize minutiae.

The other theme that stuck out is that the NBME is trying hard to be fair.

They’re not trying to trick you. The NBME shuns ambiguity. The questions should be training-level appropriate. A group of content experts should be able to cover the options and agree on the answer.

We’ve all heard med students complain, “the USMLEs are worthless. There is no way this will help me become a better doctor.” (Note: in my experience, it’s almost always people disappointed with their score who claim this.)

I’ve taken these tests and dissected every NBME practice question. Now having read the rationale behind the USMLEs, I can see how remarkable a product the exams are.

Are there random genetic diseases that you’ll never see in your clinical practice? For sure. Are they proportionally rare on the exam? You bet!

The NBME wants you to interpret and apply clinical data.

Sounds like a noble goal to me.

What are your thoughts? Let us know in the comments!

Reference:

Paniagua,M.A., & Swygert,K.A.(Eds.). (2016). Constructing Written Test Questions for the Basic and Clinical Sciences (4thed.). The National Board of Medical Examiners.Retrieved from https://www.nbme.org/publications/item-writing-manual.html

Photo by Kristina Flour