Automated Medical Quiz Generation Research: DailyMed, November 2024

The study

This is a study that built a pipeline for automatically generating medical multiple-choice quizzes. The paper appeared on medRxiv overnight.

paragraphs

Generation and review pipeline

The workflow is more detailed than simply asking a chatbot to create a good question.

• Topic generation.
• Question generation.
• Literature search and comparison against papers.
• Difficulty assessment. Questions classified as easy or medium undergo work to increase their difficulty.
• Improvement work. Three improvement plans are created, one is selected, and it is applied.
• Validation of the question, answer choices, hint, and explanation. If the item does not pass, the process restarts from question generation; if it passes, it is stored in a database.
• Human expert review. After revisions, the final version is saved.

Evaluation and cost

The study used five evaluation criteria, which may be useful to refer back to later. Among OpenBioLLM-70B, GPT-4o, and Sonnet 3.5, GPT-4o performed best, receiving an average rating of 4.6 out of 5. This suggests that the resulting quality was sufficient.

Using GPT-4o, question generation cost 51 cents per item and took 16 seconds.

My thoughts

Many people may imagine that this can be done by simply telling ChatGPT, “Make a great question!” But practical results require a detailed pipeline like this, including iterative improvement and review processes.

I had wanted to try this kind of research this year, so I feel a little envious. I wonder whether Claude 3.6 is any better.

Links

Paper: https://www.medrxiv.org/content/10.1101/2024.11.11.24317087

The DailyMed website is also public, although registration is currently closed: https://dailymed.ai