Automating MeSH Keyword Extraction from Abstracts

Why I Built It

Medical journals commonly require authors to provide three to six MeSH keywords when submitting a paper. Many journals do not seem to check strictly whether the supplied terms are genuine MeSH keywords, but finding them carefully still takes more time and attention than expected.

Since I started working with LLMs, I have repeatedly wanted to automate this process. The pattern was always the same: “Finding keywords is annoying; should I build a feature?” followed by, “No, I’ll do it manually just this once and build it later when I have time.”

I finally completed it. Looking through the project folder, I found that I had downloaded and organized MeSH keywords in April 2025. After resuming the work nine months later, I succeeded this time.

• Tool: https://huggingface.co/spaces/sangzinahn/mesh-keywords-from-abstract
• The API cost is approximately KRW 1 per query. I decided to make it freely available and cover that cost myself.

Project Overview

MeSH (Medical Subject Headings) is a standardized medical thesaurus maintained by the U.S. National Library of Medicine and used to index PubMed papers. The 2026 vocabulary contains 31,110 descriptors and approximately 267,000 synonyms, known as entry terms.

Selecting suitable terms improves a paper’s discoverability, but finding accurate keywords in such a large vocabulary requires substantial time and domain knowledge. The project goal was therefore to recommend relevant MeSH terms from an abstract alone, maximize Recall@20—the proportion of correct terms among the top 20 recommendations—and build a lightweight system suitable for real-time web deployment.

• Initial Recall@20: 36%
• Final Recall@20: 64.4%
• Improvement: 79%
• System size: reduced from roughly 1.5 GB to approximately 15 MB
• Final API usage: two calls per request, at approximately $0.002 per request

Data Preparation

The source data were the 2026 MeSH XML files, specifically desc2026.xml, which was 312 MB and contained 31,110 descriptors. Each record included a unique UI identifier, preferred term, definition or ScopeNote, tree numbers, and synonyms.

Definitions were missing for 150 of the 31,110 records, mostly names of chemical groups. These were completed using GPT-5-mini with a 20-shot hybrid prompting approach: 10 contextual neighboring examples and 10 alphabetically distant examples. All 150 missing definitions were completed at a cost of approximately $0.22.

For example, for “Adenine Nucleotides,” the prompt was: “You are a medical lexicographer. Define: Adenine Nucleotides.” The resulting definition described nucleotides containing adenine as the nitrogenous base, including AMP, ADP, and ATP. The generated definitions were judged to be at a medical-expert level.

Experiment 1: What to Embed

The first experiment compared embedding only the MeSH term name with embedding the term name plus its definition. The embedding model was text-embedding-3-small, using 1,536-dimensional vectors.

Strategy A used only the term name, such as “Diabetes Mellitus, Type 2.” Strategy B used the term and definition, such as “Diabetes Mellitus, Type 2: A subclass of diabetes mellitus...” The initial benchmark included four papers: two clinical trials and two papers authored by me.

Term-name-only embeddings performed better: average Recall@20 was 36%, compared with 25% for term-plus-definition embeddings. Average MRR was 1.00 for Strategy A and 0.80 for Strategy B, making Strategy A 44% better on Recall@20.

Definitions were occasionally helpful. In UKPDS 34, for example, the definition’s reference to “sulphonylurea” helped detect chlorpropamide because the same term appeared in the abstract. However, definitions diluted the signal for high-level concepts. In an AI competency systematic review, Strategy A found Physicians, Ethics, Medical, while Strategy B did not.

I therefore selected term-name-only embeddings. Although definitions could assist with specific drug names, they reduced performance in general topic-term retrieval. Generating embeddings for both strategies cost approximately $0.15.

Experiment 2: Sentence-Level Embeddings

The next question was whether embedding each sentence separately would detect specific entities more effectively than representing an entire abstract as one vector. Sentences were split with regular expressions designed to handle abbreviations such as “Dr.” and “vs.” For each MeSH term, the highest similarity score across all abstract sentences was used.

The benchmark was expanded from four to 10 papers, including five papers authored by me. Whole-abstract embedding achieved an average Recall@20 of 38%, while sentence-level embedding achieved 41%, an improvement of 8.5%.

Sentence-level processing successfully detected Methimazole in an ethionamide pharmacokinetic modeling paper through the sentence “coadministered with methimazole”; Biocompatible Materials in a dental implant paper through “various biomaterials”; and Drug Induced Liver Injury in a mitochondrial DNA-DILI paper through “drug-induced liver injury.”

It also had limitations. Ethics, Medical was weak because the ethical concept was implied across the full context rather than strongly expressed in a single sentence. Pharmacokinetics was missed when the abstract used only the abbreviation “PK.” Despite this trade-off, sentence-level embedding was adopted because it was better at detecting specific entities.

Experiment 3: Dense and Sparse Hybrid Retrieval

I then combined dense vector retrieval with sparse BM25 retrieval. All 267,000 entry terms were embedded. Dense retrieval used sentence-level maximum-score aggregation, while BM25 was applied to tokenized entry terms. The combined score was calculated as 0.7 × Dense + 0.3 × BM25.

Embedding the entry terms cost approximately $0.03, but produced 267,000 vectors requiring 1.5 GB of storage. The hybrid method achieved average Recall@20 of 39%, lower than the 41% obtained with sentence-level embeddings alone. Its average MRR was 0.73.

The hybrid approach helped certain terms. For “Education, Medical, Undergraduate,” it improved the ranking from 186th to 61st. However, noise introduced by synonyms reduced overall recall. I rejected this strategy.

Experiment 4: LLM-Based Candidate Generation

To overcome the limitations of embedding retrieval, I asked GPT-5-nano to propose 20 MeSH terms and matched its proposed terms to the real MeSH vocabulary using fuzzy matching with difflib. I also tested reciprocal rank fusion (RRF) of the LLM and embedding rankings.

Embedding alone achieved average Recall@20 of 41.2%. The LLM alone reached 60.6%, a 47% improvement. The RRF hybrid reached 57.1%, which was better than embeddings but worse than the LLM alone.

The LLM uniquely found chlorpropamide and insulin for UKPDS 34; diabetes mellitus, antitubercular agents, and aged for a tuberculosis pharmacokinetics paper; machine learning for a sepsis AI paper; titanium and surface properties for a dental implant paper; and organic anion transporters and cohort studies for a rifampicin pharmacokinetics paper.

The embedding results added noise and diluted the RRF ranking, while LLM candidates were already sufficiently comprehensive. One GPT-5-nano call cost approximately $0.001, so the LLM-only approach was selected.

Experiment 5: Combining the LLM with Direct Extraction

The LLM could still miss terms explicitly present in an abstract, such as “Large Language Models.” To address this, I extracted one- to five-word n-grams directly from the abstract and performed exact matching against the MeSH vocabulary. The vocabulary was expanded to include all 267,000 entry terms, allowing matches such as “Large Language Model” to resolve to “Large Language Models.”

LLM-only recommendation achieved average Recall@20 of 60.7%. Direct extraction alone achieved 36.1%. Combining the two reached 64.4%, a 6.2% improvement over the LLM alone.

Direct extraction added chlorpropamide for UKPDS 34; headache and fatigue for a COVID-19 vaccine paper; and systematic review for an AI competency review. Because this matching runs locally, it provided the additional improvement without API cost.

Final Optimization and System Design

Directly extracted terms were boosted in ranking. Terms found by both methods received a score of 2.5, directly extracted terms a score of 1.5, and LLM-only terms a score of 1.0. This moved “Large Language Models” from rank 25 to rank 7.

Substring duplicates were removed when a more specific term was available. For example, when “Education, Medical, Undergraduate” was present, “Education, Medical” was removed. An optional LLM reranking step then sent the top 30 candidates back to the LLM for a more precise top-10 selection. This moved “Large Language Models” from rank 7 to rank 1 and removed “Abstracts,” which was noise.

Section headers such as Abstract, Purpose, and Methods are automatically removed before analysis. The final pipeline removes headers, generates 20 LLM candidates in parallel with n-gram extraction, matches candidates to the MeSH vocabulary, merges and deduplicates them, applies source-based scoring, optionally reranks the top candidates, and returns the top 20 recommendations.

Lightweight Web Service

Embedding-based retrieval was fully removed from the final system. The unnecessary embedding files were mesh_embeddings_v2.pkl at 184 MB and mesh_entry_term_embeddings.pkl at 793 MB. The necessary vocabulary file, mesh_descriptors_imputed.csv, is 14 MB. Together with a 13 KB Gradio app and a 100-byte requirements file, the system is approximately 15 MB.

The web service uses Gradio 6.0, is hosted free on Hugging Face Spaces, and uses the OpenAI GPT-5-nano API. Users can paste an abstract of up to 2,000 characters, view the top 10 terms immediately, expand the list to 20, open each term in the NLM MeSH Browser, view a definition of up to 200 characters, distinguish sources with [llm], [direct], and [both] badges, copy results, or download them as CSV.

The API key is stored as an environment variable rather than exposed in the code. The service also has a 2,000-character input limit and uses Gradio’s built-in rate limiting.

• Deployment URL: https://huggingface.co/spaces/sangzinahn/mesh-keywords-from-abstract
• Expected response time: approximately 2 seconds

Costs and Future Work

One-time development costs were approximately $5.40: $0.22 for definition completion, $0.15 for descriptor embeddings, $0.03 for entry-term embeddings, and approximately $5.00 for experiments and testing. In operation, the basic LLM recommendation requires one API call at approximately $0.001, while reranking uses two calls at approximately $0.002. At 1,000 requests per month, this would be approximately $2.00.

The next steps are to evaluate the system on a larger benchmark of more than 100 papers, support Korean abstracts through a translation layer, collect user feedback to improve recommendation accuracy, and automatically incorporate annual MeSH updates.