scientific-literature-search

Scientific Literature Search

Overview

Scientific literature search is the foundation of evidence-based research. A well-executed search maximizes recall (finding all relevant papers) while maintaining precision (avoiding irrelevant results). This guide provides a systematic approach that combines database-specific query strategies, AI-assisted synthesis, and direct content extraction, organized into a three-tiered framework that scales from targeted lookups to comprehensive landscape reviews.

Key Concepts

The PICO Framework

For clinical and biomedical questions, structure queries using the PICO framework:

• P (Population): Who are you studying? (e.g., "Diabetes Mellitus"[MeSH])
• I (Intervention): What treatment or exposure? (e.g., "Metformin"[MeSH])
• C (Comparison): What is the alternative? (e.g., placebo, standard care)
• O (Outcome): What result are you measuring? (e.g., "Cardiovascular Diseases"[MeSH])

PICO queries can be combined with publication type filters to target specific evidence levels:

"Diabetes Mellitus"[MeSH] AND "Metformin"[MeSH] AND "Cardiovascular Diseases"[MeSH] AND ("clinical trial"[Publication Type] OR "meta-analysis"[Publication Type])

Three-Tiered Search Strategy

Literature search is most effective when approached in tiers of increasing breadth:

Tier 1 -- Database-Specific Searches (Most Reliable)

Query established academic databases (PubMed, arXiv, Google Scholar) for peer-reviewed, indexed content. This is the most reliable tier and should always be the starting point.

• PubMed (via Biopython Bio.Entrez): Primary database for biomedical and life science literature. Supports MeSH controlled vocabulary and advanced field tags.
• arXiv (via the arxiv package): Preprint server for physics, mathematics, computer science, and quantitative biology. Results appear faster than peer-reviewed journals.
• Google Scholar (via the scholarly package): Broadest coverage across all academic disciplines. Note: has aggressive rate limits on automated queries.

Best for: finding specific papers, systematic reviews, clinical evidence, preprints.

Tier 2 -- AI-Assisted Web Search (Comprehensive)

Use the Claude API with the web_search_20250305 server-side tool to synthesize broader context, identify research trends, and surface recent developments not yet indexed in databases. Also use general web search (e.g. via the duckduckgo-search package) for protocols, tutorials, and software documentation.

Best for: understanding the research landscape, complex multi-faceted questions, finding recent developments, identifying key researchers.

Avoid for: specific paper lookups (use Tier 1), citation counts (use Google Scholar), systematic reviews requiring reproducibility, searches where exact query terms must be documented.

Tier 3 -- Direct Content Extraction (Deep Dive)

Extract and analyze full-text content, PDFs, and supplementary materials from identified papers using trafilatura (HTML article extraction), pypdf (PDF text), and the Crossref API (DOI → supplementary file URLs).

Best for: detailed methodology extraction, data retrieval, protocol identification, supplementary data access.

PubMed Field Tags

PubMed supports field-specific searching to improve precision:

| Tag | Description | Example | |-----|-------------|---------| | [MeSH] | Medical Subject Heading (controlled vocabulary) | "Neoplasms"[MeSH] | | [Title] | Title field only | "CRISPR"[Title] | | [Title/Abstract] | Title or abstract | "gene therapy"[Title/Abstract] | | [Author] | Author name | "Zhang F"[Author] | | [Journal] | Journal name | "Nature"[Journal] | | [Publication Type] | Article type filter | "Review"[Publication Type] | | [Date - Publication] | Publication date range | "2020/01/01"[Date - Publication]:"2024/12/31"[Date - Publication] | | [MeSH Major Topic] | MeSH term as major focus of the article | "CRISPR-Cas Systems"[MeSH Major Topic] |

Boolean Operators

Boolean operators control how search terms combine:

# AND: All terms must be present -- narrows results
results = query_pubmed("CRISPR AND cancer AND therapy")

# OR: Any term can be present -- broadens results (use for synonyms)
results = query_pubmed("(tumor OR tumour OR neoplasm) AND immunotherapy")

# NOT: Exclude terms -- use sparingly to avoid losing relevant papers
results = query_pubmed("cancer immunotherapy NOT review")

Use parentheses to group OR terms together before combining with AND.

arXiv Subject Categories

arXiv organizes preprints by subject category. Biology-related categories include:

| Category | Description | |----------|-------------| | q-bio.BM | Biomolecules | | q-bio.CB | Cell Behavior | | q-bio.GN | Genomics | | q-bio.MN | Molecular Networks | | q-bio.NC | Neurons and Cognition | | q-bio.QM | Quantitative Methods | | cs.AI | Artificial Intelligence | | cs.LG | Machine Learning |

Decision Framework

Use this tree to determine which search tier and database to start with:

What type of question are you answering?
├── Clinical / biomedical question
│   ├── Specific drug or treatment → Tier 1: PubMed with PICO query
│   ├── Disease mechanism → Tier 1: PubMed with MeSH terms
│   └── Clinical trial evidence → Tier 1: PubMed filtered by Publication Type
├── Computational / quantitative methods
│   ├── ML model or algorithm → Tier 1: arXiv (cs.LG, cs.AI)
│   ├── Computational biology method → Tier 1: arXiv (q-bio.*) + PubMed
│   └── Software tool or pipeline → Tier 2: AI-assisted web search
├── Broad research landscape
│   ├── Current state of a field → Tier 2: AI-assisted web search
│   ├── Recent developments (last 6 months) → Tier 2: AI-assisted web search
│   └── Cross-disciplinary question → Tier 1: Google Scholar + Tier 2
├── Specific paper or data
│   ├── Known paper details → Tier 1: any database by title/author/DOI
│   ├── Methodology or protocol → Tier 3: full-text extraction
│   └── Supplementary data → Tier 3: DOI-based supplementary fetch
└── Protocols / reagents
    ├── Lab protocol → Tier 2: web search for protocols.io, etc.
    └── Validated reagents → Tier 2: AI-assisted web search

| Scenario | Recommended Tier and Database | Rationale | |----------|-------------------------------|-----------| | Systematic review of clinical evidence | Tier 1: PubMed with MeSH + publication type filters | Reproducible, documented search strategy required | | Finding a preprint on a new ML method | Tier 1: arXiv with category and keyword search | Preprints appear on arXiv before journals | | Understanding the research landscape | Tier 2: AI-assisted web search | Requires synthesis across many sources | | Extracting a specific protocol from a paper | Tier 3: PDF content extraction | Need full-text access to methods section | | Finding papers across disciplines | Tier 1: Google Scholar | Broadest coverage across fields | | Identifying key researchers in a niche area | Tier 2: AI-assisted web search | Requires contextual synthesis | | Downloading supplementary data tables | Tier 3: DOI-based supplementary fetch | Direct access to supplementary files |

Best Practices

1. Use controlled vocabulary (MeSH) for PubMed searches: Free-text searches miss papers that use different terminology. MeSH terms map synonyms to a single concept, improving recall without sacrificing precision.

``python # Free text misses synonyms query_pubmed("heart attack treatment") # MeSH captures all synonyms query_pubmed('"Myocardial Infarction"[MeSH] AND "Drug Therapy"[MeSH]') ``

1. Include synonyms and alternative terms with OR: Scientific concepts often have multiple names (e.g., tumor/tumour/neoplasm). Group synonyms with OR inside parentheses to avoid missing relevant papers.

``python query_pubmed("(myocardial infarction OR heart attack) AND (treatment OR therapy)") ``

1. **Use phrase searching for multi-word c