🦖 Semantic Similarity Index

🦖 Semantic Similarity Index

Measure how isolated or connected disease research is across the global biomedical literature, using PubMedBERT embeddings on PubMed abstracts spanning 175 GBD diseases.

What it does

1. Takes a disease list (GBD taxonomy) as input
2. Retrieves PubMed abstracts (2000-2025) for each disease with quality filtering
3. Generates 768-dimensional PubMedBERT embeddings for every abstract
4. Computes four semantic equity metrics per disease:
   • Semantic Isolation Index (SII): average cosine distance to k-nearest disease neighbours; higher = more isolated, less connected research
   • Knowledge Transfer Potential (KTP): cross-disease centroid similarity; higher = more potential for research spillover
   • Research Clustering Coefficient (RCC): within-disease embedding variance; higher = more diverse research approaches
   • Temporal Semantic Drift: cosine distance between yearly centroids; measures how research focus evolves
5. Generates publication-quality multi-panel figures:
   • Panel A: Semantic isolation by disease category (boxplot)
   • Panel B: Top 20 most semantically isolated diseases (bar chart, NTD/Global South colour-coded)
   • Panel C: Semantic isolation vs research volume (scatter with regression)
   • Panel D: NTD vs non-NTD significance test (Welch's t-test, Cohen's d)
6. Produces a markdown report with all metrics, rankings, and reproducibility bundle

Why this exists

If you ask ChatGPT to "measure research neglect for diseases," it will:

• Not know which embedding model to use for biomedical text
• Hallucinate metrics that sound plausible but have no methodological grounding
• Skip quality filtering (year coverage, abstract coverage, minimum papers)
• Not handle MPS acceleration or checkpointed batch processing
• Produce a single scatter plot with no disease classification

This skill encodes the correct methodological decisions:

• Uses PubMedBERT (the gold-standard biomedical language model)
• Fetches from PubMed with exponential backoff and NCBI rate limiting
• Quality filters: year coverage >= 70%, abstract coverage >= 95%, minimum 50 papers
• Batch embedding with Apple MPS acceleration and CPU fallback
• Checkpointed processing (resume after interruption)
• HDF5 storage with gzip compression and SHA-256 checksums
• Classification against WHO NTD list and Global South priority diseases
• Statistical significance testing (Welch's t-test, Cohen's d)