Skip to content

Writing Adapters

BibMedEd uses a plug-and-play adapter pattern. Each data source (PubMed, OpenAlex, Scopus, IEEE, etc.) is a single Python file that implements the BaseSourceAdapter interface. Adding a new source requires no changes to the core application.

The Contract

Every adapter must implement two methods and declare three class attributes:

from app.adapters.base import BaseSourceAdapter, RawRecord, SearchResponse

class MyAdapter(BaseSourceAdapter):
    name = "mysource"              # unique identifier
    display_name = "My Source"     # shown in the UI
    requires_api_key = False       # if True, UI shows "(API key)" badge

    async def search(self, query: str, **kwargs) -> SearchResponse:
        """Search the source, return total count and a list of record IDs."""
        ...

    async def fetch(self, ids: list[str]) -> list[RawRecord]:
        """Fetch full records by ID, return as RawRecords."""
        ...

Drop the file into app/adapters/ and restart the worker. The registry auto-discovers it.

RawRecord Reference

Every adapter maps its source data into RawRecord — the universal format that the rest of the pipeline understands:

Field Type Required Description
source_id str Yes The source's native ID (e.g., PMID, OpenAlex ID)
source_database str Yes Adapter name (e.g., "pubmed", "openalex")
title str Yes Paper title
abstract str \| None No Abstract text
doi str \| None No DOI without URL prefix
year int \| None No Publication year
journal_name str \| None No Journal display name
journal_issn str \| None No ISSN
publication_type str \| None No e.g., "article", "review"
authors list[RawAuthor] No Author list with name, orcid, affiliation
mesh_terms list[str] No MeSH terms (PubMed-specific, empty for other sources)
keywords list[str] No Author keywords
references list[str] No IDs of referenced works
external_ids dict[str, str] No Cross-database IDs for deduplication

external_ids Convention

This is how cross-database deduplication works. Populate every ID you know about:

external_ids = {
    "openalex": "W123456",
    "pmid": "32064006",
    "doi": "10.1016/j.example.2024.01.001",
}

The dedup engine matches on DOI first, then PMID. The more IDs you provide, the better dedup works.

Optional Overrides

For sources with cursor-based pagination or large datasets, override these defaults:

search_paginated()

By default, yields all IDs from a single search() call. Override for cursor-based APIs:

async def search_paginated(self, query: str, **kwargs) -> AsyncGenerator[list[str], None]:
    cursor = "*"
    while cursor:
        resp = await self._client.get("/works", params={"search": query, "cursor": cursor})
        data = resp.json()
        yield [r["id"] for r in data["results"]]
        cursor = data.get("next_cursor")

fetch_stream()

By default, slices IDs into batches of 200 and calls fetch() per batch. Override if your API has native batch/streaming support.

methodology_label()

Returns "{display_name} API" by default. Override for a more descriptive label in the methodology log:

def methodology_label(self) -> str:
    return "PubMed E-Utilities"

Walkthrough: OpenAlex Adapter

The shipped OpenAlex adapter (app/adapters/openalex.py) is a complete real-world example. Key patterns:

  1. search() hits /works?search=... and returns the first page of IDs + total count
  2. search_paginated() overrides the default to use OpenAlex cursor pagination
  3. fetch() uses pipe-separated filters to batch-fetch records
  4. _to_raw() maps OpenAlex JSON to RawRecord, including:
  5. Extracting cross-database IDs from the ids object
  6. Reconstructing abstracts from OpenAlex's inverted index format
  7. Parsing author affiliations from nested institution objects

Read the full source: app/adapters/openalex.py

Walkthrough: CrossRef Adapter

The CrossRef adapter (app/adapters/crossref.py) shows the same pattern applied to a metadata-only source that's the canonical DOI resolver. CrossRef contributes huge cross-source dedup leverage because most other adapters carry a DOI but few normalise it consistently.

Patterns worth noting:

  1. Polite pool via mailto — CrossRef ranks polite-pool requests above anonymous ones. The adapter accepts an email constructor arg and appends mailto= to every request when set.
  2. Cursor pagination via cursor=* — search_paginated() walks next-cursor exactly as OpenAlex does; CrossRef's deep-paging API uses the same idiom.
  3. DOI normalisation — _to_raw() lower-cases the DOI both as source_id storage convention and in external_ids["doi"] so cross-source dedup matches the same DOI emitted by PubMed or OpenAlex regardless of case.
  4. Multi-fallback year extraction — published-print → published-online → issued → created. CrossRef is inconsistent about which key is populated for a given record.
  5. References get the same lowercase treatment — references list is populated from reference[].DOI and lowercased, so downstream citation analyses can cross-link to other CrossRef records without case mismatches.

Read the full source: app/adapters/crossref.py — and the tests at tests/test_adapters_crossref.py demonstrate the fixture-driven testing convention every new adapter PR should follow.

Walkthrough: Semantic Scholar Adapter

The Semantic Scholar adapter (app/adapters/semantic_scholar.py) shows the Graph API v1 pattern, including its citation graph and an optional-but-not-required API key.

Patterns worth noting:

  1. No key required, key raises the ceiling — requires_api_key = False. The free anonymous tier allows roughly 100 requests per 5 minutes per IP; passing an api_key constructor arg sets the x-api-key header and moves the adapter to the keyed tier (1 request/sec sustained), which is more predictable for large batch fetches even though its instantaneous ceiling is lower than an anonymous burst.
  2. Offset pagination with an inclusive hard cap — search_paginated() walks the offset/next fields returned by /paper/search. The upstream API caps offset at 9999 inclusive; the adapter stops once the next offset would exceed that cap rather than erroring, and search() still reports the upstream total count so callers can see when a query was capped short of the true result size.
  3. Batched detail fetches — fetch() calls /paper/batch, which accepts at most 500 IDs per request; the adapter chunks arbitrary-length ID lists into _BATCH_LIMIT-sized calls and skips any batch entry that comes back None (an ID the API couldn't resolve) instead of failing the whole request.
  4. DOI normalisation, same convention as CrossRef — _to_raw() lower-cases externalIds.DOI before storing it as doi and in external_ids["doi"], so dedup matches records from PubMed, OpenAlex, or CrossRef regardless of case. references[].externalIds.DOI gets the same lowercase treatment.
  5. external_ids populated with doi, pmid, and semantic_scholar — the paperId is stored under external_ids["semantic_scholar"] and externalIds.PubMed (when present) is mapped to external_ids["pmid"], giving this adapter three cross-reference keys instead of one.
  6. Journal metadata has two possible shapes — S2 records carry journal info in either a free-text journal object or a canonical publicationVenue object depending on the paper; _to_raw() falls back from one to the other for both name and ISSN.
  7. fieldsOfStudy doubles as keywords — Semantic Scholar doesn't expose author keywords, so the adapter maps its fieldsOfStudy list into RawRecord.keywords as the closest available signal. mesh_terms=[] is passed explicitly, same as every non-PubMed adapter.

Read the full source: app/adapters/semantic_scholar.py — and the tests at tests/test_adapters_semantic_scholar.py.

Step-by-Step: Adding a New Source

  1. Create app/adapters/mysource.py
  2. Implement search() and fetch() mapping to RawRecord
  3. Populate external_ids with every cross-reference ID the API provides
  4. Restart the worker: docker compose restart worker
  5. The new source appears in the search page dropdown automatically