Skip to content

Getting the data

Before any transformation, acquisition. The mechanism you use shapes what you can build, and the constraints are as often organizational as technical.

Bulk FHIR

The FHIR REST API is designed to retrieve resources for one person at a time. For a research data set covering a population, that is the wrong access pattern, and the Bulk Data Access specification exists to solve it.

How it works

You invoke an $export operation at one of three levels:

Level Endpoint Returns
System /$export Everything the server will give you
Patient /Patient/$export All resources for all patients
Group /Group/[id]/$export All resources for a defined cohort

Group-level export is usually what a research project wants, because your data use agreement almost certainly covers a defined population rather than the whole system.

The operation is asynchronous. You request, receive a status endpoint, poll it, and eventually receive a manifest of file URLs. Each file is NDJSON: newline-delimited JSON, one resource per line, one file per resource type. You download the files, usually with the same authorization, and you have your source data.

Useful parameters: _type to limit resource types, _since for incremental extraction, and _typeFilter for finer selection where supported.

What varies in practice

The specification is clear and implementations differ substantially. Ask these questions early, because the answers determine your project plan.

  • Which resource types are actually exported? Support is uneven, particularly for Coverage, Device, Specimen, and DocumentReference.
  • Is _since supported and correct? Incremental extraction depends on it. Some servers accept the parameter and return everything.
  • How long does a full export take? Hours is common. Days happens.
  • What are the size limits? Some servers cap the export or the file count.
  • Is the export a point-in-time snapshot or is it consistent? A long-running export against a live system can produce internally inconsistent data.
  • How is authorization handled? SMART Backend Services with asymmetric client authentication is the usual pattern, and it requires key registration with the server operator, which is often the slowest step in the whole project.
  • Does the export include deleted or amended resources? This determines whether your incremental strategy can work at all.

Do a small export first

Request a Group export for twenty people before requesting one for two hundred thousand. You will learn what is supported, what is populated, and what your parser breaks on, in an afternoon rather than a month.

The alternatives

Bulk FHIR is not always available or appropriate.

Per-person REST queries. Workable for small cohorts, particularly in prospective studies where enrollment is gradual. Rate limits make it impractical above a few thousand people.

Subscriptions. FHIR Subscription notifies you when data changes. Useful for keeping an instance current after an initial load, and it requires infrastructure to receive and process notifications reliably.

Direct database extraction. Many organizations building an OMOP instance from their own EHR go straight to the underlying database rather than through FHIR. It is often faster and gives access to content the FHIR interface does not expose.

The tradeoff is real, and stating it plainly helps: direct extraction gives you more data and a pipeline that is specific to one vendor, one version, and often one installation. A FHIR-based pipeline gives you less data and a design that transfers. Which is right depends on whether you are building for one site or for a network.

A hybrid is common and sensible: FHIR for the core clinical domains where it is well supported, direct extraction for the gaps.

Patient-mediated exchange. Where research participants authorize retrieval of their own records, individual authorization can gather data across many organizations without institution-by-institution agreements. This produces FHIR data with a different shape, since it is deep on individuals and drawn from many sources, and identity resolution across sources becomes a central problem rather than an edge case.

Before you request anything

Four questions that belong in the project plan and are frequently deferred until they block progress.

What population, defined how? Group export needs a Group, and someone has to construct it. The construction rule is a study design decision.

What resource types, and why? Requesting everything is tempting and produces volume you will not use, along with governance exposure you did not need. Requesting too little means a second export cycle.

What is the legal basis and what does it permit? A data use agreement that permits specific resource types for a specific purpose is different from broad authorization. This shapes the technical request.

What happens to identifiers? Whether direct identifiers arrive at all, and where the de-identification boundary sits, needs to be decided before data moves rather than after.

Practical notes on NDJSON

Files are large and lines are independent. Stream them. Loading a multi-gigabyte NDJSON file into memory as a JSON array will fail, and the failure mode is unhelpful.

One file per resource type is the common pattern, not a guarantee. Some servers split by size and emit several files per type.

Keep the raw files. They are your audit trail and your ability to reprocess without re-exporting. Storage is cheaper than another export cycle and much cheaper than another round of approvals.

Record the manifest. It tells you what you received, when, and from what request. Store it alongside the data.

Validate a sample before full processing. Run resources through a FHIR validator against the profiles the server claims to support. Errors found here are much cheaper than errors found after loading.

Profiling before designing

The single highest-value activity between acquisition and transformation is profiling. Before writing mapping code, answer these against your actual data.

  • How many resources of each type?
  • For each resource type, what proportion of each element you plan to use is actually populated?
  • What code systems appear in each coded field, and in what proportion?
  • What is the distribution of date precision? How many dates are year-only?
  • What status values appear, and how often?
  • How many distinct Encounter classes, and does partOf appear at all?
  • How many distinct source code systems appear that are not in the OMOP vocabularies?
  • What proportion of references resolve within the export?

This is a day or two of work and it prevents weeks of rework, because it converts your transformation design from a specification-based guess into a data-based decision. It also produces the numbers you will need later for your ETL documentation.

Next