The Data Pipeline Behind Every Page
From a raw NPPES export to an instant CSV download: how we ingest, normalize, materialize, and cache 405,211 verified dental provider records, last refreshed September 23, 2026.
The Pipeline, Stage by Stage
The raw NPPES export loads into a staging table via a bulk COPY - built to handle a full national export of tens of millions of source rows without holding it all in memory at once.
Staged rows are matched to standardized state and city entities, and NPPES taxonomy codes are rolled up through a curated mapping into one public specialty per provider - verified against a real NPPES export, not assumed from the taxonomy spec.
Every state, city, and specialty combination's counts are pre-computed into a dedicated statistics table right after each load - so browsing a page never triggers a live count over the full provider table.
Rendered pages are cached for 6-24 hours depending on type, and every cache entry touched by a refresh is invalidated as part of that same refresh - so a visitor never sees numbers older than the last load.
Source
Every record originates from the NPPES NPI Registry, the authoritative federal database maintained by CMS (the Centers for Medicare & Medicaid Services) under the National Provider Identifier program - practice addresses, phone/fax, specialty taxonomy, and active status all come from that single public source.
Normalization
Raw source rows are loaded one-per-(NPI, taxonomy) pair - a provider enrolled under several taxonomies (roughly one in nine are) appears as multiple source rows sharing one NPI. Those get collapsed into one identity record per provider plus a full set of taxonomy links, so a provider's complete specialty history is preserved even though their public-facing pages show a single representative classification.
Materialized Statistics
Nearly every count on this site - state totals, city breakdowns, specialty coverage - reads from a single materialized statistics table recomputed after each load, not a live query against the full provider table. This is a deliberate performance choice: a live count filtered by a common attribute (a specialty most providers share, for example) can fool a query planner into a badly slow execution plan at this scale, and pre-computing the aggregate sidesteps that risk entirely rather than trying to out-tune it query by query.
Query Planner Maintenance
A bulk load this size (hundreds of thousands of rows, replaced in place) leaves the database's query-planner statistics stale until routine maintenance catches up - which can take longer than the gap between two refreshes. Rather than rely on that timing, we explicitly refresh planner statistics on the affected tables immediately after every load, so query performance stays consistent from the first request after a refresh, not just eventually.
Refresh Cadence & Deactivation Handling
The full export is reprocessed on a regular cadence. A provider whose NPI shows a deactivation date with no later reactivation is marked inactive and drops out of every public count - inactive records are retained internally for audit purposes but never counted or published.
Caching & Freshness
Two independent cache layers sit in front of the database: a full rendered-page cache (so a repeat visit to a popular page skips template rendering entirely) and a query-level cache for smaller, frequently-reused lookups. Both are invalidated for every state, city, and specialty affected by a refresh, so a published page never serves stale counts for the length of an entire cache window.
Delivery
A purchased CSV is generated on demand at download time, not snapshotted when you pay - a download within your link's validity window reflects whatever the site currently shows, including any refresh since your purchase. Exports stream in fixed-size batches rather than paging through the full table, so even a nationwide, 400K+ record export starts downloading immediately instead of waiting to build in memory first.
Frequently Asked Questions
How fast is data delivered after purchase?
Instantly. CSV exports are generated on demand from current data, streamed directly to your browser - no waiting on a manual export or a support ticket, even for a nationwide-scale file.
Does a refresh cause any downtime?
No. Refreshes load into staging tables and swap in atomically, so the site keeps serving the previous data set right up until the new one is ready - visitors never see a partial or empty page mid-refresh.
How do you keep query performance consistent at this scale?
Every page you browse reads from materialized, pre-aggregated counts rather than scanning the full provider table live - and after every bulk load we refresh the database's query-planner statistics, so a common attribute (a specialty most providers share, for example) can't silently degrade a query's performance.
How fresh is the data I'd be buying?
A CSV download is generated fresh from whatever the site currently shows at the moment you download it - not a snapshot frozen at purchase time - so a download within your link's validity window always reflects the latest refresh.
Is any of this data cached in a way that could go stale?
Rendered pages are cached briefly (6-24 hours depending on page type) purely for speed, and every cache entry touched by a refresh is invalidated immediately as part of that refresh - so cached pages never outlive the data refresh that superseded them.