Clone — Enrichment reference data¶

The clone_* DAGs import public reference datasets into the data warehouse. Unlike source DAGs, which ingest actor lists shared by partners, clone DAGs feed reference data used to enrich, validate, or correct actors already present on the platform.

See also: Sources and reference data, Airflow, dbt, Enrich SIRET/SIREN.

Business context¶

The platform cross-references the actors it hosts with trusted external datasets:

Reference data	Example business uses
Annuaire Entreprises (AE)	Verify a SIREN/SIRET, detect a closed establishment, suggest a replacement via succession links
BAN	Correct a city or address, geocode
La Poste / INSEE / Koumoul	Match a postal code to a municipality, attach an actor to an EPCI
Contours administratifs	Geographic reference data (municipalities, departments, regions, EPCIs)

This data only changes when the official source publishes a new version. There is no reason to reprocess it on every enrichment run: it should be refreshed at clone cadence, not at suggestion cadence.

General workflow¶

Every clone_* DAG follows the same business chain:

        flowchart LR
    A[Download source] --> B[Load into DB]
    B --> C[Validate quality]
    C --> D[Switch in_use view]
    D --> E[Remove old tables]
    E --> F[Normalize via dbt]

1. Import a new version¶

Data is downloaded from the official URL (data.gouv, INSEE, La Poste, etc.) and loaded into a timestamped table (clone_<kind>_YYYYMMDDHHMMSS). This table holds the raw version of the reference dataset.

2. Validation¶

SQL checks verify that the imported table is usable (consistent volume, required fields present, etc.). On failure, the table is dropped and the DAG stops: the previous version remains in service.

3. Switch the `*_in_use` view¶

Downstream consumers (dbt, enrichment DAGs) never read the timestamped table directly. They go through a stable view clone_<kind>_in_use that points to the latest validated version.

After a successful clone, the view is recreated to point to the new table. dbt models and enrichment DAGs therefore keep referencing a fixed name, with no service interruption.

4. Cleanup¶

Older timestamped tables for the same reference dataset are dropped to limit disk usage.

5. dbt normalization (depending on the dataset)¶

For some clones, a dbt step runs immediately after the switch. It turns raw data into ready-to-use intermediate models (cleaning, typing, indexes, preparatory joins). See the next section.

dbt normalization: « clone first, enrich second »¶

Principle: dbt preparation of a reference dataset is triggered when that dataset changes, i.e. at the end of the relevant clone DAG. Enrichment DAGs only need to refresh their own marts models (tag:… selector without +), assuming base and intermediate layers are already up to date.

Best practice: tag models with `normalisation`¶

Systematically add the normalisation tag to every dbt model that must be recalculated after a clone. Clone DAGs select these models with a combined selector: tag:<domain>,tag:normalisation (for example tag:geo,tag:normalisation or tag:ban,tag:normalisation).

Reference data normalized at clone time¶

Clone DAG	dbt models prepared (base + intermediate layers)
`clone_ae_unite_legale`	Annuaire Entreprises `unite_legale` chain
`clone_ae_etablissement`	`etablissement` chain (+ models tagged `normalisation`)
`clone_ae_lien_succession`	`lien_succession` chain (successors of closed establishments)
`clone_insee_commune`	Geographic models (`tag:geo`)
`clone_koumoul_epci`	Geographic models (`tag:geo`), including `int_epci`
`clone_laposte_codes_postaux`	Geographic models (`tag:geo`)

The three geographic clones (INSEE, Koumoul, La Poste) trigger the same tag:geo scope. This is intentional: the run is lightweight and keeps municipalities, postal codes, and EPCIs consistent whenever one of these datasets is updated.

Clones without immediate normalization¶

Other clones only import raw data; dbt normalization happens elsewhere:

Clone DAG	dbt normalization
`clone_ban_adresses`, `clone_ban_lieux_dits`	Daily `enrich_dbt_models_refresh` DAG (`+tag:intermediate,tag:ban`)
`clone_ca_*` (contours administratifs)	Raw geographic data, consumed directly or via other models

Impact on enrichment DAGs¶

enrich_acteurs_closed, enrich_acteurs_rgpd, enrich_acteurs_villes, etc. no longer recalculate Annuaire Entreprises or geographic intermediate models. They only refresh their suggestion marts models, for example:

enrich_acteurs_closed → tag:marts,tag:enrich,tag:closed
enrich_acteurs_rgpd → tag:marts,tag:enrich,tag:rgpd
enrich_acteurs_villes → tag:marts,tag:enrich,tag:villes

enrich_siret_siren keeps a +tag:siren_siret selector because its models combine both AE reference data and platform actors.

DAG inventory¶

DAG	Source	Schedule	dbt normalization at clone
`clone_ae_unite_legale`	data.gouv (SIRENE stock)	1st of month, 01:00	Yes
`clone_ae_etablissement`	data.gouv (SIRENE stock)	1st of month, 02:00	Yes
`clone_ae_lien_succession`	data.gouv (SIRENE stock)	1st of month, 03:00	Yes
`clone_ban_adresses`	adresse.data.gouv.fr	1st of month, 04:00	No (via `enrich_dbt_models_refresh`)
`clone_ban_lieux_dits`	adresse.data.gouv.fr	1st of month, 05:00	No (via `enrich_dbt_models_refresh`)
`clone_insee_commune`	INSEE (COG)	Sunday, 00:00	Yes (`tag:geo`)
`clone_ca_commune`	Contours administratifs	Sunday, 01:00	No
`clone_ca_commune_associee_deleguee`	Contours administratifs	Sunday, 01:00	No
`clone_ca_departement`	Contours administratifs	Sunday, 01:00	No
`clone_ca_epci`	Contours administratifs	Sunday, 01:00	No
`clone_ca_region`	Contours administratifs	Sunday, 01:00	No
`clone_koumoul_epci`	Koumoul	Sunday, 01:00	Yes (`tag:geo`)
`clone_laposte_codes_postaux`	La Poste (datanova)	Sunday, 02:00	Yes (`tag:geo`)

Source and usage details: Sources and reference data.