Data Platform

Task types

• clone : get partner data to enrich ours (data.gouv, la poste, insee, annuaire-entreprise, ban…)

• source : get partner data to create new acteur (eco-organisme, cma, sinoe…)

• enrich : fix, or add datato existing actor (url, siren, siren, adresse…)

• clustering : group actor which are duplicated - shared by multiple sources

• compute acteur : to be displayed or share

• opendata : compute and share opendata

• stats : compute data quality stats

• tech : cleanup task

Some use dbt, other use pandas

• Airflow
  • Configuration runtime
  • Vue d’ensemble des DAGs
  • File organization
  • Follow the rules
  • Suggestion system
  • Identifier management
  • XCom backend and object storage
  • XCom backend testing checklist
• Ingestion sources and enrichment reference data
  • Actor sources
  • Cloned enrichment reference data
• Clone — Enrichment reference data
  • Business context
  • General workflow
  • dbt normalization: « clone first, enrich second »
  • DAG inventory
• dbt Documentation
  • Model Layers
  • Naming Conventions
  • Project Structure
  • Patterns
  • Makefile Commands
  • dbt Commands
  • Data Flow Example: Stats Stacked Acteurs
  • Best Practices
• Clustering & Deduplication
  • ⚠️ Known limitations
  • 📜 Definition
  • ➡️ State transitions: scenarios
  • 🧪 Algorithm
  • 🚤 Performance
  • 🔀 Schema
• Enrich — Acteurs SIRET & SIREN (enrich_siret_siren)
  • Cohort 1 — SIRET proposed from a known SIREN
  • Cohort 2 — SIREN proposed from a known SIRET
• Enrich — Acteurs fermés (enrich_acteurs_closed)
  • Shared pipeline — candidates and replacements
  • Cohort 1 — Replaced by a SIRET from the same SIREN
  • Cohort 2 — Replaced by a SIRET from a different SIREN
  • Cohort 3 — Not replaced, legal unit closed
  • Cohort 4 — Not replaced, establishment closed only
• Enrich — Acteurs SIRET & SIREN (lien succession) (enrich_siret_siren_lien_succession)
  • Cohort — SIREN/SIRET successor from lien succession