Modern Data

Data Platform Foundation

A lakehouse defined in Terraform, with access rules and quality checks from the first deploy.

GIST stepIterateShip
Download the PDF

Get the Data Platform Foundation overview

A short PDF to share with your team. It covers what is in the package, how it works, and a worked example.

The problem

The first months of a data program go into plumbing.

Networking, storage, identity, access policies, and deployment pipelines all have to be right before anyone ships a data product. Teams rebuild them for every program, and the security settings get decided last.

What's included

A governed lakehouse, defined in Terraform.

01

Reference architecture

Built for Azure first, with reference designs for AWS and Google Cloud and variants for Microsoft Fabric, Snowflake, and BigQuery. It includes an access model for admins, engineers, and analysts.

02

Terraform modules

Network, storage, Key Vault, Databricks, Unity Catalog, and monitoring. Dev and prod have separate settings, each with its own remote state.

03

Ingestion jobs

A Databricks Asset Bundle with templates for files through Auto Loader, incremental database loads over JDBC with a watermark, and paginated REST APIs. Data lands in bronze and is typed and tested on its way to silver and gold.

04

Quality rules and CI/CD

Quality rules in YAML send bad rows to a quarantine table and stop a load that crosses a set threshold. Production deploys log in with OIDC and wait for an approval.

How it works

What gets built on Azure.

  1. Private network

    A virtual network with private endpoints, delegated Databricks subnets, and private DNS zones.

  2. Governed lake

    ADLS Gen2 with one container per layer: landing, bronze, silver, and gold. Shared keys and public access are switched off.

  3. Secrets

    Key Vault with role-based access only, network deny by default, and purge protection in production.

  4. Compute

    A Premium Databricks workspace with VNet injection and no public IPs. Production gets a private front end.

  5. Access as code

    Unity Catalog grants. Admins get everything, engineers read and write, and analysts read silver and gold.

  6. Audit logs

    Storage, Key Vault, and Databricks audit logs go to Log Analytics.

Worked example

A weakened security setting fails in review.

Guardrail checks run on every pull request and need no cloud access, so problems show up while the change is still a diff.

What the checks cover
  • Wiring between the network, storage, and compute modules
  • Environment settings files, complete and consistent
  • Security settings from the reference design
  • Quality rules on every source load
Result

A change that loosens a security setting fails before it reaches a subscription. Production deployments still wait for a person to approve them.

Where it fits

What it runs on, and where it stops.

Works with

  • Azure first: ADLS Gen2, Azure Databricks, Unity Catalog, and Key Vault
  • Reference designs for AWS and Google Cloud
  • Microsoft Fabric, Snowflake, and BigQuery variants

Assumptions and limits

  • The static checks pass today. Validating against your own subscription is the first job on your side
  • One environment per state file. Account-level resources are set up once
  • Databricks, storage, and logging cost money to run, so we size them with you first
Ask Dex

Ask about the Data Platform Foundation

Dex answers from our own pages and links to where it found the answer.

Starting a lakehouse on Azure?

Show us your landing zone and network layout. We will show you where the modules plug in.