Data & ETL
A three-zone data lake that governs itself and runs only when data arrives. Upload a JSON file under input/ and the pipeline runs end to end.
What it is
The Data & ETL Blueprint is a governed, event-driven data lake on AWS, delivered as two repositories and deployed into the workload accounts of your BuiltForProd Baseline. Three S3 zones hold raw, processed and curated data. AWS Glue catalogs and transforms it, a Lambda function starts the work when a file lands, and AWS Lake Formation decides who can read which column.
It is deliberately serverless. A data platform that costs money while it sits idle gets switched off, and a data platform that gets switched off is not a platform. Six modules, six units and three workflows.
Three zones
Each zone has its own bucket, its own Glue catalog database and its own lifecycle.
Raw
Exactly what arrived, kept as it landed. Server-side encryption with KMS, versioning on, and lifecycle rules that move older objects to infrequent access and then to Glacier.
Processed
The cleaned, deduplicated result of the Glue job, written as partitioned Parquet and cataloged so it can be queried without moving it.
Curated
The shaped datasets analysts and downstream systems actually consume, governed by Lake Formation tags rather than by copying subsets into private buckets.
The pipeline
-
A file lands
A JSON file is uploaded under the input prefix of the raw zone. That is the entire trigger condition.
-
Lambda starts the job
A container-image Lambda function in private subnets, traced with X-Ray, receives the S3 event and starts the Glue job. It holds no schedule and no idle capacity.
-
Glue transforms it
A PySpark job on Glue 4.0 cleans, deduplicates and partitions the data, writing Parquet into the processed zone. A run costs a fraction of a dollar and stops when it is done.
-
The catalog updates
A crawler refreshes the Glue Data Catalog so the new partitions are queryable immediately, under the same Lake Formation grants as everything else.
Governance in the data layer, not around it
LF-Tags carry the policy
Resources are tagged by zone, sensitivity and domain, and grants are written against those tags. A new table inherits the policy of its zone instead of needing a new rule.
Column-level grants
Analysts can be given the dataset without being given every column in it. Sensitive fields stay out of reach without maintaining a second, redacted copy of the data.
Direct S3 access is denied
A bucket policy denies reads that bypass Lake Formation, so the catalog is the way in and access can actually be audited rather than inferred.
Encrypted and versioned throughout
Every zone uses server-side encryption with KMS, versioning, and lifecycle rules to infrequent access and Glacier, so retention is a setting rather than a cleanup project.
When teams choose this Blueprint
| Situation | What it changes |
|---|---|
| Analytics or machine learning needs a governed data platform and there isn’t one | A lake with zones, a catalog and grants from the first file, instead of a bucket that becomes one later. |
| Analysts need the data, but not every column in it | Tag-based and column-level grants, auditable, without copying subsets into separate buckets. |
| A Spark cluster sits idle most of the day | Compute starts on an event and stops when the work is done. Cost follows data volume, not the clock. |
| Every team builds its pipelines differently | One zone layout, one catalog convention and one governance model that new pipelines are written against. |
Questions about this Blueprint
What is the Data & ETL Blueprint?
The Data & ETL Blueprint is a governed, event-driven data lake on AWS, delivered as two repositories and deployed into the workload accounts of your BuiltForProd Baseline. It has three S3 zones, an AWS Glue catalog and job, a Lambda trigger and Lake Formation governance, with no always-on compute.
How does a file become a table?
Upload a JSON file under the input prefix and the pipeline runs end to end. An S3 event invokes a Lambda trigger, the trigger starts a PySpark Glue job, the job cleans, deduplicates and partitions the data into Parquet, and a crawler updates the Glue Data Catalog so the result is queryable.
What does it cost to run when nothing is happening?
Close to nothing, because nothing is running. There is no cluster to keep warm: the compute is a Lambda function and a Glue job that both start on an event and stop when the work is done. In the demo sizing the lake costs under five dollars a month at rest, and AWS charges are billed by AWS.
How is access to sensitive columns controlled?
Through AWS Lake Formation. The Blueprint tags resources with LF-Tags for zone, sensitivity and domain, grants access against those tags including at the column level, and applies a bucket policy that denies direct S3 access, so the catalog is the way in and every grant is auditable.
Can we use our own transformation logic?
Yes. The PySpark job that ships is a real, working transformation on Glue 4.0, and it is meant to be replaced by yours. The catalog, the trigger, the zone layout and the governance model stay the same whatever the job does.
What does it need from the Baseline?
The workload account, the private subnets the Lambda function runs in, the KMS keys and the pipeline identity all come from the Baseline, and the Blueprint reads them from Parameter Store rather than creating its own.
See the whole catalog, the Baseline it deploys into, or the controls and evidence the foundation produces around it.
Find out what production would take.
Tell us what you are running and what is coming: a funding round, an audit, a first enterprise customer, a migration. We will tell you what we would build, what it costs, and whether we are the right fit.