The data lake that started as one shared bucket
It began with an export, a notebook and a folder called raw. Now three teams depend on it, an analyst has a copy on a laptop, and nobody can answer which columns contain personal data. Data platforms do not fail loudly. They become ungovernable quietly.
How the lake became a swamp
Every step was reasonable. The pipeline was a script, the schema was implied, and access was granted by adding people to the bucket.
Analytics, reporting or machine learning now depends on data that lives in an ungoverned bucket, and someone has finally asked who can read what.
-
The schema lives in whoever wrote the transformation script.
Downstream queries break without warning when an upstream field changes, and nobody finds out until a report looks wrong.
-
Access is all or nothing at the bucket level.
Giving an analyst the data they need means giving them every column, including the ones legal would rather they never saw.
-
“We can't give analysts the whole bucket.”
So teams copy filtered extracts into new buckets, and the number of uncontrolled copies grows every quarter.
-
“We pay for a data cluster that sits idle most of the day.”
Batch workloads with peaky arrival patterns run on always-on compute, which is the most expensive way to run them.
-
Nobody can say which files are raw, processed or safe to publish.
Consumers read from wherever they found data once, so the pipeline has no boundary and no contract.
What an ungoverned lake costs
The cost is rarely storage. It is trust, duplication and the day someone asks a question about personal data.
- Data you cannot vouch for
- Without zones and a catalog, no one can state where a number came from, so decisions are made on data that nobody will defend.
- Copies you cannot track
- Coarse access control produces extracts, and extracts produce copies in places your controls do not reach.
- Compliance exposure
- A customer, auditor or regulator asking which columns contain sensitive data needs an answer from the catalog, not from a search of the transformation code.
- Idle compute
- Clusters provisioned for the busiest hour bill for every other hour, whether work arrives or not.
- Rework
- Retrofitting governance onto a lake that grew without it means reprocessing history and renegotiating access with every consumer.
What we do about it
We deploy a data platform with the boundaries and the governance in place from the first file, and an event driven pipeline with no always-on compute.
-
Three zones with real boundaries
Raw, processed and curated storage, each encrypted with managed keys, versioned, and moved to colder classes on a lifecycle policy as data ages.
-
A catalog per zone
Crawled schemas and a data catalog per zone, so consumers query a table with a contract rather than a folder with a convention.
-
Governance at the column level
Access granted through data-lake permissions with tags for zone, sensitivity and domain, and column-level grants, so an analyst can be given the columns they need and not the ones they do not.
-
Direct access closed off
A storage policy denies reading around the governance layer, so the catalog is the only way in and the access record is complete.
-
A pipeline that runs when data arrives
An upload triggers a function, which starts a batch job that cleans, deduplicates and partitions the data into columnar files. There is no cluster to keep alive between runs.
The outcome: Data lands in a zone, is catalogued, and becomes queryable under access rules you can describe in a sentence. The platform costs what it processes rather than what it might process.
What that is, in the portfolio
-
BuiltForProd Blueprints
Opinionated, production-ready reference architectures for real workloads, deployed into your Baseline. Each one is a working system, not a diagram.
-
BuiltForProd Baseline
A multi-account cloud foundation in OpenTofu and Terragrunt, customized for your organization, deployed into your accounts by our engineers, then handed over with the repositories and the documentation.
What you get
-
A zoned data lake in your accounts
Raw, processed and curated storage with encryption, versioning and lifecycle rules, defined as infrastructure code in your repositories.
-
A working pipeline, end to end
A sample transformation job, a crawler and an event trigger, deployed and demonstrated: drop a file in the input prefix and the pipeline runs.
-
Column-level access control
Tag-based governance across zone, sensitivity and domain, with grants that reach individual columns rather than whole buckets.
-
Query-ready output
Cleaned, deduplicated, partitioned columnar data, catalogued so analysts can query it with the tools they already use.
-
No always-on compute
The pipeline is event driven, so cost follows the data that arrives rather than the capacity someone reserved.
-
Documentation for data engineers
How to add a zone, a table, a job or a consumer, with the governance model explained rather than implied.
Every repository and every cloud account stays yours. We keep no standing access after handover. See security and ownership for how that is enforced, and pricing for how an engagement is quoted.
What is in the Blueprint
A working data platform in two repositories.
| What | Figure | Where it comes from |
|---|---|---|
| Storage zones | 3 | The Data and ETL Blueprint: raw, processed and curated, each encrypted with managed keys and versioned. |
| Catalogs | One per zone | Data and ETL Blueprint. A crawler populates the catalog so consumers query tables, not prefixes. |
| Governance tags | 3 dimensions | Data and ETL Blueprint. Zone, sensitivity and domain tags drive permissions, including column-level grants. |
| Direct storage access | Denied | A bucket policy in the Data and ETL Blueprint denies reads that bypass the governance layer. |
| Always-on compute | None | Data and ETL Blueprint. The pipeline is triggered by an upload event; nothing runs between batches. |
| Code delivered | 6 modules, 6 units | The Data and ETL Blueprint repositories, with three pipelines for deployment and jobs. |
The same evidence is documented page by page in the product documentation and measured against the BuiltForProd Standard.
How it works
Four steps from the first conversation to a platform your team owns.
-
Describe the data, not the tools
What arrives, how often, how sensitive it is, who consumes it and which questions the business asks of it.
-
Design zones, tags and access
The sensitivity model, the domains, who gets which columns, and how data ages through the lifecycle. Scoped and priced before work starts.
-
Deploy into your workload accounts
Our engineers deploy the lake, the catalog, the governance model and the pipeline, then run real data through it end to end.
-
Your team runs a pipeline
In the handover, your data engineers add a table, grant a column-level permission and trigger a run themselves. Then the repositories are yours.
Yes, but…
The objections we hear on the first call, answered plainly.
We already have a warehouse. Does this replace it?
No. This is the lake and the governance in front of it. A warehouse is a consumer of curated data, and teams commonly run both, with the lake holding raw history and the warehouse serving analytics.
Our data volumes are small. Is this overkill?
Possibly, and we will tell you if so. The argument for zones and a catalog at small volume is governance, not scale: retrofitting access control after three years of copies is the expensive path.
Do our analysts have to change tools?
Usually not. Output is standard columnar data registered in a catalog, so the query engines and business intelligence tools your team already uses can read it, subject to the grants you set.
Is this machine learning infrastructure?
It is the data layer underneath it: governed storage, a catalog and a pipeline. Model training and serving are workload architecture, and worth a separate conversation rather than a claim on a landing page.
Questions
What is the Data and ETL Blueprint?
The Data and ETL Blueprint is a production-ready reference architecture for a governed data lake, delivered as two repositories and deployed into your own workload accounts. It includes zoned storage, a data catalog, tag-based and column-level governance, and an event-driven batch pipeline.
Does it need a Baseline first?
It deploys into workload accounts with identity, networking, encryption keys and logging already in place. If that foundation does not exist, it is deployed first, and both can be part of one engagement.
How is sensitive data handled?
Through tags and grants rather than through separate copies. Sensitivity is a tag on a table or a column, and permissions are granted against those tags, with direct storage access denied so the record of who read what is complete.
What does it cost to run?
That depends entirely on data volume and run frequency, and it is your cloud bill rather than ours. The design point is that there is no idle compute between runs, and storage moves to colder classes as data ages.
Can you help us migrate existing data?
Yes, as scoped work. Backfilling history into the zone model and registering it in the catalog is usually the largest piece, and it is quoted per engagement.
More answers are in the FAQ and the support center.
Govern the data before it governs you.
Tell us what data you hold, who consumes it and what you have been asked to prove about it. We will tell you what we would deploy and what should stay as it is.