A data platform defined in code
Reports are built from manual exports, and the cloud was set up by hand. We put the infrastructure into Terraform and Ansible, build tested pipelines on top of it and hand over a platform the client’s team can run.
The problem
Every report starts with someone exporting files from the order system, the warehouse system and the finance tool, then joining them in a spreadsheet. Two people get two different numbers for the same week.
The cloud account grew the same way. Servers and databases were created by hand in the console over several years. Nobody can say exactly what is running, and nobody could rebuild it. The company wants to use AI on its operational data, but there is no reliable data to give it.
What we build
One platform, with the infrastructure and the pipelines in the client’s repository.
- Infrastructure as code. Networks, access, storage and compute are defined in Terraform. The few servers that remain are configured with Ansible. A new environment is created from the code, not from memory.
- Ingest. Data is loaded from the source systems on a schedule. Each load is recorded: what arrived, when and how much.
- Transform. Raw data is cleaned and joined in layers, from raw to ready for use. The business rules are written once, in dbt, where everyone can read them.
- Test. Each pipeline has checks on its data: missing values, duplicates, late arrivals. A failed check stops the load and sends an alert.
- Deliver. Every change goes through review and a CI/CD pipeline. It is tested first, then released the same way to each environment.
- Observe. Dashboards show whether the pipelines ran, how fresh the data is and what the platform costs.
How it runs
- The platform is deployed in the client’s cloud account.
- Changes to infrastructure and to pipelines are pull requests. The plan is reviewed before it is applied.
- Secrets are kept in the cloud secret manager, not in the repository.
- A runbook describes what to do when a pipeline fails, and who is called.
What the client gets
- Their infrastructure in Terraform and Ansible, in their own repository.
- Tested data pipelines with one definition for each business number.
- CI/CD, monitoring and alerts.
- Documentation, a runbook and a team that has worked on the platform with us.
The result
Reports come from one place and show the same numbers. A new environment can be created from code. The data is ready for the agents and AI systems the company wants to build next.