A mid-sized Central European manufacturer of packaging machinery had a reporting problem that showed up at 06:00. Its overnight refresh of plant-floor and ERP data ran for roughly nine hours on a US hyperscaler, and on heavy nights it overran into the morning shift. At the same time, cloud egress costs and per-query charges kept pushing the monthly bill up. The client wanted to leave its US hyperscaler for a manufacturing data platform it could predict, in EU datacenters.

The people who felt it first were the plant managers. They opened the morning meeting with figures from two shifts back, or with a half-finished report and a note that the job was still running. Quality engineers could not trust the previous shift’s scrap and rework numbers until mid-morning.

The challenge: three sources, one overnight window

The client’s reporting drew on three kinds of data. Machine data came from the controllers on its assembly and test lines, quality data from inspection stations and the quality management system, and commercial data from the ERP. Each source landed in a different store, and the nightly job pulled all of them into a reporting layer on the hyperscaler.

Two things made the bill and the runtime grow together. Every report that joined machine data against ERP orders moved data between services and out to the analysts’ tools, which is where the egress charges came from. And the job reprocessed far more history than it needed: a full recompute of the joins each night, because nobody could say with confidence which partitions had changed.

We want to be fair to the hyperscaler here. The platform did what it was built to do, and its elasticity could have absorbed the nine hours with a bigger bill. The client’s objection was to the pricing model for this workload (constant internal movement of the same data) and to having its production and order data under a US parent.

What we built: a manufacturing data platform in one place

We scoped the work with the client’s controlling, quality and plant IT teams, then built and now run a Data Driven platform for them. It sits in the EU datacenter site the client chose, as an isolated tenant that belongs to them alone, with backups included. Orchestration, a data lake, notebooks and dashboards all run on our infrastructure, so data moving between those layers stays inside one network and does not cross a billing boundary.

The design had four parts.

  • One landing zone for all three sources. Machine, quality and ERP data arrive in the data lake in the same raw form, with the same timestamps and plant and line identifiers, so a join no longer starts with a reconciliation step.
  • Incremental processing. The orchestration layer tracks which partitions changed since the last run and recomputes only those. The full nightly recompute is gone.
  • Ordered stages. Ingestion, cleaning, joins and report tables run as separate stages with explicit dependencies, so a late ERP extract delays only what depends on it.
  • Dashboards on prepared tables. Plant managers query tables that are already built, rather than triggering heavy joins when they open a report.

The ERP reporting migration

The ERP reporting migration was the riskiest piece, because finance and controlling relied on the existing report definitions. We did not rewrite them in one pass. We rebuilt the reports in the new platform, ran them in parallel with the hyperscaler versions for several weeks, and compared outputs line by line.

Differences were logged and traced to a cause: a rounding rule, a currency conversion date, a treatment of cancelled orders. The client’s controllers decided which behaviour was correct for each, and we encoded that decision once in the transformation layer. Only after the parallel run agreed did the old reports switch off.

A nine-hour job is rarely a compute problem. It is usually the same data being moved and recomputed every night because nothing records what changed.

Plant floor analytics at start of shift

With incremental processing and a single landing zone, the overnight refresh now finishes in under two hours, down from roughly nine. Plant managers have the previous shift’s figures when they arrive: output per line, first-pass yield, downtime by cause, and open orders against the plan. That is the practical gain in plant floor analytics for this client: the morning meeting starts from the same numbers for every plant.

The shorter window also changed how the client works with the data. Quality engineers use the notebooks on the platform to look at inspection results from the same night, rather than waiting for the next day’s refresh. A rerun after a late or corrected ERP extract is no longer a decision to skip; it takes under two hours instead of consuming the morning.

Cloud egress costs and the monthly bill

The client’s monthly data-platform spend fell by about a quarter. Part of that is structural: reporting traffic between the lake, the processing stages and the dashboards stays inside one environment, so there are no egress charges on the client’s internal reporting. Part of it is the work no longer done, since incremental runs process a fraction of the data the full recompute did.

The spend is also easier to forecast. The client can see what the platform costs without modelling how many queries analysts will run or how much data each one will move.

What the client can do now

  • Open every plant’s morning meeting with the previous shift’s figures.
  • Rerun a report after a late or corrected ERP extract without losing the morning.
  • Export its data from the platform whenever it chooses, with no switching or egress charges, so the next change of platform is not a repeat of this one.
  • Hand the running of the platform to us, which it did, while keeping the report definitions under its own controllers’ control.

The client’s production and order data now sits in EU datacenters we operate ourselves, with only authorised DIVERSITY staff in the EU able to reach the infrastructure, and with no parent company above us on which a foreign order could be served.

If you run reporting on a hyperscaler and the overnight job is the first thing you check each morning, we can review the pipeline with you under NDA and tell you where the hours go.