01 · Case study · Data Platform
35% faster pipelines. ~20% lower compute cost.
A production data platform running across Microsoft Fabric and Databricks, serving finance and operational workloads. Over time it had accumulated orchestration and compute overhead — slow to run, expensive to keep running, still responsible for the same datasets on the same schedule.
35%
faster pipeline execution
~20%
lower compute cost
50+
stakeholders served
Context
The platform grew organically. Pipelines, schedules, and compute choices piled up over time. Sources included ERP, leasing, and finance systems; downstream, 50+ stakeholders depended on the datasets for daily decisions.
Constraint
Runtime and cost had to come down without touching reliability. Same datasets, same schedule, same guarantees — less waiting, less spend.
Architecture
- SourcesERP / leasing / finance systems
- Fabric ingestionScheduled + event-driven loads
- Delta / OneLakeSingle governed storage layer
- Transformation / SparkSimplified orchestration
- Quality + reconciliationChecks before anything ships
- Analytics / downstream users50+ stakeholders
Decision
Orchestration was redesigned and compute simplified: fewer moving parts, less redundant processing, workloads sized and scheduled for what they actually needed. Nothing exotic — mostly removing work the system didn't need to do.
Result
Pipeline runtime down 35%. Compute cost down ~20%. Same datasets, same reliability, a third less waiting.
Nothing exotic — mostly removing work the system didn't need to do.
Lessons
- Overhead accumulates silently on platforms that work. Nobody schedules it; it just compounds.
- Removing unnecessary work beats optimizing necessary work.
- Cost and runtime are engineering constraints, not finance's problem to discover later.
Stack
Microsoft Fabric · Databricks · Spark