Case Study
From 3 to 11 Squads
Automating DataOps at Scale
Implementing CI/CD and DataOps on Snowflake to scale data modelling from 3 squads to 11, and clear the backlog within weeks
The Client
A Major UK Retailer
One of the UK's leading retail companies had mobilised WhereScape 3D across their data team to build a Data Vault model on their Snowflake platform. With the initial build well underway, a critical new requirement emerged: automate code deployment using GIT and CI/CD to facilitate rapid team growth and eliminate a growing backlog of requests that was throttling business agility.
"We had 3 squads and a growing backlog. Manual deployments were our biggest bottleneck, we needed to automate before we could scale." Client Data Engineering Lead
Retail
United Kingdom
Snowflake, DataOps, CI/CD, WhereScape 3D
The Challenge
Scaling without the scaffolding
Manual Deployment Drag
Manual deployment processes slowed down development cycles, creating friction that compounded as the team tried to move faster.
Scaling 3 to 11
The data team needed to grow from 3 to 11 squads rapidly, but without automation, every new squad would multiply inefficiency.
Growing Backlog
A mounting backlog of requests had built up due to limited automation and inconsistent processes across the existing squads.
Standardisation Gap
Data models, mappings, and lineage documentation varied across squads, there was no consistent standard to onboard new teams against.
The Journey
Automate, standardise, and scale
DataOps Foundation
- Implemented DataOps for Snowflake using GitHub and AWS virtual machines
- Designed a CI/CD pipeline integrating GIT for auditability and version control
- Automated code deployment with approval processes built into the workflow
- Built trust through transparent architecture decisions shared with all squads
Standards & Scale
- Established consistent data modelling standards across all existing and new squads
- Enabled rapid onboarding, expanded from 3 to 8 squads within 4 weeks
- Reached full scale of 11 squads within just 6 weeks of go-live
- Tailored the solution to the client's specific Snowflake and WhereScape environment
Training & Handover
- Provided comprehensive training to client team members on the DataOps solution
- Equipped the team to manage and maintain the CI/CD pipeline post-go-live
- Reduced the backlog within 8 weeks as automated deployment removed bottlenecks
- Ensured long-term self-sufficiency with clear documentation and runbooks
The Transformation
From manual bottleneck to automated pipeline
3 squads, manual deployments, growing backlog, inconsistent standards across teams.
11 squads, automated CI/CD, backlog cleared in 8 weeks, standardised data models.
Ready to Transform?
Let's automate your data pipeline
By implementing a DataOps solution with CI/CD, Engaging Data enabled the client to automate, standardise, and scale their data modelling processes. This transformation reduced turnaround times, streamlined operations, and provided a scalable framework for future growth based on business demand.