Cloudera Enterprise Ingestion Hub Migration
Challenges
Ageing infrastructure delivered poor performance, incurred high maintenance costs and could not support modern platform upgrades. Complex interdependent workflows and large-scale ingestion and replication processes increased migration risk and complexity. An aggressive migration timeline required a seamless transition with minimal disruption to business-critical analytics and reporting operations.
Outcome
Achieved nearly 15% infrastructure cost savings. Successfully migrated 416 workflows and completed migration within a record 3-month delivery timeline.
Solution
Cloud
Challenges
Solution
Technology Stack
Outcomes
Over 400 critical data workflows. More than 1,100 replication dependencies. Just three months to migrate a business-critical platform before a major data centre shutdown. For one of the UK's largest multinational brands, the migration of its Enterprise Ingestion Hub (EIH) was a business-critical priority. The platform was crucial for enterprise reporting, analytics and operational decision-making. This was far more than a lift-and-shift migration. The platform contained complex interdependencies, ageing infrastructure and outdated technologies that increased both cost and risk.
Cloudaeon took ownership of the challenge and modernised the environment on Cloudera CDP, improved performance and created a future-ready foundation for the organisation's cloud-first strategy.
Client Problem
A UK-headquartered multinational brand relied on its Enterprise Ingestion Hub (EIH) as a business-critical platform for ingesting. Like structured, semi-structured and unstructured data into Hadoop before enriching and replicating it into the Enterprise Analytics Hub (EAH) on Azure. The platform powered enterprise reporting, analytics and operational decision-making across the organisation. As part of its broader cloud-first transformation strategy, the enterprise had announced the decommissioning of its Stockley Park Data Centre by January 2025. This required all applications, including EIH, to be migrated to the new infrastructure by September 2024.
The migration was not a simple infrastructure move. It was central to reducing dependency on physical infrastructure and lowering operational costs. Moreover, it was to prepare the organisation for the rollout of new Azure-based datasets planned for March 2025.
The existing environment presented several technical and operational challenges:
The ageing infrastructure at Stockley Park was too costly to maintain and delivered poor performance.
Legacy workflows contained numerous redundant and unused pipelines that consumed compute resources unnecessarily.
Existing hardware could not support modern operating systems or upgraded platform components, increasing security and performance risks.
The platform contained complex interdependent workflows requiring careful migration with minimal operational disruption.
Large-scale replication and ingestion dependencies increased migration complexity significantly.
Many vendors hesitated to take ownership of the migration due to the scale, platform complexity and aggressive timeline.
Root Cause Analysis
The primary issues started from architectural and operational limitations within the legacy on-premise environment. The Stockley Park infrastructure was built on ageing hardware, HP Gen 8 Servers. It lacked the ability required for modern data engineering workloads. These inefficiencies increased operational costs and introduced performance bottlenecks with higher maintenance overhead.
Several ingestion and processing workflows had evolved without adequate lifecycle governance, leading to:
Unused and redundant pipelines continued executing unnecessarily.
Workflow orchestration complexity increased operational overhead.
Legacy dependencies created migration and upgrade risks. -
Older software versions introduced compatibility constraints across Python, SQL and Impala workloads.
The existing environment also relied on legacy messaging components such as Flume, which lacked flexibility and were hard to maintain. Additionally, older operating systems and outdated platform components restricted the organisation’s ability to adopt newer capabilities.
The scale of the platform led to several migration challenges:
Over 400 workflows required migration.
Critical replication dependencies needed validation.
Business continuity requirements demanded near-zero disruption.
Tight decommissioning deadlines left minimal room for migration delays or architectural redesigns.
Due to the lack of a structured rehosting and modernisation strategy, the migration risked workflow instability, replication failures and reporting issues. It also increased the risk of missing critical decommissioning timelines.
Solution Architecture
Cloudaeon designed and executed a structured rehosting and platform modernisation strategy for the Enterprise Ingestion Hub. The target architecture involved migrating all ingestion and processing workflows from the Stockley Park environment to a newly built, higher-capacity cluster in the Swindon Data Centre. The new environment was built on Cloudera CDP Private Cloud Base.
The modernised environment included:
Expanded compute and storage capacity with additional nodes and disks.
Latest Linux operating systems for improved compatibility and security.
Upgraded Cloudera CDP stack for enhanced scalability and operational reliability.
Updated Python and Impala environments for compatibility with newer processing frameworks.
Migration of messaging workflows from Flume to Apache NiFi for improved orchestration and maintainability.
Continued orchestration through Oozie with ETL processing using Hive and Impala.
Preservation of existing Python and SQL frameworks to minimise migration risk and accelerate delivery timelines.
Repointing and validation of Evangelosoft replication services between the Swindon and Beam environments.
How We Delivered
Cloudaeon initiated the project with a detailed discovery and assessment phase to evaluate the complete workflow ecosystem, including the infrastructure dependencies and migration scope.
During the assessment, the actual migration scope expanded significantly from the initially planned 350 workflows to 416 workflows.
Platform Migration & Rehosting
Cloudaeon migrated all workflows from Stockley Park to the Swindon cluster using Cloudera CDP Private Cloud Base. To minimise risk and accelerate migration timelines:
Existing Python and SQL frameworks were preserved rather than redesigned.
Workflow logic was reused and adapted wherever possible.
Existing orchestration patterns were retained to ensure operational continuity.
Technology Modernisation
As part of the platform upgrade plan:
Messaging workflows were transitioned from Flume to Apache NiFi.
SQL scripts were updated for compatibility with newer Impala releases.
Deprecated SQL keywords were replaced.
Query syntax was aligned with upgraded platform requirements.
Latest Linux operating systems and upgraded Python environments were implemented.
Replication & Connectivity Validation
Cloudaeon repointed the Evangelosoft application between the Swindon and Beam environments and validated 1,134 replication IDs within just two weeks.
This ensured replication stability and continuity across the migrated ecosystem.
Testing & Validation
Extensive functional regression testing was conducted across all migrated workflows to ensure zero disruption to downstream operations. That involved workflow validation testing, replication validation, continuous performance monitoring with failure remediation.
Technology Stack
Cloudera CDP Private Cloud Base
Hadoop
Apache NiFi
Apache Flume
Apache Hive
Apache Impala
Apache Oozie
Python
SQL
Linux OS
Azure Enterprise Analytics Hub (EAH)
Outcomes
Successfully migrated 416 workflows, exceeding the original target of 350 workflows.
Completed migration within a record 3-month delivery timeline.
Achieved nearly 15% infrastructure cost savings.
Delivered migration before the decommissioning deadline.
Validated 1,134 replication IDs within just 2 weeks.
Ensured zero disruption to business operations during migration.
Improved scalability and performance through deployment on the latest Cloudera CDP stack.
Accelerated delivery timelines by reusing proven Python and SQL frameworks.
Established a future-ready platform capable of supporting upcoming Azure data expansion initiatives.
POD & Managed Ops Transition
Following successful migration delivery, Cloudaeon’s engagement evolved into a broader operational support and platform management model. Cloudaeon continued supporting Cloudera environments across the Swindon Data Centre through a POD-based operating structure focused on platform stability, operational continuity, workflow management and ongoing optimisation. This ensured the migrated EIH platform remained stable and aligned with M&S’s long-term cloud-first transformation strategy.
Conclusion
Cloudaeon didn't just move workloads from one environment to another. The team eliminated technical debt, modernised critical components and simplified a complex data ecosystem while keeping operations running seamlessly. The organisation now has a future-ready platform that is more cost-efficient and better equipped to support its cloud-first strategy. Ready to modernise your data platform? Talk to an expert today.