How M&S Achieved Seamless Data Migration in Just Two Weeks
Marks & Spencer streamlined ETL migration with Prophecy, Airflow and Databricks, ensuring data integrity.
What is Lorem Ipsum?
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged.
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged.
Background
Marks & Spencer, one of the UK's most recognised retail brands, embarked on a data transformation initiative to enhance its analytics capabilities. The company needed to integrate a newly onboarded data quality tool into its Synapse Analytics environment. To achieve this, an efficient ETL pipeline was required to ensure seamless data migration while maintaining data integrity and governance.
Cloudaeon was engaged to implement and support the end to end ETL process using Prophecy, a low code data engineering platform, alongside Airflow and Databricks.
Challenges
The project presented a critical deadline: migrating data from the new data quality tool into Synapse Analytics had to be completed within two weeks before the existing data quality tool was decommissioned. The team needed a streamlined approach to implementing Prophecy while ensuring minimal disruption to existing analytics workflows.
Technologies Deployed
Prophecy: A low code data engineering platform for visual ETL pipeline development.
Apache Airflow: Workflow orchestration for managing the end to end data pipeline.
GitHub Actions: Automation for version control and deployment.
Databricks: Compute infrastructure for executing transformations and data processing.
Solution
Cloudaeon designed and deployed an efficient ETL pipeline leveraging Prophecy’s low code interface to transform and migrate data. Airflow orchestrated Prophecy generated pipelines to ensure structured, dependent execution. Databricks was used for executing Prophecy generated wheel (.wbl) files, ensuring scalability and performance.
This approach not only accelerated development but also eliminated the need for complex custom ingestion frameworks, reducing development time and operational overhead.
Outcome
Marks & Spencer successfully migrated and validated data between the old and new data quality tools within the set deadline. The integration of Prophecy and Airflow enabled a fast, reliable ETL build, reducing deployment time to just two weeks. Bypassing legacy ingestion frameworks, the project delivered a scalable, future proof data pipeline that streamlined analytics operations.
This case study highlights how Cloudaeon’s expertise in ETL modernisation helped Marks & Spencer achieve a seamless data transition, reinforcing the importance of innovative, agile ETL solutions in the retail sector.
