M&S Cuts Migration Time 70% with Unity Catalog Accelerator

M&S, a global retailer, was in the process of migrating from the Databricks environment from the legacy Hive Metastore to the Unity Catalog. The retailer deals with huge volumes of data every day. In such a situation, maintaining data governance and ensuring fine grained access control was a big challenge.

M&S had already initiated the migration process. However, they soon encountered technical and operational challenges that slowed down the process, which led to inconsistencies and production failures.

To accelerate the migration process and ensure reliability, M&S urgently needed expertise in the Unity Catalog migration from those who were proficient in Databricks and automation. Cloudaeon were that expert. With in-depth knowledge and experience in Databricks Unity Catalog, Hive Metastore, migration processes and automation.

Collaborating with Clouadeon, M&S successfully completed the Unity Catalog migration process with the help of Cloudeaon's Unity Catalog Migration Accelerator, a purpose built method to expedite the migration process cost effectively.

Challenges

Manual refactoring of notebooks

While migrating notebooks from a legacy workspace (Hive Metastore) to Unity Catalog, identifying the notebooks that can be migrated as is and identifying and replacing the deprecated functions was a challenge. Manual identification and migration of the notebooks were time consuming and error prone.

Inconsistent development and production environments

While M&S was executing the Unity Catalog migration in house, their developers faced a lot of challenges due to inconsistencies in the development and production environment codes. The development (lower) environments were not in sync with the production (higher) environment, which led to breaking production workloads.

Lack of visibility and tracking

Due to poor access control, there was no centralised mechanism to track the changes. What changes were made? By whom? There was no visibility. This impacted the environments and audits were difficult to execute.

Extended downtime

Migration steps like table scanning, refactoring and testing for notebooks were executed sequentially, which was time consuming, and correlated workflows required all dependent notebooks to be shut down, leading to extended downtime.

Solutions

Planning and execution

Planning and executing the entire project was a big challenge, as the scope of the project was huge. Approximately 500+ pipelines were to be migrated. The project was planned and executed in phases so that the timelines were not missed. These pipelines were from multiple domains, for easy and quick migration, they were categorised and prioritised based on their complexity.

Also, resource allocation was done wisely, where the right skillset resource was allocated the right task.

This minute level planning helped Cloudaeon to execute the project hurdle free within the time promised and agreed budget.

Cloudaeon implemented its Unity Catalog Migration Accelerator, a solution tailored specifically by Databricks experts aiming for automation and speedy migration.

Automated notebook scanning and refactoring

With the help of Cloudaeon’s Unity Catalog Migration Accelerator, all the notebooks (or assets to be migrated) were scanned in the GitHub repository to identify deprecated functions through an automated process. It automatically replaced functions where applicable and flagged the deprecated ones that required manual intervention.

This automation was not only saving a significant amount of time but also reducing manual efforts, further eliminating errors.

CI/CD integration via GitHub

Once the deprecated functions were replaced, Cloudaeon leveraged GitHub CI/CD to automate Notebook deployment across environments. This ensured consistency in the propagation of changes from development to production environments, thereby eliminating inconsistencies.

Audit and rollback mechanism

Unity Catalog Migration Accelerator ensures tight access control with crystal clear visibility. Every change made by anyone triggered a pull request. This ensured all updates were reviewed and approved before merging, further providing an audit trail allowing easy rollback if needed.

Parallel processing to reduce downtime

To reduce downtime caused by sequential processing of notebooks, Cloudaeon’s Databricks Untiy Catalog migration experts executed the migration tasks, like refactoring, in batches. This reduced the downtime significantly and accelerated the overall process.

Impact

The Cloudaeon’s Unity Catalog Migration Accelerator enabled the client to:

Reduce migration time by over 70%

By leveraging automation and parallel batch processing, the time required for tasks like scanning, refactoring and deploying notebooks across environments was

reduced by 70%.

Enterprise grade governance and audit compliance

Databricks Unity Catalog enabled fine grained access control and standardised tracking, providing top notch governance and audit compliance.

Prevent production failures

Cloudaeons Databricks Unity Catalog experts ensured consistency across environments and caught errors early and quickly through automated scans, avoiding disruptions and production failures.

Future proof data architecture

By successfully migrating to Unity Catalog, M&S is in an improved situation with easy scalability and higher security. With a broader perspective, by collaborating with Clouadeon, M&S's data architecture has been built for Databricks’ future roadmap.