Cloud Information Warehouse Migration a Group Effort for UK’s Co-op



It’s not unparalleled for purchasers to spend a number of years and tens of tens of millions of {dollars} attempting emigrate from one information warehouse to a different. It’s additionally not unusual for these workouts to fail. However for the Co-op Group within the UK, the migration from a Teradata warehouse to Azure SQL Information Warehouse not solely got here in on time and below funds, but in addition amid the worst pandemic in a century.

The Co-Op Group is a cooperatively run group in the UK that was initially based in 1844. At present, the Co-op (because it’s recognized within the UK) spans greater than 7,000 places throughout the nation, offering an array of grocery, retail, insurance coverage, authorized, and funeral providers. In 2020, it generated revenues of £11.5 billion (or $15.4 at present alternate charges) and had 4.3 million energetic members.

Like several fashionable group, the Co-op depends on a mixture of IT to automate its enterprise processes. For years, it has used a Teradata information warehouse to tell selections in its retail operations. The system held 25 years’ price of knowledge detailing the who, what, how, and when of Co-op’s buyer actions.

“It was just about the engine that ran our retail operation,” says Co-op’s Director of Information, Digital and Loyalty, Charlotte Lock.

Nonetheless, the on-prem Teradata system was approaching end-of-life for Co-op, and so the organizations information workforce began exploring new choices within the 2018-2019 timeframe. The cloud loomed massive for Co-op, because it has for a lot of corporations which can be searching for extra flexibility and the facility to scale up their information warehousing operations in a brief period of time.

“The choice emigrate was to reap the benefits of the cloud,” Lock tells Datanami. “So actually, processing velocity, price, and storing the information that’s been sustaining on prem methods, but in addition to have the ability to crunch tens of millions of rows of knowledge–billions of rows of knowledge typically–in a fraction of the time that we’d have wanted to try this in our on-prem options.”

Then COVID-19 hit in early 2020, and that ramped up the time horizon significantly, Lock says. If the Manchester-based group was contemplating a leisurely five-to-six yr migration earlier than COVID-19, that actually was now not a part of the plan afterward.

“We may see even earlier than COVID–and naturally COVID accelerated it–that the requirement for much more information to have the ability to carry out real-time motion as we tapped into, for instance, e-commerce,” Lock says. “We didn’t actually have a considerably e-commerce operation earlier than COVID. However really the demand meant that we needed to set one thing up actually swiftly.”

A part of the priority with the outdated system was that the Co-op lacked complete views of its clients, which meant that buyer exercise in retail couldn’t simply be matched with buyer exercise in authorized or insurance coverage. That restricted the cross-selling and up-selling alternatives which can be so vital for ecommerce operations, just like the one which the Co-op was looking for to construct in a brief period of time. It additionally restricted the Co-op’s capability to function in an omni-channel trend, which is the place it was headed.

“The place group that had been predominantly nose to nose…increasingly, we’re seeing our clients eager to transact by means of omni-channel,” Lock says. “It’s the coincidence of transferring to the cloud with our enterprise ambition of turning into extra digital and extra private.”

The mixture of those elements–the larger processing flexibility of the cloud and COVID-‘s omni-channel calls for–drove Co-op to the choice to retire its Teradata system and substitute it with Microsoft Azure Synapse Analytics in early 2020, Lock says.

Nonetheless, contemplating how essential the Teradata system was to the Co-op’s retail operations, it couldn’t simply pull the plug. It additionally couldn’t languish with a ceaselessly migration program that will or could not ever absolutely pan out. That’s when the corporate thought-about bringing in one other participant to assist with the migration: Datometry.

Datometry is a San Francisco firm that develops a database virtualization product that may make one database appear to be one other. Its emulator, referred to as Hyper-Q, was developed to assist corporations migrate off two of the most well-liked analytics databases in the marketplace: Teradata and Oracle.

Datometry CEO Mike Waas described how Hyper-Q works in a July interview with Datanami:


“We intercept the speaking from the appliance, unpack the requests, take out the SQL, after which do what successfully is sort of like what the higher half of any database does, that means construct a complete algebraic mannequin for the incoming request, optimize that, after which synthetize what the optimized SQL means for that vacation spot,” Waas stated.

This was simply what the physician ordered for the Co-op. Because it didn’t should rewrite its Teradata code, the group may basically simply take all 10,000 tables and 800 to 900 automated processes that it has developed for Teradata, and transfer them immediately into Azure Synapse, and let Hyper-Q deal with any minor incompatibilities or incongruencies between the 2 methods.

“It does virtualize it to a fairly profitable diploma, which implies that we had been capable of preserve our Teradata system,” Lock says. The Datometry software program allowed Co-op to “construct this emulation layer in parallel, after which be capable of shut down Teradata and successfully have the identical code working.”

The precise migration from Teradata to Azure Synapse was finished in batches, beginning with the largest, most complicated batch runs. After thorough consumer acceptance testing (UAT) to verify the brand new system is giving the identical reply because the outdated system, and is assembly service degree agreements (SLAs), it might go on to a different batch, and so forth, till the entire thing was moved.

“We invested rather a lot in high quality assurance, as a result of what you don’t need to do is sort of change off the backup plan till you’re actually, actually assured,” she says. “So we went again and again and over.”

With help from the Microsoft Azure workforce and Datometry, the Co-op was capable of full the migration to Azure Synapse Analytics in only a matter of months. It’s now working its information warehouse absolutely within the Azure cloud. The Teradata system has but to be decommissioned, nevertheless it’s now not within the loop, Lock says.

Whereas it didn’t must, the group took the chance to repoint about 15% of the Teradata code to run extra effectively within the Azure Synapse setting, Lock says. “Not as a result of it wasn’t working within the engineman essentially,” she says. “However as a result of the groups are spending an terrible lot of time in that information, and really when these processes had been written within the first occasion, it was a special setting.”

The return on funding on the migration has been good, in accordance with Lock, who says the group has saved tens of millions of kilos by working in Azure versus working with Teradata on prem. The flexibility to ramp up information crunching operations to make sure that stories (designed in PowerBI or MicroStrategy) are able to go and on analysts desks at 7 o’clock Monday morning is one other issue altogether.

“In retail, it’s worthwhile to have information at your fingertips. It’s a really actual and responsive setting,” Lock says. “And it actually meant that that lack of service interruption [and] the power to hurry issues up is a large benefit for the effectivity of the information workforce. It didn’t essentially imply that our stakeholders noticed an enormous benefit. It meant that our workforce may run extra effectively.”

The truth that Co-op was capable of full the migration on time and funds throughout the COVID-19 lockdown is one thing that Lock says she may be very happy with. When the undertaking was accomplished, a few of the workforce really obtained collectively, which made it further particular.

“There haven’t been any huge dramas. I haven’t needed to ask for any extra funds. I haven’t needed to lengthen the workforce so as to ship the transformation,” she says. “So it’s been a really sort of clean course of, which I feel is partly testomony to the partnership with Microsoft and Datometry. But in addition I feel additionally you realize being real looking concerning the period of time the UAT wanted to take, not attempting to transition and migrate too quickly. I feel that’s the essential factor.”

Associated Gadgets:

Is Now the Time for Database Virtualization?

Microsoft Debuts AI-Primarily based Semantic Search on Azure

Cloud Information Warehousing: Understanding Your Choices