Smoothing the migration path
When Xero opted to move to Amazon Web Services, the implications were enormous: (as we stated earlier there were more than 59 billion records, 3,000 apps and 120 databases representing all data and applications for its 862,000 customers globally at the time), the sheer scale of the migration was epic. When talking about the data warehouse, Griffiths says “It meant a huge rearchitecting of all infrastructure and systems which involved changing from SQL Server to AWS Redshift as the database platform.
It was a major undertaking”. The data warehouse was part of this exercise; built over time it represented a considerable investment as it was created and constantly modified to meet changing business information demands. But Xero had a couple of aces up its sleeve which would ease the transfer of this structure to the new platform.
Solution - steps to the cloud
The first step was an engagement with NOW Consulting (now Qrious), which performed a proof of concept to show that it was technically feasible to move from one platform to another while preserving the logic, processes and code of the data warehouse. “NOW delivered on that aspect successfully, then provided additional manpower for the actual migration,” says Griffiths.
One of the major challenges faced was what he describes as ‘a hefty SQL Server’ hosting the existing data. “We just could not recreate a system of that size in Amazon; we couldn’t just lift and shift, so we needed a new solution in Redshift which was an unknown.”
“There was a fair bit of risk and a lot of work”, says Griffiths. “With NOW’s help and with WhereScape RED as the tool to do it, we were able to reduce the complexity and risk to a point where it was feasible.”
There was a little pressure, too. “There wasn’t any question of it not being done as we had to have a solution in place; the consequences of no solution would be a lot of internal analytics and reporting would have ceased to work, it would impact our Customer Experience teams, our sales and marketing teams and even our product teams.”
NOW developed the necessary functionality within WhereScape RED which enabled connecting the existing data warehouse to the AWS Redshift database. “This was a new feature at the time called Templates; instead of code generated in WhereScape itself, the template has variables specifically for Redshift which allowed extremely flexible code generation. This was the key to moving to the new platform,” says Griffiths.