Data Integration. It sounds fairly dull doesn’t it, DDL, DML, SQL , databases, tables, source to target mapping, modelling the list goes on.
Seems a strange place far from the cut and thrust of Analytics where life altering decisions are made at the speed of thought. So what is this thing, why is it important and who are these strange pale acolytes muttering odd incantations and weirding out your Sales and Marketing team?
Although it’s not sexy, data integration is perhaps the most important and influential aspect of any data centric project. Think about it, how do my analysts produce these beautiful graphs and engaging KPI’s if not with trusted consolidated data. And it’s obvious it must be trusted, not necessarily perfect but fit for purpose, or no one will trust their promotion on it. And it stands to reason that it must also be in a central place, so those highly trained analysts can easily find it.
Think trusted and consolidated data sounds easy? Guess again.
We’re database guys and gals and even we’d agree there’s nothing attractive or interesting about a database or a database table or SQL. The data? Well that’s a different matter entirely. In most businesses this sexy data is everywhere, in databases, all guarded by IT border guards and generally stored in such a way as to make it impossibly difficult to get and combine. (trade secret - most software vendors do this intentionally). This is an awkward situation. Can’t we just keep things simple and extract data from a few sources we already know and trust? The truth is this won’t work unless the analysis you’re doing is anything other than trivial.
Let’s take an example. If we have a sales system we could extract some simple datasets, provided we could find the data, as it’s usually labelled something helpful like LZNUF. But assuming we can get the data we can do some simple sales analysis. Great. But now some clever exec wants to see profit, we can’t answer that from sales alone, we need costs, that’s in the finance system. Well it turns out the finance system was written by another software company and the data doesn’t nicely fit with the format of our sales data. Oh dear, now I’m thinking I need something like Microsoft Access, and if that sounds familiar boy you’ve got a problem brewing.
It doesn’t end there, what if my Sales Manager moves sales regions mid year? I don’t want all his sales history to move with him, well if I’m just extracting raw data from my sales system that report probably just exploded. What happens when the exec team need the report first thing and the extract fell over, or my developer leaves? The list goes on. If this sounds like your environment and you’re starting to panic then relax, you’re in good company, most businesses are in the same boat.
OK I get it, Data Integration it’s important, how do I do it right?
Actually you’re halfway there just by recognising data integration is a thing you should do right. There are a couple of other little rules of thumb to help you on your way:
Don’t crush a nut with a sledgehammer or try with a feather. There is a range of Data Integration approaches from data wrangling to building an enterprise data warehouse. Choose the one that matches your problem.
Don’t boil the ocean, data integration can be a little like a black hole sucking in new scope and requirements until there’s nothing left. Don’t fall into that trap, get started and pick a problem that is doable with your skills and experience.
Choose a technology platform that will scale with your needs and your pocket.
Automate what you can to keep your Data Integration acolytes productive and free them up to talk to your sales and marketing team rather than scaring them. You might find the exchange yields some surprisingly positive results.
And finally your data. Centralise and consolidate it, manage it, make it trusted. Put it to work.
And if you struggle with any of this, don’t worry, help is at hand. We’ve been through this a hundred times in a hundred different environments. Like with any problem, a little bit of experience goes a long way.
Remember a happy analyst is one who is rolling in data and they might just find something that changes your business.