Get faster results with an agile data warehouse

When clients come to us with business problems, such as acquisition, retention, revenue, or churn, we know a data warehouse capable of surfacing up insights is an essential part of the solution. By delivering these insights we help our customers identify the solutions to their problems.

However, for most people the term data warehouse equals “here we go – this is going to take forever and cost me a fortune”, usually accompanied by the glazing over of eyes.

The sad truth is, their thinking isn’t wrong. At least, not historically wrong. The traditional waterfall method of building a data warehouse meant a ton of upfront work – typically 6-12 months and 80% of the project budget, just to integrate the data. The problem is that you can’t guarantee the insights you’ll get from that data, and after that amount of time the business has usually given up and moved on!

Buzzword bingo

Before I go much further, I need to make a confession…

As a BI consultant I’ve acronyms and bits of jargon to burn. AWS, BPDaaS, Azure, DPM, BI, and Big Data are all part of my everyday language. But I’ve learned the hard way that using them in front of a client guarantees dull looks and a slowly dwindling engagement. If nothing else, it tends to kill the BBQ chat…

So before I go on, let me clarify a couple of key terms:

Data Warehouse (DW): This is simply a database of information from one or more sources, brought together to support analysis and reporting. The sources of data can come from a mix of internal systems (e.g. sales, marketing, or supply chain data) as well as external information (e.g. purchased lists, Facebook, or census data).

Business Intelligence (BI): These are the different tools used to interact with data stored in the database to surface up insights. These might include mechanisms such as dashboards, reports, and predictive modelling.

So where does Agile come in?

My key focus, every day, is to build data warehouses using Big Data platforms and BI tools. In particular, I apply Agile methodology to the creation of data warehouses and in turn, the outputs they can generate.

For businesses to remain relevant in this fast-paced world, they simply can’t wait 12 months to get answers to their problems. By taking an Agile approach to building a data warehouse I focus on the needs of the business, instead of the needs of the data. Traditional, data-driven methods are complex, time-consuming, and expensive, and typically result in poor adoption due to the sheer amount of time taken to see any results.

A business-driven methodology, which is the focus of an Agile data warehouse, takes an iterative approach, which reduces complexity as well as risk. This means that we can start to deliver results much quicker and continually test them against the business problem to ensure that we are on the right track.

Agile data warehouse development

Agile is not a new concept, but it is relatively new in data warehousing, where the impact of its methodologies has been revolutionary. Agile focuses on delivering an MVP (Minimum Viable Product) as opposed to a fully built solution, with business needs then driving refinements and innovations to that MVP. By building in this way speed is increased, with a more relevant data solution replacing the older, expensive (and often redundant) warehouses delivered under the traditional methodology.

Using Agile methodology in data warehousing, and then extending Agile to the analysis of the data, takes organisations to another level. Business performance can be measured faster and with a richer data set. An iterative, Agile approach also encourages experimentation, offering better outcomes. By gathering and analysing data at an Agile pace, organisations can quickly reach decisions on implementation or discarding of experiments. By using new methodologies and tools, organisations can quite literally begin to “know what they don’t know” and get answers much faster.

How does an Agile data warehouse build stack up against a traditional build? Here are a few starters:

  1. Real time – Unlike the waterfall methodology (where the delivered solution is the result of significant time and expense), Agile warehouses are developed in real time, and are responsive to changing business needs. This is so important in today’s fast-moving world.

  2. Stakeholder involvement – A successful Agile data warehouse is delivered through collaboration. By allowing all stakeholders to play ongoing roles in the build, the end solution is delivered fully understood by the business.

  3. Early refinement – By delivering an early MVP, which is then refined based on data, a solution alive to the nuance of the business is created, resulting in a better designed and utilised system.

  4. Continuous improvement – The nature of Agile means that the data warehouse is always improving and evolving, giving you an ever changing and ever refining view of the business.

But it’s not all roses… in fact, a common mistake is to associate Agile development with freewheeling and unplanned build methodology. We all know that chaos does not lead to success, and in fact, the real key to successful Agile builds lies in governance.

Keys to Agile data success

There are a few common traits I see in successful Agile data warehouse builds.

Speed to delivery
Remember you are building an MVP. The solution you are delivering is designed to deliver on business outcomes, but needs to evolve during your build as you learn more about your business needs. And post-delivery it needs to be flexible enough to adapt to your ongoing and evolving business problems. Version 1.0 does not need to be a perfect, finished solution – that’s the old, expensive, redundant way – one that will ensure high complexity, high risk and low adoption before you even get started. Perfection can be achieved as development continues.

Agile teams
Get yourself a good, adaptive, flexible team. Agile is applicable from development through to analysis – that means Agile should be part of your DNA, from developers through to your scientists and analysts. Be collaborative – this is a team sport – the closer you can get your developers and analysts, the more chance you have of success.

Make mistakes
Failing fast, identifying dead ends, and throwing resource into the opportunities this identifies… this is the lifeblood of Agile. If you are not failing, and fast, you’re not doing it right.

Collaboration
At the build stage collaborate across the business, make sure you understand your user stories, and build to deliver on these. Keep your users and analysts totally in touch, collaborating and iterating as you go.

From the manager making buying decisions based on data, to the call centre using churn risk scores, and all the way to the C-suite setting deeper strategic direction for the organisation – everyone can contribute to and profit from a collaborative and Agile approach to your data.

Just remember, it’s not about the need for perfection – it’s all about the need for speed!

Subscribe to the blog