How I help companies turn data into insights

I’ve been working with data for over 25 years now and in BI specifically for around 10. Currently I’m a BI consultant and help companies realise their projects and turn their data into insights.

But what exactly does my job entail? What does a BI Consultant actually do?

Well, as with any consultant, the client is looking to solve a problem but doesn’t have the requisite expertise to do so. So they’re looking to ‘consult’ an expert.

For me, as a BI consultant, there are 2 typical scenarios that account for the majority of the jobs I work on. A client will contact me either because a) they don’t have, at the time, any BI in place (that’s to say they’re not yet analysing their data in any meaningful way) or b) they already do some kind of reporting but they want to level up from Excel or whatever basic BI they do have to something more automated, interactive and shareable.

The approach to take when dealing with either of these 2 scenarios is pretty much identical so I’ll go over the process with you from where to start through to eventually delivering the project to the client.

Where to start?

You might think that the best place to start would be to do some kind of data audit and see what data the client has, where and how it’s stored etc. But, in fact, the best place to start this kind of BI project is actually at the end. To get the client to describe to you, in as much detail as they can, what the end result of their project looks like to them. So questions like, what do the reports or dashboards look like? What are the KPIs they want to monitor and track? Then, what kind of interactivity options would they like to be built into the reports? And who are they going to be sharing the reports with? How? And How often?

It’s also at this stage where I do 2 other important tasks. Firstly, I evaluate the client’s expressed requirements for feasibility, both technically and in terms of budget.

Sometimes, clients want to achieve things that aren’t even technically possible, either given the way their data is set up or based on the limitations of the BI tool they’ve chosen. Sometimes a client’s choice of tool is based on their budget which is what I mean when I say feasibility in terms of budget.

So I guess you’d call this ‘managing expectations’. The other thing I do is to use my experience of having worked on lots of different projects to suggest improvements or additional functionalities that the client may not have thought about, or even knew existed. This is where you can really earn your money. You can take a client’s basic, limited initial requirements and turn their project into something all singing all dancing. Which makes for a very happy client.

So, once I’ve let the client explain what their completed project looks like to them, I’ve managed expectations and proposed improvements or additional functionalities, we end up with what is, essentially, the overall scope of the project.

For those of you familiar with project management, you’ll know that scope is one of the 3 elements that make up what’s often called “the iron triangle”. The other two being time and cost.

In some cases where the client hasn’t chosen a particular BI tool already, defining the overall scope helps determine which tools are suitable for the project and which can be excluded — in fact some clients only need you to recommend the best BI tool for their needs. And, as I said before, some might have chosen a tool thinking it will do everything they need it to but only once you’ve understood their needs do you find that it doesn’t. So, you see, it’s very important to define scope first before going any further.

Scope and Spec

Once the overall layout of the project has been agreed upon, it’s time to dig a little deeper and define the scope in more detail. What this usually means is to spec out the project by making a list of all the queries that will be contained in each dashboard or report and where the data for each of these queries comes from. If it comes from more than one data source, how are they joined together? It’s kind of like a roadmap and data shopping list all in one.

There’s actually a spreadsheet I created to do this detailed spec and if you’re interested in getting your hands on a copy, I’ll leave a link below at the end of the article.

Once the project has been spec’d out, you can then start to calculate the time it will take to carry out and deliver the project and, based on this, its cost.

Now, I always recommend discussing the budget with the client even before any of the spec’ing is done. Because it’s pointless to go through all of that work when the budget just isn’t there. So making sure the client is aware of your basic daily or hourly rate is essential beforehand.

Calculating cost

But how exactly do you calculate the cost of the project to the client? Well, it isn’t just a case of multiplying the number of queries or visualisations you’ll need to build by a specific length of time. Instead, it’s much more complex than that with a lot more variables to include.

Before even starting the project itself, you have to take into account things like any pre-sales work you may have done, initial calls with the client, reviewing the project to check the feasibility of certain queries and data joins, that kind of thing. And then there’s the pre-contract admin (creating the project spec, writing the proposal etc).

Before you start building dashboards, there’s usually (although not always) work to be done on preparing the data sources. This could be as simple as just connecting the sources to the BI tool and making sure the data is presenting as you’d expect. Or you may need to write complicated SQL queries to join data together and build ‘views’ of databases. There may also be the need to build a full data warehouse that serves the project.

In my experience, I’d say that, on average, this data part of the BI project makes up around 50% of the entire project. Creating a query behind a data visualization, customising it and adding it to a dashboard can take just a few minutes or clicks. But to get to the stage where the data is set up to deliver the results you need can take hours or even days.

In fact, one project I worked on for, let’s say, a multinational soda company was just a simple scorecard with lots of single-figure KPIs on it. It only took me around an hour to build and design. But the data set up was in lots of different databases in different departments that, for security reasons, we couldn’t connect directly to. So probably 95% of the project’s time was taken up sorting out the data part. It’s a great example of how BI projects aren’t just about building dashboards. If you’d like to know more about this particular project perhaps I could write a case study on it? Let me know.

Other things to take into account when costing a project are things like estimating the number of times you’ll need to meet with the client during the project to monitor progress and check that everything is how the client wants it. And then you’ll need to account for any corrections that might need to be done after delivering the initial draft of the project, as well as any contingencies that may not have been foreseen. Finally, are there any post-project materials that you’ll need to deliver like training or best practice guides for getting the most out of the reports etc.? All this needs to be considered.

Basically, you can safely take the time you think it would take to produce the project deliverables and multiply this by around 1.6 to 2, based on the complexity of the project and the amount of detailed information about the project available at the time. If the information is a bit sketchy and thin on details, there’s a lot more room for error or misinterpretation which could result in having to re-do certain things. You get the idea.

If you’re being asked for a ballpark figure by the client then this gives you a good guide. If you can’t work out a figure as a starting point then one really unscientific way to go would be to count the number of different charts, graphs and tables on the reports the client wants you to build and multiply this by 20 or 30 minutes. This is of course assuming they’re looking to upgrade their existing reports and they can show these to you. Otherwise you’ll just have to do your best guesstimate.

Choosing the right approach

So. So far we’ve spec’d out the project and produced a costing for the client. Now, in some cases, it’s possible that the client will come back with some hesitations or reservations about the cost. In these cases there’s a very simple solution that will put the client at ease and also make your life slightly easier as well. And that is to propose a sprint-based project.

If you’re not familiar with what sprints are, they’re part of agile project management methodology. Basically, you take the whole project and break it down into different coherent parts, or mini-projects. Then you finish one sprint within a set amount of time before moving on to the next one. The obvious benefits are that the client gets to see your work and the way you manage the project without having to commit to investing all of their budget at once. So essentially, reducing their risk. The client may also want to change their mind about some aspects of the project after the first sprint which, if you were working on the full project, would mean more time to correct or rejig things.

Overall, there’s less pressure on both sides to get everything right from the off, leaving more breathing room to let the project develop organically.

In fact, I would recommend proposing a sprint-based project to the client from the off (unless it’s not a very large project on the whole). It may be an approach that they haven’t themselves considered and they might see the benefits of such an approach.

You do, however, always run the risk of shooting yourself in the foot with sprints. Why? Well, because once you’ve delivered the first sprint (let’s say a specific dashboard). The client can, if they’re so inclined, go into the tool and see behind the scenes how the dashboard has been set up. They might think to themselves that it’s actually not that complicated and decide to do the rest themselves. Especially if you’ve already done all of the complicated work on building the data sources. So this can leave you a bit out in the cold and earning less from the full project than you expected, which can be a bit annoying and frustrating.

Ok, so once everything has been agreed upon (scope, cost and time), you can then start the project. It’s always a good idea to, at least at the very beginning, meet with the client regularly to present what has been produced so far and to clear up anything you might not be 100% sure about in the spec. And clients always feel reassured when you keep in close contact.

Other than that, you just get on with producing the deliverables. Because you’ve been keeping in touch with the client along the way, it’s not like some kind of “big reveal” at the end. It’s usually more a question of confirming everything that was left to confirm and handing over the project to the client, along with any post-project materials. The client can then check over everything and sign off on the project.

Don’t get stuck

This last step is very important. If you don’t get a final sign-off for what you’ve been commissioned to do, there’s always the risk that the client will come back with things like, “would you mind just adding this extra chart to the dashboard?” or “we’ve decided we need to change the order of some things on the reports. It’ll only take a minute and you’ve already done the work.” When all is said and done, it’s really down to you whether you don’t mind doing these little add-ons after the project has been delivered but I believe it’s always best, instead of doing this for free, to try to agree with the client some kind of project maintenance agreement. A recurring monthly or quarterly fee that would include these little extras as well as making sure everything is working as it should. For me, it’s by far the more professional way of working.

Get your hands on my project specification spreadsheet here!