Nope. This post is not about Hadoop. It’s about agile – about how I learned some of agile’s key principles and about how to apply agile to the work of Business Intelligence.
In 2009 I attended an agile hands-on workshop led by Alistair Cockburn, one of the writers of the original Agile Manifesto. He indicated that we would be coding in pairs. My partner had not brought a computer, and I had spilled water on my keyboard a few minutes earlier – so we didn’t have a computer or a fancy IDE like everyone else. This limitation actually served to instruct us: Do what you can with what you have. I pulled out my 3-year old T-Mobile MDA (Phone with Windows Mobile 6) and readied a text editor and a browser. While everyone thought I was somewhat crazy, Mr. Cockburn smirked at its adequacy. We were to code a calculator. Unlike traditional software development, the specs were not fully outlined for us at the beginning. Rather, our dear instructor started with: “I want to be able to input something and you have to show me what I put in. You have 3 minutes.”
You see, elephant carpaccio is about eating a large animal one (very thin) slice at a time (carpaccio is an Italian hors d’oeuvre consisting of thin slices of raw beef or fish served with a sauce). Alistair Cockburn puts it this way:
Agile projects don’t do away with project management. In a sense, they rely more heavily on it. Someone has to do the work of breaking large ideas into bite sized, DELIVERABLE pieces. That’s alot of work. Without this, a project is simply not agile. It is people trying to deliver a beheamoth all at once.
Business Intelligence ideas begin when someone wants to acquire useful business information. This one usually knows where to get the data and has been manually retrieving it, transforming it and decorating it in order to best understand it. Business intelligence projects begin here.
There’s no one way to skin an elephant, but here is an attempt at breaking up one business intelligence project for illustration purposes.
- The biggest mistake is to start providing a solution right off the bat. The first thin slice of this project should be to understand and document the manual process that already exists. That documentation is validatable and deliverable.
- The next small step can be to begin to automate one of the steps of the manual process – say, the retrieval of the data. Upon doing this, data can be delivered and demoed prior to transforming it and decorating it. Together with the last step, this step surfaces sourcing issues. By delivering the raw data to a group of end users, they may discover that different ones relied on different sources for the same type of data. If the raw data is not delivered at this point, future critical Business Intelligence slices for streamlining data flow are hampered because they may lack proper business analysis.
- Another small step is to relieve the manual data gatherer/reporter by delivering the newly automated reporting system. The report delivered in the new system, however, is not so much the deliverable, as is the platform and framework. No real improvements are seen in the report besides the time element that it takes to deliver it. Now it’s automated. And that’s it. The automation is the deliverable. It is validatable at this point and the business is fully apprised of the progress of the Business Intelligence project.
- Value-added features such as heat maps, drill-throughs, charts, graphs – all the cool stuff that we were sold on with “Business Intelligence” aren’t delivered yet. Each one of those may constitute separate deliverable slices of the elephant.
Another way to break up Business Intelligence projects is to create separate small sprints that get delivered to different customers.
BI has DBA customers, Business Analyst customers, Data Governance Counsel customers, and perhaps even Operational users as customers. The truth about Business Intelligence is that data is a company’s asset. Different individuals or groups of people want it at different levels. Some want raw data sets, some want detailed reports, and some want top-level KPIs. If you think of your BI customer as one individual or one department, you are probably thinking of the elephant that has to be delivered in 6 months. If you consider the data as an asset that the Business Intelligence team needs to quickly make available to different people in different ways, then you can come up with countless thin slices that are deliverable and that bring value to an organization.
Perhaps today we make a restaurant of datasets with a menu that will satisfy a small number of do-it-yourselfers. Perhaps tomorrow we agregate some of that data and pump it into a spreadsheet for a die-hard Excel user. Perhaps the next day we analyze various redundant data sources and deliver a process for curating data.
Here are the keys I hope you take away from this post.
- Deliver – and collaborate. Do this as often as possible. End-users who validate your work give you the feedback you need and you ensure that they have skin in the game. The first version does not have to have everything. It just needs to be deliverable. Trust me, that little thing you just did, that’s valuable to your customer. Don’t make them wait for it. Give it to them now! Do this with each feature and do it often. There’s nothing wrong with delivering value daily.
- Get validated. Customers will be happy to see that what you delivered is correct. Leave bells and whistles for later.
- Give them credit. Most end users know their domains well. If they have data, they can derive business information, ideas, and decisions themselves.Therefore, start by giving them data. They will thank you for it and will be more willing to work with you and entrust you with business logic that you can build into reports in subsequent sprints.
In Conclusion (Yes, this part is for you, Business Intelligence Professional)
If you have a BI framework in mind that has many layers and many customers, part of your job is to break that up into much smaller deliverable slices. That’s the way to continuously add value through software development and through business intelligence. Don’t be fooled, though. This work is hard. Business intelligence professionals must gain the necessary skill: learn to write micro- user stories from the use cases so your agile business intelligence teams can work in micro-increments. It doesn’t come for free. BI project managers and team members will have to work closely with one another and with their customers to mature their agile discipline.
I delivered my javascript calculator on that old smartphone and learned that value can be had in micro sprints. It’s more than doable. It relieves hunger and is tasty, just like a thin slice of Carpaccio.
