Agile Business Intelligence
Business Intelligence (BI) projects that incorporate key aspects of Agile processes dramatically increase the probability of a successful outcome.
I wonder why business intelligence (BI) projects have a reputation for being slow, painful and ineffective - and why do they often fail to deliver on the promise to improve data-driven decision-making? I believe part of the answer is in the approach: the waterfall, linear, command and control model of the traditional System Development Life Cycle (SDLC) that is still pervasive in most technology projects today. There is a better way!
One of the core principles of Agile is embracing a realistic attitude about the unknown. It is interesting that at the beginning of a traditional technology project, when the least amount is actually known about an organization and its business rules, environment, variables, players, questions and requirements, that the greatest amount of effort is made to lock in the scope, the cost and the schedule. It's understandable that we want to limit risk, but in reality the pressure to protect ourselves can lead to excessive time spent on analysis, which often still results in unclear requirements, leading to mismatched expectations, change orders and cost overruns. This is a well known phenomenon - at the very point where we have the least amount of information, we are trying to create the most rigid terms. See the “Cone of Uncertainty” concept.
I think part of the reason for this paradox stems from an intrinsic lack of trust. Steven M.R. Covey explains in his book, “The Speed of Trust”, that trust has two components: character and competence. In each situation in which you are asked to trust, you must have both. For example, if your best friend is a CPA, you might trust them as a friend, have complete confidence in their character and trust them to handle your taxes, but you will not trust their competency to perform surgery on a family member. It’s the same in business. We might have confidence in a vendor’s base software product, but not trust their ability to understand our needs or implement the solution well. And trust has to be earned. Once an organization has trust, the speed at which change can be communicated and accommodated dramatically increases. And this increase in speed translates into an improved outcome and a reduction in the cost, both of which are a by-product of the clear communication that is possible when trust is present.
What does all this have to do with business intelligence? I believe BI projects lend themselves to an agile, iterative approach, and this approach requires trust in order to work. I’m not a big fan of some of the Agile terminology - terms like “product backlog” (doesn’t “backlog” sound negative?) and “sprint” (is it a race?) But I do fully embrace the concept of working solutions vs. endless analysis, communication and collaboration instead of rigid process enforcement, responding to change vs. “hold your feet to the fire” denials of needed change requests. In general, it’s the concept of “outcome” vs. “output” that is so inspiring to me about Agile. I’ve seen examples where a technology project met all of the formal “outputs” specified in the contract, yet sadly failed to deliver the most important thing – the “outcome” that the organization was really trying to achieve. For example, the CRM implementation that was delivered on time and on budget but that none of the staff would use, or the BI project that resulted in dashboards that measured the wrong things. These are not examples of successful projects because the true desired outcome was not achieved.
How can Agile concepts be used in BI?
- Identify an initial high profile “win” and complete a small but important aspect of the project to inspire the team, generate enthusiasm, engagement and feedback
- Facilitate data discovery : create a hypothesis -> investigate and experiment -> learn -> ask new questions and repeat the process
- Value the learning and the teamwork that is intrinsic to the process and which builds trust and speeds the ability to adapt to change
In a future post I’ll debunk some of the common myths that surround the topic of agile processes.