The Business Intelligence Department

The executive who was running the BI department wanted to increase productivity of the department and the quality of their output. The bottom line was, they were too slow and cost too much — yes I bet you never heard that before. This was a Fortune 100 company so there was plenty of bureaucracy and change does not always come easy but never the less, she was driven to make a difference and determined to fight through obstacles. I couldn’t have asked for a better advocate.

The BI department consisted of approximately one hundred and twenty five staff members in 2 main locations within the USA and another one-hundred external staff members from their sourcing partner working primarily from India. The Charter of the organization was to provide services across 6 areas:

• Data - Aquire and integrate internal and external data into the enterprise systems, secure the data and provide extracts and/or API access.

• Analytics - Support business analytics teams throughout the organizaiton. This was a team of statisticians, data scientists and internal consultants who provided analytics and statistical consulting support to departments across the company.

• Reporting - Support and provide standard company reports as well as ad hoc reporting services. This area was largely outsourced and eventually would be winnowed down through self-service and automation.

• Data Science - Support the big data ecosystem via data analytics to drive insight into corporate strategy and direction.

• Self-Service - This group was dedicated to providing and supporting self-service reporting and analytics tools.

• Consulting - This was the internal consulting group that worked with senior leaders on the “art of the possible”, evangelizing BI capabilities and working with leaders to demonstrate how data can be leverage in new ways.

I will discuss a few of the important decisions and challenges along the way and possibly our solutions could help projects faced with similar issues.

Which Flavor of Agile

I have delivered software using and been trained in some of the major Agile Methodologies - DevOps, Scrum, SAFe, and LeSS — up to this time I had used these methodologies within traditional software development departments not in a Business Intelligence delivery model.

It seemed that everyone had an opinion on which flavor of Agile we should use ultimately it came down to SAFEe, Scrum, Extreme Scoping or a SAFe/Scrum hybrid that part of the organization was using. We eliminated Extreme Scoping, it is an Agile methodology that didn’t get much traction but it had the advantage of being designed explicitly for BI/EDW organizations. While the focus on BI was of course a major benefit finding resources familiar with it — including the outsourcing vendor — was impossible, so we passed. I have always thought that SAFe has advantages over Scrum for large companies and I am no fan of “hybrid” models - that is when companies start picking and choosing or combining models — it is usually a mistake. However, in this company’s environment we choose the company standard or at least as close to a standard as they had and we opted for the SAFe/Scrum hybrid. The BI sourcing vendor was very familiar with it, tons of training materials existed, internal staff were familiar, and it was relatively easy to find external resources if needed.

Distributed Agile Delivery Models

When ever the topic of distributed Agile comes up, my business partner always says “distributed Agile it’s a varsity sport”. I have to agree, that is an apt metaphor. This is exacerbated when the teams are offshore, vendor supported and in this case a Agile is a new way of working. The challenges implementing distributed Agile were significant, I will outline some of the obstacles and lessons learned may help others in similar situation.

Communications

This was without a doubt the primary hurdle we faced. Agile typically depends on small teams communicating closely while developing iteratively. In a distributed environment this is more difficult. The delivery model you choose to implement has to address the communication challenge. We put a tools in place to assist in lowering barriers and foster connections - these were put in place before we started not after we had the issue. Video conferencing, robust messaging platform (e.g. MS Teams). We made cultural shifts to work in this new mode.

We started with a basic structure that kept Product Owners onsite and the remaining team member off. This was the easiest model to implement. We had to ensure Product Owners felt like part of the team and were not left out of any meetings or decision making as they tended to be isolated. As the teams matured we implemented, in a POC fashion a “follow the sun” methodology. We experimented with colocated teams for Product Owners, Developers onshore and test teams 100% offshore. We also tried the approach where work was simply broken up as 100% offshore or 100% onshore - the issues here were mostly around the offshore teams loosing focus and drifting of course so in the end we opted for one of the two split models described above.

Motivating staff - this goes hand in hand with communications. You have to be cognizant of the fact that the offshore team needs to be recognized for their work. I have seen the offshore group get ignored when it comes time for accolades or take the lion’s share of the blame when something goes wrong. You need a leader who understand the One Team concept and success is shared.

Tool Choices

We spent a significant amount of time thinking about what we needed to be successful and matched tools to that need. All too often I see firms buy tools and “figure it out later”. First and foremost made sure that we picked tools in context of our needs and determined how the tool(s) under consideration would work in our environment. Some of the other areas we had to be cognizant of were - corporate standards, industry adoption (we didn’t want to pick an obscure tool and then have to train external staff), integration with our technical environment, and self-study as well as traditional training availability. We also wanted to take a best in suite approach to limit the number of different vendors and integration issues we could face. Some tools we choose by category are listed below:

• Application and operational analytics - AppDynamics

• Agile project management - this was a close choice between Atlassian vs IBM Engineering Workflow Management. In the end Atlassian Jira/Confluence won the day. Which wanting to stay in a suite we used Atlassian for CI/CD

• Test Automation - Our test teams used both Jira and IBM test tools. QA tracking was maintained in Jira

Organizational Change Management

We knew from the beginning that to be successful that a substantial amount of time needed to be spent on Organizational Change Management (OCM), for us this included a marketing campaign to sell the benefits and communicate successes, rigorous training for all team members - this included coaches, internal and external classes, mentoring programs, and external consultants to bridge the staffing gap. We also rolled in team member rotation plans into the OCM umbrella, this helped to keep the staff fresh, motivated and eager for new challenges. Technical staff often forget the OCM component but it is vital to the success of a new Agile rollout. We had to pay special attention to training because while I was used to dealing with developers who even if they didn’t have hands on Agile experience they were almost always familiar with the concepts and usually have studied the topic. BI staff members in contrast had little direct Agile experience so training and mentoring was a must.

Moving from a waterfall methodology isn’t for the faint of heart it is doable with some planning and patience.