Data centric projects & user stories

I've been able to work on a couple of types of data centric projects. One was creating an ODS based on consolidating customer data from multiple source systems. This involved finding the data, extracting, transforming, & loading, as well as matching customer records, de-dupping, address matching and data cleansing. Now I've been working with some teams who are focused on moving data into a data warehouse. When first working with data teams a typical reaction is that we can't do user stories. Or we have to complete the whole data model done up front. Or we have to load all the data at once otherwise we're duplicating work, ... Basically I get a reasonably long list of why we can't do incremental development for data centric projects.

Well the truth is different. I agree with the data guys that some upfront modeling is required. We need to know the lay of the land. We also need to make sure the new data coming in and the proposed data model changes meet the enterprise data governance standards set up by the IT Architecture group. That means we need to spend some time up-front doing analysis and design work. We don't need a "complete" data model with every detailed identified. What we need is the team to understand agile data modeling and in particular evolutionary data modeling. Scott Ambler provides a thorough description of agile data modeling, so if you're interested in some additional detail check it out here.

Another obstacle is when trying to determine the work to be completed in an iteration. My experience has been that teams are reluctant to break stories up, so that partial end-to-end functionality is completed. Instead the teams have wanted to break the stories up by activity (e.g., analysis, design, development, QA) and it's been a bit of struggle to get them something different.

By way of a made up example let's say the business wants to know it's most profitable products for a product line. Let's assume part of the problem is that the data needed is in multiple systems. Here's one of the things the business wants:

• As a product manager I want to see product line products ordered by profit so I can manage and adjust product offerings to improve profitability for a product line

Let's say the team estimates this story to be too large to start and complete within a single iteration. The team might suggest we break it up by activity, but why not get some functionality completed and show that to the busines. Maybe we could break that larger story into 2-3 stories.

• As a product manager I want to see product line products ordered by cost so that...

This would give the business a look at all products within a product line showing the costs associated with each product. They might find value in this, especially if similar products have high degrees of variation in costs. Other possible stories:

• As a product manager I want to see the revenue by product so that ...
• As a product manager I want to see the profit by product so that ...
• ...

These smaller stories provide value to the business and allow the team to complete working software in smaller chunks without have to have all data extracted, transformed and loaded before we begin providing the business with valuable functionality.

The point is to keep the stories about functionality and not fall into the trap of activity based stories or, so called, technical stories that really just represent technical tasks that need to be completed.