Wednesday, 20 July 2016

Hidden Assumption of Agile

Agile cures common problems that we experience in software development, however, there are limitations to Agile.  It may seem like a silver bullet, but there are circumstances under which Agile is not the best choice for development, or at a minimum, not to develop the entire project.

The problems with the waterfall model are well known and understood by most by now.  The waterfall methodology at a high level appeals to executives because it provides a clean way of thinking about a problem.

After all:
  • you can't design something before you have the requirements
  • you can't code something before you have the design
  • you can't test something before you do the coding
These facts are true in Agile as well.  The only difference is that you don't try to do all of any phase as a big bang.  We understand that unless we use some kind of iterative process we can't return to a previous phase and fix anything.  So the Agile process looks as follows:

That is, instead of doing all of one phase as a big bang, we do a bit at a time.  Some methodologies call the code that results from each sprint as a 'brick', and through building bricks, we eventually complete the wall.  So Agile is essentially slicing up the waterfall methodology, but all of the work that needs to be done is still done.
So why on earth would we use a waterfall process for ANYTHING?

Well one of the hidden assumptions of Agile is that you can build all software projects one brick at a time.  But, when was the last time that you saw contractors show up and just start building a skyscraper without a plan?

I mean, isn't it just a matter of building the skyscraper one layer at a time, why bother with planning?

Even though a skyscraper will be built one floor at a time from the bottom up, there still needs to be some architectural planning that happens first.  The materials in the support structure of the building together with the common components that provide common services to each floor (plumbing, electricity, air conditioning) need to be planned out in advance. 

If insufficient planning is put into the architecture, then insufficient plumbing is put in place to drive water to the top of the building, insufficient breakers are purchased to support the electrical needs, insufficiently strong materials might be put in place to hold the weight of the building.

To some degree software is malleable and you can add things after the fact.  But there is often a cost to add things after the fact which is much higher than if the element was designed up front.  For example, you could add add a parking garage to the Empire State building, but it would be cost prohibitive. If they had wanted a garage, it should have been built when the building was constructed.

So any project that will require serious architecture should pause and plan for the structural and connective elements that will be required by the project.  The requirements that are tied to the architecture should be developed first so that the architecture of the project is developed first.  Of course, the architecture can be developed using Agile or the traditional waterfall approach.

Once the architecture is in place, then the rest of development can be done using Agile knowing that all the structures and connective elements are in place.  This is akin to building the skyscraper one floor at a time once the architecture is planned.

The alternative to building out the architecture first is that you find yourself in the position of having architected a building of 10 floors and then realize that you need to have 30 floors. The initial architecture which came together quickly will be snowed under by the work arounds that you need to as you get above 10 floors.  You might get to 15 or 20 floors, but you will never finish the project.

Things that would be expensive to add after you have built a few floors:

  • Needing an extra elevator
  • Needing an extra floor in the parking garage
  • Needing the entire building cabled with fiber optics
Most projects do not require advanced architectures, just like most 2 story houses don't need to have architects.  But if your project is large and has serious architectural considerations then you are best suited to plan for the architecture up front.

Make sure to plan the architecture before doing Agile development

Friday, 8 July 2016

User stories or use cases?

Alistair Cockburn says "A user story is to a use case as a gazelle is to a gazebo".

If you are extremely familiar with both techniques then this makes sense.  If you are not familiar with both then suffice it to say that you can put a gazelle in a gazebo but not vice versa.

A user story is of the form As a <type of user>, I want <some goal> so that <some reason> (see here).  

A use case for withdraw money has many more components to it, and a skeleton of such a use case can be found here.  I develop this skeleton use case in a four part tutorial:
  • Part 1: Use case as a dialog
  • Part 2: What is an actor?
  • Part 3: Adding interface details
  • Part 4: Adding application context
The above user story is only a part of the entire use case.  For an ATM, a user story might be As a user, I want to withdraw money so that I can have physical cash.  This user story is only the summary of the withdraw money use case without the details.

User stories come from the Extreme Programming methodology. The assumption was that there will be a high degree of interaction between the developers, and the end customer and that QA will largely be done through test driven development.  It is important to realize that Extreme Programming does not scale.

Once your development team gets large, i.e. you have 3+ agile teams and your code base gets large enough to warrant a formal testing environment, then you will outgrow user stories as your only method of capturing requirements.

You can still use user stories for new modules developed with an end customer to take advantage of the light weight and rapid nature of user stories, but at some point those user stories should transition to use cases.

Unfortunately, there are quite a few ways to build use cases and not all of them are effective.  Alistair Cockburn's method (found here) is an excellent way to capture use cases for more sophisticated systems.  There is no doubt that use cases are heavier weight than user stories, but consider this:
  • Use cases can more easily be turned into test cases by QA
  • With use cases you more easily prove that you have all the requirements
  • Use cases annotated with screens and reports can be used to collaborate with remote off site customers
  • Well written use cases can easily be broken up into a sprint back log
  • Use cases will work if you are not using Agile development methods
So don't forgo the speed and dynamic nature of user stories, just recognize that there are limits to user stories and that you will need to transition to use cases when your project team or application grows.

Friday, 17 June 2016

Ready, Fire, Aim: How most gather requirements

Connecting with your customers and delivering value depends on understanding your customer's requirements and selling the correct product or solution that solves your customer's problems.

Commonly, the requirements gathering process is done hastily or not at all in the rush to get the sale. After all, the faster you can make the sales, the faster the money is in the bank -- correct?

The more that product customization is required, the more it is important understand the customer's actual problem.  Long lead times to build and implement a solution means will not only lead to difficult implementations but also to a more disappointed customer and a loss of future revenue.

If you build software, which has long lead times, it is even more important to make sure that what you are building will satisfy the customer's requirements as changing the final software solution will be nearly impossible and very costly.

Why do we continually misunderstand and sell the wrong solutions and building the wrong software?

Human Nature

Time pressures to make a sale put us under pressure and this stress leads to making quick decisions about whether a product or solution can be sold to a customer.  We listen to the customer but interpret everything he says according to the products and solutions that we have.

Often the customer will use words that seem to match exactly the products we have.  But then after the sale once we have to implement we often find that either we deliver a poor solution to the customer or a solution is infeasible and we must refund the customers money.

However, behind every word that the customer is using there is an implied usage, and understanding that implied usage is where we fail to gather requirements. For example, suppose the customer says I need a car.

Suppose that you sell used cars:
  • the customer asked for a car
  • you sell cars
Ergo problem solved!
  • What if the customer needs an SUV but you don't have any?
  • What if the customer really needs a truck?
  • What if the customer needs a car with many modifications?
  • What if the car the customer needs has never been built?
We hear the word car and we think that we know what the customer means.  The order-taker sales person will spring into action and sell what he thinks the customer needs. Behind the word car is an implied usage and unless you can ferret out the meaning that the customer has in mind, you are unlikely to sell the correct solution.

If you don't understand how the customer will to use your solution then you don't understand the problem.

Comedy of Errors

For products that require customization, the sale will get transferred to professional services that will dig deeper into the customer's requirements.  At this point you discover that the needs of the customer cannot be met.  This leads to sales people putting pressure on professional services and product management to find a solution, after all, losing the sale is not an option.

Sometimes heroic actions by the product management, professional services, and software development teams lead to a successful implementation, but usually not until there has been severe pain at the customer and midnight oil burned in your company.  You can eventually be successful but that customer will never buy from you again.


As WIlliam Ralph Inge said, There are no rewards or punishments -- only consequences. The consequence of selling the wrong solution to a customer is:
  • Whether you lose the sale or not, the customer loses faith in you
  • The sales person is perceived as incompetent in the rest of the organization especially by professional services and software devleopment
  • Your cost of sales and implementation is much higher than expected
  • Your reputation is damaged


Selling the correct solution to the customer requires that you understand the customer's problem before you sell the solution.

When customization is required, good sales people engage resources that can capture the customer's requirements accurately and assess that you can deliver a solution to the customer.

Slowing down to understand the customer requirements and how you will solve his problems is the key.  By understanding the customer's requirements and producing the correct solutions you become a trusted adviser to the customer.

See also

  • Shift Happens, effect of scope shift in a software development project.

Monday, 12 October 2015

Do Project Managers need Domain Experience?

Opinions vary on whether a project manager needs to have domain experience.  Certainly project managers that do not have domain experience will be the first to say that domain experience is not necessary as long as they have access to excellent subject matter experts.

I would advocate a more nuanced position; that is, a project manager does not need domain experience IF his subject matter experts understand the risks and dependencies that are inherent to the domain.

Let's go through a couple of personal projects that I have been involved with where the project manager did not have domain experience.

Telco Project

I am currently involved in a project that involves a LAN/WAN/WIFI upgrade of a large customer for a large telecommunications company.  The project manager does not have domain expertise in networks and is counting on the subject matter experts to provide him sufficient input to execute the project.

The subject matter experts are so advanced in their knowledge of networks that they no longer understand what beginners (i.e. the project manager) do not know.  They have assumed that when they indicate things to the project manager that he understands what they mean and will take appropriate actions.

The project manager is continually running into situations where he did not understand the implications of certain risks and dependencies.  This has caused a certain amount of rework and delays.

Fortunately, this is not a project with tremendous amounts of risk or dependencies so the project will be late but will succeed.

Mobile Handset Project

In the distant past, I was part of a team that was building a mobile POS terminal that worked over cellular (GSM, CDMA).  The project manager in this situation did not have domain experience and was counting on the subject matter experts.  In this case, the subject matter experts were very good at general design, but not experts in building cellular devices.

Because the subject matter experts were not specialists, they knew most of the key principles of designing mobile handsets but did not understand all the nuances of handset design.  There were several key issues required by practical handset manufacturing that were overlooked by the generalists and ended up creating such a strong cost over-run that the start-up went out of business.


In the first project, the subject matter experts were extremely good, however, the project manager failed to understand the implication of some of their statements and this introduced large delays in the project.

In the second project, the subject matter experts were generalists and did not understand all the risks and dependencies of the project.  The project manager (and start-up) were doomed to fail because "you don't know what you don't know". Both these projects show that a project can be delayed or fail because a project manager does not have domain experience.


So if a project does not have many uncertainties and dependencies then it is extremely likely that the project manager does not require domain experience and can rely to some degree on his subject matter experts.

However, if the project has complex uncertainties and/or dependencies then a good project manager without domain experience is likely to find himself in a position where the consequences of not understanding the uncertainties and dependencies will either introduce serious rework or torpedo the project.

See also