More on Scrummerfall

A couple of comments have led me to think that I didn't explain what it is.  Let's review waterfall phases:

• Requirements specifications
• Design
• Implementation
• Integration
• Testing
• Deployment

Each of these has a gated exit, such that to exit one phase, you need to meet certain criteria.  For example, to leave "Design" phase, you have to have your detailed design with estimates signed off by developers, analysts, and the customer.  You cannot enter Implementation until you have finished design.

Scrummerfall still uses a phase-based methodology, but uses iterations for the "Design" and "Implementation" phases.  Testing is done as unit tests during development, but QA is not involved until the actual Testing phase later.

Scrummerfall is easier to introduce in companies heavily invested in waterfall, as it's only one group (developers) that are actually changing how they work.

Scrummerfall also makes two assumptions that become more invalid and costly the larger the projects are:

• Requirements don't change after Design
• Integration, Testing, and Deployment are best done at the end

Now just because Agile doesn't have gated phases for these activities doesn't mean they don't happen.  Design and requirements gathering still happen, as do release planning, testing, deployment, etc.  The difference is that all of these activities happen each iteration.

This is very tough in fixed-bid projects, which assume that requirements, cost, and deadline don't change.  There are alternatives to fixed-bid projects, which I won't cover here, that provide the best of both fixed-bid and time-and-materials projects.

With Agile, you don't do a "Testing" iteration and a "Design" iteration.  That's lipstick on a pig, you're still doing waterfall. 

So how do you avoid Scrummerfall if you're trying to introduce Agile into your organization?  The trick is to sell the right ideas to all of the folks involved.  If it's only developers leading Agile adoption, chances are you won't get too far past TDD, continuous integration, pair programming, and the rest of the engineering-specific XP practices.

Get buy-in from an analyst, a PM, a tester, your customer, and your developers.  You won't have to convert all of the analysts and PMs, just the ones working on your project.  Remember, each person needs to see tangible business value from the changes you are proposing.  I tend to target management for Scrum and developers on XP, because although it's easy to get everyone to agree on values and principles, the concrete practices vary widely between the different roles.

For the record

This is not Scrum:

• Planning
• Release planning
• Backlog creation
• Architecture and high-level design
• Development sprints
• Design
• Code
• Test
• Conclusion
• System integration
• System test
• Release

This is Scrummerfall, where we still do a phase-based waterfall model, but do iterations during the "development" phase.  I hear the word "hybrid" thrown around a lot in the above model.

This isn't the cool kind of hybrid, like a Prius or a Liger (pretty much my favorite animal).  It's more of the Frankenstein hybrid, where it looks good on paper, but in the end you need the village mob with pitchforks and torches to drive it away.  Afterwards, the villagers have a bad taste in their mouth regarding science, so they banish it instead of booting out the mad scientist.

One of the key benefits of agility is the ability to respond to change through feedback.  If the backlog is set in stone before my "development sprints" start, how do I change the backlog once business requirements and priorities change, which they inevitably will?  There is no mechanism to respond to change in Scrummerfall, dooming your project to quick failure, but now you get to blame Scrum.

Another disaster waiting to happen is waiting until the end to do system integration.  I think everyone learned that "it works on my computer" doesn't fly very far once you start collecting a paycheck.  Customers demand the software works on their machine, not yours.  So why wait until the very end to do the riskiest aspect of development, when cost of failure is at its highest?  It pretty much guarantees failure and some exciting blamestorming meetings.

I think these models come about from those who think Agile and Scrum are just another process, a cog to switch out.  Agile isn't a process, it's a culture, a mindset, a belief system, a set of values, principles, and practices.  Treating it like just another process gives rise to tweaking and leads to hideous hybrid Frankenstein monsters, roaming the countryside and spreading failure.

Intention-concealing interfaces: blob parameters

When someone is using your code, you want your code to be as explicit and easy to understand as possible.  This is achieved through Intention-Revealing Interfaces.  Evans describes the problems of opaque and misleading interfaces in Domain-Driven Design:

If a developer must consider the implementation of a component in order to use it, the value of encapsulation is lost.  If someone other than the original developer must infer the purpose of an object or operation based on its implementation, that new developer may infer a purpose that the operation or class fulfills only by chance.  If that was not the intent, the code may work for the moment, but the conceptual basis of the design will have been corrupted, and the two developers will be working at cross-purposes.

His recommendation is to create Intention-Revealing Interfaces:

Name classes and operations to describe their effect and purpose, without reference to the means by which they do what they promise.  This relieves the client developer of the need to understand the internals.  The name should conform to the Ubiquitous Language so that the team members can quickly infer their meaning.  Write a test for a behavior before creating it, to force your thinking into client developer mode.

Several times now I've run into blob parameters (I'm sure there are other names for them).  Blob parameters are amorphous, mystery parameters used in constructors and methods, usually of type ArrayList, HashTable, object[], or even params object[].  Here's an example:

public class OrderProcessor
{
    public void Process(object[] args)
    {
        Order order = (Order) args[0];
        Customer customer = (Customer) args[1];

        // etc.
    }
}

The intentions behind this code are good, but the results can lead to some frustrating results.  Furthermore, this pattern leads to "Intention-Concealing Interfaces", which is style completely opposite from the Intention-Revealing style Eric proposes.  Clients need to know the intimate details of the internal implementation of the component in order to use the blob parameters.

Wrong kind of extensibility

Blob parameters lead to the worst kind of coupling, beyond just "one concrete class using another concrete class".  When using blob parameters, the client gets coupled to the internal implementation of unwinding the blob.  Additionally, new client code must have intimate knowledge of the details, as the correct order of the parameters is not exposed by the blob method's interface.

When a developer needs to call the "Process" method above, they are forced to look at the internal implementation.  Hopefully they have access to the code, but otherwise they'll need to pop open Reflector to determine the correct parameters to pass in.  Instead of learning if the parameters are correct at compile-time, they have to wait until run-time.

Alternatives

Several alternatives exist before resorting to blob parameters:

• Explicit parameters
• Creation method
• Factory patterns
• IoC containers
• Generics

Each of these alternatives can be used separately or combined together to achieve both Intention-Revealing Interfaces and simple code.

Explicit Parameters

Explicit parameters just means that members should ask explicitly what they need to operate through their signature.  The Process method would be changed to:

public class OrderProcessor
{
    public void Process(Order order, Customer customer)
    {
        // etc.
    }
}

The Process method takes exactly what components it needs to perform whatever operations it does, and clients of the OrderProcessor can deduce this simply through the method signature.

If this signature needs to change due to future requirements, you have a few choices to deal with this change:

• Overload the method to preserve the existing signature
• Just break the client code

People always assume it's a big deal to break client code, but in many cases, it's not.  Unless you're shipping public API libraries as part of your product, backwards compatibility is something that can be dealt with through Continuous Integration.

Creation Method/Factory Patterns/IoC Containers

When blob parameters start to invade constructors, it's a smell that you need some encapsulation around object creation.  Instead of dealing with changing requirements through blob parameters, encapsulate object creation:

public class OrderProcessor
{
    private readonly ISessionContext _context;

    public OrderProcessor() : this(ObjectFactory.GetInstance<ISessionContext>())
    {
    }

    public OrderProcessor(ISessionContext context)
    {
        _context = context;
    }
}

Originally, the ISessionContext was a hodgepodge of different object that I needed to pass in to the OrderProcessor class.  Since these dependencies became more complex, I encapsulated them into a parameter object, and introduced a factory method (the "ObjectFactory" class) to encapsulate creation.  Client code no longer needs to pass in a group of complex objects to create the OrderProcessor.

Generics

Sometimes blob parameters surface because a family of objects needs to be created or used, but all have different needs.  For example, the Command pattern that requires data to operate might look like this before generics:

public interface ICommand
{
    void Execute(object[] args);
}

public class TransferCommand : ICommand
{
    public void Execute(object[] args)
    {
        Account source = (Account) args[0];
        Account dest = (Account) args[1];
        int amount = (int) args[2];

        source.Balance -= amount;
        dest.Balance += amount;
    }
}

The ICommand interface needs to be flexible in the arguments it can pass to specific implementations.  We can use generics instead of blob parameters to accomplish the same effect:

public interface ICommand<T>
{
    void Execute(T args);
}

public struct Transaction
{
    public Account Source;
    public Account Destination;
    public int Amount;

    public Transaction(Account source, Account destination, int amount)
    {
        Source = source;
        Destination = destination;
        Amount = amount;
    }
}

public class TransferCommand : ICommand<Transaction>
{
    public void Execute(Transaction args)
    {
        Account source = args.Source;
        Account dest = args.Destination;
        int amount = args.Amount;

        source.Balance -= amount;
        dest.Balance += amount;
    }
}

Each ICommand implementation can describe its needs through the generic interface, negating the need for blob parameters.

No blobs

I've seen a lot of misleading, opaque code, but blob parameters take the prize for "Intention-Concealing Interfaces".  Instead of components being extensible, they're inflexible, brittle, and incomprehensible.  Before going down the path of creating brittle interfaces, exhaust all alternatives before doing so.  Every blob parameter I've created I rolled back later as it quickly becomes difficult to deal with.

Stop the madness

I've been extending a legacy codebase lately to make it a bit more testable, and a few small, bad decisions have slowed my progress immensely.  One decision isn't bad in and of itself, but a small bad decision multiplied a hundred times leads to significant pain.  It's death by a thousand cuts, and it absolutely kills productivity.  Some of the small decisions I'm seeing are:

• Protected/public instance fields
• Inconsistent naming conventions
• Try-Catch-Publish-Swallow
• Downcasting

The pains of these bad decisions can wreak havoc when trying to add testability to legacy codebases.

Public/protected instance fields

One of the pillars of OOP is encapsulation, which allows clients to use an object's functionality without knowing the details behind it.  From FDG:

The principle states that data stored inside an object should be accessible only to that object.

Followed immediately by their guideline:

DO NOT provide instance fields that are public or protected.

It goes on to say that access to simple public properties are optimized by the JIT compiler, so there's no performance penalty in using better alternatives.  Here's an example of a protected field:

public class Address
{
    protected string zip;

    public string Zip
    {
        get { return zip; }
    }
}

public class FullAddress : Address
{
    private string zip4;

    public string Zip4 
    {
        get
        {
            if (Zip.Contains("-"))
            {
                zip4 = zip.Substring(zip.IndexOf("-") + 1);
                zip = zip.Substring(0, zip.IndexOf("-"));
            }
            return zip4;
        }
    }
}

There was an originally good reason to provide the derived FullAddress write access to the data in Address, but there are better ways to approach it.  Here's a better approach:

public class Address
{
    private string zip;

    public string Zip
    {
        get { return zip; }
        protected set { zip = value; }
    }
}

I've done two things here:

• Added a protected setter to the Zip property
• Changed the access level of the field to private

Functionally it's exactly the same for derived classes, but the design has greatly improved.  We should only declare private instance fields because:

• Public/protected/internal violates encapsulation
• When encapsulation is violated, refactoring becomes difficult as we're exposing the inner details
• Adding a property later with the same name breaks backwards binary compatibility (i.e. clients are forced to recompile)
• Interfaces don't allow you to declare fields, only properties and methods
• C# 2.0 added the ability to declare separate visibility for individual getters and setters

There's no reason to have public/protected instance fields, so make all instance fields private.

Inconsistent naming conventions

Names of classes, interfaces, and members can convey a great deal of information to clients if used properly.  Here's a good example of inconsistent naming conventions:

public class Order
{
    public Address address { get; set; }
}

public class Quote : Order
{
    public void Process()
    {
        if (address == null)
            throw new InvalidOperationException("Address is null");
    }
}

When I'm down in the "Process" method, what is the "address" variable?  Is it a local variable?  Is it a private field?  Nope, it's a property.  Since it's declared camelCase instead of PascalCase, it led to confusion on the developer's part about what we were dealing with.  If it's local variable, which the name suggests, I might treat the value much differently than if it were a public property.

Deviations from FDG's naming conventions cause confusion.  When I'm using an .NET API that uses Java's camelCase conventions, it's just one more hoop I have to jump through.  In places where my team had public API's we were publishing, it wasn't even up for discussion whether or not we would follow the naming conventions Microsoft used in the .NET Framework.  It just happened, as any deviation from accepted convention leads to an inconsistent and negative user experience.

It's not worth the time to argue whether interfaces should be prefixed with an "I".  That was the accepted convention, so we followed it.  Consistent user experience is far more important than petty arguments on naming conventions.  If I developed in Java, I'd happily use camelCase, as it's the accepted convention.

Another item you may notice as there are no naming guidelines for instance fields.  This reinforces the notion that they should be declared private, and the only people who should care about the names are the developers of that class and the class itself.  In that case, just pick a convention, stick to it, and keep it consistent across your codebase so it becomes one less decision for developers.

Try Catch Publish Swallow

Exception handling can really wreck a system if not done properly.  I think developers might be scared of exceptions, given the number of useless try...catch blocks I've seen around.  Anders Heljsberg notes:

It is funny how people think that the important thing about exceptions is handling them. That is not the important thing about exceptions. In a well-written application there's a ratio of ten to one, in my opinion, of try finally to try catch. Or in C#, using statements, which are like try finally.

Here's an example of a useless try-catch:

public class OrderProcessor
{
    public void Process(Order order)
    {
        try
        {
            ((Quote)order).Process();
        }
        catch (Exception ex)
        {
            ExceptionManager.Publish(ex);
        }
    }
}

In here we have Try Catch Publish Swallow.  We put the try block around an area of code that might fail, and catch exceptions in case it does.  To handle the exception, we publish it through some means, and then nothing.  That's exception swallowing.

Here's a short list of problems with TCPS:

• Exceptions shouldn't be used to make decisions
• If there is an alternative to making decisions based on exceptions, use it (such as the "as" operator in the above code)
• Exceptions are exceptional, and logging exceptions should be done at the highest layer of the application
• Not re-throwing leads to bad user experience and bad maintainability, as we're now relying on exception logs to tell us our code is wrong

Another approach to the example might be:

public class OrderProcessor
{
    public void Process(Order order)
    {
        Quote quote = order as Quote;
        
        if (quote != null)
            quote.Process();
    }
}

The only problem this code will have is if "quote.Process()" throws an exception, and in that case, we'll let the appropriate layer deal with those issues.  Since I don't any resources to clean up, there's no need for a "try..finally".

Downcasting

I already wrote about this recently, but it's worth mentioning again.  I spent a great deal of time recently removing a boatload of downcasts from a codebase, and it made it even worse that the downcasts were pointless.  Nothing in client code was using any additional members in the derived class.  It turned out to be a large pointless hoop I needed to jump through to enable testing.

Regaining sanity

The problem with shortcuts and knowingly bad design decisions is that this behavior can become habitual, and the many indiscretions can add up to disaster.  I had a high school band director who taught me "Practice doesn't make perfect - perfect practice makes perfect."

By making good decisions every day, it becomes a habit.  Good habits practiced over time eventually become etched into your muscle memory so that it doesn't require any thought.  Until you run into a legacy codebase that is, and you realize how bad your old habits were.

Bizarro-tive development

Anyone familiar with Superman also knows about Bizarro, a doppelganger of Superman.  Bizarro looks like Superman, but is opposite in every way.  Instead of saving people, he kills them.  Instead of eloquent speech, he talks like Tarzan.  He's not from Earth, but from Htrea (Earth backwards, clever).  And so on.

A colleague reminded me again of the challenges of explaining iterative development to a manager who had been living with waterfall for many years.  The timeboxing concept came through, but iterative and incremental, not so much.

We drew our original six-month schedule on the board and chopped it up into monthly iterations, saying we would deliver at the end of each month.  We should have been more specific on what "deliver" meant, because this was what the manager then suggested:

• Iteration 1: Envisioning
• Iteration 2: Planning
• Iteration 3: Development
• Iteration 4: Refactoring
• Iteration 5: Testing
• Iteration 6: Stabilization

Each iteration is timeboxed, which is good, but this is just slapping arbitrary exit points on waterfall.  Not iterative, but bizarro-tive development.  Once we explained further, this was the next suggestion:

• Iteration 1
  • Week 1: Envisioning/Planning
  • Week 2: Development
  • Week 3: Refactoring
  • Week 4: Testing/Stabilization
• Iteration 2 - lather, rinse, repeat

Still not exactly what we were talking about.  True iterative development has most or all aspects happen every day, and some points in the iteration some things will happen more than others.  But we're never doing just one aspect, and slapping phases into iterations isn't iterative development, but it's what Bizarro might do.

Myth of the isolated production fix

While in WCF training this week, I heard once again the argument why config files are great - your IT staff can change them without a recompilation.  Sounds great right?  But what exactly does this imply?

Sure, there's no recompilation, but do the changes get tested?  I can't imagine someone modifying the production environment without testing those changes first.  If something goes wrong, who's fault is it?  It's the application team's responsibility to ensure a tested, reliable application, but they're not the ones making this change.

How exactly does this change happen?  Does the IT staff manually edit the configuration files?  What happens if they make a mistake?

Flirting with disaster

Statements like that, "change without a recompilation", absolutely makes me cringe, as it usually implies that these changes are small, isolated, easy, and therefore don't need to be tested.

The problem is that making configuration changes can sometimes lead to unexpected behavior changes because something we didn't anticipate is dependent in some way on the configuration file.  Even if the development team is consulted about these changes, how can they say with confidence the change is small or isolated without actually testing those changes?

Every change, no matter how small, that could potentially affect the behavior of the system, needs to be tested in a variety of systems and environments.  Automated deployments to, and testing of, clones of production environments gives the team confidence in their changes.  Anything else is a wild and potentially hazardous guess.

A responsible SCM process

The first step in any reasonable SCM process is continuous integration.  Until the build is repeatable, automated, and tested, the team can't have much confidence in the reliability or quality of a build.  Even small production fixes should start at this phase, as there's never any guarantee a configuration change doesn't affect business logic without regression testing.

After a CI build and possibly a nightly deployment, we typically have a set of gated environments where builds are put through more and more tests until they are certified as production-ready.  Only builds labeled as production-ready are allowed to be promoted to the production environment.  Although every build in CI processes might be labeled "production-ready", sometimes longer regression tests need to occur before certifying a build ready for the next environment.

But why are small changes allowed to subvert this process?  The only time this process should be allowed to be subverted is in the event of a critical failure, and the business is losing money because of downtime.  If it takes three weeks for a build to move through the pipeline, that's three weeks where the business is losing money.  The only time we compromise on the quality of the build is when we absolutely have to push it out right away (i.e., in hours).

That's not to say testing doesn't happen, but it happens in a targeted area.  After the hotfix is pushed out, we still go through the gated promotion process, just to make sure we didn't miss anything.  We might even throw out the changes in source control for more sound fixes.  That hotfix is still considered suspect and isn't treated as production-ready, but temporary.

Refuse to compromise values

Hotfixes are a rare occurrence and must go through a promotion process themselves, where the severity of the bug must have a certain level of negative effect on business.  The temptation to push out lower severity bugs through hotfixes becomes higher when a few successful hotfix deployments occur.  The business asks "well, if it was that easy, why don't we do that all the time?".  This is just playing production roulette, and eventually it will catch up to you.

I don't feel terrible occasionally compromising on practices when the urgency of a hotfix requires it, but it's important for the team never to compromise on their core values.  When hotfix patches are demanded on a regular basis, the team should learn to say "no", push back, and volunteer a more responsible approach.

Dialing up quality

Quality is not a light switch, it can't be flipped on overnight, or even in six months.  Although the term "quality" differs from person to person, I rather like James Shore's description of Quality With a Name.  He defines quality as having a great design, and that a great design is easy to change.

Furthermore, he concludes:

Great designs:

• Are easily modified by the people who most frequently work within them,
• Easily support unexpected changes,
• Are easy to modify and maintain,
• and Prove their value by becoming steadily easier to modify over years of changes and upgrades. 

Now that your team agrees on what great design is, reality sinks in when they go back to the monolithic flying-spaghetti-code-monster system and see how far away they are from great design.

Great design and high quality can be achieved, but it takes many individual victories to reach the final goal.  A team can dial up quality, one principle/practice at a time, until the team and the system are one smoothly running machine.

Low hanging fruit

So how do we decide where to begin?  The easiest is the low hanging fruit, pain points that affect developers every day.  Here's some common pain points/smells I run in to:

• Everyone is afraid to get the latest version
• "It works on my machine" is your team's mantra
• Every source code file has at least 3 different coding styles
• It takes forever to reproduce a defect locally
• No one knows if the defect is still fixed next week, month, or six months from now

And the list can go on and on.  Once pain points are found, take the top pain point and set a goal to tackle it in the next week or month.  Then schedule a brown-bag lunch to show the problem, how you tackled it, and the principles that led you to these actions.

Show the value

When I want to correct some of these quality issues, I can't tackle the issue in one fell swoop.  Often new ideas have to be introduced a little at a time, where I prove the value as I go.

I can't force unit tests or pair programming on a team if no one sees the value.  If I try it out, show the value, then team members will start to come onboard.

A plan of action might be:

1. Automate the build locally
2. Automate the build of the latest version on a build server
3. Remove all warnings
4. Add some automated tests to reproduce a defect
5. Get these automated tests running with the nightly build
6. Introduce unit tests on new code, running with the nightly build
7. Introduce some key FxCop rules where we're seeing the most varying coding styles
8. Gradually introduce other FxCop rules to hit other pain points

Once initial buy-in for values like "feedback" and "maintainability" happens in your team, it's that much easier to introduce other practices that enforce those key values.  Perfect quality is never achieved, but as my dentist told me, excellence is the pursuit of perfection.

Baby steps to excellence

It's easy to get discouraged trying to get 100% test coverage on a legacy app.  A better intermediate step is to get 80% coverage on the class or classes that have the most defects.  Lessons learned from that experience can then be applied to the rest of the app.  But if we tried to tackle the entire system all at once, we wouldn't get far enough to realize the value, and the team would discard the value as impossible to achieve.

By taking small steps toward realizing our goals, dialing up quality one step at a time, we can create a pattern of success take the system to a level of quality that otherwise would not have been imagined to be possible.

Daily routine with continuous integration

I chuckled quite a bit after reading the Top 5 Signs of Discontinuous Integration, though I think "dysfunctional integration" is a better word.  So what's my routine?

Start of the day

1. Check if build is green
2. Get latest if it is
3. Fix build if it isn't

Coding

1. Code/write tests
2. Run local build, make sure code compiles and tests pass
3. Get latest
4. Run local build again
5. Check in, with decent comments
6. Wait for CI build to finish
7. If build is red, drop everything and fix
8. Otherwise, go back to step 1

I'll get latest several times per day.  The more often I integrate (the "continuous" part), the easier it is do so.  If you're scared to get latest version when the build is green, you probably have some dysfunctional integration issues.  If you don't know what "the build is green" means, well, that's a whole other ball of wax.

Values over vendors

Many times when I start explaining Scrum or Agile for the first time, one of the first questions I get asked is "what tool do I use for <project function>"?  I fully understand needing to standardize on a platform such as .NET, Java, RoR, etc.  But how you deliver shouldn't be nearly as important to the business as what you deliver.

One of the most important aspects of true Agile teams is a self-organizing team, which determines how best to deliver and develop what the business needs.  If the business can't trust the team to choose how, how in the world can the business trust them to deliver anything at all?

A self-organizing team cultivates a set values, principles, and practices to deliver software.  The values come first, and everything else is built on top of that.  If you start with tools and try to fit values around the tools, you'll find the team is only going to care about fulfilling the corporate policy and much less about the value behind the tool.  It's far easier to build policies on values than to derive values from policies, as policies built without values will seem arbitrary and pointless to the team.  If there is a need for a corporate policy, involve as many of the stakeholders as possible into those decisions.  I've seen several teams become disinterested and disheartened when a tool is forced upon them without their input or approval.

So instead of forcing the organization to use a certain build, bug tracking, or requirements gathering technology because of political or vendor-lock-in reasons, cultivate a set of values like "Simplicity", "Feedback", "Quality", etc.  Values define principles and practices, practices reinforce values, and practices can be followed through tools.  By focusing on the values instead of tools, we can broaden our scope of practices to reinforce our values.  Tools can enforce practices, but tools can never define values.  If we focus on tools, our value system becomes warped and twisted towards the tool vendor's values (selling their tool) or those pushing the tool internally (political, ego, or personal gain).

Agile cheat sheet

I already have the ReSharper cheat sheet and a Smells to Refactorings quick reference.  For those who have every cheat sheet taped to their walls, Dave Laribee created a nice Agile cheat sheet, that includes both the manifesto and the principles:

Agile cheat sheet

Too often introductions and conversations about Agile jump straight to specific processes and tools, so it's nice to have a nice little reminder of our team's goals visible to whoever walks by.  It also serves as a good introduction to anyone that stops by that's not familiar what Agile is all about, or has only heard of it through processes like Scrum or XP.  I just love the visible information these cheat sheets provide...

Agile references for PMs

By "PM", I'm referring to Project Managers.  Adopting Agile can be a scary proposition for those entrenched in waterfall processes.  I have a lot of sympathy for PMs whose dev team decides to switch to Agile out from under them.  PMs need not be left behind, and in fact, have a very valuable role in Agile development, just not what they might be used to.  I see it as a re-education on the reality of software development, that Gantt charts don't define reality, they distort and mislead those trying to make decisions based on reality.  Here are a list of references that will help those on the PM side try to make sense of those crazy developers and their Agile ideas.

• Intro to Agile and overview
• Quick intro into Lean, an Agile management process, from Fred George, a VP at ThoughtWorks
• Agile & Iterative Development: A Manager's Guide, Craig Larman
• A great book on the history of iterative development and an overview of several Agile processes
• Agile Project Management, Jim Highsmith
• Guide for PMs to learn or adapt Agile
• Agile Manifesto
• Also check out the list of principles
• Specific processes
• Scrum process guidance from the VSTS process template, Scrum for Team System
• Great presentation and overview of Scrum
• Agile Software Development with SCRUM, Ken Schwaber
• Explanation of key Scrum roles, artifacts, and processes in more detail
• Agile Project Management with Scrum, Ken Schwaber
• Case studies from PM and other role's viewpoints on many different projects and scenarios
• Enterprise Scrum, Ken Schwaber
• Scaling Scrum out to the enterprise level
• Lean Software Development, Mary Poppendieck, Tom Poppendieck
• More information on Lean, which evolved from the Toyota production system
• Other PM-related activities
• Agile Estimation and Planning, Mike Cohn
• Guide for creating realistic, accurate estimates for use in any Agile process
• User Stories Applied, Mike Cohn
• Or, why "The system shall" style requirements are evil

Eliminating the PM role is ultimately a mistake for a dev team moving to Agile, as someone eventually has to answer the $$$ questions.  Putting the onus on the development team/organization to determine costs, staffing, direction, etc., can drag their focus away from delivering business value.  Not having a PM on your team (or reducing the role of the PM) is the quick-and-dirty fix to a dev team's Gantt chart nightmares, but eliminating that role won't address the business needs of having the role in the first place.

Some pairing tips

One of the stranger things I noticed about pair programming was that I felt I was more efficient working with a pair than working on task alone.  Mostly I felt it was because not only did I have someone to help out when I got slowed down, but interruptions had a bigger impact as there was someone sitting next to me, waiting for the interruption to "go away".

I know it's a shocker, but interruptions can have a huge impact on productivity, as we can't multitask, and too much email slows us down and stresses us out.  In addition to some basic pairing etiquette rules, here are some general pairing tips to help improve your pairing productivity:

• Close email (both webmail AND Outlook)
• Close any RSS readers/aggregators
• Close any browser windows irrelevant to current task (i.e., that eBay item you're sniping)
• Set any IM clients to "Away" or with a message of "Pairing"
• Set your phone to "vibrate"

By forcing ourselves to minimize distractions, we also had to set aside parts of the day to deal with the distractions.  It wasn't that we ignored emails and such, it's that we would only allow ourselves to address those distractions outside of pairing sessions.  This lets us get closer to the 5-6 ideal engineering hours per day that we used for capacity calculations.

Importance of collocation

Jeremy Miller mentioned that Fred George is blogging now, and that he'd be worth reading.  Wow, Jeremy wasn't lying.  One of the first posts really resonated with me, on collocation.  I've now had the opportunity to work in a variety of different office types:

• Individual offices
• Shared room, facing walls
• Shared room, facing each other, away from walls
• Cubes

It's something I noticed but couldn't quite quantify or measure at the time, but collocation had a drastic positive effect on our team communication.  Basically, if I have to stand up and walk to talk to someone, it's not going to happen very often.  If I don't even need to turn my head to have a conversation, communication happens all the time.

Since most inefficiencies I encounter during work are because of a lack of communication, it follows that we should optimize our environment for communication.

A rough measurement

So how many conversations did I have per day given each office type?  This is based purely on my recollection, so obviously it's skewed and a bit off, but here's what I remember:

Office Type	Conversations per day
Individual office	1-3
Shared room, facing walls	20-25
Shared room, facing each other, away from walls	50-60
Cubes	5-10

When it came to the shared rooms, I had to estimate in conversations per hour, because communication happens so often.  When I was pairing, it was a continuous conversation, so I'd have to start measuring in length instead of number.  Even when I wasn't pairing, I would get involved with maybe a dozen conversations per hour.

Communication is also something that training can only take so far.  If the training is working against what the environment naturally encourages, the success rate isn't going to be too high.

Tweaking the environment

When I was in a shared room, facing walls, our setup looked something like this: (apologies for the crude drawing)

We had several issues with this layout:

• People talked to each other without looking anyone in the eye (losing valuable visual communication)
• No wall space for whiteboards 
• People usually only talked to those in their immediate vicinity
• Those on the left never heard those on the other side of the room

I should point out that our room wasn't this small, it does look a bit cramped in there in my drawings.  To address these issues, we played around with our desks until we arrived at our final layout:

We found several advantages with this layout:

• Everyone was facing each other
• Everyone could hear all conversations and jump in if needed
• Wall space was freed up for whiteboards, to put up status, do some modeling, etc.
• Whiteboards were visible to everyone in the group
• We became a more cohesive, trusting team

The biggest impact was the last point, since communication builds trust. With trust in place, we can achieve true shared responsibility, and finger pointing was reduced to almost nil.  If finger pointing did happen, the entire team knew about it and recognized the problem, and the issue would come up and get resolved in our next daily stand-up.

As one of the biggest Agile values is communication, collocation is absolutely essential in building a smooth running Agile team.  Without collocation, the Agile team can become fractured, distant, and slip back in to the siloed, deferred responsibility paradigm that waterfall processes enforce.

Some Behave# news

Joe already announced this, but it doesn't hurt to get the word out some more, right?  We're going to combine our efforts into Behave#, a behavior-driven design framework.  For more information about our framework, check out Joe's posts here and here, as well as my announcement.

I had already released a pre-alpha version of Behave# before Joe and I decided to combine our projects, but we'll soon be formalizing our efforts with a vision statement.  We may not have angels singing yet, but I hope to get there soon.  If you already have some features in mind that you'd like us to address, we'll be using the issue tracker in CodePlex for determining feature priority (though our votes override all others :) ).  I'm really looking forward to seeing how this all turns out.  Stay tuned!

Continuous Integration book now out

I just read from Martin Fowler's Bliki that a new book in his signature series is out, "Continuous Integration: Improving Software Quality and Reducing Risk".  If you're not familiar with Fowler's signature series, check out the books on that list:

• Patterns of Enterprise Application Architecture, Martin Fowler
• Test Driven Development: By Example, Kent Beck
• Beyond Software Architecture: Creating and Sustaining Winning Solutions, Luke Hohmann
• Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions, Gregor Hohpe, Bobby Woolf
• User Stories Applied: For Agile Software Development, Mike Cohn
• Refactoring to Patterns, Joshua Kerievsky
• Refactoring Databases: Evolutionary Database Design, Scott W. Ambler, Pramodkumar J. Sadalage
• Implementing Lean Software Development: From Concept to Cash, Mary Poppendieck, Tom Poppendieck
• xUnit Test Patterns: Refactoring Test Code, Gerard Meszaros
• Continuous Integration: Improving Software Quality and Reducing Risk, Paul Duvall, Steve Matyas, Andrew Glover

That's quite an impressive list of books to be in one series.  Many of these books follow the "Duplex Book" pattern, where the book is split into two sections.  The first is a smaller section designed to be read cover-to-cover.  The next section(s) provide prescriptive guidance that can be read end-to-end, or in bits and pieces as necessary.

As an aside, I've felt that nothing impacts or enables success quite as much as continuous integration.  In my (admittedly limited) experience, CI seems to open the doors to other Agile practices like "Whole Team", unit testing and test-driven development, pair programming, and others.  CI is the low-hanging fruit that solves many obvious and common problems in development, while subtly introducing the team to several core Agile values, like communication and feedback.

Motivation for iterative development

I've been asked by many teams now, "why iterative development?"  I have a lot of strong, but not well-formed ideas why I like iterative or Agile processes over waterfall processes.  But in Craig Larman's book
"Agile and Iterative Development: A Manager's Guide", he provides a nice list of motivations (with explanations and evidence) for choosing iterative development:

• Iterative development is lower risk; the waterfall is higher risk
• Early risk mitigation and discovery
• Accommodates and provokes early change; consistent with new product development
• Manageable complexity
• Confidence and satisfaction from early, repeated success
• Early partial product
• Relevant progress tracking; better predictability
• Higher quality; less defects
• Final product better matches true client desires
• Early and regular process improvement
• Communication and engagement required
• "I'll know it when I see it" required

I'm not a fan of force-fed processes, whether they are Agile or waterfall.  Any kind of process needs buy-in from all stakeholders involved in a product or project, from the business to the developers.  Craig's book supplied the ammunition I needed to gain buy-in from business and development people who were only used to waterfall processes.

For those interested in introducing iterative development to your team, I'd start with Craig's book, then more detailed books about specific processes you're interested in.

Can your dev team handle Agile/iterative development?

Too often development teams or organizations push straight into an Agile/iterative process like Scrum without considering the implications of iterative development.  Iterative development forces regular feedback and regular deliveries, but how do we know if the dev team is able to handle feedback and deliver more often?  If any of these criteria match your team, then you are going to have some major challenges adopting an Agile process:

• Testability is not the top design concern
• Unit tests are optional or non-existent
• Builds happen less often than once a day
• Deployments to test environments happen even less often, if at all
• "Quick and Dirty" is your team's motto
• Refactoring is considered taboo and dismissed as "over-engineering"
• Email is the primary method of communication between team members
• You don't see the person giving you requirements for weeks or months at a time
• You don't see the testers until just before releasing

Feedback isn't useful if your team or application can't act on that feedback.  Without solid engineering practices in place, adopting an iterative process will likely end in disaster, as the application wasn't built to accept changes and feedback on a regular basis.

Introducing Behave#

I really like the idea of executable requirements.  Executable requirements is the idea that I can express requirements from the business in a human-readable, executable format.  Part of fulfilling the requirement is that the "executable requirement" part can execute successfully in the form of acceptance tests.

Recently, I checked out a couple of API's for executable requirements in the form of behavior-driven design (BDD).  For more information on BDD, check out these articles from Dan North's blog:

• Introducing BDD
• What's in a Story?

The API's I checked out were NBehave and Joe Ocampo's NUnit.Behave.  NBehave was rather cumbersome to set up, and didn't have much of a fluent interface.  NUnit.Behave was much easier to use, but forced a dependency on NUnit.  I didn't want to be forced to use NUnit, nor did I want to inherit from certain classes or implement certain interfaces.  Heavily influenced by NUnit.Behave, I created Behave#.

What is Behave#?

Behave# (http://www.codeplex.com/BehaveSharp) is a fluent interface to express executable requirements in the form of acceptance tests with a distinct BDD grammar.  Stories follow the "As a <role> I want <feature> so that <benefit>" template.  Stories are made up of scenarios, which follow the "Given When Then" template.  Through a fluent interface, Behave# shapes your acceptance tests to closely match the original user story.

What would a user story look like in code?

 The fluent interface I settled on looks like:

new Story("Transfer to cash account")
    .AsA("savings account holder")
    .IWant("to transfer money from my savings account")
    .SoThat("I can get cash easily from an ATM")

    .WithScenario("Savings account is overdrawn")

        .Given("my savings account balance is", -20)
            .And("my cash account balance is", 10)
        .When("I transfer to cash account", 20)
        .Then("my savings account balance should be", -20)
            .And("my cash account balance should be", 10);

I really like this syntax as it can be read very easily and there aren't a lot of other objects or frameworks getting in the way.  So how do we actually execute custom actions in our story definition?  Here's a full-fledged example, using NUnit to provide assertions:

[Test]
public void Transfer_to_cash_account()
{

    Account savings = null;
    Account cash = null;

    Story transferStory = new Story("Transfer to cash account");

    transferStory
        .AsA("savings account holder")
        .IWant("to transfer money from my savings account")
        .SoThat("I can get cash easily from an ATM");

    transferStory
        .WithScenario("Savings account is in credit")

            .Given("my savings account balance is", 100, delegate(int accountBalance) { savings = new Account(accountBalance); })
                .And("my cash account balance is", 10, delegate(int accountBalance) { cash = new Account(accountBalance); })
            .When("I transfer to cash account", 20, delegate(int transferAmount) { savings.TransferTo(cash, transferAmount); })
            .Then("my savings account balance should be", 80, delegate(int expectedBalance) { Assert.AreEqual(expectedBalance, savings.Balance); })
                .And("my cash account balance should be", 30, delegate(int expectedBalance) { Assert.AreEqual(expectedBalance, cash.Balance); })

            .Given("my savings account balance is", 400)
                .And("my cash account balance is", 100)
            .When("I transfer to cash account", 100)
            .Then("my savings account balance should be", 300)
                .And("my cash account balance should be", 200)

            .Given("my savings account balance is", 500)
                .And("my cash account balance is", 20)
            .When("I transfer to cash account", 30)
            .Then("my savings account balance should be", 470)
                .And("my cash account balance should be", 50);

    transferStory
        .WithScenario("Savings account is overdrawn")

            .Given("my savings account balance is", -20)
                .And("my cash account balance is", 10)
            .When("I transfer to cash account", 20)
            .Then("my savings account balance should be", -20)
                .And("my cash account balance should be", 10);

}

To perform actions with each "Given When Then" fragment, I pass in a delegate (anonymous in this case, but much prettier with lambda expressions in C# 3.0).  This delegate is executed as the story execution proceeds, and is cached by name, so each time the story sees "my cash account balance is", it will execute the appropriate delegate.  I don't need to define the custom action every time.

The result of the execution is the story results outputted in a human readable format (by default, to Debug):

Story: Transfer to cash account

Narrative:
   As a savings account holder
   I want to transfer money from my savings account
   So that I can get cash easily from an ATM

   Scenario 1: Savings account is in credit
      Given my savings account balance is: 100
         And my cash account balance is: 10
      When I transfer to cash account: 20
      Then my savings account balance should be: 80
         And my cash account balance should be: 30

      Given my savings account balance is: 400
         And my cash account balance is: 100
      When I transfer to cash account: 100
      Then my savings account balance should be: 300
         And my cash account balance should be: 200

      Given my savings account balance is: 500
         And my cash account balance is: 20
      When I transfer to cash account: 30
      Then my savings account balance should be: 470
         And my cash account balance should be: 50

   Scenario 2: Savings account is overdrawn
      Given my savings account balance is: -20
         And my cash account balance is: 10
      When I transfer to cash account: 20
      Then my savings account balance should be: -20
         And my cash account balance should be: 10

This format closely resembles the original user story I received from the business.

Conclusion

I like being able to verify deliverables in the form of automated tests.  When these tests closely resemble the original requirements as user stories, I'm more confident in the deliverables.  Having executable requirements in the form of acceptance tests in a fluent interface that matches the original requirements can bridge the gap between what the business asks for and what gets delivered.  Behave# gets us one step closer to bridging that gap.

Refining daily stand-ups

I rediscovered a great article that helped my old Scrum team to refine and improve our daily stand-ups:

http://www.martinfowler.com/articles/itsNotJustStandingUp.html

Most of the smells mentioned manifested themselves at one time or another, but frequent abuses included:

• Reporting to the Leader
• Stand-up Meeting Starts the Day... Late
• I Can't Remember
• Problem Solving

I only found this article the first time after our 10th sprint or so, unfortunately.  So many activities and artifacts, whether it's a process, project management, developing, code, etc. have recognizable smells, it can be difficult to recognize these smells when they exist.

I think I need a new general, all-encompassing smell of "smell ignorance".  That's when you don't know the smells of a particular area, and not knowing the smells limits your ability to improve and succeed.

VSTS Scrum process templates

There have been some rumblings around that some in my company might be interested in Scrum and Team System, so I thought I'd compile a list of Scrum process templates and some highlights (and lowlights).  The three Scrum process templates I've found are:

• Scrum for Team System
• Microsoft eScrum
• VSTS Scrum Process Template, via CodePlex

Each process template adds custom work items and reports related to Scrum, but they all have their quirks and niceties.

Scrum for Team System

This process template was originally released a year ago by a company called Conchango.  You can find this template on the Scrum for Team System website at http://www.scrumforteamsystem.com/.  I've personally used this template for about 10 months covering about a dozen Sprints.

Pros

• Great website with thorough process guidance and free training videos
• Mature, with several updates to the template
• All sprint artifacts present, with automatic rollup calculations
• Good reports, including:
  • Sprint Burndown
  • Product Burndown
  • Product Backlog Composition
  • and about a dozen more
• Portal reports, a set of smaller reports designed for the project SharePoint portal
• Support through forums
• Widely adopted
• Includes tool to update warehouse (critical for up-to-date reports, as Team System only updates the warehouse ever hour or so)

Cons

• All artifacts created and managed through Visual Studio, which not all team members may have
• Reports have a lot of custom code, making them difficult to tweak
• Does not plug in to the Areas and Iterations constructs already present in Team System
• Only one active project per Team Project
  • i.e., everyone in the same Team Project will use the same sprints, work items, etc. with no good way to partition them
  • This forces every new team to have a new Team Project

Microsoft eScrum

This one was just released from Microsoft, and from the description, looks like it's been used internally at Microsoft.  I found this on a post on Rob Caron's blog.  That post links to a download on Microsoft's downloads site here.  I should note that I tested all of these process templates using a Team System VHD, so I didn't have to get access to our corporate Team System server.

Pros

• Fantastic web portal for managing sprints and sprint artifacts.  It's all Ajax-y too.
  • Pages for managing daily sprints, reports, etc.
• Context-sensitive help in web portal
• Dynamic capacity calculations in portal
• All sprint artifacts present
• Ability to have multiple "Products" in one Team Project in source control, allowing multiple teams to use Scrum for one Team Project
• Allows definitions of each role (Project Member, Product Owner, etc.)
• Some better options on each work item type, such as categories
• Integrates with Areas and Iterations
  • Areas are Products
  • Iterations are Sprints

Cons

• New, released only on June 12
• No support through forums, or anywhere else online (Google only found 2 relevant pages)
• Painful setup, lots of manual steps
• Not as many reports (~half a dozen)

VSTS Scrum Process Template from CodePlex

I also found this one on the post on Rob Caron's blog.  At the bottom of the post, it links to the CodePlex project.  This project was intended to improve on the Scrum for Team System process template by taking advantage of Areas and Iterations.  It's being developed by a handful of TFS MVP's.

Pros

• Lightweight, fits in well with Areas and Iterations
• Good list of reports, some of them quite different than the other templates
  • Unplanned work
  • Quality Indicators
  • Project Velocity
  • Builds
• Supports basic Scrum/Agile work items (User Story, Backlog Item, etc.)
• Custom work item for reviews
• Open source, so it's updated frequently

Cons

• Open source, so don't look for great support
• Still in beta
• Not a lot of people using it
• No project portal
• No installer

Summing it up

The Conchango process template is by far the most mature, so I'd usually go with that one, but the awesome portal site and the integration into Areas and Iterations make the eScrum process template a compelling alternative.  As for the CodePlex template, it looks promising, but I'll reserve judgement until a final version is released.  Doesn't look ready for prime time quite yet.  The great thing about process templates is that you can edit them after you create the Team Project.  If there's a report missing you want, it's pretty easy to look at one of the other process templates and see what others are doing, and add whatever you need.