GrabBag<T>: NBehave

Scott recently voiced his opinion on the validity of story runners (i.e. the xBehave tools) in an agile shop. First, let me say that I sincerely appreciate the passion Scott has for BDD, and it's that passion that will drive the community forward. Second, in recognizing that passion, I won't respond to comments about technology fetishes and code-sturbation for now, but I do honestly understand the concerns behind them.

Scott sees a tool in search of a problem, and in completely disagreeing with the problem it's trying to solve, questions the motivation of those developing the tool, which he sees as seemingly selfish and egotistical reasons. From that point of view, again, I completely understand Scott's concerns, and luckily I have a Bellwarese translator to get to the heart of those concerns.

I did feel it unfortunate that the BDD conversation revolved around tools. BDD is far too nascent to argue tooling, and instead the discussion should have focused getting the language, values, and concepts right.

So Scott's core question is:

Are Story Runners appropriate for executable requirements?

As an aside, not once in my experience have I had a BA come up to me and say "hey, while you're delivering this, it would be great if you had some automated tests for it too". I've never had anyone from the business ask me to use any kind of automated testing tool, such as NUnit, FitNesse, NSpec, or NBehave. These tools will always be pushed by geeks, as it is the geeks that see the value of these tools first.

For me personally, capturing stories in code was never about traceability. Specifically, the tangible benefits Joe and I saw were:

Providing a more complete description of the behavior of the system

Unit tests too granular
Even specifications can be difficult to organize

Stories provide a better overall, macro view of the system
Executable stories remind us of our final goal

For us, the final goal isn't the unit tests, but that the story is satisfied.
Capturing in code helps direct us towards that final goal

Executable stories can help capture the conversation
Executable stories can help shape our domain model and infuse the ubiquitous language into the system

The more appropriate venue for our discussion was Jeremy's topic Sunday on executable requirements. Stories are part of BDD, as we can all agree, but should stories be used as executable requirements?

The result from the discussion was "let's see, as nothing else has worked great so far". Start with stories, end with stories. Stories -> Scenarios or Aspects -> Specifications (NSpec) -> Unit tests (NUnit) -> Integration Tests (FitNesse/NUnit) -> Functional/Acceptance Tests (FitNesse/NBehave).

I do appreciate this dialogue, as only through debate with the truly passionate can clarity be achieved.

p.s. please don't suggest exploding laptops, we've had enough problems with that here...

As Joe already mentioned, Behave# has merged with NBehave. The merged NBehave will still be hosted on CodePlex, and the old project site will redirect to the new CodePlex NBehave site. With the announcement of the merge comes the first release of our merging efforts (0.3), which you can find here.

The new release added quite a few features over the previous release, including:

Pending scenarios
Console runner
- Decorate your tests with "Theme" and "Story" attributes
- Scenario result totals
- Dry run output

The major addition is the console runner feature. One problem we always ran into was that as developers, we wanted to get pass/fail totals based on scenarios (not tests) so that we could have more meaningful totals that matched our original stories. A single story could have several scenarios, but would only report as one "Pass" or "Fail" to NUnit. Additionally, this is the first release that compares fairly evenly with the first release of rbehave.

So how can we use NBehave to accomplish this? As always, we'll start with the story.

The Story

I've received a few comments for a different example than the "Account" example. I'll just pull an example from Jimmy Nilsson's excellent Applying Domain-Driven Design and Patterns book (from page 118):

List customers by applying a flexible and complex filter.

Not story-friendly, but a description below lets me create a meaningful story:

As a customer support staff
I want to search for customers in a very flexible manner
So that I can find a customer record and provide them meaningful support.

I'm playing both developer and business owner, but normally these stories would be written by the customer. Otherwise, so far so good. What about a scenario for this story? Looking further into the book, I found a suitable scenario on page 122. Reworded in the BDD syntax, I arrive at:

Scenario: Find by name

Given a set of valid customers
When I ask for an existing name
Then the correct customer is found and returned.

Right now, I only care about finding by name. It could be argued that the original story is too broad, but it will suffice for this example. I'm confident those conversations will take place during iteration planning meetings in any case.

Now that we have a story and a scenario, I can author the story in NBehave. First, I'll need to set up my environment to use NBehave.

Setting up the environment

I use a fairly common source tree for most projects:

All dependencies (i.e. assemblies in the References of my project) go into the "lib" folder. All tools, like NAnt, NUnit, etc. that are used as part of the build go into the "tools" folder. For NBehave, I've copied the "NBehave.Framework.dll" into the "lib" folder, and the entire NBehave release goes into its own folder in the "tools" folder. For more information about this setup, check out Tree Surgeon.

Now that I have NBehave in my project, I'm ready to write some NBehave stories and scenarios.

The initial scenario

Before I get started authoring the story and scenario, I need to create a project for these scenarios. If I have a project named MyProject, its scenarios will be in a MyProject.Scenarios project. Likewise, its specifications will be in a MyProject.Specifications project. You can combine the stories and specifications into one project if you like. Finally, I create a class that will contain all of the stories in my "customer search" theme.

I don't name the class after the class it might be testing, instead I name it after the theme. The reason is that the implementation of the stories and scenarios can (and will) change independently of the story and scenario definition. Stories and scenarios shouldn't be tied to implementation details.

After adding a reference to NBehave and NUnit from the "lib" folder, Here's what my solution tree looks like at this point:

Note that I named my file after the theme, not the class I'm likely to test (CustomerRepository). Here's my entire story file:

using NBehave.Framework;

namespace NBehaveExample.Core.Specifications
{
    [Theme("Customer search")]
    public class CustomerSearchSpecs
    {
        [Story]
        public void Should_find_customers_by_name_when_name_matches()
        {
            Story story = new Story("List customers by name");

            story.AsA("customer support staff")
                .IWant("to search for customers in a very flexible manner")
                .SoThat("I can find a customer record and provide meaningful support");

            story.WithScenario("Find by name")
                .Pending("Search implementation")

                .Given("a set of valid customers")
                .When("I ask for an existing name")
                .Then("the correct customer is found and returned");
        }
    }
}

A few things to note here:

Theme classes are decorated with the Theme attribute
- Themes have a mandatory title
Story methods are decorated with the Story attribute
The initial story is marked Pending, with an included reason

The attributes are identical in function to the "TestFixture" and "Test" attributes of NUnit, where they inform NBehave that this class is a Theme class and it contains Story methods. NBehave finds classes marked with the Theme attribute, and executes methods marked with the Story attribute.

Now that we have a skeleton story definition in place, I can run the stories as part of my build

Using the console runner

New in NBehave 0.3 is the console runner, which runs the Themes and Stories and collects metrics from those runs. To run the above stories, I use the following command:

NBehave-Console.exe NBehaveExample.Core.Scenarios.dll

From the console runner, I get the following output:

NBehave version 0.3.0.0
Copyright (C) 2007 Jimmy Bogard.
All Rights Reserved.

Runtime Environment -
   OS Version: Microsoft Windows NT 5.2.3790 Service Pack 1
  CLR Version: 2.0.50727.1378

P
Scenarios run: 1, Failures: 0, Pending: 1

Pending:
1) List customers by name (Find by name): Search implementation

I only have one scenario thus far, but NBehave tells me several things so far:

Dot results
- A series of one character results shows me I have one scenario pending (similar to the dots NUnit outputs)
Result totals
- Includes total scenarios run, number of failures, and number of pending scenarios
Individual summary result
- List of failing and pending scenarios
- Name of story, scenario, and pending/failing reason

Let's say I want a dry-run of the scenario output for documentation purposes:

NBehave-Console.exe NBehaveExample.Core.Scenarios.dll /dryRun /storyOutput:stories.txt

I've set two switches for the console runner, one to do a dry run and one to have a file where the stories will be output. Story output can be turned on regardless if I'm doing a dry run or not. Here's the contents of "stories.txt" after I run the statement above:

Theme: Customer search

	Story: List customers by name
	
	Narrative:
		As a customer support staff
		I want to search for customers in a very flexible manner
		So that I can find a customer record and provide meaningful support
	
		Scenario 1: Find by name
			Pending: Search implementation
			Given a set of valid customers
			When I ask for an existing name
			Then the correct customer is found and returned

This output provides a nice, human-readable format describing the stories that make up my system.

Now that I have a story, let's make the story pass, using TDD with Red-Green-Refactor.

Make it fail

First, I'll add just enough to my story implementation to make it compile:

[Story]
public void Should_find_customers_by_name_when_name_matches()
{
    Story story = new Story("List customers by name");

    story.AsA("customer support staff")
        .IWant("to search for customers in a very flexible manner")
        .SoThat("I can find a customer record and provide meaningful support");

    CustomerRepository repo = null;
    Customer customer = null;

    story.WithScenario("Find by name")
        .Given("a set of valid customers",
            delegate { repo = CreateDummyRepo(); })
        .When("I ask for an existing name", "Joe Schmoe",
            delegate(string name) { customer = repo.FindByName(name); })
        .Then("the correct customer is found and returned",
            delegate { Assert.That(customer.Name, Is.EqualTo("Joe Schmoe")); });
}

All I've done is removed the Pending call on the scenario and added the correct actions for the Given, When, and Then fragments. The "CreateDummyRepo" method is just a helper method to set up a CustomerRepository:

private CustomerRepository CreateDummyRepo()
{
    Customer joe = new Customer();
    joe.CustomerNumber = 1;
    joe.Name = "Joe Schmoe";

    Customer bob = new Customer();
    bob.CustomerNumber = 1;
    bob.Name = "Bob Schmoe";

    CustomerRepository repo = new CustomerRepository(new Customer[] { joe, bob });

    return repo;
}

I compile successfully and run NBehave, and get a failure as expected:

F
Scenarios run: 1, Failures: 1, Pending: 0

Failures:
1) List customers by name (Find by name) FAILED
  System.NullReferenceException : Object reference not set to an instance of an object.
   at NBehaveExample.Core.Specifications.CustomerSearchSpecs.<>c__DisplayClass3.b__2() in C:\dev\NBehaveExample\src\NBehaveExample.Core.Specifications\CustomerSearchSpecs.cs:line 28
   at NBehave.Framework.Story.<>c__DisplayClass1.b__0()
   at NBehave.Framework.Story.InvokeActionBase(String type, String message, Object originalAction, Action actionCallback, String outputMessage, Object[] messageParameters)

Now that I've made it fail, let's calibrate the test and put only enough code in the FindByName method to make the test pass.

Make it pass

To make the test pass, I'll just return a hard-coded Customer object:

public Customer FindByName(string name)
{
    Customer customer = new Customer();
    customer.Name = "Joe Schmoe";
    return customer;
}

NBehave now tells me that I have 1 scenario run with 0 failures:

.
Scenarios run: 1, Failures: 0, Pending: 0

The dot signifies a passing scenario. Now I can make the code correct and put in some implementation.

Make it right

Since CustomerRepository is just sample code for now, it only uses a List<Customer> for its backing store. Searching isn't that difficult as I'm not involving the database at this time:

public Customer FindByName(string name)
{
    return _customers.Find(delegate(Customer customer) { return customer.Name == name; });
}

With this implementation in place, NBehave tells me I'm still green:

.
Scenarios run: 1, Failures: 0, Pending: 0

I can now move on to the next scenario. If I had additional specifications for CustomerRepository not captured in the story, I can go to my Specifications project to detail them there.

Where we're going

With NBehave's console runner, I can now easily include NBehave as part of my build. I'm not piggy-backing NUnit for executing and reporting tests, as I'm writing Stories and Scenarios, not Tests. This option is still available to me and I can create stories inside of tests, so we're not forcing anyone to use the Theme and Story attributes if they don't want to.

It's a good start, but there are a few things still lacking:

Integration with testing/specification frameworks
- The story for authoring the "Then" part of the scenario still isn't that great
Features targeted for automated builds/CI
- XML output
- A nice XSLT formatter for XML output
- HTML output of stories in addition to raw text
- Integration with CC.NET
Setup/Teardown for stories/themes, with appropriate BDD names
- Not sure, since having everything encapsulated in the story can direct my API better

Happy coding!