Myth of the isolated production fix

While in WCF training this week, I heard once again the argument why config files are great - your IT staff can change them without a recompilation.  Sounds great right?  But what exactly does this imply?

Sure, there's no recompilation, but do the changes get tested?  I can't imagine someone modifying the production environment without testing those changes first.  If something goes wrong, who's fault is it?  It's the application team's responsibility to ensure a tested, reliable application, but they're not the ones making this change.

How exactly does this change happen?  Does the IT staff manually edit the configuration files?  What happens if they make a mistake?

Flirting with disaster

Statements like that, "change without a recompilation", absolutely makes me cringe, as it usually implies that these changes are small, isolated, easy, and therefore don't need to be tested.

The problem is that making configuration changes can sometimes lead to unexpected behavior changes because something we didn't anticipate is dependent in some way on the configuration file.  Even if the development team is consulted about these changes, how can they say with confidence the change is small or isolated without actually testing those changes?

Every change, no matter how small, that could potentially affect the behavior of the system, needs to be tested in a variety of systems and environments.  Automated deployments to, and testing of, clones of production environments gives the team confidence in their changes.  Anything else is a wild and potentially hazardous guess.

A responsible SCM process

The first step in any reasonable SCM process is continuous integration.  Until the build is repeatable, automated, and tested, the team can't have much confidence in the reliability or quality of a build.  Even small production fixes should start at this phase, as there's never any guarantee a configuration change doesn't affect business logic without regression testing.

After a CI build and possibly a nightly deployment, we typically have a set of gated environments where builds are put through more and more tests until they are certified as production-ready.  Only builds labeled as production-ready are allowed to be promoted to the production environment.  Although every build in CI processes might be labeled "production-ready", sometimes longer regression tests need to occur before certifying a build ready for the next environment.

But why are small changes allowed to subvert this process?  The only time this process should be allowed to be subverted is in the event of a critical failure, and the business is losing money because of downtime.  If it takes three weeks for a build to move through the pipeline, that's three weeks where the business is losing money.  The only time we compromise on the quality of the build is when we absolutely have to push it out right away (i.e., in hours).

That's not to say testing doesn't happen, but it happens in a targeted area.  After the hotfix is pushed out, we still go through the gated promotion process, just to make sure we didn't miss anything.  We might even throw out the changes in source control for more sound fixes.  That hotfix is still considered suspect and isn't treated as production-ready, but temporary.

Refuse to compromise values

Hotfixes are a rare occurrence and must go through a promotion process themselves, where the severity of the bug must have a certain level of negative effect on business.  The temptation to push out lower severity bugs through hotfixes becomes higher when a few successful hotfix deployments occur.  The business asks "well, if it was that easy, why don't we do that all the time?".  This is just playing production roulette, and eventually it will catch up to you.

I don't feel terrible occasionally compromising on practices when the urgency of a hotfix requires it, but it's important for the team never to compromise on their core values.  When hotfix patches are demanded on a regular basis, the team should learn to say "no", push back, and volunteer a more responsible approach.

Ruby-style loops in C# 3.0

Ruby has a pretty interesting (and succinct) way of looping through a set of numbers:

5.times do |i|
  print i, " "
end

The results of executing this Ruby block is:

0 1 2 3 4

I really love the readability and conciseness of this syntax, just enough to see what's going on, but not a lot of extra stuff to get in the way.  In addition to the "times" method, there's also "upto" and "downto" methods for other looping scenarios.

Some slight of hand

With extension methods and lambda expressions and C# 3.0, this form of loop syntax is pretty easy to do.  Not really a great idea, but at least an example of what these new constructs in C# 3.0 can do.

So how can we create this syntax in C#?  The "times" method is straightforward, that can just be an extension method for ints:

public static void Times(this int count)

Now this method shows up in IntelliSense (when I add the appropriate "using" directive):

Now that the Times method shows up for ints, we can focus on the Do block.

Adding the loop behavior

Although I can't declare blocks in C# 3.0, lambda expressions are roughly equivalent.  To take advantage of lambda expressions, I want to give the behavior to the Times method in the form of a delegate.  By declaring a delegate parameter type on a method, I'm able to use lambda expressions when calling that method.

I didn't like passing the lambda directly to the "Times" method, so I created an interface to encapsulate a loop iteration, faking the Ruby "do" block with methods:

public interface ILoopIterator
{
    void Do(Action action);
    void Do(Action<int> action);
}

Now I can return an ILoopIterator from the Times method instead of just "void".  Now the final part is to create an "ILoopIterator" implementation that will do the actual looping:

private class LoopIterator : ILoopIterator
{
    private readonly int _start, _end;

    public LoopIterator(int count)
    {
        _start = 0;
        _end = count - 1;
    }

    public LoopIterator(int start, int end)
    {
        _start = start;
        _end = end;
    }  

    public void Do(Action action)
    {
        for (int i = _start; i <= _end; i++)
        {
            action();
        }
    }

    public void Do(Action<int> action)
    {
        for (int i = _start; i <= _end; i++)
        {
            action(i);
        }
    }
}

public static ILoopIterator Times(this int count)
{
    return new LoopIterator(count);
}

I let the "LoopIterator" class encapsulate the behavior of performing the underlying "for" loop and calling back to the Action passed in as a lambda.  It makes more sense when you see some client code calling the Times method:

int sum = 0;
5.Times().Do( i => 
    sum += i
);
Assert.AreEqual(10, sum);

That looks pretty similar to the Ruby version (but not quite as nice), but it works.  Compare this to a normal loop in C#:

int sum = 0;
for (int i = 0; i < 5; i++)
{
    sum += i;
}
Assert.AreEqual(10, sum);

Although the "for" syntax is functional and about the same number of lines of code, the Ruby version is definitely more readable.  Adding additional UpTo and DownTo methods would be straightforward with additional ILoopIterator implementations.

Feature abuse

Yeah, I know this is more than a mild case of feature abuse, but it was interesting to see the differences between similar operations in C# 3.0 and Ruby.  Although it's possible to do these similar operations, with similar names, this example highlights how much the syntax elements of the static CLR languages can get in the way of a readable API, and how much Ruby stays out of the way.

Fluent interface endgame

In a conversation on BDD on the altnetconf message board, the topic switched to language-oriented syntax in the CLR, to which Scott notes:

When IronRuby gets here, I think we should at least stop and consider the value to the community of encouraging them to try do language-oriented specification with tools and programming languages that don't quite hit the mark.

Or maybe we should be turning our focus to Boo or IronPython for achieving the solubility in specification code that we can't have in C# and VB.

I've felt this for quite a while now, though I have reservations about focusing on Boo's Specter or IronPython for the time being.  My response was:

I think there's an assumption among anyone developing language-oriented tools in CLR using extension methods, fluent interfaces, etc. that it's all a poor man's substitute for the clearer syntax that Ruby inherently provides. Once IronRuby hits some sort of beta/RC status, I don't see much point at all continuing to try to wrestle a pig into a yoke while the farmers laugh "use an ox you moron".

I do like the readability the Boo macros provide, but I'm always reminded of that scene from King of the Hill when Hank meets his new neighbor Khan (who is from Laos) for the first time:

Hank: So are you Chinese or Japanese?
Khan: No, we are Laotian.
Bill: The ocean? What ocean?
Khan: From Laos, stupid! It's a landlocked country in Southeast Asia
between Vietnam and Thailand, population approximately 4.7 million!
Hank: ... So are you Chinese or Japanese?
Kahn: D'oh

As someone who wants to use BDD more, Specter looks like a promising CLR tool, but as a proponent of BDD, Boo might set the bar too high when there's only Chinese or Japanese in the greater community. Then again, if curly-braces dilute a concept that is supposed to lie very close to the conversations BDD starts with, maybe it's better to draw a line so the value isn't diminished.

I don't really have any answers yet. Make it pure or make it accessible, tough choice...

Again, I don't have anything against Boo, but I just don't know how successful these efforts will be once IronRuby comes out.  For example, why should I care about the Boo Build System when I can use Rake in Visual Studio when IronRuby ships?

Until these languages are officially supported by Microsoft, they are for the most part eliminated as an option for one reason or another, whether it be company policy, lack of adoption, or just apathy for learning a new language.  Once they are supported, I will happily ditch my fluent interface and extension methods for a true natural-language friendly programming language.

Double-edged sword of InternalsVisibleTo

I've had some conversations with both Joe and Elton lately about the InternalsVisibleTo attribute.  From the documentation, the assembly-level InternalsVisibleTo attribute:

Specifies that all nonpublic types in an assembly are visible to another assembly.

This attribute was introduced in C# 2.0, and allows you to specify other assemblies that can see all types and members marked "internal".  In practice, all assemblies are signed with the same public key, and you'd specify that the unit test assembly can see that assembly.

• MyProject.Core -> has the "InternalsVisibleTo" attribute defined in the AssemblyInfo.cs file, pointing at the below assembly
• MyProject.Core.Tests -> is signed with the same public key as the Core assembly

Notice that the "Core" project knows about the "Tests" project, but the actual project dependency is the other way around.  It's definitely better than using reflection to access private members for testing, but there are some definite pros and cons with this approach.

Pros

• Allows your test libraries to access internal classes and methods for additional testing and coverage
• Keeps your public API limited to what you want to publish
• Provides greater flexibility for internal refactoring and backwards compatibility
• Reduces the surface area of your public API

Cons

• Easily abused, so things usually marked "private" are now marked "internal"
• Potential loss of encapsulation
• Decision about what should be public could be wrong
• Essentially two levels of "public" visibility that have to be managed
• Enforces bi-directional dependencies between assemblies

Personally, I always felt like marking something "internal" was cheating just a little bit, and I have trouble deciding when to make something internal or not.  But unless you're delivering a public, published and documented API as part of your product, using the "InternalsVisibleTo" attribute would probably be overkill.

However, if you are delivering an API, you should consider using this attribute to keep a high level of coverage and reduce the surface area the API for your customers.  You could try starting by making everything "internal", then shape the public API based on specific use cases.

Dialing up quality

Quality is not a light switch, it can't be flipped on overnight, or even in six months.  Although the term "quality" differs from person to person, I rather like James Shore's description of Quality With a Name.  He defines quality as having a great design, and that a great design is easy to change.

Furthermore, he concludes:

Great designs:

• Are easily modified by the people who most frequently work within them,
• Easily support unexpected changes,
• Are easy to modify and maintain,
• and Prove their value by becoming steadily easier to modify over years of changes and upgrades. 

Now that your team agrees on what great design is, reality sinks in when they go back to the monolithic flying-spaghetti-code-monster system and see how far away they are from great design.

Great design and high quality can be achieved, but it takes many individual victories to reach the final goal.  A team can dial up quality, one principle/practice at a time, until the team and the system are one smoothly running machine.

Low hanging fruit

So how do we decide where to begin?  The easiest is the low hanging fruit, pain points that affect developers every day.  Here's some common pain points/smells I run in to:

• Everyone is afraid to get the latest version
• "It works on my machine" is your team's mantra
• Every source code file has at least 3 different coding styles
• It takes forever to reproduce a defect locally
• No one knows if the defect is still fixed next week, month, or six months from now

And the list can go on and on.  Once pain points are found, take the top pain point and set a goal to tackle it in the next week or month.  Then schedule a brown-bag lunch to show the problem, how you tackled it, and the principles that led you to these actions.

Show the value

When I want to correct some of these quality issues, I can't tackle the issue in one fell swoop.  Often new ideas have to be introduced a little at a time, where I prove the value as I go.

I can't force unit tests or pair programming on a team if no one sees the value.  If I try it out, show the value, then team members will start to come onboard.

A plan of action might be:

1. Automate the build locally
2. Automate the build of the latest version on a build server
3. Remove all warnings
4. Add some automated tests to reproduce a defect
5. Get these automated tests running with the nightly build
6. Introduce unit tests on new code, running with the nightly build
7. Introduce some key FxCop rules where we're seeing the most varying coding styles
8. Gradually introduce other FxCop rules to hit other pain points

Once initial buy-in for values like "feedback" and "maintainability" happens in your team, it's that much easier to introduce other practices that enforce those key values.  Perfect quality is never achieved, but as my dentist told me, excellence is the pursuit of perfection.

Baby steps to excellence

It's easy to get discouraged trying to get 100% test coverage on a legacy app.  A better intermediate step is to get 80% coverage on the class or classes that have the most defects.  Lessons learned from that experience can then be applied to the rest of the app.  But if we tried to tackle the entire system all at once, we wouldn't get far enough to realize the value, and the team would discard the value as impossible to achieve.

By taking small steps toward realizing our goals, dialing up quality one step at a time, we can create a pattern of success take the system to a level of quality that otherwise would not have been imagined to be possible.

ALT.NET-itis

Has anyone else that attended ALT.NET feel under the weather this week?  I was fine before the conference, then felt like crap afterwards.

I have a feeling that it was because of the germ-sponge getting passed around at the closing session:

Martin holding the orange plague-dough

It was orange at the beginning, but quickly took a brown hue as it went around the room.  I'll bring some rubber gloves next time...

Subscribing to your Google Calendar in Outlook 2007

I've been in love with Google Calendar for a long time now.  I have several calendars I'm viewing at the same time, including my personal calendar, UT football schedule, and a few others.  I always hated having to view two calendars at work, one for my work schedule and one for my Google calendars.

Scott Hanselman mentioned that he views multiple external calendars inside Outlook 2007.  Several of the external calendars are Google calendars.  Outlook 2007 supports any ICAL source, which most online calendar applications provide.  It's easy to set up, and the instructions can be found here:

Subscribe to your Google Calendar 

Making your Outlook calendar show up in Google calendar is a bit more tricky, and I haven't found anything that does actual synchronization and not just import/export.

Now I have all of my calendars in one convenient location, with each calendar color coded from their source.  However, I found I have to think of more creative excuses on why I forget things my wife's calendar...