BDD Everywhere - don't underestimate your target audience

I have recently moved from doing TDD to BDD, using Scott's excellent SpecUnit.Net with, I have to say, some real success.

BDD was the subject matter being discussed at May's London Altnet beers. After the talk I was discussing with some people about their approach to doing BDD. What was being presented was the idea that BDD specs are useful only at the outer layers of the application and once you get more low level, you should revert to normal style TDD tests.

I don't agree. I have been writing BDD specs for every single class I write, and have recently upgraded our build system (rock on TeamCity) to make the spec reports. Saying BDD is just for the outer layers is ignoring the fact that the users of application know a lot more about their business than you do.

I'll give an example. We are building a system for the traders to view their trades - in what is known in the financial world as a "blotter". Being a WPF application, we build the ViewModels first and then build some sort of data access component. So the requirement is that all the trades made today should be displayed on the blotter when the user refreshes it. Here is a set of specs which say that

when the user refreshes the blotter and no trades have been made today
- should display a message saying there are no trades
- refresh button should be enabled

when the user refreshes the blotter and two trades have been made today
- should be two trades displayed in the blotter
- should display the earlier trade on the first row
- should display the later trade on the second row

So now we write some tests for that, and flesh out our implementation. During this time we create an interface ITradeProvider with a method GetTradesForToday(), and use a mocking framework to stub out the calls to that method using our favourite mocking framework.

Our user/customer/business analyst would definitely find those specs useful, and may even give some feedback or changes, which is great. But what about the TradeProvider implementation? Is that worth doing it for? Some would say no, but I would say. We could start off with something like this:

when loading todays trades
- should load todays trades
- should not load trades from a previous date

That's not really saying anything useful is it. Or is it? Imagine you show this to your users and/or business analysts. They're gonna ask:

"What on earth does that mean? When loading today's trades, should load today's trades? It doesn't make sense! We're a bank, we have loads of trade systems, which one are you getting the trades from?"

"Oh right, well I was going to get them from the MYZ team's Oracle database which we got the trades from on the last project I worked on"

"Ok, that's fine, but that's only going to work for trades outside of Asia. For the trades on the Asian stockmarkets, we're going to have to get the data from the AYN team located in Singapore. They have a web interface for looking at the trades and I think they provide some sort of service for you to retrieve it from"

"Oh OK, thanks! I'll update the specs and get back to you to review them again!"

You may think this is a fairly contrived example, but it's the sort of thing you will come up with time and time again. As developers, we move from project to project all the time, and we often make assumptions based on previous projects. Users, on the other hand, are getting you to build a system to provide value to their business. They are likely to know it a lot better than you! Business Analysts especially have a very detailed knowledge of all the internal systems and what their responsibilities are (though they may not have the kind of deep technical knowledge you have of communications protocols, etc).

So think twice about abandoning BDD for the lower layers of the system. Write the specs, share them, invite everyone you possibly can to review them. The earlier problems are found in a system, the cheaper they are to fix. That's been proven through countless studies.

Evidence Based Scheduling

A co-worker just sent me this article from Joel Spolsky: http://www.joelonsoftware.com/items/2007/10/26.html.

As I'd expect from Joel, it's a very well written article, and definitely worth a read. Has anyone got any experience doing this? I've always thought that tracking individual velocity was a bit fascist, but perhaps it could prove useful?

Also, has anyone tried running a project using FogBugz. Is it any good?

Extension methods make your code more readable

There has been a long standing argument for and against using extension methods. Recently I am on the quest to write code which is as readable as possible. That really is my prime aim - I want someone who is simply scanning my code to be able to, at a high level, understand exactly what is going on in that function. All other aims come second to that (e.g. until a requirement comes along which says otherwise).

Well, six months ago, I would never have made this statement - "Extension methods are a massive help here!". Honestly, they just are. I know people hate them because you can't really tell where the method comes from, and it looks wierd, and it breaks OO because you should inherit to add functionality (well actually, I disagree with that statement being made blindly)...

So I have a function which takes in some dates and returns a status depending on how the dates compare. Here is a simplified version of the function:

public Status GetStatus(DateTime now, DateTime modifiedOn, DateTime createdOn)
{
  if (modifiedOn.Date == now.Date)
    return Status.Amended;
  else if (createdOn.Date == now.Date)
    return Status.New;
  else
    return Status.Unchanged;
}

So what's wrong with that? Well you have no idea what the if statement is actually doing! Tell me, if you scan that quickly, can you tell what the business rule is here? I am pretty sure most people will have to scan that a few times to get what is going on there. I considered adding a comment, but comments are a definite code smell (see DRY) so this is how I changed it (using an extension method)

public Status GetStatus(DateTime now, DateTime modifiedOn, DateTime createdOn)
{
  if (modifiedOn.SameDayAs(now))
    return Status.Amended;
  else if (createdOn.SameDayAs(now))
    return Status.New;
  else
    return Status.Unchanged;
}

Simple but effective methinks. I think it's definitely clearer (though could perhaps be made even better). Do you agree?

A shout out to Benjamin Mitchell

Benjamin has recently started blogging (and about time too!). I have worked as a developer on one of his teams before, and he is a good guy to work for and really knows his stuff. He's one of those PMs who is always looking to improve his team's knowledge, skills and processes.

I hope to see postings about Agile techniques and Lean Development, and hopefully some good anecdotes too!

I am on Twitter

Well, after much nagging by my good friend Mr Norton, followed by a number of people at the AltNet Beers the other night ("what, you aren't on Twitter mate???"), I have finally succumbed and signed up. You can find me at http://twitter.com/nmosafi. I hope to be posting about all my fun and frustrations at work, and hopefully will avoid doing so whilst completely hammered!

Last night's AltNet Beers

Firstly, thanks to Seb for organising the ALT.NET beers conference last night. I had a great time and met some very interesting people who I hope to see at future conferences.

The first topic was on two related questions, which were something along the lines of "why would I use a non-relational database" and "how would I architect a database differently if I was writing more than I was reading".

Personally, I wasn't really getting the question here. The whole thing about "writing more than I read" made no sense to me - why bother writing it if you're never going to read it. The speaker was unable to really identify where his pain points are with his system. I felt that there was an interesting discussion to be had, but the conversation didn't seem to inspire many people and it wasn't carried too far. I would welcome this discussion continuing elsewhere and if it does I would be keen to take part in it.

We talked a bit about document databases here, Ayende naturally getting involved as he has designed (not built, mind) a document database before. The idea is that if you want to store fairly unstructured data, you would use a document database. Again, this isn't something I know much about and I wasn't sure what the benefits of a document database are over, say, storing the document on a file system and then indexing the information you want to search separately (e.g in a relational database for querying, or search appliance for searching).

Note: I was going to discuss this further outside with Ayende but the topic changed to BDD which is something I have recently starting doing so I wanted to hang around for that one! If you're reading then perhaps we can discuss it next time?

So as I said, we moved onto BDD which Ian Cooper summed up excellently with a great analogy, which I won't repeat here. What it bogs down to is that whilst test driven development is about ensuring that we do things right, behaviour driven development is about ensuring that we do the right thing. Given that we had a bunch of Lean/Kanban enthusiasts there, I thought it quite appropriate topic. With BDD you minimise rework by getting things done right the first time, this will naturally help to you can increase throughput and decrease wastage, which is what Lean's all about.

So having recently started a new job (which explains my lack of recent blog posts) I am trying to introduce BDD to the team, using SpecUnit.Net. They don't do TDD at all, and I think BDD is not only easier to adopt but also adds much more value. I am actually new to BDD, having consistenly read about it I'd not had the chance to actually try it on a proper project. I am impressed with the effect and the way it makes you think about your code, so much so that I couldn't see myself not developing in this way ever again!

Anyway I actually had some interesting conversations after the event about BDD. I will try to gather my thoughts about this soon and perhaps capture them on a future blog post.

Silverlight 3… why WPF any more?

Just reading about Silverlight 3 and amazed by the amount of new features which are being added… just look at the list!

It begs the questions is there actually anything left in WPF that Silverlight now can’t do?  It now even supports running outside of the browser!

COMET: The Bayeux Protocol and ASP.NET support

In my previous post I discussed the merits of the COMET technique for pushing data to the browser over standard web protocols.

The basic idea behind COMET is that the browser keeps an HTTP connection open and the server uses it to send data back.  This contrasts with HTTP’s Request/Response paradigm where the browser is specifically requesting data when it needs it.  There are two main types of COMET – Streaming and Long-Polling.  For Streaming COMET, the browser opens a single connection and keeps it open indefinitely, receiving events from the server when they are ready to be sent and immediately processing them.  With Long-Polling, a connection is opened and is kept open until the server has data to send to the browser (or data is sent immediately if waiting).  Once the data is received, a new connection is immediately opened which again blocks until data is ready to be received, and so on.

Over the last several years a protocol for COMET, known as Bayeux, has been formalised by the dojo foundation.  The protocol specifies the wire format for messaging between client and server as well as the transport mechanisms involved, and supports both streaming and long-polling over standard HTTP transports.  It’s essentially a specification for a pub/sub message bus, where clients can publish messages to channels and subscribe to messages on those channels.  A number of client libraries have support for Bayeux style comet messaging, most notably the Dojo Toolkit.

There are also a number of server frameworks for COMET out there, including one from Dojo themselves – cometd - which implements Bayeux and integrates with their client framework.  There are also commercial COMET servers out there, such as LightStreamer and Liberator, which actually offer .NET integration and support for integrating with enterprise messaging systems such as Tibco RV.  However, searching around I have been unable to find any that run within ASP.NET (well definitely not free ones anyway) and I think it’s a very important thing to support within ASP.NET - custom servers are fine, but you can’t exactly run them on shared hosting providers!

One of the main challenges to developing a COMET server which runs within IIS is the threading model of ASP.NET (I believe Apache has the same issues, but not sure).  With ASP.NET, each request is serviced by a single thread from the thread pool.  This basically rules out being able to support Streaming COMET, as we would have to keep a thread aside for each client, and there a limited number of threads in the thread pool.  So this leaves the Long-Polling option.  This is wholly possible within ASP.NET as it has support for asynchronous processing, meaning we can put the thread back on the thread pool until we actually have data to send back to the client.

And this is what I am working on now. The project is called AspComet and is available at the AspComet Google Code Website via SVN.  It’s pretty basic at the moment and it just supports a small test case, however it’s built against the Bayeux spec and therefore can be used with any client library which supports Bayeux (I am testing with dojo of course).

In the next post, I will describe a bit about how it works and walk through a sample simple chat application built using it.

COMET: Pushing to the web browser

A technology which is making a lot of advances in the world of the web is COMET. COMET is the name given to a group of techniques for pushing data to the browser using JavaScript. Being a Request/Response architecture, HTTP is not really designed for sending data to the browser on demand, COMET enables this.

So a bit of background here. Traditional “AJAX applications” often continually poll for changes and update the screen accordingly.  In fact, in ASP.NET, a very easy solution is to combine an asp:Timer class with an asp:UpdatePanel and simply re-render the changing content every few seconds and send back the updated HTML.  Taking this to a more advanced level, it’s possible to send data back to the browser (typically as JSON) and then use DOM manipulation to update the elements viewed on the screen.

To give a more concrete example, imagine a simple “chat room” application.  A simple solution could be to render a div with the list of chat messages in it, wrapped in an UpdatePanel, and then provide a text box for entering chat messages.  When the user presses “enter” within the textbox, an AJAX request is made to send the message back to the server, and the server adds this message to the list of chat messages.  The update panel can then be refreshed using a timer.

What are the problems with this?  Two things – scalability and latency.

Firstly, we are sending back the rendered HTML each time - this is not the most efficient use of bandwidth (i.e. each chat message is likely wrapped in a few elements).  Secondly, every time the UpdatePanel is refreshed, the entire chat history has to be sent back.  As this grows this will be become problematic as the amount of data being processed and sent over the network will increase exponentially with the number of users contributing to the chat.  Finally, as we are polling (say once every 5 seconds) there is a lag involved between when a user enters a chat message and when another user receives the message.  These are potentially not major issues, but in this day and age the average web user’s expectations are quite high - they expect lightweight, fast and responsive web user interfaces!

We can solve each of these problems in turn.  To resolve the first issue, we can remove the update panel, and replace it with an AJAX call made using a standard JavaScript “setTimeout()” function.  Instead of sending back HTML using an UpdatePanel, we send back an array of JavaScript objects encoded as JSON.  Each object represents a single chat message and could have the following properties – ID, Source, Message.  The AJAX response handler would render the appropriate HTML from the JSON objects (for example, using ASP.NET 4’s client side templating).

Once you have JSON objects coming back to the browser, the second issue becomes easy to solve.  The browser can send back the ID of the last message it received, and the server only needs to send back those messages which were received after that.

It’s when we try to resolve the latency issue that the this technique becomes problematic.  The only real solution using plain old polling is to increase the polling frequency, which leads to a huge waste of server resources.

So we come back to COMET, which is a technique (some might say hack, but I disagree) of pushing data to the browser using standard HTTP requests.  There are alternatives to this approach, namely using browser plugins.  Flash and Silverlight both have support for raw sockets, and Silverlight has the “polling duplex” WCF transport (which I may discuss in a later post).  However, COMET is a solution which has been designed from the ground up to use standard web technologies and run over HTTP.  I will discuss the technical details of COMET further in the next post… this post is really just an introduction and to point out that I have started working on a project which will enable COMET via ASP.NET hosted in IIS!

More to come…
Neil

Another great error message from MS

Spent the last 2 hours trying to diagnose an error message.  I am writing a batch file which simply executes a SQL script using sqlcmd.  This is the command:

sqlcmd -S %SQLServerName% -E -i InstallSql.sql > InstallSql.txt

For the life of me, I have no idea what is going wrong.  The error message is incredibly useful

Sqlcmd: Error: Microsoft SQL Server Native Client 10.0 : Syntax error, permission violation, or other nonspecific error

Ok, so it could be one thing, or maybe it’s another, oh actually it could be anything at all really.  Essentially they're saying YOU ARE ON YOUR OWN WITH THIS ONE, trial and error is the only way.

Anyone got any ideas what this could be? I’m gonna throw my laptop out of the Window soon!

Azure Event Log and log4net

I have been experimenting with Windows Azure recently for hosting a small application I wrote.  Thus far I am fairly impressed; despite a few periods of downtime and the odd bug with deployments, things seem to be stabilising fairly quickly and it’s all working well now (well I can’t complain as it’s currently free!)

Azure’s hosting platform provides a number of services which allow you to interact with the platform.  These are mostly through the Microsoft.ServiceHosting.ServiceRuntime.RoleManager class.  For example, you are able to retrieve configuration settings from the service configuration file, access local non persistent resources, and log messages to the Azure Event Log, via the RoleManager.WriteToLog method.

Messages can be copied from the log to any Azure Storage account simply by clicking a button on the configuration menu of the hosting instance:

These logs are copied to the “blobs” area of your chosen Azure Storage account (there are also tables and remote queues) and Microsoft have provided a fairly simple REST API for managing the data stored within the account.  There is also a .NET api which wraps the REST API but I am yet to look at it (I will do once I build a persistence layer for my application).

A few tools have been written for viewing the log files – Azure Storage Explorer seems to be the best that' I’ve tried, those there may be more which I’ve not found yet.  You simply put the URLs of the REST endpoints into the config file and you get a nice viewer which lets you see all the messages and delete

Now all my applications tend to standardise on log4net for logging messages, which provides configurable and customisable routing of logging messages to pretty much any target you want.  I wrote a simple log4net appender which can be used to route messages to the Azure Event Log.  Here is the source code

public class AzureAppender : AppenderSkeleton
{
    protected override void Append(LoggingEvent loggingEvent)
    {
        RoleManager.WriteToLog(GetLogName(loggingEvent), loggingEvent.RenderedMessage);
    }

    private static string GetLogName(LoggingEvent loggingEvent)
    {
        if (loggingEvent.Level == Level.Critical)
            return EventLogNames.Critical;

        if (loggingEvent.Level == Level.Error)
            return EventLogNames.Error;

        if (loggingEvent.Level == Level.Warn)
            return EventLogNames.Warning;

        if (loggingEvent.Level == Level.Info)
            return EventLogNames.Information;

        if (loggingEvent.Level == Level.Verbose)
            return EventLogNames.Verbose;

        return EventLogNames.Error;
    }

    private static class EventLogNames
    {
        public const string Critical = "Critical";
        public const string Error = "Error";
        public const string Warning = "Warning";
        public const string Information = "Information";
        public const string Verbose = "Verbose";
    }
}

Note that log4net has lots more log levels than Azure supports, so I just default to Error if I get a message which is not one of the matching levels.