One of the first questions non-technical entrepreneurs who have an idea ask me is, “How much would it cost me to build an application and how soon can you build it?” And for someone, who is not familiar with software development, it could be very difficult and even frustrating to get a realistic estimate with a timeframe. You might be surprised to learn that a lot of developers and managers don’t understand how to estimate software either! I’ve personally been a part of large scale development efforts, with ballpark estimates, coming from management, where projects never met the deadlines and were delivered way over budget. Hopefully after reading this post you will have a better understanding of what it takes to get a good estimate.

Software Estimation Story

Building software is much like building a house. Imagine you want to build a house and have a friend who is an architect, so you ask him if it’s possible to build a three bedroom house for around $100,000. He thinks about it and says yes, but it depends on the type of house and what’s going to be in it. In your friend’s mind he imagines a simple house, with no thrills, that he could design and build for that amount. But you might have some other ideas. You want to have a nice swimming pool, a playground, a three car garage, bamboo floors, granite everywhere, a sauna, gold plated mirrors, high ceilings, and etc. You also might want a house on a hill with a view. At this point the house might cost ten times the original amount.

There may also be many times during the project when you forgot to mention or thought of some other important feature, like an entertainment room, that you feel you must have! But your friend has already started working on the house and built the frame. And you might have to redo the entire design and knock down walls to accommodate, it which will make the project off budget, and add a lot of additional time.

It’s not always necessary to know everything up front, but it is important to anticipate large features so that your friend can design a plan that will accommodate changes later.

Steve McConnel, in his great book “Rapid Development”, gives a great summary of the software estimation story that all customers and managers must understand:

The basic software-estimation story is that software development is a process of gradual refinement. You begin with a fuzzy picture of what you want to build and then spend the rest of the project trying to bring that picture into clearer focus. Because the picture of the software you are trying to build is fuzzy, the estimate of the time and effort needed to build it is fuzzy, too. The estimate can come into focus only along with the software itself, which means that software-project estimation is also a process of gradual refinement.

In other words, it’s impossible for a programmer to estimate the project until he has a detailed understanding of what the customer wants. Until a specific feature is understood in detail, the rough estimate could be a factor of sixteen differences for that feature.

Another difficult part is that unless you’ve previously built the same house, you probably won’t have all the details, hence you have a fuzzy picture. Software development is a process of discovery and gradual refinement. Developers work with customers to unearth intricate business rules, features, and capabilities. One way you can bring a picture into the focus is by going through the process of defining a product, thinking through the requirements, and creating a design. Let’s go through what we need to have to better understand what we are trying to build.

Defining a product

Detailed understanding means that a developer can take the product definition and create a model of it in code. Most likely a developer is not a domain expert of the business and he needs a lot of help from the customer to understand it. Imagine that you get a job at an unfamiliar organization, where you had to learn how the business works before you can be productive. You will need a lot of time to get up to speed, and so will a developer in order to build your product. It’s also much cheaper to be make mistakes and have invalid assumptions at this stage, because no code is written and it doesn’t cost anything to correct it.

For the product owner at this stage, a good place to start is to define a clear problem/solution statement, along with user stories, tasks, and use cases. User stories is a simple description of a feature told in the following way:

As a <type of user>, I want <some goal> so that <some reason>.

For example, it may look like this: As an incentive compensation specialist I would be able to set up compensation wrappers so that I can create goals for payees. Does this story make a lot of sense to you? If you are not working in the incentive compensation business, I don’t think it will. Stories like these are an invitation for conversations between developer and business experts. During the conversation the developer will identify terms used, a vocabulary for that business domain, and know ubiquitous language in Domain Driven Design. It will also lead to a better understanding of a product, elements, subsystems and entities.

At this stage any additional information that a business expert can present would be helpful. Examples are screenshots of similar application, high level flowcharts, wireframes, onboarding manuals, etc. A developer can write a documentation summary about the problem/solution domain identifying all the entities, business vocabularies, definitions, subsystems, interactions and actors.

Requirements Specification

After defining the product and learning about the business, we can see a better picture, but it’s still not perfect; at this moment there could be a factor of eight differences between the high and low estimate. The next step is to zoom in even closer and define a list of requirements. The amount of features software can have has a tremendous impact on the design, time, and cost of the application. Even if you practice agile methodologies, you still need to gather a minimal set of requirements so you can plan for the future. Just like in the definition stage, it’s much easier, and cheaper, to make changes during requirements rather than realizing you’ve missed an important feature down the line, and have it take months to redesign the application. (Knock down walls).

Input at this stage from the business side should be a list of important features that this project must have. While a developer’s job is to understand how these features fit together and to create a plan that will accommodate future expansions.

Product Design

We are getting much closer to a better estimate at this point; the difference might be a factor of three between high and low estimates. Over my career I’ve learned that nothing brings software into a picture like creating a high level and detailed design. This is the stage where a developer takes everything learned about the product, and starts modeling the domain. During this stage she will make a lot of mistakes and flush out incorrect assumptions. Again, much cheaper to make mistakes here than during coding. Starting with a high level design of the major subsystems and defining a clear communication between them, then modeling your domain and define major objects and abstractions. (For a more detailed analysis on how to model a domain, see Eric Evan’s book, “Domain Driven Design”).

What if I don’t know what I want?

I think the best way to approach this problem is to hire a User Experience (UX) designer who can take vague ideas and create throw away sketches of the product. Each sketch can then be reviewed, refined, or redrawn. This process will flush out the requirements that will crystalize in an initial prototype, and can be tested on users before you start the expensive software development. A good UX designer in this situation will save you a lot of money because she will bring the requirements and prototype into focus, for the fraction of the cost of building software! By sketching, creating multiple prototypes, and then testing it on users, you know exactly what works and there is no chance of building something that your customers can’t use.

I see a tremendous amount of benefit of hiring a UX designer early in any software project. There is a lot of power in sketching, defining, and user testing multiple prototypes without actually building anything. It’s costly to experiment by building software and I don’t think anyone can get the user experience right the first time.

Getting an estimate

After the picture is more clear, a developer will create a list of low level details with a range estimate for how many man-hours it would take to complete each task. You must get estimates on low levels of details, this way you get a more accurate estimate and you can gauge how the project is progressing. A good developer will also provide additional information about the costs that are associated with each task and other alternatives that could be more cost effective. Like a good contractor who would present and educate you about different counter tops, their benefits, and costs, you want your developer to do the same for you when providing a list of estimates.

It’s also very important not to overlook other important tasks that are not coding, but have a significant impact on schedule and budget, like QA, documentation, deployments, setting up dev/production servers/databases, configuration, integration, holidays, training, and sick days. In addition there should be allowance with working with new languages, working with unfamiliar technologies, and frameworks. There is always a learning curve when doing something new and this should be accounted for.

Make estimation part of your plan Estimation should be part of your plan and you need to spend a significant amount of time on it; don’t rush it.

Word on Agile Software Development

I am a big believer in iterative development, rapid delivery, minimal planning, and many other principles Agile Manifesto stands for. Unfortunately, I’ve noticed some developers use Agile as an excuse to have no planning, design, or requirements analysis at all, which results in the costliest approach of them all: code-like-hell. It’s common, and you want to be wary of ballpark estimates with use of an Agile as an excuse to not do to any planning or design.

There are mission critical pieces in any business application that might call external services via some sort of an external API over the local network or Internet. The problem with delivering messages over the network is that networks sometime have a tendency to drop messages and cause timeouts. There is a great paper on Kyle Kingsbury’s blog where him and Peter Bailis provide a list of evidence that failures can happen in many levels of the network or operation system. And it’s not only network that can fail, the receiving 3rd party applications might also be down, under load and slow to respond.

Processes, servers, NICs, switches, local and wide area networks can all fail, and the resulting economic consequences are real. … The consequences of these outages range from increased latency and temporary unavailability to inconsistency, corruption, and data loss.

Therefore, if we want to have a reliable communication with external services we need to implement some kind of a retry mechanism that can redeliver messages and recover from faults.

Solving Guaranteed Delivery

Message Queue is one of the solutions that can address this problem and guarantee delivery. Rather than calling an external API within the application process, like a web request, you place a message into the reliable and durable message queue that guarantees that your message will be delivered to the consumer at least once. Since putting a message on the queue is usually a fast operation, it also speeds up your application performance. Most message queues guarantee delivery by providing some sort of a mechanism of acknowledging if message has been received by the consumer. And if consumer doesn’t respond after a period of time the message gets returned into the queue so it can get processed again. This basically guarantees that a message will get delivered or retried for a certain pre-configured number of times.

There are numerous alternative Message Queue solutions with each addressing certain problems in it’s own way. Since the main specification for calling critical services is guaranteed delivery and we are not building something like a high throughput trading application, the Amazon SQS provides the best alternative to the self-hosted MQs. One of the benefits is that you don’t have to administer message queue servers and spend a lot of time figuring out how to setup redundant clusters for reliability and worry about network partitions. Of course you loose on speed of placing a message into the queue. But a 20ms average put call to SQS is also good enough for this problem.

Until you figure out if you actually need something faster and spend all the time to learn and setting up the MQ cluster, I think Amazon SQS provides the best bang for your time. It’s easy to understand, has a great SDK and it’s ready to go. It’s also not very expensive. A million calls costs $1. Yes you do have to poll the queue, but you can also use the long polling and that will reduce the number of calls of one consumer to 1 call every 20 seconds (if no activity), which is about 120k calls per month, or about 12 cents.

Putting a message into the queue is trivial and doesn’t need more explanation. What is not trivial is creating a reliable application with workers to process the messages. We also want the ability to start multiple workers so they can process messages from the queue in parallel rather than one at a time. Since many workers can be started on one node and run in parallel we need our service efficiently use the CPU resources meaning not blocking threads while waiting for I/O.

I wasn’t able to find an open source application that can always listen to the queue and process messages as they come in. So I needed to write my own. A good candidate for this task is a Win32 Service, since it provides a platform for always running service that can also self restart itself on fault and boot up with windows automatically.

Creating Message Processor Win32 Service

The windows service must always be running, meaning that each worker will have a main while loop that will continue indefinitely. Also you need to start multiple workers so you have to use some sort of multi threaded solution. My initial version was to new up multiple Threads that invoke an asynchronous method. Like this:

protected override void OnStart(string[] args)
{
      for (int i = 0; i < _workers; i++)
    {
      new Thread(RunWorker).Start();
    }
}

public async void RunWorker()
{
  while(true)
  {
    // .. get message from amazon sqs sync.. about 20ms
    var message = sqsClient.ReceiveMessage();

    try
    {
        await PerformWebRequestAsync(message);
        await InsertIntoDbAsync(message);
    }
    catch(SomeExeception)
    {
        // ... log
        //continue to retry
        continue;
    }
    sqsClient.DeleteMessage();
  }
}

And it was working fine, however there is a problem with this code. The initial threads that were created would exit once the first execution of the method would hit first await. So there was really no point of creating those threads. In addition, I wasn’t passing a cancellation token to threads so I could not signal it to shut down whenever I wanted to gracefully exit the service. Thanks to Andrew Nosenko who pointed out a better and cleaner way of accomplishing the same goal using tasks.

Rather than starting threads manually you start each task and add it to the List collection. This way the threadpool is efficiently managing your threadpool threads and schedules it according to the system resources.

List<Task> _workers = new List<Task>();
CancellationTokenSource _cts = new CancellationTokenSource();

protected override void OnStart(string[] args)
{
  for (int i = 0; i < _workers; i++)
  {
    _workers.Add(RunWorkerAsync(_cts.Token));
  }
}

And inside of the RunWorkerAsync’s while loop you call token.ThrowIfCancellationRequested(); that will throw OperationCancelException and exit the thread when the cancel is requested.

With windows service when you start a service the main Win32 Service thread gives you some time to start your processes and it must return quickly, meaning not to get blocked. So your OnStop method is where you have to call your Task.WaitAll(_workers) which blocks the current thread until all workers have completed their tasks. So once the OnStop method begins you signal the cancellation token to cancel the tasks, and then you call Task.WaitAll and wait until all tasks run to completion. If all tasks have been completed prior to calling WaitAll it would just continue so there is no risk that it could finish faster. The OnStop method looks like this:

_cts.Cancel();
try
{
    Task.WaitAll(_workers.ToArray());
}
catch (AggregateException ex)
{
    ex.Handle(inner => inner is OperationCanceledException);
}

It uses an AggregateException.Handle method which will throw any unhandled exceptions after it finished running. And since we are only expecting OpearationCanceledException it will just return.

Polishing it up

One problem with windows service application is that it’s hard debug, you cannot just attach the debugger to the running win32 service. To work around this problem we will use a topshelf project. Topshelf allows you to run your windows service just like you would run a console application with the ability to debug and step through the code. It also make it easier to configure, install and uninstall the service.

Here is a quick sample code that will make a message processor console application into a Win32 service.

public class MessageProcessor
{
    List<Task> _workers;
    CancellationTokenSource _cts;
    public MessageProcessor()
    {
        _workers = new List<Task>();
        _cts = new CancellationTokenSource();
    }
    public void Start() {  //.. same as above }
    public void Stop() { //.. same as above }
}

public class Program
{
    public static void Main()
    {
        HostFactory.Run(x =>
        {
            x.Service<MessageProcessor>(s =>
            {
                s.ConstructUsing(name=> new MessageProcessor());
                s.WhenStarted(tc => tc.Start());
                s.WhenStopped(tc => tc.Stop());
            });
            x.RunAsLocalSystem();

            x.SetDescription("Amazon SQS Message Processor");
            x.SetDisplayName("AmazonSQSMessageProcessor");
            x.SetServiceName("AmazonSQSMessageProcessor");
        });
    }
}

Once you build an executable, you can run MessageProcessor.exe install from command line and the service will get installed, additional -help will show you all the commands that you can do.

Summary

Incorporating queues in your application architecture can help with guaranteed delivery of the business critical messages. It can also speed up your application process since it will offload the work to the external process. On the downside, your application becomes dependent on another application running in a separate process and it’s more code to maintain and deploy. To ensure you message processor doesn’t become a single point of failure, you will also need to have at least 2 nodes running this windows service for redundancy. However, if your business requires guaranteed delivery for the mission critical API calls, the overhead of maintaining message queue solution is worth it’s weight.

The problem is that most software engineers suck at sales. I personally felt like it was never my job, and never bothered to learn it. Only to discover later that if I ever wanted to grow a successful business I would have to eventually take the bull by the horns and do it. And it’s not only business, being a great salesman can help to build a successful career as well.

Having lack of knowledge and experience in selling, it was very important for me to hear a real story from a person in software industry, who was actually successful at selling his or her consulting services. Luckily, I came across this great interview on The Eventual Millionare podcast where I learned about Ian Altman . He built and sold software consulting businesses before, and he has coined his own method of selling that actually made a lot of sense to me.

I’ve always had this bias about salespeople being sleazy and dishonest, but what Ian found out is that it doesn’t have to be that way. He explains that if you focus on FIT (Finding Impact Together) you actually become a problem solver who works together with the client rather than against him. You focus on results and impact, and if your service doesn’t bring value to the client you walk away.

Enter “Same Side Selling”

I was eager to learn more about his approach and I picked up Ian’s book right away. Since I never closed a sale before, I found Same Side Selling book very informative. Further, I learned that rather than explaining all the things that you offer, if you focus on actual problems, your client would not need to translate it into his own language. And if what you are describing is an actual problem that his business is having, he will be much more likely interested in what you have to say.

The book is full of wisdom on how to sit down with a customer to identify the problem, and figure out if it’s worth solving. Author brings up analogy of puzzle pieces where you need ask a series of well formulated questions to learn about the pieces that your customer has and see if yours fit. Unless you actually get to the underlying issue you won’t know for sure that your solution will bring the results.

Honesty and focusing on the outcome rather than a sale is the best way to get on the same side with a client, and it’s much more likely to benefit both parties in the long run.

This approach of problem solving fits my personality very well, and I will definitely practice it in the near future.

I will have to read this book over again, especially before a meeting with a client, so I can refresh and tune in on the same side selling method. I will let you know how it goes.

Honest Solution

This book has opened my eyes that selling doesn’t have to be adversarial. That figuring out a problem with a client and getting his business should be mutually beneficial. Honest selling helps building reputation and gets repeat business.