Releasing Software Fixes

My day job consists of performing software maintenance on a suite of software applications. That means I fix bugs. The customer is a government organization that we usually do not talk about. We have process controls to ensure that any changes to the software are controlled and managed. Today I thought I would go over the process of releasing bug fixes to give you an idea of what all is involved at my work.

Let's suppose that the customer has opened a trouble ticket on a problem in our software. To start with, our Help Desk team enters the information in our bug tracking tool. Then the Help Desk assigns the ticket to the team lead of one of our teams. The team lead often does a little analysis on the problem. Then the ticket is assigned to a developer on their team. These days I fill the team lead role for the application developers. But I am also a developer myself (sometimes I assign problems to myself).

While you are working on a trouble ticket, you are to enter your progress in our bug tracking tool. This allows the Help Desk to provide status to our customers. Let us suppose that I get lucky enough to figure out the problem and determine the code changes needed to correct it. Here are the steps I would take to get the software fix to our users:

• Document all code changes in a Microsoft Word document
• Generate a unit test plan to verify the fix
• Obtain a peer review of the code changes and unit tests
• Check the code changes into source code control
• Request a release number from the configuration management team
• Check the release number information into source control config files
• Label all files that are to be included in the release
• Generate the documentation that accompanies the release
• Submit a request ticket to the configuration management team to build the software
• Review the configuration management team work for technical problems
• Forward the software release to the internal test team

This is the list of steps that are taken for every single software change that corrects a bug in our system. Things get more complicated when there are more than one defect corrected in any given release. Other complications include the fact that some of our code is in a C++ client application, while some other code is in database stored procedures. The above list does not take into account all the additional steps that occur after the application development team is done their part in the release.

Hacking Software

I run a small informal software business on the side of my day job. Recently I have been writing small programs which perform tasks that Internet hackers might do. The programs have been released on my Black of Hat blog. The interesting thing about this effort is that I have employed very little process in developing my own software.

For example, my latest program "Crawl" is one that scans a web page and extracts the URL links. I thought about the problem for a couple days. Then I went directly into coding this morning and developed version 1 of the software. Late in the morning I built my release version of the application. Finally I posted this app on my blog.

So far my application appears to be working. I have received no reports abouts bugs in it or it not working on different systems. There was not rigorous process applied to this application. I did do some thinking before coding. But I did not write down and specifications or designs. I really did not perform any formal testing on the application. And the software was developed in record time with not many problems.

In retrospect, this may have been a special case of software development. The application was not very complex or large in scope. The development team was just one person. And the developer (me) was the same of the person who determined the requirements. Things get more complicated and may require process overhead when they scale up. However for my side software business, the real question is whether I would gain any benefit it an added layer of formal process was imposed on top. I am still mulling this question over in my head.

Perhaps I shall try adding a formal process to the next application I develop for my side business. I can track how much additional time I spend on the process part of development. And perhaps I can try to objectively rate the quality or amount of problems in a process controlled application release. The results are not some theory that will get documented in a paper. They will help decide how to run my own software business which I hope to succeed.

Disorganization

My company is trying to close out the project we are working on. This is due to the fact that we lost the contract to perform the work. There are a number of personnel still working on the project. If they are not busy on other tasks, they are supposed to be working on locating, organizing, and storing our documentation. I assigned our least busy developer to work full time on this task. I told him to go find out what needs to be done, and do it.

In retrospect, this may not have been the best direction to give. But I did not want to get bogged down in the details as I have other priorities to attend to. There is an individual who is assigned to the documentation task. And there are many people working on the task. But after working for a while, we had a meeting where it became clear to me that this task was in a state of chaos.

I directed a few key members to come meet me in my office. At this follow-on meeting, I declared that we needed to finalize and document a folder structure in which all our application team documents needed to go. The person who was in charge of the task said it was their responsibility to determine what needed to be done. I concurred. However I also pointed out that it was evident there was not method to the madness of the task. Later that afternoon, the task leader put together a document demonstrating a folder structure for organizing documents for each team. In addition, the actual goals of the task we finally articulated in writing.

Sometimes you need to sit down, figure out what you need to do, and document the plan. This is required when the task at hand requires more than one person, and absolutely needed when it involves more than one team. One would think that tasks such as document collection and organization are nothing like rocket science. But without a plan, and more importantly a documented plan, you have all the ingredients for utter chaos in the workplace. Why must we have to relearn these lessons year after year?

Document Organization

Let's suppose that you have actually documented the main processes used. How does one access this information? When I first came to my current project, you got handed a huge three ring binder full of information. This was good in that I could read up without even having a computer or access yet. However this got to be a pain when the documents needed to by updated. And as staff transitioned through the project, the binder and documents seemed to get lost.

More recently our application development team put together a small Microsoft Access database. This contained all kinds of information pertinent to our team. It was nice because you could search it based on keywords. Updated were easy. There was some trouble when two or more developers tried to update the same document at the same time. But this was an infrequent occurrence. Since this was a side project, the server where the database was hosted was not up 100% of the time. Still it was a useful project.

Over the last year or two, the project has tried to organize and assemble all documentation in a single place. The goal was to have a common structure for each team. This was a monumental task. A couple people have tried to accomplish the task. They have failed. Now we have a full team working on this. I do not want to be negative, but I am not expecting much from this. The scope is large. And you usually don't get your best people assigned to a task like this. Such a shame.

Process Matters

My team performs software maintenance on a legacy product. Basically we respond to software trouble tickets. These tickets are tracked closely by the customer. Our company's help desk team has been recently hounding us for "estimated resolution dates" for each ticket. So development has been figuring out, given what we know about the problems, what is the most likely date when we can produce a fix.

Something did not seem right about the Help Desk stating they were being hounded by the customer for estimated resolution dates. So I decided to participate in the next meeting with the customer. It was here that I found out what the customer was looking for. They wanted to know what was the date that we could expect a user to be able to experience the fix and verify that the trouble ticket had been resolved.

That about blew my mind. There is often a huge gap between the time we release a fix to our customer, and when the user actually experiences the fix. Our customer has a lengthy configuration management process. Our fixes go through their quality control first. Then somebody has to actually implement the fix. This can take time, especially when pushing workstation fixes out to the multitude of our users. There are also some processes which run once a week. Depending on the timing, users may not see the process run for up to a week.

I was glad when our project manager also expressed concern about the definition of estimated resolution date. We had to amend the way we accounted for the days that make up this metric. It seems we had been shooting ourselves in the foot by using the wrong definition of this date. You need the right process in place. And that process needs to be continually monitored to ensure it is still relevant and correct. Otherwise you are potentially marching down the path to doom.

Write It Down

A basic tenet of a good software process is that it needs to be documented. This may seem obvious. But you would be surprised how many time the new guy comes in, and must spend a lot of time asking people how things are done. This does not mean a lot of detail needs to be written. And it does not have to be very formal. It just has to been accessible and practical.

I have seen another problem with software process documentation. Often times there are huge documents that are supposed to document the software process. However when read, these documents are actually full of fluff that do not correspond to the actual process being used. This is worse than no documentation. It is lip service to the software process.

In the past I have lead software teams. Some times I had implemented a hands-on approach where I oversaw all aspects of my team's work. In other cases I just wrote out a handful of well-documented process pages, and let the team follow them as I spent more time coding. Guess which option was more fun for me?

Let it Begin

I am a firm believer that a good software process can be a major component of development success. This becomes more important when you have a staff of mediocre programmers. Left to themselves you can get a software disaster very quickly. But a documented process can help guide the sheep to deliver in a predictable manner. I use the term sheep lightly. However software is a challenging profession. There are a lot of developers who are not here but are not entirely up to the challenge.

Even outstanding developers can benefit from a well defined software process. This let's them concentrate on the more important tasks of development like solid architectural design. For the rest of the mundane tasks, they can just follow the process. Their minds will be freed up to continue to be productive on solving the truly hard problems.

If instituting a good process was simple, more software project would be successful. But creating and enforcing high quality processes is in itself a significant task. The great thing about really good processes are that they are self-correcting (process improvement). Hard work up front can result in less headaches later. I plan to explore such processes in this blog.