tag:blogger.com,1999:blog-30976904395274556042024-03-12T19:47:27.413-04:00Software ProcessA study of how a strong software process can drive success in development.Software Processhttp://www.blogger.com/profile/17308836734575362181noreply@blogger.comBlogger22125tag:blogger.com,1999:blog-3097690439527455604.post-47930092243981720182013-04-23T10:58:00.000-04:002013-04-23T10:58:54.252-04:00Falling Apart<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-HWxQqQLIPlM/UXago3rw56I/AAAAAAAAAB8/FjGdO7aMyv0/s1600/FallingApart.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" dua="true" height="320" src="http://3.bp.blogspot.com/-HWxQqQLIPlM/UXago3rw56I/AAAAAAAAAB8/FjGdO7aMyv0/s320/FallingApart.gif" width="247" /></a></div>
I have been working on my current project for a long time. We are talking over 10 years now. So I have seen the project change a lot. I focus mainly on the development side of things. That is where I have the most insight.<br /><br />
We used to do a lot of good things in development. There were daily builds to make sure nobody broke anything. There were efforts to document the system in a formal way. We kept track of metrics on bugs to see how we were doing overall. Towards the middle of the project, we started writing white papers. We also attempted to make our documentation more accessible with a Wiki. We used our bug tracking tool to manage work.<br /><br />
Now most of that has been thrown out the window. Ad hoc processes, if you could call them that, are being used in place. Sometimes software gets reviewed. Sometimes it does not. There is some upfront documentation on new features. But that is about it. We no longer use a tool to manage work. We use email. Oh joy.<br /><br />
Of course we have a hit in quality. And what is more depressing, it takes a lot more effort just to move things along now. Where did we go wrong? Well different companies have taken over the project now and then. The management style is starkly different between these companies. And the results flow down hill to the rank and file.Software Processhttp://www.blogger.com/profile/17308836734575362181noreply@blogger.com0tag:blogger.com,1999:blog-3097690439527455604.post-29351340463267591912010-08-18T13:34:00.002-04:002010-08-18T13:38:56.373-04:00Continuous<a href="http://1.bp.blogspot.com/_2ZvMWcMIvD4/TGwZpfAZl7I/AAAAAAAAABc/KtUM1GL8DyA/s1600/Continuous.jpg"><img id="BLOGGER_PHOTO_ID_5506804644855781298" style="FLOAT: left; MARGIN: 0px 10px 10px 0px; WIDTH: 320px; CURSOR: hand; HEIGHT: 121px" alt="" src="http://1.bp.blogspot.com/_2ZvMWcMIvD4/TGwZpfAZl7I/AAAAAAAAABc/KtUM1GL8DyA/s320/Continuous.jpg" border="0" /></a>Oh boy. It has been a long time since I posted an entry to this blog. Well let's get right to it. I want to talk about continuous delivery. In the old days, our team would produce one major software release a year. Then it grew to 2. This year we did 3. I think you see the pattern.<br /><br />Incremental <a href="http://susops.blogspot.com/2010/05/more-build-woes.html">builds</a> were a way to integrate changes in the system faster. Less changes. So there is less to search through when something goes wrong. When taken to the extreme, this is continuous delivery. You do a release each time you do a build. Damn.<br /><br />This level of delivery almost requires you to automate the heck out of everything. Problems get exposed right away. You need to be ready for this level of release. Let's get one thing straight. There is going to be pain. We are not ready for this on my project. There is pain all right. However continuous builds will make all the pain come to fruition at once. We can't handle that. Not yet at least.Software Processhttp://www.blogger.com/profile/17308836734575362181noreply@blogger.com0tag:blogger.com,1999:blog-3097690439527455604.post-23254982120101207862009-06-11T18:10:00.002-04:002009-06-11T18:10:55.148-04:00Lessons from 37 Signals<a href="http://4.bp.blogspot.com/_2ZvMWcMIvD4/SjGA1JzkZ_I/AAAAAAAAABU/5VO8sskajNA/s1600-h/37.jpg"><img id="BLOGGER_PHOTO_ID_5346195883319126002" style="FLOAT: left; MARGIN: 0px 10px 10px 0px; WIDTH: 207px; CURSOR: hand; HEIGHT: 207px" alt="" src="http://4.bp.blogspot.com/_2ZvMWcMIvD4/SjGA1JzkZ_I/AAAAAAAAABU/5VO8sskajNA/s320/37.jpg" border="0" /></a>37 Signals is a successful <span class="blsp-spelling-corrected" id="SPELLING_ERROR_0">start up</span> company. They have a number of rapidly developed small products which they sell. I have read a white paper from them on the lessons they have learned from their experience with software development.<br /><br />They recommend you keep projects very short. We are talking about the order of 2 weeks or so. They advise against project plans which are set in stone. 37 Signals does not believe in design documentation or functional specifications. That seems like a radical idea. They also tend to focus on what is real as opposed to abstract theories.<br /><br />Regarding content, 37 Signals likes to focus on things that are constant. They want to solve problems which will be encountered for the next 10 years. They also believe in sharing ideas with the public so that others may learn. Here is an idea which is not new. 37 Signals believes that you should “question everything”. I could use that advice myself. Having programmed for many years, I carry a lot of baggage in the form of untested assumptions.<br /><br />37 Signals believes that there is a time when you should give up on a problem which is hard. They also back the idea of working 32 hours a week instead of 40. When you think about it, that is not so radical when compared to the Tim Ferris 4 hour work week. In terms of timing, 37 Signals thinks you should release the best half of your application first.<br /><br />According to this company, client work is depressing. I second that motion. You can work hard to solve a specific problem for a client. However in the end, all you get to do is charge for those hours. You are not building a cash cow. I would like to end on a funny perspective from 37 Signals. They have no formal training program. Their method is to just drop people in the pipeline and hope they figure things out.Software Processhttp://www.blogger.com/profile/17308836734575362181noreply@blogger.com0tag:blogger.com,1999:blog-3097690439527455604.post-1654070369450780332009-05-16T14:01:00.002-04:002009-05-16T14:02:31.705-04:00SCM<a href="http://1.bp.blogspot.com/_2ZvMWcMIvD4/Sg7_m1IlmnI/AAAAAAAAABM/oa3j-P_pxQY/s1600-h/SCM.jpg"><img id="BLOGGER_PHOTO_ID_5336483651043826290" style="FLOAT: left; MARGIN: 0px 10px 10px 0px; WIDTH: 320px; CURSOR: hand; HEIGHT: 191px" alt="" src="http://1.bp.blogspot.com/_2ZvMWcMIvD4/Sg7_m1IlmnI/AAAAAAAAABM/oa3j-P_pxQY/s320/SCM.jpg" border="0" /></a>There is a book online by Eric Sink about source control. You would think this guy knew what he was talking about. He founded a company that produces source control software. I thought I would share some of the things I learned or recalled while reading his book.<br /><br />The first order of business is defining source control. It is also called version control or <span class="blsp-spelling-error" id="SPELLING_ERROR_0">SCM</span>. This let’s developers work in parallel on the same code base. You must learn how to do this. You don’t get taught this at college. Some example of source control systems are <span class="blsp-spelling-error" id="SPELLING_ERROR_1">SourceSafe</span>, Subversion, <span class="blsp-spelling-error" id="SPELLING_ERROR_2">CVS</span>, and Perforce. The product created by the author Eric Sink is Vault.<br /><br />Source code gets stored in a repository. You can get a copy of the code to view. You can also check out a copy of the code. The repository will automatically merge your changes in with changes from other developers. How is this repository maintained? It can store the whole copy of every version. However that is not very space efficient. Instead I believe most repositories only store the deltas between each version.<br /><br />You normally have the ability to name a certain version of a file or your whole code base. This name is called the label. Another idea in source control is branching. This is where you have two or more copies of your code base at the same time. The good source control systems are integrated right into your <span class="blsp-spelling-error" id="SPELLING_ERROR_3">IDE</span>. I must confess that, even though my own source code tools integrate with my <span class="blsp-spelling-error" id="SPELLING_ERROR_4">IDE</span>, I do not use this functionality. I prefer to do things manually myself.Software Processhttp://www.blogger.com/profile/17308836734575362181noreply@blogger.com0tag:blogger.com,1999:blog-3097690439527455604.post-17746262784084942892009-05-13T20:34:00.002-04:002009-05-13T20:35:03.266-04:00Complex Requirements<a href="http://1.bp.blogspot.com/_2ZvMWcMIvD4/SgtnJLl_WUI/AAAAAAAAABE/FFwjc_AkSHE/s1600-h/Complex.jpg"><img id="BLOGGER_PHOTO_ID_5335471590979098946" style="FLOAT: left; MARGIN: 0px 10px 10px 0px; WIDTH: 320px; CURSOR: hand; HEIGHT: 319px" alt="" src="http://1.bp.blogspot.com/_2ZvMWcMIvD4/SgtnJLl_WUI/AAAAAAAAABE/FFwjc_AkSHE/s320/Complex.jpg" border="0" /></a>I recently checked out an article by Scott Ambler in Doctor <span class="blsp-spelling-error" id="SPELLING_ERROR_0">Dobbs</span> magazine. And I paid attention because his name sounded familiar from the past. I think he may have written about C++ in the past. From his bio, he is into process and agile methods. His article affirmed that requirements are complex, but applying Agile to them can help.<br /><br />We do not want a heavy level of documentation for requirements. Take an example from Scrum. They have something called a backlog which prioritizes requirements. This is a good methodology to manage simple requirements.<br /><br />You would think that complex requirements could be broken down. The problem is that if you do break down the requirements, the smaller pieces might not make sense by themselves without the context. You need to try to group the smaller sets of requirements in a smart fashion for this to work.<br /><br />Just like traditional object oriented analysis and design, you should break the requirements into use cases and scenarios. The use cases should be simple. There should also be very little documentation.Software Processhttp://www.blogger.com/profile/17308836734575362181noreply@blogger.com0tag:blogger.com,1999:blog-3097690439527455604.post-5510194715129438582009-03-05T14:39:00.002-05:002009-03-05T14:44:48.253-05:00Agile Evangelism<img id="BLOGGER_PHOTO_ID_5309790758104508898" style="FLOAT: left; MARGIN: 0px 10px 10px 0px; WIDTH: 320px; CURSOR: hand; HEIGHT: 317px" alt="" src="http://2.bp.blogspot.com/_2ZvMWcMIvD4/SbAqj-B4qeI/AAAAAAAAAA8/kFk5yHK0g3w/s320/Wendy.jpg" border="0" />I read an editorial lately about the relentless evangelism by the Agile tribe. Their attack on traditional waterfall methods is <span class="blsp-spelling-corrected" id="SPELLING_ERROR_0">vicious</span>. Sure many software project fails. But don't Agile projects fail as well? I doubt anybody is keeping track of this. I put a picture of Wendy <span class="blsp-spelling-error" id="SPELLING_ERROR_1">Friedlander</span> up for your amusement. She is one of the agile evangelists out there.<br /><br />Software projects worked before Agile came around. and they will continue to work after Agile has come and gone. So what is there to truly tie any success to the Agile methodology? As you can tell, I have a lot more questions than answers here.<br /><br />I am big on unit testing myself. That does not mean I advocate Agile methods. Unit testing itself is not a new technique created by the Agile movement. I will say that I like some ideas behind Test Driven Development (TDD).<br /><br />One of the blogs I faithfully read is Joel on Software (<span class="blsp-spelling-error" id="SPELLING_ERROR_2">Jos</span>). Most of the good stuff comes from comments posted by <span class="blsp-spelling-error" id="SPELLING_ERROR_3">JoS</span> readers. It is funny that Joel has blasted Agile for the main part. He says that you cannot accomplish much when you are writing endless unit tests. And if you don't know Joel, he is big on getting things done.Software Processhttp://www.blogger.com/profile/17308836734575362181noreply@blogger.com0tag:blogger.com,1999:blog-3097690439527455604.post-5553507183623287302008-11-17T21:42:00.001-05:002008-11-17T21:43:52.155-05:00Life at Google<a href="http://4.bp.blogspot.com/_2ZvMWcMIvD4/SSIryDwrEKI/AAAAAAAAAA0/gMrrZbB8rZY/s1600-h/Goog.jpg"><img id="BLOGGER_PHOTO_ID_5269822652979810466" style="FLOAT: left; MARGIN: 0px 10px 10px 0px; WIDTH: 320px; CURSOR: hand; HEIGHT: 241px" alt="" src="http://4.bp.blogspot.com/_2ZvMWcMIvD4/SSIryDwrEKI/AAAAAAAAAA0/gMrrZbB8rZY/s320/Goog.jpg" border="0" /></a>One of my favorite <span class="blsp-spelling-error" id="SPELLING_ERROR_0">bloggers</span> is Steve <span class="blsp-spelling-error" id="SPELLING_ERROR_1">Yegge</span>. He initially worked at Amazon. Now he is with Google. A few years back he wrote a post bashing Agile. That was not the real story. His advice for developers is to work for a company like Google instead of using Agile methods. This was a firsthand look into the life of Google developers. Yes I have heard about the 20 percent side project work at Google. But Steve really gets down to the details of what it is like to work there. With the falling stock price, Google may not be the optimal location for the best developers now. But it is still interesting to take a look at a day in the life of the Google developer.<br /><br />There are managers at Google. However they all write code. Furthermore they do not tell developers what to work on. Developers are self organizing. There are incentives to be part of a successful project. You have the potential to make giant bonuses and get stock options galore. This often translates into developers who feel like they owe Google. However there is also a drive to gain the respect of your peers at Google. Your peers also happen to be very smart if you work at Google.<br /><br />Google has some environmental differences than your average development shop. It is very quiet there. You get free meals. Meetings are scheduled for the middle of the day so you don’t have to come in too early. It is kind of like a <span class="blsp-spelling-corrected" id="SPELLING_ERROR_2">start up</span> mentality. Google offers a lot of training. Some of it is formal. Most of it is not. There are apparently technical presentations going on all the time at Google.<br /><br />On the development side, Google takes software engineering very seriously. There is a lot of emphasis placed on unit testing, design documentation, and code reviews. Some people just build developer tools at Google.<br /><br />As usual, Steve generated a lot of comments from readers. Many people state that Google is unique in that it does not have paying customers who need software developed and shipped. Also Google has a lot of smart people. Google projects are financial failures. They just get a lot of money from ads in search. Google has deep pockets. If the Google model is so good, why does not everybody use it?Software Processhttp://www.blogger.com/profile/17308836734575362181noreply@blogger.com0tag:blogger.com,1999:blog-3097690439527455604.post-85608635215017594332008-09-10T20:39:00.002-04:002008-09-10T20:45:07.809-04:00Capturing Requirements<a href="http://3.bp.blogspot.com/_2ZvMWcMIvD4/SMho45QtsgI/AAAAAAAAAAs/L6yAwGBzFrE/s1600-h/Write.gif"><img id="BLOGGER_PHOTO_ID_5244557092725830146" style="FLOAT: left; MARGIN: 0px 10px 10px 0px; CURSOR: hand" alt="" src="http://3.bp.blogspot.com/_2ZvMWcMIvD4/SMho45QtsgI/AAAAAAAAAAs/L6yAwGBzFrE/s320/Write.gif" border="0" /></a>I have written about a <a href="http://susops.blogspot.com/2008/09/coupon-savings.html">requirements gathering</a> meeting with out client in my <a href="http://susops.blogspot.com/">Software Maintenance</a> blog. So I will not repeat a lot of the details here. However I did want to call to attention some details that might affect the success of our work.<br /><br />The customer knows what the want business wise. However they do not know how to design or code the changes. That is why they have us. I keep hearing the customer try to understand the existing system, and design changes so their needs would be met. I continually attempt to focus our client into explaining to us what they business needs are. They are, after all, experts in their business.<br /><br />Our customer has a lot of ideas about business needs in their head. At the meeting I attended, they were able to articulate many of those needs. This meeting was a precursor to the client formally requesting that we make a lot of changes to their software. I could not stress enough that they needed to write down the business rules for us. Unfortunately our requirements analysis team does not understand the customer yet. The requirements people are new. So we need the customer to write down their business needs. That is the only way we will have a chance to get the job right. The needs are extensive enough that just telling a few folks from our company about them will not do.Software Processhttp://www.blogger.com/profile/17308836734575362181noreply@blogger.com0tag:blogger.com,1999:blog-3097690439527455604.post-49785832334127792502008-07-29T21:06:00.012-04:002008-07-30T01:42:20.836-04:00Code Review Redux<a href="http://bp3.blogger.com/_2ZvMWcMIvD4/SI_7U9Ynq2I/AAAAAAAAAAk/Pwa5pW3h548/s1600-h/AlexMartelli.jpg"><img id="BLOGGER_PHOTO_ID_5228674029894216546" style="FLOAT: left; MARGIN: 0px 10px 10px 0px; CURSOR: hand" alt="" src="http://bp3.blogger.com/_2ZvMWcMIvD4/SI_7U9Ynq2I/AAAAAAAAAAk/Pwa5pW3h548/s320/AlexMartelli.jpg" border="0" /></a>James Turner recently interviewed Alex <span class="blsp-spelling-error" id="SPELLING_ERROR_0">Martelli</span> at the <span class="blsp-spelling-error" id="SPELLING_ERROR_1">O'Reilly</span> Open Source Convention (<span class="blsp-spelling-error" id="SPELLING_ERROR_2">OSCON</span>). Alex is well known in the Python community. He is also a tech lead at Google. The subject of the interview was Code Reviews, which you know I have an interest in. So I thought I would go over and comment on the Alex's points.<br /><br /><p>The purpose of a code review is to catch errors. However a byproduct of reviews is also for developers to learn. Some things a code review can accomplish are to fix mistakes, ensure a uniform style of coding, check for good variable names, and factor our common functionality.</p><p></p><p>Obviously there are some shortcomings to the current method of conducting code reviews. Junior developers get intimidated by senior ones, especially when the old salts have big egos. In other scenarios the developers say nothing during a review. They have the attitude that they want to get out as soon as possible, so they state that everything looks fine. At the other end of the extreme are developers who nitpick every little thing they can with the code reviewed. Finally there is a lack of good tools to help code reviews.</p><p></p><p>Alex had some advice on how to improve the code review situation. He recommends that reviewers ask questions instead of criticize the developer. All people involved with code review should treat the activity as a way to learn. It helps immensely if the developers are <span class="blsp-spelling-corrected" id="SPELLING_ERROR_3">humble</span> and have integrity. And using software to assist the process is valuable. A tool which allows a reviewer to read the code, click points of interest, enter comments, and respond to comments is invaluable.</p><p></p><p>Alex also provided some examples of code review from the commercial community. Apaches uses an empty bug list as the signal to enter code review. Test Driven Development (TDD) seems to be taking the place of code review at many commercial development firms. However such techniques will not find some subset of problems such as race conditions and/or deadlocks. Pair programming also replaces code review in other locations. But after two developers have paired up for a while, it becomes hard to conduct independent review.</p><p></p><p>Prior to the interview, I had never heard of Alex <span class="blsp-spelling-error" id="SPELLING_ERROR_4">Martelli</span>. However I decided to listen to the video interview since he worked for Google. And I had respect for him having heard the insight he provided on code reviews. In closing he had mentioned the use of defect injection, where defects are randomly introduced into a system. The code review process is then judged by how many such defects are caught. Perhaps I shall try to introduce this technique into our own software development team. It might be a starting point to fix the movement away from code reviews on my project. I will let you know how that goes.</p>Software Processhttp://www.blogger.com/profile/17308836734575362181noreply@blogger.com0tag:blogger.com,1999:blog-3097690439527455604.post-76053392291898361472008-07-25T23:12:00.002-04:002008-07-25T23:13:40.634-04:00Document Your Work<a href="http://bp0.blogger.com/_2ZvMWcMIvD4/SIqWMgRmCSI/AAAAAAAAAAU/sYkQakSKZyc/s1600-h/Document.jpg"><img id="BLOGGER_PHOTO_ID_5227155459083208994" style="FLOAT: left; MARGIN: 0px 10px 10px 0px; CURSOR: hand" alt="" src="http://bp0.blogger.com/_2ZvMWcMIvD4/SIqWMgRmCSI/AAAAAAAAAAU/sYkQakSKZyc/s320/Document.jpg" border="0" /></a>We have an oral process on my project that you must document your progress on trouble tickets in our bug tracking system. Recently a lot of the development staff have not been following the rule. This causes a lot of overhead, especially for members of the Help Desk. It is a pain to have to document your every step in a defect tracking system. When you are on the hunt for the source of a problem, you want to keep focused on the problem at hand. Documenting progress may slow you down and cause you to lose your focus.<br /><br />There are some benefits to documenting your progress. It prevents other people from having to call you up or e-mail you to find your status. This prevention has a rippling effect. If the customers can view your progress, they do not have to talk to people who will only talk to other people to get to the answer. Sometimes just the evidence that somebody is making updates helps calm a customer that is encountering a lot of problems.<br /><br />I work on a lot of trouble tickets. And I like the work. Personally I make sure I log my progress in frequent intervals. I make sure I write my findings for a broad audience. However I also add enough detail so that at any point, another developer can come in, read my comments, and continue the research. This has made redistribution of work very easy for the people I work for and with.<br /><br />Another incentive for writing down your progress in the defect tracking system is that it can lead to less meetings. That means less meetings amongst developers. It also means less meetings with the customer. When somebody is not confident that progress is being made on trouble tickets, meetings are called to order. This is a vicious cycle. The meetings take up time and prevent you from actually fixing the bugs. Or they make you stay late when there are no meetings to find out what the problems are.<br /><br />Right now our help desk has to attend trouble ticket meetings. The help desk personnel are trying to rally developers to enter their updates in the bug tracking system. I think I will join them and spread the word. Document your work already. You will only be helping yourself.Software Processhttp://www.blogger.com/profile/17308836734575362181noreply@blogger.com0tag:blogger.com,1999:blog-3097690439527455604.post-35449929827133479702008-07-23T17:53:00.002-04:002008-07-23T17:54:48.863-04:00Code Review<a href="http://bp3.blogger.com/_2ZvMWcMIvD4/SIeoiCTaynI/AAAAAAAAAAM/vqqG7uRjMTM/s1600-h/Review.gif"><img id="BLOGGER_PHOTO_ID_5226331195274087026" style="FLOAT: left; MARGIN: 0px 10px 10px 0px; CURSOR: hand" alt="" src="http://bp3.blogger.com/_2ZvMWcMIvD4/SIeoiCTaynI/AAAAAAAAAAM/vqqG7uRjMTM/s320/Review.gif" border="0" /></a>The Little Tutorials web site has a recent entry that takes a <span class="blsp-spelling-error" id="SPELLING_ERROR_0">contrarian</span> point of view for Code Reviews. The author concludes that they are mostly a waste of time. He contends that people are already busy enough, and that forced code reviews frequently degrade to something unproductive. The author does make some recommendations if you absolutely must conduct code reviews. You should run your code through the debugger first before presenting it for peer review. I share some of the sentiments from this article. But I also have some code review success stories that sway me a different way in the end.<br /><br />I work on a rather complex business application suite at work. We have, for the most part, subjected maintenance changes to the code to code reviews. This has indeed come at a high development cost. But the reviews we perform are not your normal type where a lot of developers get together in a room to go over the code. Ours are conducted independently by one other member of our <span class="blsp-spelling-error" id="SPELLING_ERROR_1">dev</span> team. The author puts together documentation that show all the code changes. The author also produces a unit test plan. These documents are reviewed. Comments are entered on a standard peer review sheet. The result is sent back to the author via email.<br /><br />The surprising fact is that I have found our code review to be very useful. But the effect may not be as you would expect. Just the fact that we must prepare for a code review influences the quality of the work that is performed. If you know that you must document your code changes, you become more aware of your changes as a whole. And you must thoroughly understand what you are doing. Otherwise it will come up at review time since the reviewer must be able to also understand the changes based on the documentation you product. This works the same way with development testing. Simply the act of having to write down a unit test plan forces the developer to actually think about, plan, execute, and write down the steps steps and scenarios.<br /><br />Our code reviews occasionally also produce the positive effect that you might expect a normal review to do. Some developer who did not understand the true requirement for the change gets enlightened. A developer gets encouraged to add more comments to promote maintainability of the code. A reviewed finds alternate paths that needs to be unit tested before the fix gets shipped.Software Processhttp://www.blogger.com/profile/17308836734575362181noreply@blogger.com0tag:blogger.com,1999:blog-3097690439527455604.post-157493570426826672008-07-15T16:37:00.000-04:002008-07-15T16:38:29.624-04:00Exceptions to the RuleI recently read a blog post by <span class="blsp-spelling-error" id="SPELLING_ERROR_0">Chadran</span> Nair emphasizing the occasional need to override the existing process when circumstances dictate it. In fact, the author went on to say that companies which understand and implement this policy get his business. I wonder whether this is applicable in the software development realm. I do not think processes are to be taken lightly. And the breaking of normal process should also be a serious and seldom used resort.<br /><br />We had a process to compute statistics for how well our software did each year. A couple years ago our customer requested the process be followed to produce input for an important report. This task came to me. So I followed the process and got some unusual stats. They were way too big. I could not imagine how we were coming up with the numbers I was getting. The process dictated that I follow the rules and produce the results.<br /><br />I inquired to my manager whether I should blindly follow the process and publish the wild results. Like most managers at the time, my manager decided to protect himself and told me to work it out with our customer. Now I had a good relationship with most members in the customer community. So I went to the top technical man in our customer organization. I shared my concerns with him. He definitely told me to research the numbers and find out why the results were off the charts. I was able to skirt by the normal process and found a data point that was messing up all the statistics. In the end, I was told to report a set of numbers that blindly followed the process, as well as another set which discounted the wild data point.<br /><br />Perhaps this example illustrates the right way to decide when to bypass the normal processes. It should not be done in isolation. And it should be done sparingly. You should get management (or customer) buy in first. But there comes a time when reason needs to prevail. Once in a while you cannot be a drone and follow the documented process when it is going to lead you astray. Yes you can defend yourself by stating you followed the process. However as the old saying goes, rules are meant to be broken.Software Processhttp://www.blogger.com/profile/17308836734575362181noreply@blogger.com0tag:blogger.com,1999:blog-3097690439527455604.post-84219822112547029842008-07-08T06:53:00.001-04:002008-07-08T06:53:50.230-04:00Change ControlOur system has a huge database of information that it stores. Some database tables have 100 million plus rows. And there are hundreds of database tables used. Development makes changes to the schema throughout the year. These changes need to be tightly controlled to prevent chaos from spreading.<br /><br />In the past, the <span class="blsp-spelling-error" id="SPELLING_ERROR_0">DBA</span> team implemented a process to control all changes to the database. Requests would be made for the changes. All the teams would approve the changes. The <span class="blsp-spelling-error" id="SPELLING_ERROR_1">DBA</span> team would promote the changes to development, test, and then production. Changes were grouped into releases to minimize the work to implement the changes. This arrangement worked well most of the time.<br /><br />Somewhere along the line, a team of consultants came in to help re-architect the system. Unfortunately this re-architecture was a complete disaster. The customer eventually pulled the plug on the new system. Development was tasked with restoring the old system. Through some miracle the consult team remained on the project. And now they are working to take over the change management for the database.<br /><br />Previously the <span class="blsp-spelling-error" id="SPELLING_ERROR_2">DBA</span> team had a good rapport with the rest of the development team. So the database change management ran smoothly. Now there is a fight for the control. There was no clear edict that appointed the consultants as the maintainers of the database. In fact, they do not actually implement any changes. The <span class="blsp-spelling-error" id="SPELLING_ERROR_3">DBA</span> Team does that. The result is that there is not a clear direction on how to initiate changes for the database.<br /><br />The management team is aware of the current situation. And they have their own objectives to achieve on this project. At about this time every year, we need to start making monthly changes to the database in preparation for next year’s release. The database change process needs to be ironed out and the control issues resolved before we can perform this task properly. It seems that politics is getting in the way of technical progress once again.Software Processhttp://www.blogger.com/profile/17308836734575362181noreply@blogger.com0tag:blogger.com,1999:blog-3097690439527455604.post-806012914924724942008-07-08T06:52:00.001-04:002008-07-08T06:52:57.719-04:00DocumentationOur team currently has a task to update our design documentation based on the changes we made when migrating to a new tool set. The amount of documentation change is small. Our goal was to ensure that functionality remained unchanged during our tools upgrade. We only had to modify one small part of report invocation. Unfortunately we found out that nobody knew the process to perform the update.<br /><br />Part of the confusion stems from the fact that the software maintenance is being performed by a new company that recently won the contract. In addition, the customer decided to become more involved with the documentation management. This just adds complexity to an already complicated task. Previously the development team used to make design changes in Rational Rose. We would use Rational <span class="blsp-spelling-error" id="SPELLING_ERROR_0">SoDA</span> to extract design information from Rose to produce the design document. But <span class="blsp-spelling-error" id="SPELLING_ERROR_1">SoDA</span> has some problems which we were unable to resolve. So the process changed to involve <span class="blsp-spelling-error" id="SPELLING_ERROR_2">SoDA</span> generation with some additional markups by hand. Things got hectic when the previous company that did maintenance was wrapping up their contract. The <span class="blsp-spelling-error" id="SPELLING_ERROR_3">SoDA</span> generation was dropped and a Microsoft Word document was edited exclusively to incorporate design changes.<br /><br />Currently one individual in our client organization is tasked with managing the latest version of all documents. This includes our large design documents. One would think that development could go to this individual to get the latest version of the design doc for update. But we have been cautioned that this individual sometimes gives out the wrong version of the document. Furthermore I do not think any of the procedures are documented. And if they are, the development team does not know about them. I sense a severe weakness in the maturity of the process here. This will most likely come out if we were to be audited.<br /><br />I think somebody is working on these problems. This impacts the day to day work when a developer needs to document some design changes. In general, coders do not like to perform documentation tasks. So nobody is sweating this problem just yet. However we are embarking on the design for a number of new features requested by the client. Therefore things are about to get very hectic real quick in our documentation world. Good luck to us.Software Processhttp://www.blogger.com/profile/17308836734575362181noreply@blogger.com0tag:blogger.com,1999:blog-3097690439527455604.post-19206166660602540512008-06-27T23:02:00.000-04:002008-06-27T23:03:23.881-04:00Duplicate CodeWe have a suite of applications that we maintain for our customer. Each application has its own build script. This is pretty good since there is a code controlled way to generate a release. Any developer should be able to log into a configuration controlled machine and run the build script. Our project takes this further in that no programmers actually run the build. A separate configuration management team runs the build.<br /><br />Recently we upgraded the tools we use for development. This has created the need to modify our build scripts to use the new tools to build the software. The script modification task has been distributed among a couple developers. It is strange that a number of the developers have asked me where exactly the build script code is located. These guys are not slackers. They did the work and searched our source code control system. The problem is that they found 3 separate copies of the build script code checked in.<br /><br />Off the top of my head I did not know which of the 3 versions was the real build script code used for Production. Maybe this is written down somewhere. But it is definitely not documented in any well known location. I gave the developers my best guess as to where the build script code would be. But this has identified some break downs in our build process. The same code should not be duplicated in multiple locations within our source control system. And more importantly, the location of the code should be documented somewhere along with other details about the build process.<br /><br />I truly do not know whether we will be correcting these process problems any time soon. It appears we are now in a rush to get the build scripts modified and working with the new tools. There is a schedule to keep that is critical. This reminds me of some similar problems in the actual code base of the applications. We have a number of functions that are duplicated in our code base. When the function is needed, one of the copies is called at run time. You can imagine the maintenance problems this will incur. If you fix a bug in one of the location, you need to know that the code is duplicated in many places, and track down and fix the code there as well. A lot of times this problem was created due to schedule constraints.Software Processhttp://www.blogger.com/profile/17308836734575362181noreply@blogger.com0tag:blogger.com,1999:blog-3097690439527455604.post-82717417286974486162008-04-16T10:19:00.003-04:002008-04-16T11:34:12.474-04:00Releasing Software FixesMy day job consists of performing software maintenance on a suite of software applications. That means I fix bugs. The customer is a government organization that we usually do not talk about. We have process controls to ensure that any changes to the software are controlled and managed. Today I thought I would go over the process of releasing bug fixes to give you an idea of what all is involved at my work.<br /><br />Let's suppose that the customer has opened a trouble ticket on a problem in our software. To start with, our Help Desk team enters the information in our bug tracking tool. Then the Help Desk assigns the ticket to the team lead of one of our teams. The team lead often does a little analysis on the problem. Then the ticket is assigned to a developer on their team. These days I fill the team lead role for the application developers. But I am also a developer myself (sometimes I assign problems to myself).<br /><br />While you are working on a trouble ticket, you are to enter your progress in our bug tracking tool. This allows the Help Desk to provide status to our customers. Let us suppose that I get lucky enough to figure out the problem and determine the code changes needed to correct it. Here are the steps I would take to get the software fix to our users:<br /><ul><li>Document all code changes in a Microsoft Word document</li><li>Generate a unit test plan to verify the fix</li><li>Obtain a peer review of the code changes and unit tests</li><li>Check the code changes into source code control</li><li>Request a release number from the configuration management team</li><li>Check the release number information into source control config files</li><li>Label all files that are to be included in the release</li><li>Generate the documentation that accompanies the release</li><li>Submit a request ticket to the configuration management team to build the software</li><li>Review the configuration management team work for technical problems</li><li>Forward the software release to the internal test team</li></ul><p>This is the list of steps that are taken for every single software change that corrects a bug in our system. Things get more complicated when there are more than one defect corrected in any given release. Other complications include the fact that some of our code is in a C++ client application, while some other code is in database stored procedures. The above list does not take into account all the additional steps that occur after the application development team is done their part in the release.</p>Software Processhttp://www.blogger.com/profile/17308836734575362181noreply@blogger.com0tag:blogger.com,1999:blog-3097690439527455604.post-78260451747067191122008-03-22T19:35:00.003-04:002008-03-22T19:43:25.904-04:00Hacking SoftwareI run a small informal software business on the side of my day job. Recently I have been writing small programs which perform tasks that Internet hackers might do. The programs have been released on my <a href="http://black-of-hat.blogspot.com/">Black of Hat</a> blog. The interesting thing about this effort is that I have employed very little process in developing my own software.<br /><br />For example, my latest program "Crawl" is one that scans a web page and extracts the URL links. I thought about the problem for a couple days. Then I went directly into coding this morning and developed version 1 of the software. Late in the morning I built my release version of the application. Finally I posted this app on my blog.<br /><br />So far my application appears to be working. I have received no reports abouts bugs in it or it not working on different systems. There was not rigorous process applied to this application. I did do some thinking before coding. But I did not write down and specifications or designs. I really did not perform any formal testing on the application. And the software was developed in record time with not many problems.<br /><br />In <span class="blsp-spelling-corrected" id="SPELLING_ERROR_0">retrospect</span>, this may have been a special case of software development. The application was not very complex or large in scope. The development team was just one person. And the developer (me) was the same of the person who determined the requirements. Things get more complicated and may require process overhead when they scale up. However for my side software business, the real question is whether I would gain any benefit it an added layer of formal process was imposed on top. I am still mulling this question over in my head.<br /><br />Perhaps I shall try adding a formal process to the next application I develop for my side business. I can track how much additional time I spend on the process part of development. And perhaps I can try to objectively rate the quality or amount of problems in a process controlled application release. The results are not some theory that will get documented in a paper. They will help decide how to run my own software business which I hope to succeed.Software Processhttp://www.blogger.com/profile/17308836734575362181noreply@blogger.com0tag:blogger.com,1999:blog-3097690439527455604.post-51386063616885568042008-03-13T00:46:00.003-04:002008-03-13T00:54:14.159-04:00DisorganizationMy company is trying to close out the project we are working on. This is due to the fact that we lost the contract to perform the work. There are a number of personnel still working on the project. If they are not busy on other tasks, they are supposed to be working on locating, organizing, and storing our documentation. I assigned our least busy developer to work full time on this task. I told him to go find out what needs to be done, and do it.<br /><br />In retrospect, this may not have been the best direction to give. But I did not want to get bogged down in the details as I have other priorities to attend to. There is an individual who is assigned to the documentation task. And there are many people working on the task. But after working for a while, we had a meeting where it became clear to me that this task was in a state of chaos.<br /><br />I directed a few key members to come meet me in my office. At this follow-on meeting, I declared that we needed to finalize and document a folder structure in which all our application team documents needed to go. The person who was in charge of the task said it was their responsibility to determine what needed to be done. I concurred. However I also pointed out that it was evident there was not method to the madness of the task. Later that afternoon, the task leader put together a document demonstrating a folder structure for organizing documents for each team. In addition, the actual goals of the task we finally articulated in writing.<br /><br />Sometimes you need to sit down, figure out what you need to do, and document the plan. This is required when the task at hand requires more than one person, and absolutely needed when it involves more than one team. One would think that tasks such as document collection and <span class="blsp-spelling-corrected" id="SPELLING_ERROR_0">organization</span> are nothing like rocket science. But without a plan, and more importantly a documented plan, you have all the ingredients for utter chaos in the workplace. Why must we have to relearn these lessons year after year?Software Processhttp://www.blogger.com/profile/17308836734575362181noreply@blogger.com0tag:blogger.com,1999:blog-3097690439527455604.post-82236421930534909572008-03-08T20:22:00.002-05:002008-03-08T20:27:47.908-05:00Document OrganizationLet's suppose that you have actually documented the main processes used. How does one access this information? When I first came to my current project, you got handed a huge three ring binder full of information. This was good in that I could read up without even having a computer or access yet. However this got to be a pain when the documents needed to by updated. And as staff transitioned through the project, the binder and documents seemed to get lost.<br /><br />More recently our application development team put together a small Microsoft Access database. This contained all kinds of information pertinent to our team. It was nice because you could search it based on keywords. Updated were easy. There was some trouble when two or more developers tried to update the same document at the same time. But this was an infrequent occurrence. Since this was a side project, the server where the database was hosted was not up 100% of the time. Still it was a useful project.<br /><br />Over the last year or two, the project has tried to organize and assemble all documentation in a single place. The goal was to have a common structure for each team. This was a monumental task. A couple people have tried to accomplish the task. They have failed. Now we have a full team working on this. I do not want to be negative, but I am not expecting much from this. The scope is large. And you usually don't get your best people assigned to a task like this. Such a shame.Software Processhttp://www.blogger.com/profile/17308836734575362181noreply@blogger.com0tag:blogger.com,1999:blog-3097690439527455604.post-73846209079089334532008-03-07T23:04:00.002-05:002008-03-07T23:11:37.171-05:00Process MattersMy team performs software maintenance on a legacy product. Basically we respond to software trouble tickets. These tickets are tracked closely by the customer. Our company's help desk team has been recently hounding us for "estimated resolution dates" for each ticket. So development has been figuring out, given what we know about the problems, what is the most likely date when we can produce a fix.<br /><br />Something did not seem right about the Help Desk stating they were being hounded by the customer for estimated resolution dates. So I decided to participate in the next meeting with the customer. It was here that I found out what the customer was looking for. They wanted to know what was the date that we could expect a user to be able to experience the fix and verify that the trouble ticket had been resolved.<br /><br />That about blew my mind. There is often a huge gap between the time we release a fix to our customer, and when the user actually experiences the fix. Our customer has a lengthy configuration management process. Our fixes go through their quality control first. Then somebody has to actually implement the fix. This can take time, especially when pushing workstation fixes out to the multitude of our users. There are also some processes which run once a week. Depending on the timing, users may not see the process run for up to a week.<br /><br />I was glad when our project manager also expressed concern about the definition of estimated resolution date. We had to <span class="blsp-spelling-corrected" id="SPELLING_ERROR_0">amend</span> the way we accounted for the days that make up this metric. It seems we had been shooting ourselves in the foot by using the wrong definition of this date. You need the right process in place. And that process needs to be continually monitored to ensure it is still relevant and correct. Otherwise you are potentially marching down the path to doom.Software Processhttp://www.blogger.com/profile/17308836734575362181noreply@blogger.com0tag:blogger.com,1999:blog-3097690439527455604.post-8551439188229209032008-01-15T01:09:00.000-05:002008-01-15T01:14:05.668-05:00Write It DownA basic tenet of a good software process is that it needs to be documented. This may seem obvious. But you would be surprised how many time the new guy comes in, and must spend a lot of time asking people how things are done. This does not mean a lot of detail needs to be written. And it does not have to be very formal. It just has to been accessible and practical.<br /><br />I have seen another problem with software process documentation. Often times there are huge documents that are supposed to document the software process. However when read, these documents are actually full of fluff that do not correspond to the actual <span class="blsp-spelling-corrected" id="SPELLING_ERROR_0">process</span> being used. This is worse than no documentation. It is lip service to the software process.<br /><br />In the past I have lead software teams. Some times I had implemented a hands-on approach where I oversaw all aspects of my team's work. In other cases I just wrote out a handful of well-documented process pages, and let the team follow them as I spent more time coding. Guess which option was more fun for me?Software Processhttp://www.blogger.com/profile/17308836734575362181noreply@blogger.com0tag:blogger.com,1999:blog-3097690439527455604.post-29301458371421998042008-01-12T00:37:00.000-05:002008-01-12T00:44:16.622-05:00Let it BeginI am a firm believer that a good software process can be a major component of development success. This becomes more important when you have a staff of mediocre programmers. Left to themselves you can get a software disaster very quickly. But a documented process can help guide the sheep to deliver in a predictable manner. I use the term sheep lightly. However software is a challenging profession. There are a lot of developers who are not here but are not entirely up to the challenge.<br /><br />Even outstanding developers can benefit from a well defined software process. This let's them concentrate on the more important tasks of development like solid architectural design. For the rest of the mundane tasks, they can just follow the process. Their minds will be freed up to continue to be productive on solving the truly hard problems.<br /><br />If instituting a good process was simple, more software project would be successful. But creating and enforcing high quality processes is in itself a significant task. The great thing about really good processes are that they are self-correcting (process improvement). Hard work up front can result in less headaches later. I plan to explore such processes in this blog.Software Processhttp://www.blogger.com/profile/17308836734575362181noreply@blogger.com0