Geeks With Blogs
Mark Pearl

I live in a world of continuous integration servers, extensive unit, integration & regression tests and a whole bunch of other checks and balances to insure the quality of the systems we develop.

I live in this world because I know things can go wrong and I want to have a safety net to make sure I am not going to mess things up.

I didn’t always have these checks in place – I remember the cowboy days where we would click build, if it compiled we were good to go and would release to the world. They were scary days, but also if I look back at them, because I knew we had no safety net, I was always extra vigilant when releasing updates to solutions and had very few situations where we would make a major mistake or something would go wrong – mainly because we knew that it was so easy for things to go wrong.

My world has changed, for the better – however it can also put one in a false sense of security. Today I would like to warn those of you that live in the same world of me of being careful not to be in a false sense of security.

For the last few months we have been confidently churning out code on a solution, thinking that our CI Server would pick up any issues or breaking changes if our unit tests failed, or integration tests failed etc.

Usually this is the case, but someone somewhere made the mistake of not removing old files as part of the build process. Thus when someone else was tweaking the jobs on the build server, they pointed the unit tests job to a legacy copy of the unit tests dll instead of the correct file – resulting in our unit tests happily passing regardless of the actual state of the solution because the ci server was testing something 4 months old.

The only way we picked this up was I got suspicious that something was wrong when the unit tests were failing on my local machine, but passing on the CI server, resulting in me writing a failing test and pushing it through to the build server where it passed happily! Very worrying.

After a few hours of investigation we picked up the issue and rectified it, but the lesson has been learnt…

  1. Run tests on your local machine
  2. Run regular fire drills on the CI Server

I am becoming more and more of a believer in what I call fire drills of the system. Once every few weeks one runs through a series of diagnostic processes to ensure that the safety nets are working correctly. In this case a fire drill for the build server would be to pass through a failing test on purpose and verify that it failed.

So, have the safety nets, they are good – but question them once in a while to verify that they are working correctly, or the net result can be a lot of pain.


Posted on Thursday, January 5, 2012 8:19 AM Misc | Back to top

Comments on this post: Why my CI server gives me a false sense of security

# re: Why my CI server gives me a false sense of security
Requesting Gravatar...
At a previous company, we used to reimage and rebuild the build agents regularly (once a month, maybe once a week -- I forget). This ensures that there's no old detritus hanging around on the machines, and also forces you to think about dependencies you inadvertently add to the build. Of course, we had an automated rebuild script to do it. It would have been painful to do it by hand each time.
Left by Roger Lipscombe on Jan 05, 2012 1:18 PM

# re: Why my CI server gives me a false sense of security
Requesting Gravatar...
@Roger.... yes, I think it is those old detritus hanging around that has been our biggest headache... I think I might do what you suggest... thanks for the comment
Left by Mark Pearl on Jan 05, 2012 1:43 PM

# re: Why my CI server gives me a false sense of security
Requesting Gravatar...
@Roger, did you test the automated rebuild script? Uh uh, recursion ;-)
Left by Steve Martin on Jan 06, 2012 11:10 AM

Your comment:
 (will show your gravatar)

Copyright © MarkPearl | Powered by: