If you could deploy to prod every day, would you be more agile?

In my previous project we developed server-side software, that could only be deployed to production once a week. On Tuesday night you submit the binaries to Beta environment, and on Saturday they go to production, unless critical bugs are found during Beta testing. No binaries on Tuesday → no deployment on Saturday. Anything  that did not make the cut by Tuesday night would not go into the next release, end of story.

Interestingly, this lack of flexibility made our project more “agile”, because it created natural sprint boundaries. Extending the sprint by a day or two was not an option. Constantly adding new changes to the release candidate was not an option either. Deadline was king.

Some developers that just joined the project could not comprehend that this is for real, and there are virtually no exceptions. “But this is just a very small change” would not work. “But I was only an hour late” would not work either. If it was not tested and deployed to Beta by 5PM Tuesday, it is not going to production Saturday. I am not going to make people stay late, test your stuff and handle emergency tickets just because you procrastinated. Case closed.

I can’t really blame those developers; we are surrounded by ostensibly strict rules that are rarely or never enforced if the violation is minor. The vast majority of car drivers break the speed limit and roll at stop signs several times a day, but very small fraction actually gets punished. In fact, it is unsafe to follow all road rules to the letter, because people expect you to break them. Taking a pen or a box of staples from work for personal use would hardly raise any eyebrows, even though technically it is stealing. Even if discovered, it is unlikely to lead to anything bigger than a verbal warning. Stealing the same pen from a store could easily get you a criminal conviction with unpleasant long term consequences.

Going back to software development: my current project is more client oriented, with a CDN-like delivery system. So, the release schedule is much more flexible: theoretically you can whip up a new universally available version every hour or two. You may or may not call it “production”, but any client would be able to run it within minutes.

Counter-intuitively, this actually takes us farther away from agile development than weekly releases. The notion of sprint gets diluted. Everyone gives lip service to agile and Scrum, but making last minute updates to the release candidate is too easy and too tempting. Various excuses are brought forward to defend such behavior: business expects us to be flexible, we are not a software company, our software has limited distribution and extensive support, and the like. In reality, of course, this “flexibility” does not lead to faster delivery of features to the majority of clients. On the contrary, as the official final releases are delayed, most clients get their features later than expected.

Introducing non-negotiable artificial deadlines causes significant pushback. People refuse to believe that living by the book and voluntarily giving up ability to update the release candidate at any time is the good thing; that it is not only possible, but beneficial to the project. Since they have never seen it actually done, they think of it as a purely theoretical academic concept that has no place in real life. Ironically, inflexibility built into he system by “higher forces” makes it easier to eradicate procrastination and enforce true agile development.

4 Comments


  1. 1) We can deploy PostJobFreeWeb and PostJobFreeService ~5 minutes after new feature is committed (it takes about 5 minutes to build solution and run 1000+ autotests).
    That DOES makes us more agile.
    We actually do deploy several times per day.
    That means new features are hitting production faster, but more importantly, bugs are fixed much, much faster.

    2) What was the point of 4 days delay between Tuesday test deployment and Saturday production deployment?
    Couldn’t that 4 days work be replaced by autotests?

    Reply

    1. What does your planning process look like? Do you have a notion of sprints/versions?

      3 days Beta-testing was a religious dogma given to us from above. I think 1 day would be enough. We had over 1,000 unit and integration tests, but the coverage was not universal: I would not trust them blindly to release the software to production without a manual test.

      Also, when you integrate with external systems, testing against real-life counterparts may be difficult due to security and organizational concerns. You can test against mocks of course, but then you are testing against what you *think* those system would do; in reality they may behave differently.

      Reply

  2. > Do you have a notion of sprints/versions?

    No.
    We just have individual features that we want to implement or bugs that we want to fix.
    As soon as they are ready – we deploy.

    Bugs most of the time have priority over features.
    So we solve the most critical problems first (and immediately deploy them).
    If we have time – we work on less critical things (and deploy them when they are ready).

    > I would not trust them blindly to release the software to production without a manual test.

    It is NOT “blindly”.
    End users do manual testing for you as well.
    If your turnaround time is quick – you can rely on end users testing more.

    However if bug reoccurs – this is a clear sign that an automated test is in order here.

    > testing against real-life counterparts may be difficult due to security and organizational concerns

    Are you implying that manual testing is somehow easier in that difficult situations?
    Generally, if testing is hard – it should be automated.

    Reply

  3. In my experience “inflexibility built into the system” only forces 30-50% of developers (depends on the company) to deliver/test on schedule. The rest would just quietly release half-done features and/or fake manual test results.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *