Pragmatic Agile Development: How to develop fast & contain Technical Debt

By Rod Newing On January 6, 2012 At 1:24 pm

Mr Brown and Mr Black lived next door to each other. They both needed to build new houses for themselves to give more space for their families and a better living environment.

Mr Brown hired a group of waterfall builders. After a year he asked how they were getting on. They assured him that they had completed the family’s needs assessment and produced a detailed design. It would only take another three months of checking and validating before they could start work. Three months later Mr Brown got a job in another part of the country and his wife was pregnant with twins, so they needed a bigger house in another part of the country.

Mr Black hired some agile builders. The foundations were in within two weeks for his approval and every two weeks he reviewed progress and agreed what would be built in the next two weeks. The house was quickly completed and he moved in. Unfortunately the house sank on its foundations with the weight of the furniture and collapsed. “You screwed up by not digging deep enough foundations,” Mr Black cried. “No we didn’t,” they retorted. “It is entirely your fault because you signed off the foundations at the end of week two!”

A group of senior IT executives met over dinner in London to discuss a pragmatic approach to agile. One that can rapidly deliver working software while managing technical debt. These were their recommendations:

Technical debt

Technical debt will always exist and it is unrealistic to expect to reduce your legacy debt all at once. However, it is not inevitable for agile development to add to technical debt, rather it is a result of ignoring operational requirements. By itself, it is not a problem, as long as the organisation knows it is creating technical debt and understands the consequence of it on the customer experience.

Very few organisations measure it, or if they do, they measure it in the wrong area, such as stable legacy systems that haven’t changed for years and are not causing problems. They should be measuring the .NET or Java developments, which tend to accrue massive amounts of technical debt.

Service providers often know more about an organisation’s technical debt than they do. If technical debt is measured on a continuous basis, it becomes a quality improvement mechanism, as developers learn to make better choices.

Hybrid approach

In order to provide performance and resilience, a hybrid approach is needed. A project must start by drawing up an overall blueprint of the final system. This to define the operational requirements of an effective underlying platform. It covers the overall architecture, main component capabilities, scaling factors and interoperability standards. Some components may already exist and involvement of architects is absolutely critical. There is no shame in this waterfall-type initial phase, as long as it is not detailed.

An agile development approach is then used to deliver rules, interfaces and functional outcomes, including existing components. Infrastructure is delivered by providers, but in line with the sprint ‘heartbeat’ to make it work effectively. Every sprint must review progress on acceptance of the operational requirements.

Short sprints are good

Short sprints, typically two weeks, act like a large metronome or drumbeat for the IT organisation. Working against immovable deadlines encourages everybody to by hyper-successful, not chaotic. The idea is to ‘fail fast and recover quickly’. Errors in approach can only last two weeks and must be fixed in two weeks. The level of rework from a sprint is a lot less than that which is massively under-reported on large waterfall developments.

Culture

Success depends on changing the culture of the organisation. It starts with being really honest about the overall company culture and how agile the development can really be. Getting top-level support at the outset is essential.

Developers must become much more open and find a way to communicate that problems exist, so that they can be reworked. They must understand the risk of technical debt and be encouraged and incentivised to disclose it.

They must also understand that the single most important user or customer function is that the functionality is actually available and works efficiently when they want to use it. Use the phrase ‘operational requirements’ or ‘operational imperatives’, not ‘non-functional’, which automatically makes them lowest priority. They are not directly customer-facing, but they must be done for the functionality to work.

Similarly, the latest update in scrum, the first in ten years, has replaced the word ‘prioritisation’ with ‘ordered backlog’. Prioritisation focused the mind on the ‘sexy’ functionality, like spinning logos.

Switch something off

Software development projects offer an opportunity to replace outdated systems and remove their associated technical debt. This also gets rid of the hidebound thinking that caused the debt. It is generally faster, cheaper and easier than trying to retrofit new requirements onto old system that the organisation doesn’t fully understand.

People are deterred from replacing old systems by fear and risk. They also cite the cost of changing, such as tens of millions for a system that only costs £50,000 a year to run. However, they should consider the risk to the business, which could cost it £60 million.

Functionality can be migrated out of monolithic systems into common services on a phased basis. Replacing the eventual residue should be straightforward.

Skill up

There is a steep learning curve in agile development and a high failure rate is to be expected. The number of developers who really understand agile development is quite low. Hire the best people or consultants you can afford who really know what it is about.

- By Rod Newing

Add your comment

XHTML : You may use these tags : <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

This is a Gravatar-enabled website. To get your own globally-recognized avatar, please register at Gravatar.com




Comments

  1. Rod, your article is weakened by your flawed “building” analogy at the beginning. Software is not like building physical structures. It’s quite possible (and not really difficult) to upgrade the “foundations” as you go. Agile is NOT more likely to incur technical debt than waterfall.

    No matter what the project lifecycle, the avoidance (removal, really) of technical debt depends largely on the skills of the developers.

     — Reply
    • Lev

      George, please accept my apologies in advance for diametrically disagreeing with you. At least as applies to large, mission critical IT systems.

      Once you’ve built a foundation that’s integrated to adjacent systems, into the data structures, and is architected in a particular way, it is actually very difficult to redesign and to fix problems in the architecture. That said, it’s true that it’s not so difficult to go and fix bad coding practice that stinks up the whole codebase.

      Also, while Agile practices are actually very quality-focused and rigorous about refactoring, most software in IT that’s written under the Agile rubric seems to be rushed to market and never refactored. I don’t know that I would blame the quality of developers’ skills, but rather the distorted, dilbertesque process that they are put through.

      My 2 agile cents on this topic.

       — Reply
  2. Welcome George – thanks for pitching into the debate!

    Please excuse my tongue-in-cheek introduction. It was just an idea that went through my mind as the experts discussed the problems in the early stages of their agile developments.

    It was never my intention to suggest that agile development is more likely to incur technical debt than waterfall. I was merely trying to set the scene for the subsequent discussion, about the need to avoid the extremes and take a pragmatic view.

     — Reply