Generative AI: get ready to multiply your Technical Debt
Or why you need to be mindful in adopting GenAI for development
The biggest cost in software development is software maintenance, and right now approaches to GenAI are almost exclusively focusing on the former and ignoring its impact on the latter. This is going to hit hard the more code that gets written or supported using GenAI. There is a very real chance that uncheck and unmanaged use of GenAI utterly balloons the support cost of applications
I’ve written before on why you need to think about testing with GenAI development.
But here lets look at why unchecked and uncontrolled use is a disaster and what you should be doing.
Maintenance is most of the costs
The first piece is to repeat a factual statement:
Maintenance Costs are bigger than development costs
Now it doesn’t matter whether its 3x the development cost, 1.5x the development cost or 9x the development cost, the point is that the maintenance part is what drives the TCO for a system, if you can get it down at 1.5x you’re doing great, if its 9x… well not good.
This means that building code for maintainability is incredibly important. If your code is super “clever” and therefore only brilliant people can maintain it, then you need brilliant people to maintain it, if however you are smart enough to write straight forward code that anyone can maintain, then you can get anyone to maintain it. Sometimes you have to write clever code, for instance a framework, so the majority of code can be straight forward. That means having a split in support, or you ended up with a framework that collapses, and everything goes badly wrong.
To do some maths here. If you hand code everything and make it super straightforwards and it costs you $1m to do that and then $1.5m to maintain it over the next 5 years then the TCO for that system is $2.5m.
If however you create a system that you get into production for $300k but it costs 9x to maintain it, then your TCO is $3m, a pretty significant increase that is going to only keep getting bigger the longer the system is there.
This was when people were writing the code, GenAI is about to make that get a whole lot worse.
Generate the code, maintain by hand, balloon the budget
Now here is the problem:
It is really easy to use GenAI to create complex code that even the developer doesn’t understand.
For instance if you are Python dev using OpenCV and need something that is lower memory footprint and maybe a bit faster, then you could get an LLM to create a C++ version of your code… code you would not understand. You could then run it and test it (after learning how to compile it), so now we have an application where the developer doesn’t actually understand the code, like checking in binaries rather than source code and expecting people to maintain the binary.
This code could be fabulous, in the sense that it works for the current challenge, but its ratio to maintainability is going to be significantly higher that 9 to 1. If the developer who created it doesn’t understand it, how on earth will you maintain it? It might have only cost you $50,000 to create the code but give it is now practically unmaintainable you could end up with millions in support costs.
Scale that across lots of applications where you cannot maintain them and suddenly you are blowing the entire budget you thought you’d saved, and an awful lot more. New applications being thrown into the landscape to patch issues, rather than fixes being applied, because people are not able to write the fixes.
Even if you could fix it, its still more expensive to fix the code
“Ah, but Steve, we can totally maintain that code, we have the skills and are willing to deploy them for support”
Awesome, so you’re using GenAI to create the code, but the great news is you understand the code, you have the skills and now you can manually fix the issues in the code.
So now you have, thanks to Generative AI, developers creating amount twice or three times the number of lines/function points/[complexity metric of your choice] which means you need a bigger team to support that increase in complexity, so you’ve again even in the best case skewed the costs towards maintenance yet again. If you’ve generated 1,000 lines of code and are then maintaining it as code, then you’re equivalent comparison for maintenance is how much it would have cost you to develop. So if it would have cost you $10k to develop that code, even though it only cost you $200 the maintenance cost is related to the $10k.
If you are maintaining the code then it doesn’t matter how little it cost to develop
How to control costs with Generative Development
To control costs you need to be much more explicit in how and where you are using Generative AI. Like with your career there will be places where GenAI hugely automates the task, some places it supports the task and some places it really won’t help.
So the first step within the design of an application (remember design?) is to clearly identify the three classes of Generative AI development you are going to do
- Fully Generative AI — these are parts of the application where you will use 100% Generative AI
- Generative Assisted — These are parts of the application where you want to use GenAI to augment the developers
- Pure Code — These are parts of your application where you don’t want to use GenAI
I’ll go into approaches on these in more detail in subsequent posts, but lets start with the headlines
Fully Generative — Treat it like a compiler
So if you are doing fully generative areas, don’t say “just this function” make it a whole code file, so the entire thing can be dumped and recreated, the maintenance needs to be of the prompt not the code. That means you need to create prompt files where you detail the prompt you used, why you used it, and any elements you tried that produced bad results.
In this world every generative file has an associated “.prompt” file that details out the prompt. This is just the prompt, its the changelog for the prompt. Some areas (testing, integration, migration) are absolutely going to go this way.
Generative Assisted — be clear on the assistance
If you are allowing people to use Generative AI to assist in their coding, then you need to get them to mark out the sections where generative helped and comment where they edited the code from what was generated. This means there should be a clear marker saying “This bit was generated from this prompt” and if edited the lines should identify what the change was. Top tip here is to use your version control system between the generative version and the final version so you’ve got a full record of those changes.
This means that when someone comes to fix a section they can potentially use prompt modification to do the edit, and then reapply the specific change that was done, or update the edit if that is quicker. I’d expect IDEs in future to start helping mark out the generative sections and their edits, plus the human code holding it all together.
Pure Code — human all the way
If you are building something new, or don’t want to risk any IP issues, particularly if you are building a new product and want the IP to be absolutely clearly your own. Then there will be code you want to create yourself in a 100% human form. Mark this clearly and audit clearly to make sure it doesn’t get polluted. In maintenance you need to review whether the restrictions still apply, and if they do, then manually maintain it, if they don’t then decide which of the two categories above it moves into.
Fail to plan, generate your fail
Simply put you can’t afford to “experiment” on development with GenAI without a clear plan, a plan that carefully categorizes work and a plan that includes investment in testing industrialization (an effort that will more than pay itself back).
To not do so is to ensure that you are left with a hideous mountain of tech debt, the likes of which no mere human could create.
