What is Tech Debt?
On a software engineering project, Tech Debt is any bad issue that would not be in the project if you started today and had CI (Continuous Integration) set up properly to block these issues. However, even with these preventative CI measures in place, Tech Debt always accrues as time passes. Tech Debt can waste 23% of a developer’s time1, so it’s something every company should make a priority.
So, what is Tech Debt specifically? Below is a typical list of items that would be considered Tech Debt and the negative consequences associated with each type:
Warnings
Every platform generally has tools that do some sort of static analysis on the codebase and will flag warnings when it detects them. If these warnings are ignored, the developer may miss critical ones.
For example, Android has resource strings that have parameter substitution. If one language has an incorrect number of parameters, it will warn you. If you miss this warning, and this bug gets into the field, the app will crash for that language.
Baselines
Typically, legacy projects (ones that haven’t had prevention systems added as part of their CI) take this approach to set a baseline when those prevention systems are added. Baselines list all the issues that are ignored while new issues are prevented by CI. However, SDK and Platform library upgrades can increase the tech debt if the person performing the upgrades doesn’t have time to address any changes needed by the library update and just regenerates the baselines.
Deprecated APIs
APIs that should not be used are usually marked as “deprecated” (aka, these are APIs that should not be used or there are replacement APIs that are better) for one or two cycles of upgrades, but are then removed. A removed API is why we cannot perform an upgrade of a library. Also, if you continually update to the latest libraries, the other end of the API may be disabled eventually, which can force a small fire drill and unplanned upgrade.
Code Coverage
Code coverage is a measurement of how well unit tests exercise the existing code paths. Lack of code coverage increases the probability of bugs when developers change existing code; they may inadvertently break the logic in other parts of the code which are missing tests.
Old Feature Toggles
Feature toggles are generally used for controlled feature launches, A/B testing, and emergency disabling of a feature. In the first two cases, if a feature toggle is forgotten (i.e. made the default), there is unnecessary extra code to maintain; developers may not know they have to make changes to the non-default version of the code. The only valid feature toggles that are long lived in any app should be the emergency disables.
Who Owns Tech Debt?
Now that we understand what Tech Debt is, we can ask “Who should fix Tech Debt?” The responsibility typically falls to:
- Individuals obsessive about a clean code base - these people hate having Tech Debt in the app and strive to fix it. This usually occurs in smaller companies where everyone pitches in to do what’s needed.
- “Platform” team - this is a team dedicated to keeping apps and the app platform updated, so cleaning up Tech Debt and upgrading libraries falls into this team’s jurisdiction. You find this type of team at larger companies.
- Feature/Pod/Squad teams - ideally, the folks working in the code (the feature/pod/squad teams) take care of Tech Debt in their code area. This can happen in small and large companies but is a team effort to keep the codebase clean.
So, which is best? Ideally, you have a mixture of (2) and (3) with some (1) in each. On large projects, an individual can’t reduce Tech Debt effectively without devoting a significant amount of their time doing, what I call, “app janitor” work. Teams should have a strong sense of ownership over their code so they keep it clean (Google libraries and build tools are generally the domain of the Platform team).
How Do You Measure Tech Debt?
Sonarqube is one of the most common tools used to measure Tech Debt in companies. It ties into data from numerous static code analysis sources and there are various plugins to upload data to it (e.g. Jacoco for JVM-based languages). It’s most mature for JVM based languages and will even run static analysis tools like Checkstyle, Findbugs, and PMD on Java code. Some of the shortfalls include support for native app languages - Kotlin support sometimes lags behind new language features and Swift code coverage takes extremely long to run.
Despite Sonarqube’s weaknesses, it is useful because of its API. The Sonarqube API lets you get issues and their severity as well as code coverage by “module” (where a module is generally defined as a folder with code in it). There is a dashboard that shows code coverage by module as well.
Unfortunately, Sonarqube doesn’t show/measure all the different types of Tech Debt we mentioned. In order to cover those, you have to write custom scripts to generate issue counts of other issues that might be important to you: e.g. Android lint counts per module, deprecations for various SDKs, old library versions, etc.
For this information to be actionable, it still needs to be assigned to appropriate developers. This is where we’ve spent time mapping code ownership. As you can imagine, modules need to be associated with feature teams before issues can be fixed. This is done using a two step tree:
Modules are always associated with an app feature. Features are functional sections of code (e.g. onboarding, login, cart, checkout, notifications, deeplinks). Each feature can be broken down into modules, such as cart-ui, cart-repository, cart-api, cart-network. Features don’t change with re-organizations. However, feature ownership can change with re-organizations.
How Do You Show Tech Debt?
This Dilbert cartoon is almost a perfect illustration of why you need to make Tech Debt visible so that it can get fixed. It’s not just the developers that need to know that it’s there. Other stakeholders must understand that this is an ongoing issue of any software development project, no matter the age. Tech Debt needs to be kept in check with continuous maintenance or the development process will slow down or hit road bumps.
So, who are the stakeholders? Project or Product Managers are the first group because they have to know when fixing Tech Debt might cause slowdowns on a project. Senior management is another, because they have to allow fixing Tech Debt to be included in the priorities for a project. Also, development teams are a stakeholder because not all developers are aware that there is Tech Debt.
Reports to these stakeholders should show:
- Ownership: which team has the highest Tech Debt so they can prioritize fixing it
- Baselines: a snapshot has to be made so that progress can be tracked against it; this can be quarterly or annually
- Graphs of progress and history
Below is an example report that can be shared with non-developer stakeholders (note the color coding that shows teams with “worse” Code Quality Scores than expected; team names are blurred out for privac):
And here is an example of a progress tracking report that can be shared with non-developer stakeholders who will want to see this metric improve:
How Do You Measure Effectiveness of Fixing Tech Debt?
Baselines and historical graphs are the obvious ways to measure progress. You can track reductions in bugs, and Sonarqube issues for starters. However, some issues are of a higher severity than others and Sonarqube tries to group these together for you.
But how do you get a singular measure for a team or codebase that is easy for all stakeholders to understand and track? That’s where we’ve come up with a Code Quality Score (CQS). This is basically a weighted value composed of the metrics we can retrieve from Sonarqube: #issues, lines of code, and code complexity. This CQS value’s smallest granularity is at the module level which is what is provided by Sonarqube and what developers find most useful. However, other stakeholders need this at the Feature or Team level to understand the level of Tech Debt owned by each team or which part of the codebase has the most Tech Debt.
There are other measurements of customer and developer happiness, or team velocity that can correlate to decreases in Tech Debt and indicate progress. However, correlations are not always present since many external factors may cause these metrics to change. For example,
- Reduction in customer complaints may be a result of pricing changes or changes in consumer sentiment
- Developer happiness while working in the codebase could be impacted by schedules or features they’re building
- Sprint speeds (%bug fixes vs. feature points) can vary depending on features that are built
As a result, we track these but do not use them in calculating our CQS.
Taking Action on Tech Debt
The final step in a full plan to address Tech Debt is to help teams take action on Tech Debt over time. There are two primary processes to help teams take action on tech tech:
- block Pull Requests (code changes) with tighter restrictions on new code
- Automate creation of tickets in backlogs for existing issues
Sonarqube has the ability to block PRs based on code coverage. This is the simplest way to slowly increase code coverage and encourage a culture of testing. The code coverage limit can start at whatever level a legacy project has and then new PRs can be blocked if they don’t have enough of a higher level of code coverage. This provides gradual improvement of a code base that is prone to bugs when code is changed.
The other thing we’ve done is automate ticketing for Sonarqube issues. By taking advantage of APIs for Sonarqube and JIRA, we add Tech Debt epics to feature team backlogs and keep the epics filled with the N highest priority tickets to address Tech Debt, but not enough that their entire backlog is overwhelmed with issues.
Here is an example of tracking ticket fixing history that can be shared with a feature team (team name is blurred out). The team can see how many issues are still pending (the SonarPending graph) which is an indication that they are decreasing Tech Debt.
What’s Next?
Our current Code Quality Effort has been successful, but we still see ways to improve on what we have built.
- For example, with the CQS, we found that people were confused with the score because higher numbers were worse than lower numbers, even though we color coded the CQS.
- We currently only measure Sonarqube issues and code coverage, but the other issues mentioned at the start of this article haven’t been measured and included. We’ll cover CQS v2 in another blog.
- The current mapping of modules to feature to team to backlog is maintained manually and prone to errors because it has no automated checks and gets outdated quickly.
In another blog, we’ll cover how we’re automating this and making it a self-checking process with more data integrity via a system we call AppDirectory.
References
1
https://research.chalmers.se/publication/518318/file/518318_Fulltext.pdf
