I’ve been around long enough to see many different efforts to improve software quality fail. Often the ideas are fairly one-dimensional rather than wholistic. I’ll give you one example.
Let’s say there’s a decision that everyone must use continuous integration in the next release. Even if it’s not sold as a panacea, there seems to be an implication that it’ll definitely lead to significantly improved quality. And it’s not to say that it couldn’t. But the problem is that for continuous integration to be very successful, it requires other things like good development practices and perhaps a suite of unit tests.
Having said all this, many could be done in isolation.
Better source code management practices
Most of this revolves around source code management. A common way to use tools like git is to do the following:
- Branch for a bug fix
- Branch for feature work
- Branch for releases as soon as they exist
This makes testing significantly easier. One can test a fix by looking at a bug fix branch and not have to mess with roots. Or do regression testing. Or with features under development. It also makes it very clear the intent of every check-in. If I merge one of my branches into one called 1.1, well, that’s the branch for version 1.1. I can regress this against earlier named branches easily. Dumping everything in master, which is quite common, causes significant problems:
- If master breaks, anyone at the company that wants to check out and build your project will fail, wasting their time
- A check-in to master is unclear what release it was intended for, what feature it was for, or what bug it fixes. Sure, you can always put that in the commit notes but it’s not an elegant solution. I’ve seen so many cases where someone commits to master thinking the fix will go in a certain release, but earlier that day, someone did a release branch off master. Then you ship without the fix.
- It’s sometimes very difficult to back out work on one feature or bug if commits are intermixed with commits for other features or bugs
With a branch-based system, every commit message clearly indicates the intent of the commit and every branch clearly describes its purpose. Either the bug it’s for or the feature it’s for or, after a merge, what release it’s intended (at least initially) to go into.
As an addendum, if you use Stash (this may work on other source control systems), you can create release branches and then set them to auto-forward to each other. So, for example, let’s say you have three releases that are being worked on:
Perhaps 1.0 is nearing completion, a bug fix release is underway and everyone not working on that is working of 2.0 stuff. You can set 1.0 to forward to 1.1 and then to 2.0 in Stash. Then, when you commit to 1.0, for example, it automatically ends up in the other releases. It’s clear what version you wanted your code to be committed to and you don’t need to worry about committing to multiple versions.
Bug fixing attached to features
This is used by some companies. Essentially, you work on a feature, but before it can be signed off on, the bugs targeted for this release must be fixed. The big wins for quality are:
- Features are much more testable rather than having to wait months until bug fix time begins
- It’s quicker and easier to fix bugs right after the code was written to create the feature, improving efficiency
Fix showstoppers for lower priority bugs
The way companies generally manage releases is that they stop working on lower priority bugs first, then medium, then high. So at the end of the cycle is when you fix your showstoppers.
I’ve honestly never seen it done any other way, but if seems more logical to me that this process be inverted.
First, what is a showstopper?
- Something you believe is so important that we’d stop the release for it
- Something that is likely very risky
- Something that could be a significant change
When’s the best time to fix these? Months before the release when we have time to live on the changes or right at the very end? I would propose that showstoppers be kept close to zero for the entire release cycle. When you hit a zero point, then you start fixing medium priority, and so on. In the end, it also seems like you’d fix more showstoppers and less medium priority bugs. Isn’t that a good outcome? Would love to hear from anyone that employs such a system.
Less reporting, more testing
As a company gets bigger, the amount of time spent doing reporting appears to go up exponentially. I’ve personally worked with teams that spend at least 3 hours a week preparing reports for upper management. That doesn’t even count other red tape. Instead of making one tester spend three hours creating charts and graphs, drop by their office and ask them how things are going. Or perhaps ask them to write a brief summary email. We’ve forgotten about subjectivity and we trust reports more than the opinions of those in the trenches.
Also, anecdotally, I once ran a web server that hosted reports from engineers. Guess how often they actually got read. I’ll give you a hint. It’s very depressing.
No special treatment for bugs reported by higher-ups
This will never, ever happen, but I find the idea to be conceptually interesting.
We’re all familiar with the bug coming down from on high that we drop everything to fix. It’s a nice perk for higher-ups but terrible for product quality. We’re all in this together and the importance of a bug has nothing to do with the importance of the person filing it.
If management is living on builds that have all their edge cases fixed, they have an inaccurate picture of the current state of quality. They may decide you don’t need as much bug fixing time as was alloted.
I’m not suggesting they not file bugs. but that minor bugs are punted unless a lot of duplicates are filed. Super serious bugs are showstoppers and just like bugs filed by anyone else, they get maximum priority.
The question is how to actually execute this, other than just convincing your management chain that it’s the best thing for the company even if it may be a downside to them personally.
Shorter feedback loop
This is a generic concept that encompasses both the previous and the next line item and other topic areas as well. In general, though, the idea is that an engineer should be informed of a bug as soon as possible after it has been introduced. If it can be 10 mins instead of hours because the QA person is doing per-commit testing. It can be days instead of months if the above item is adopted. Every context switch is a loss of productivity. The more efficiency an engineer has, the more bugs they can fix and the higher quality can be.
Fast continuous integration
Continuous integration systems I’ve seen often have one thing in common: They are so slow that you might as well have a QA person build and run unit tests manually. This is usually due to always building a project clean every time. If you have a small project, then just skip to the next item, but the difference between a clean and a dirty build on a large project can be tremendous, especially if frameworks and custom libraries are involved.
The different between a few minutes and 30 minutes is the difference between an engineer with the same source file they were working on already open versus several context switches they may have made since they introduced the problem.
Engineers and QA have more power than they think they do. If someone tells you to check in code that’s absolute crap, push back. Fight. Fight for your own code or for your own testing. Don’t sign off on your testing just because there is a deadline. Be honest. Say no, it’s not good enough to ship.
Better testing practices
You can’t blame testers for poor quality software, but the way testing takes place and what types of testing are emphasized can have huge impacts on quality.
Seems like software companies especially either go under or grow fast. Being able to scale better improves software quality as productivity can improve for everyone.
Often the same mistakes are repeated in a new project that existed in the early stages of the old projects. Being prepared for what those might be can help budget adequate time to address them for the first few releases.
Improve value of dogfood testing
One of the most important things you can do is to improve the quality of builds given to dogfood testers.