What the worst projects say!
Have you heard any of these:
- What's a test?
- I don't have the time to write a test!
- We have enough internal knowledge, we don't need to document.
- I'd rather be working on code, not documenting!
When I talk to people about their projects or trying to decide on which my third-party project dependencies to rely on, I often look at testing and documentation methodologies.
The projects that I come across without adequate testing and documentation are the ones I find problematic over time to deal with. There are certainly other things that make a project bad to rely on, but these two are usually sure signs.
And the engineers always have excuses for why they don't have these things, but I find that it usually comes down to a single thing:
Writing tests and documentation sucks.
Occasionally, I have met the individuals who love writing tests. I am not one of them. Most programmers I have met would much rather be building something instead of validating that something works or writing documentation. The best programmers I have met play with ideas, but start writing their mental models into documentation before code starts becoming a project.
Like wearing a seatbelt is critical for safety, writing tests and documentation is critical for long-term success and velocity. That I don't like wearing a seatbelt, writing documentation or writing tests has no effect on the validity of that statement.
Wanna watch velocity die? Don't write tests
So I will occasionally hear about how writing tests cause velocity to slow. I always get a chuckle out of this.
There are two things that you can do to guarantee that your code velocity will die:
- Use a dynamically typed language like Python or Node.js
- Don't write tests
I'm not going to spend time on dynamic typing here, but basically, you are just waiting for runtime bugs to occur that could have been caught in compile. Yeah it's fast at first. That doesn't last.
Without tests though, how do you know your functions and methods even work? When someone makes a change to the code, how do you know that the 500 lines work the way you think it should?
When the new person comes in and makes a change, how do you know the change they made is good? If you are in a dynamic language, hell you can't even be sure they are returning the right type! If you are in a typed language, they might be returning the right type, but doesn't mean they are returning the right value.
Can you look at complicated code from 3 years ago you wrote and be sure in a review, without tests, that the code change is good? If you said "yes", well you are certainly better than me. Matter of fact, you are better than all the programmers I've met, and I've met some of the best.
Now there may be things you can't test or the time to write a test vs. payoff is too low. Game engines probably have display code that is really hard to test that isn't worth it. But I have also seen game programmers write virtually no tests for "velocity". Having met some others from the game industry, this seems like a norm (certainly haven't done a study, its just hearsay with a few people's observations).
The long-term results are always the same, each change keeps accumulating small bugs that would have been caught with adequate testing. The larger the code base, the more of these that will creep in. And velocity will go off a cliff.
Don't document? What's that smell? Oh right, burning money
There are two types of documentation, both are important:
- Code documentation
- Software documentation
These two very separate beasts but are critical to understanding software.
Code documentation refers to comments within the code that can be extracted to understand libraries that make up software. Godoc/Pydoc/... offer structure to write this type of documentation and extract it to allow understanding of library structure.
Software documentation is a level up from code documentation. It explains the overall architecture of the software, how to use the software, etc...
If you are a small company, these can be critical. Hiring new people or replacing people on a project becomes a long onboarding exercise without software/code documentation.
If you are a large company, this makes handing off software between teams difficult. And it incurs the same costs as a small company, though these are typically hidden by company scale. But nevertheless, it is costing you money.
Some engineers like the idea of being the only person who can fix a problem or use lack of documentation as a way to have control. Managers sometimes also encourage this behavior. It is like job security through obscurity. These are typically bad behaviors for the engineers and the company that should be discouraged.
You have documentation, but it is gathering dust
This is one of the most common problems I see. Large companies, small companies, ... documentation gets out of date.
Usually, this is at the software documentation level. If the initial documentation was good, then this usually isn't because the team didn't care. Usually, it is because it is an afterthought.
This is a hard problem to solve. My general belief is that to fix this, you have to put documentation next to the code. Both have to be checked into version control.
This does a few things which I think are important:
- Documentation is always seen. To ignore updates, usually, the engineers have to make a conscious choice.
- Allows you to create documentation metrics alongside code change metrics.
The first brings awareness of the documentation to the engineer. Otherwise, this usually is a secondary thought, if thought of at all. The code reviewer (and if you don't have these, you are in trouble already) also have the opportunity to stop the submit without a documentation update.
The second is critical. Without tracking of metrics, teams cannot be aware when large deviations between code updates and software updates have occurred. This can be leveraged for company-wide tracking of documentation.
Incentivizing
I once witnessed a new management system for an operations group come online. This system required adding lots of assets manually. But adding these entries didn't help the engineers in any way.
I tried to explain to the manager in charge of integrating the system into our processes that this wasn't going to work. The system required the engineer to do twice the work with no benefit to them. The manager replied, "they will have to because management is dictating it must be done". I looked back at the guy like he was nuts and he looked at me as if this was a sure thing. One of my failures at that time was not having the skills to communicate with him in a way that would have lead to the right outcome. By this time there was a lot of pressure to get the system out the door. To have gotten this right the software team would have required another 1 year of work.
Looking back, the politics of delaying any launch would not have been in his favor. It was easier to believe that mandates would work. If the system failed to be used, the blame would not fall on him. If he caused delays on the launch, however, he would suffer repercussions. The dev team would have been upset that the correct requirements had not been pushed to them. The Ops team members who were supposed to be shepherding development would have been on the hook for some blame. So...
The system came online and within a week the diff between the live state and the recorded state was thousands of pages. This mistake on human behavior on the part of management and the software designers cost the company tens of millions of dollars over the years. Not only in the hacks that were required to keep the data up to date but the huge cost of removing the hacks (which took years of work).
Human behavior often dictates the success or failure of something within an organization, not technology. Not that mandates cannot help a situation, but it certainly won't guarantee it.
The key to any organization change is encouraging the right behavior. Management saying "you must document" or "you must test" will not get you the desired outcome. The people writing your software must see the value and be incentivized to do it.
I've always liked the saying you can have something "good, fast or cheap. Choose two!" If you want to get good documentation, you will have to incentivize it and loose the "fast" or "cheap" option.
How you incentivize will be organizationally dependent. But using metrics and adding in compliance checking can help. However only if you are giving the right incentives and promoting the right software culture. If you keep asking for "fast" and "cheap", you will keep reaping the same long-term problems.
So what can you do?
Well, you have a few problems to solve:
- Increasing the visibility of documentation to encourage regular updates
- Incentivizing documentation updates and tracking documentation drift
- Incentivizing testing and providing metrics around code coverage
- Changing the culture
The first one is easy and has multiple ways to do it. If I was to give a recommendation, I would use markdown and store it close to the code. Multiple languages can render it in whatever style you choose, it is easy to index, and for every project you know where the documentation is.
This allows you to track how often documentation changes alongside code changes. This can give you warning signs on documentation drift.
The second and third is harder and requires some personalization for the particular company. Having design documentation as part of promotion consideration could be one way (though sometimes this can incentivize the wrong thing, I've seen this go sideways). Preventing product launches without up to date documentation and code testing coverage of say 70% can be another. Additional bonuses based on documentation/testing metrics could add another early incentive. Hiring in people with strong positions on these things can also help turn a culture around. Remember that the idea is to create the culture and eventually it will self-reinforce.
The key thing to understand is:
The long-term viability of a project can often be tracked to documentation and testing.
Whatever you can do to make that better will help you in the long run.