The testing-wall: A cautionary tale

I just recently came back from parental leave. And it just so happens that my first week was rather exhausting. Getting back to the rythm of the eight hour work-day is always difficult after a long break. Add a baby into that… rough. I found new respect for single parents.

My first week also came with a story I find worth telling: The tale of the testing-wall.

My team is developing a testing-framework, tiger. Within this framework we rely heavily on mockserver for intercepting and recording traffic. In my absence a new version was released and it fell onto me to migrate our framework to this new version. A colleague had already done the heavy lifting in one branch but couldn’t get it working just right. And since I had more experience in that area I inherited the branch.

I noticed one missing feature in mockserver that we needed, wrote a PR, awesome. This is when I came upon another issue, which would prove to be quite a bit more challenging. When forwarding requests without a content-type header, mockserver assumed the requests to be of type string. And, of course, when you convert a complex binary body into a string and back into binary, you will not get back the original body.

I found the issue and said to myself: „What is one more PR?“ and went on to fix it. The issue itself, after all, was just one line. I fixed this line, wrote a unit test explaining the edge-case at play here and hoped to be on my way.

Two days later I understood that I was never just that easy. This line had implications into nearly every corner of the system. Why, you might ask?

The answer lies in the tests: Of course most REAL LIFE requests do have a content-type. And changing the default behavior to be more agnostic will have no (or no negative) consequence in the externally observable behavior of the component. It is another story for the API: The signature of various commands, and with that many, many tests do use that behavior (They are tightly coupled).

It became so bad that after two days (don’t worry, I did other stuff on the side. But I only programmed on this) that I, after changing serialization, many constructors and about 50 tests I passed one module, the build progressed into the next (integration-tests…) and I had several hunderd red tests. Several hundred tests which have this behavior hard-coded into them.

This is takeaway one from this story for me: Unit tests do NOT always do good. As, for example, Robert Martin points out mainting and constantly refactoring the tests (and deleting unnecessary ones) is elementary. Every line of code is a potential liability, be it production or test. The less code you need to solve your problem, the better. I have more and more adopted the habbit of writing less actual unit tests and explaining my needs shorter and more concise in higher level tests. This generally leads to less tests, less rigidity in case of internal refactorings and more readable tests.

When confronted with the hundreds of failing unit tests I lost my motivation. „Frick that stuff“ I thought to myself. I task switched to something else, something easy and not as important. Weekend was close already, but there was still enough time for 1 1/2 quick wins. Very good for motivation, leaving me with a good feeling come monday.

In the evening I had my son to myself, meaning hours of feeding and consoling and diaper changes… a lot of time to let your mind wander. And during that time it hit: What an absolute idiot I am! The bugfix would not have to be in the mockserver itself. There was a very easy option to include it in our testsuite itself. And since I know our tests very well and we don’t rely on mockserver API heavily I know fixing tests would not be an issue. Oh man…

As it so often turns out, and this is the second takeaway: Hard problems often have a easy solution. And you need to take a step back to see this solution. Taking this step and investing the time is often worthwhile.

I found this to be true so many times and I still often fall into this trap: Productiv and busy are not the same. So I really will strive to internalize this reflex to stop and rethink once I hit a certain timebox.