Thoughts on quality engineering

Throughout my career, I’ve noticed that quality/test engineering is usually seen as a bottom-of-the-barrel discipline, something that someone should want to be promoted out of rather than someplace to end up. I find that really strange.

It takes a lot of skill to look at other peoples' code and write tests to exercise it and determine correctness, and to do it well. And to have exacting standards about code quality and testability of the code in the first place.

Nearly everywhere I’ve worked, though, test engineers have been incredibly junior and not particularly skilled. Which made it part of a self-fulfilling vicious cycle; test engineers do poor-quality work (and don’t seem to bring much value to the actual product development), so low-calibre programmers end up being put in those roles, and so then they continue to do poor-quality work. Test engineering seems to be treated as glorified QA in most places.

The most positive test engineering experience I ever had was at HBO, where my team’s build engineer was also in charge of setting quality standards, especially regarding tests, and he did a lot of instrumentation and cat-herding around it. I didn’t really appreciate it initially, but over time I came to understand and even cherish his influence. (The fact that even our build scripts were unit-tested was interesting! Annoying, but interesting, and it seemmed unnecessary until suddenly it wasn’t.)

But the actual QA/testing departmment at HBO (which our buid/test/infra engineer wasn’t part of) was… well, pretty typical.

Amazon’s mindset was basically wish-it-were-devops; put engineers on the line for supporting their own code, the reasoning went, and they’ll have an incentive to make sure the code is correct. But what really ended up happening was engineers would build something that was just Good Enough to roll out, and then find a more lucrative opportunity within the company (or even elsewhere), and then leave the rest of their team holding the support burden, which was often immense.

When I was tech lead on the media services team I decided I would Do Better and actually insist on 100% coverage of my code and full automation/CI/CD and everything. One of the new projects I was working on alone was responsible for raising the team’s overall coverage by a significant factor! But the support burden of older, less-well-factored, not-properly-tested code wore on me and I burned out.

When I was at HBO and actively participating in candidate interviews, one of our common panel questions was, “What makes good code?” Most candidates would say things like, “It needs to be correct, and tested, ideally with automated unit tests.” We’d ask, “And what qualifies as good coverage?” and most candidates would flounder a bit, or say “Oh, 100%, except for exception handling of course,” or whatever. “Why no exception handling?” we’d ask, wondering why the one thing you wouldn’t want to test is how things handle failures…

Anyway, one candidate said something that stuck with me: “It’s all well and good to have 100% coverage, but what’s really important is to make sure it’s the right 100%. You can easily construct tests that get your coverage numbers up while proving absolutely nothing about the code.” This impressed everyone on the panel. I don’t recall how the rest of the interview went, though, or if we ended up extending him an offer. Still, it’s definitely something that we talked about for the next several days.

But where I’m going with this: engineers are very bad at writing tests against their own code. They want to prove that it works, and when they have an idea that something is correct they’re only going to test it based on the assumptions that went into it. And I mean, this is a very very common thing across most humans, as Derek from Versitasium demonstrates:

Being a good test engineer means finding things that will break code, and testing the code in ways that wasn’t anticipated working. Good test coverage not only touches every line of code but every possible combination of conditionals. And it should test things that are outside of the expected range of inputs.

There are a number of jokes about test engineers which are along the lines of:

A test engineer walks into a bar and orders a beer.

A test engineer walks into a bar and orders 4,294,967,296 beers.

A test engineer walks into a bar and orders

A test engineer walks into a bar and orders -1 beers.

A test engineer walks into a bar and orders 🐄 beers.

A test engineer walks into a bar and orders ;drop table orders.

A test engineer walks into a bar and orders ⒊1

A test engineer walks into three bars simultaneously and orders two different beers from each one.

A test engineer walks into a bar and ‮orders an empty glass.2

and so on.

It might seem silly but real life is full of edge cases and things that go weird. Almost everyone has something in their life which falls into a trap of someone making assumptions. I seem to be a magnet for them, personally:

  • Legally changed name, use a different name in my day-to-day life depending on context; in some contexts I use first-initial middlename lastname but expect to be referred to by first-initial. In other contexts I’m just “fluffy.”
  • Legally changed gender, but only as a less-bad option and would much rather have an X or something on my identification (even though nobody knows how to handle that); my choice of pronouns depends primarily on who I’m talking to at the time.
  • My home is a condo that is a parcel of a historical landmark which didn’t use to be residentially-zoned. The parceling scheme is different than the unit scheme, and the current mailing address is different than the historical street address. There are four distinct, perfectly-valid ways of writing my complete address, and it’s anyone’s guess as to which one the validation system is going to go with.
  • I went to the same university for undergrad and grad school, but not contiguously.
  • I worked at Amazon twice, multiple years apart. (This messes up a lot of job sites, and also messed up Amazon’s own system internally!) At least both times were under the same name; if I were convinced to go back a third time, who knows what would happen… (I mean, systems-wise. I know what would happen to me emotionally.)
  • I got my driver’s license on my 15th birthday (when that was the legal driving age in New Mexico). A few months later, New Mexico raised the driving age to 16. Car insurance providers often reject my license history because that is not a valid age to have started driving, but if I put in the wrong year it doesn’t match the history so that causes a flag too. Fun3
  • My email and primary website have a .biz domain and I am, amazingly, not a spammer.

Anyway. Early on in my career I never really appreciated test engineering (or even testing in general), and it took me a while to get on board with it. And these days I absolutely appreciate test engineering as a distinct discipline. But I notice all around me that most of the industry is still stuck in the mindset that testing is the drudgery that should be offloaded onto unskilled engineers. Test jobs get significantly lower salaries than product engineering, and have experience and requirements in keeping with junior, entry-level engineers, even though this is a part of the industry that a lot of other things rely on. Internet services in particular are constantly under attack from people who care deeply about exploiting whatever minor issues they can find. Correctness of code also makes a huge difference with keeping private data private, and ensuring that the right things are being cached for the right people, and that business requirements are being met. But test engineers don’t seem as important as product engineers.

I asked other people who appreciate test engineering about this and the most reasonable take on it was that testing isn’t seen as making new features or shipping code, it just sort of gets in the way of things being released, and in agile shops in particular the prevailing attitude is “move fast and break things.” But that’s how we end up with an Internet where everything is insecure by default, malware spreads like wildfire, and peoples' private information becomes not-very-private while their machines get taken over by bitcoin miners.

In my own personal projects I have been generally pretty remiss in the testing process. Especially now that Publ is getting to a point where it’s handling private data that varies by viewer, I’m finding little issues now and then where proper testing and planning around it would have saved me a lot of trouble at times.4

My next major Publ milestone is all about testing and documentation. Backfilling these shortcomings is easily going to be as much work as writing Publ so far has been. And it’ll feel so, so satisfying.


  1. That wasn’t a three. 

  2. For a good time, try selecting ‮this text. 

  3. Incidentally, it looks like NM has changed the rules again, and good luck to anyone who wants to come up with a simple validation engine that covers that! 

  4. Fortunately, I haven’t had any major privacy oopses (that I know of!), just a couple of case-sensitivity problems and some link targets being mis-cached. I’ve at least biased towards fail-safe operation where post privacy is concerned. 

Comments

Before commenting, please read the comment policy.

Avatars provided via Libravatar