The Joel Test, Updated

A year ago, I found out that Joel on Software migrated from its original system to a WordPress site, and I read through all the top blog posts for every single role. Why did I do that? Same reason I read through every single one of Paul Graham's essays. Just because.

While reading Joel's blog I found the Joel Test, a series of perspective-enhancing questions a software engineer could ask a company in order to garner insights into the engineering culture there. They were, as follows:

  1. Do you use source control?
  2. Can you make a build in one step?
  3. Do you make daily builds?
  4. Do you have a bug database?
  5. Do you fix bugs before writing new code?
  6. Do you have an up-to-date schedule?
  7. Do you have a spec?
  8. Do programmers have quiet working conditions?
  9. Do you use the best tools money can buy?
  10. Do you have testers?
  11. Do new candidates write code during their interview?
  12. Do you do hallway usability testing?

The Joel Test is more than 17 years old, as I'm writing this post. I thought to update the list for today's world and today's software engineers. Apparently Dale Myers at Microsoft already did this, but it's a worthy exercise nonetheless. So without remembering any of the stuff in his post, let's go through Joel's points again:


1. Do you use source control?

If the company you're interviewing at passes zip files via email to each other with an email subject being a timestamp, or something crazy like that, run. Don't walk, run. That is not a serious company. These days, tools like git are commonplace and expected. You're pretty much expected to have a version control system with continuous integration (a point I will get to later), so this question is not that relevant anymore.

REMOVE


2. Can you make a build in one step?

Depending on the size of the project or company you're interviewing at, this may be scoped to a project or the entire company. In either case, this is invaluable. Great companies take pride in their build and deployment architecture, and make it very easy to get set up as a new developer. This question is also relevant because many companies do not have great build tools or environments, and it's not terribly evident from the outside. I would rephrase this question slightly.

CHANGE TO: "Would it be possible to see the README or build instructions for one of your core projects?"

Great companies have READMEs that specify exactly how to build a deployment properly. This question is better phrased in my opinion because it will be obvious to you whether the build instructions actually exist and plausibly work. You may or may not get the answer you're looking for here, but the substance of this question should be answered in the next question.


3. Do you make daily builds?

Today, most professional-grade software employs a form of continuous integration and delivery, where every commit triggers a litany of tests, linting, and building. Again, instead of a yes/no question, we can do something like this:

CHANGE TO: "Can I see your continous integration pipeline in action?"

I have asked the above question while interviewing, and great companies are very happy to share with you their pipelines as it demonstrates interest in software quality.


4. Do you have a bug database?

The gist of this question is whether developers or somebody writes down a bug and the exact steps to reproduce it when a bug is encountered. We use Atlassian JIRA, an issue tracker, to keep track of the bugs, and write down the steps in order to reproduce it. When fixing bugs, I personally write down the problem in the software and the patch applied in the issue before closing it out. This question is still highly relevant, but the exact text can be updated.

CHANGE TO: "Do you have an issue tracker?"


5. Do you fix bugs before writing new code?

This is so important. Literally. Without doing this, your software can't scale because you don't know if functionality is working or breaking because of new features or an existing bug. If your software can't scale, your company is fucked. Absolutely, positively make sure that this answer is a yes.

LEAVE AS IS


6. Do you have an up-to-date schedule?

I'm not exactly sure how schedules are made (honestly, we did a mix of waterfall and Agile before, and now we don't do either), but this question is pretty important too. Software is highly intangible, but you can plan out your process and you can make it better. I would update the text:

CHANGE TO: "How do you guys schedule new features or a new project, and how do you update your schedule over time?"


7. Do you have a spec?

This is probably more important at larger companies than it is at smaller ones. Paul Buchheit was famously tasked by Larry Page to "build an email thing", which became Gmail. If they give you a spec, great; code to spec. If not, do some exploration and come up with the parameters you wish to fulfill. This is assuming you are self-driven. Which you are, because you found this blog somehow.

REMOVE


8. Do programmers have quiet working conditions?

Cubicles are dead. Today, it's either open offices, shared offices, or remote working. In order of quietness, you should go remote first, shared second, and open third. I personally would never work anywhere with an open foosball or ping pong table, and you shouldn't either. However, you shouldn't need to ask this question; walking around the office should do.

REMOVE


9. Do you use the best tools money can buy?

This is things like using MacBook Pros vs. something you found in the garbage, or PyCharm vs. Atom, or TravisCI vs. buildbot. I personally notice the difference between my personal mid-2012 MacBook Pro with my corporate 2015 MacBook Pro every day. I would rephrase this question:

CHANGE TO: "What toolset do software engineers use here? Is there a financial budget for developer tools and time budget for training?"


10. Do you have testers?

So so so important. Software engineers should be expected to write and run unit, functional, and integration tests. But without testers (manual and otherwise), there's no feedback loop to check and improve your product's conceptual integrity.

LEAVE AS IS


11. Do new candidates write code during their interview?

I have interviewed at one startup where they did not give me any coding problems. They declined to place me because they hadn't budgeted for an additional software engineer before interviewing me. In other words, I dodged a massive bullet. You don't need to ask this question; if they don't give you any coding challenges or problems at all, don't work there, because it's guaranteed that incompetent people will make it in.

REMOVE


12. Do you do hallway usability testing?

This is likely more important for smaller companies, as more advanced products should have a UX team with a design philosophy, not to mention {internal, alpha, beta, canary} releases. Best way to see this is just to try out their product before the interview, and ask for a demo if it isn't consumer-facing.

REMOVE


This leaves us with 8 questions so far:

  1. Would it be possible to see the README or build instructions for one of your core projects?
  2. Can I see your continuous integration pipeline in action?
  3. Do you have an issue tracker?
  4. Do you fix bugs before writing new code?
  5. How do you guys schedule new features or a new project, and how do you update your schedule over time?
  6. What toolset do software engineers use here?
  7. Is there a financial budget for developer tools and time budget for training?
  8. Do you have testers?

I don't have an affinity to yes/no questions. I have been lied to my face before about these and I find questions that trust but verify to be much more reliable. In addition, by the time you're asking these questions, you're probably at an onsite anyways and the company will devote significant time and energy to assure you that they're not stupid. Good companies are proud of that.

I would add some of my own personal favorites:

  1. How important is recruiting and onboarding here?

Great companies make recruiting their number one priority. Not two, not three, but one. Bad software engineers will cost a company its future. Great software engineers will expand and accelerate it. It will be at least somewhat obvious during the interview process, because great companies have their own internal recruiters, practical but difficult coding problems, and engineers devoted to the task. Great companies also have a clear onboarding schedule, with prompt orientations, helpful codelabs, and friendly mentors.

  1. What coding standards do you follow?

Great companies will use linting, such as ESLint for JavaScript and PyLint for Python, and make them as strict as possible in order to have the most consistent code. Great companies will also have a clear code review process where checks not caught by tests or linting will be caught by other developers before changes are merged into wherever they need to go.

These are just suggestions, but strongly recommended ones. A good company will score 11 or 12 on the Joel Test. A bad company may score as low as 3. Knowing what to look for in a company may save you years of pain.