Ignition: Canary tests

Before you go on that road trip, you will likely perform a set of basic checks before hitting the road as a responsible driver. These include but are not limited to checking the tire pressure, lights, fuel, engine, levels of various fluids in the vehicle, etc.

A canary test is similar to those checks but a lot simpler and less time-consuming. If done correctly, you'll spend less time scratching your head and more time writing actual tests. I learned this technique from Dr.Venkat Subramaniam from one of his code retreats.

Now, take a look at the following test.

@Test
fun `junit and truth are setup`() {
  assertThat(false)
    .isTrue()
}

At first glance, this test could look pointless, and that's okay. Let's run the test and then break it down.

After the run, we see an error on the console. But it is not what we expected. Instead of an assertion failure, the build failed because the test framework could not find any tests.

FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':test'.
> No tests found: [CanaryTest.junit and truth are setup]

On further investigation, it turns out we haven't configured the build system to run tests. Let's fix that by configuring Gradle to run JUnit 5 tests.

tasks.test {
  useJUnitPlatform()
}

Now, we rerun the test and see the assertion failure we initially expected on the console.

> Task :test FAILED

CanaryTest > junit and truth are setup() FAILED
  com..truth.AssertionErrorWithFacts at CanaryTest.kt:10

1 test completed, 1 failed

FAILURE: Build failed with an exception.

Great, this failure revealed a couple of things.

1. The test failed! It means we can trust this test. We can't trust tests that don't fail.
2. The configurations for JUnit (the testing framework) and Truth (the assertion library) are correct, and both are working as expected.

It is understandable if you haven't come across bugs 🐛 in testing frameworks or libraries. I first found the value of canary tests when a framework spektacularly failed, not once but twice (two different versions). The tests kept passing even though I was asserting false to be true.

Canary tests can prevent us from chasing ghosts in production or test code due to a tooling failure or improper build configuration. It also comes in handy while troubleshooting tests on a different machine.

Now that we have verified what we wanted, we have to fix this test. We can't have failing tests in our suite.

@Test
fun `junit and truth are setup`() {
  assertThat(true) // <== changed false to true!
    .isTrue()
}

With this change, the test should pass unless something else is amiss. Rerun the test, and you should be able to see the following output on the console.

> Task :test
CanaryTest > junit and truth are setup() PASSED
BUILD SUCCESSFUL in 8s

You can use canary tests beyond JUnit and Truth (or any of you favorite combo). Let's try to extrapolate the idea to a different library - Mockito. Here's how a canary test for Mockito could look.

class CanaryTest {
  @Test
  fun `mockito can mock classes`() {
    assertThat(mock<X>())
      .isNotNull()
  }

  class X
}

When we run this test, we see the following error on the console.

Mockito cannot mock this class: class CanaryTest$X.
Cannot mock final classes with the following settings :
 - explicit serialization (e.g. withSettings().serializable())
 - extra interfaces (e.g. withSettings().extraInterfaces(...))

It looks like we need to setup mock maker inline to hit the road. Setting it up should make this test pass and I'll leave that as an exercise.

When to use?

Adding canary tests can be helpful in the following situations.

• Creating a new project.
• Before adding actual tests to a project that doesn't have any tests.
• Bringing in new testing tools/libraries into a project that already has tests.
• Adding a new module to your multi-module project.
• Bumping up major versions of your existing test framework/libraries.

The recipe

1. Write a minimal failing test using the test framework/library of your choice.
2. Run the test and ensure the test fails due to an expected assertion failure.
   • If the failure is due to an environment or a build misconfiguration, fix it and rerun the test until you see the expected failure.
3. Now, change the assertion in the test to make it pass.
   • If the test doesn't pass, look at the failure message and make the required changes until it passes.
4. Check in the code and proceed to write your actual tests.

Closing note

The fundamental idea behind canary tests is to do the bare minimum to verify if your testing environment is ready for actual tests. The concept applies to your integration tests too. For instance, a canary database integration test may try to establish a connection with the database. An Android instrumented test may check if it is able to acquire the Context to the application under test.

In case your codebase has multiple modules, I recommend having canary tests in each of those modules. Every module, after all, has its own configurations.

Canary tests are valuable tools in my testing toolkit and save me time and effort. I hope they make their way into yours too!

BTW, kudos for making it this far. If you want to learn more about building high-quality software, you can follow me on Twitter to stay informed.