Test Coverage

Learning objectives

You know what test coverage is and you know how to get test coverage for tests.

In addition to the testing functionality, Deno provides facilities for checking test coverage. Test coverage refers to the proportion of different execution paths covered by the tests, measured through e.g. covered lines of code.

Extracting test coverage data

Deno's testing functionality allows extracting test coverage using the --coverage flag, which is given a folder into which the test coverage information is stored. Let's look at extracting test coverage using our previous example. In the previous example, the helloService.js was as follows.

let message = "Hello service world!";

const getHello = () => {
  return message;
};

const setHello = (newMessage) => {
  message = newMessage;
};

export { getHello, setHello };

And the helloService_test.js was as follows.

import { assertEquals } from "https://deno.land/std@0.222.1/testing/asserts.ts";
import { getHello, setHello } from "../../services/helloService.js";

Deno.test("Function getHello returns 'Hello service world!'", () => {
  assertEquals(getHello(), "Hello service world!");
});

Deno.test("Message set using setHello returned from getHello", () => {
  setHello("hello world!");
  assertEquals(getHello(), "hello world!");
});

The helloService_test.js was placed in a folder services under a folder tests, while the file helloService.js was placed in a folder called services. The project structure is as follows.

tree --dirsfirst
.
├── services
│   └── helloService.js
└── tests
    └── services
        └── helloService_test.js

When we run the tests using the command deno test, followed by the flag coverage which is given a folder -- test_cov in this case -- into which the coverage information should be stored, we see the following output.

deno test --coverage=test_cov
running 2 tests from ./tests/services/helloService_test.js
Function getHello returns 'Hello service world!' ... ok (1ms)
Message set using setHello returned from getHello ... ok (0ms)

ok | 2 passed | 0 failed (149ms)

Now, when we check the file structure again, we see that there is a new folder called test_cov that contains plenty of files.

tree --dirsfirst
.
├── services
│   └── helloService.js
├── test_cov
│   ├── ...
│   └── ...
└── tests
    └── services
        └── helloService_test.js

Printing test coverage

Once the test coverage data has been produced by giving the flag --coverage and a folder for the test coverage data for the deno test command, we can print the test coverage with Deno's coverage command. The command is followed by the folder that contains the test coverage data -- in our case, test_cov. Let's look at the test coverage data for the previous example.

deno coverage test_cov
cover file:///path-to-project/services/helloService.js ... 100.000% (12/12)

In this case, the tests cover 100% of the lines in the file helloService.js. The output also lists the lines that are not covered by the tests.

Identifying non-covered lines

Let's continue with the example and add some conditional logic to our service. Modify the helloService.js to match the following:

let message = "Hello service world!";

const getHello = () => {
  if (message === "secrets") {
    return "Here are the secrets";
  }
  return message;
};

const setHello = (newMessage) => {
  message = newMessage;
};

export { getHello, setHello };

Now, when you run the tests, the tests still pass.

deno test --coverage=test_cov
running 2 tests from ./tests/services/helloService_test.js
Function getHello returns 'Hello service world!' ... ok (1ms)
Message set using setHello returned from getHello ... ok (0ms)

ok | 2 passed | 0 failed (134ms)

Although the tests pass and the functionality is correct, the tests do not cover all the lines in the file helloService.js. When we run the deno coverage command, giving it the folder with the test coverage data, we now see that the test coverage is not at 100%.

deno coverage test_cov
cover file:///path-to-project/services/helloService.js ... 80.000% (12/15)
   4 |   if (message === "secrets") {
   5 |     return "Here are the secrets";
   6 |   }

As we can see from above, in addition to highlighting that the test coverage is not at 100%, we also see the lines of code that are not covered by the tests. In particular, the if statement of the getHello -- on lines from 4 to 6 -- is not covered. Adding a new test case that checks the case resolves the issue.

import { assertEquals } from "https://deno.land/std@0.222.1/testing/asserts.ts";
import { getHello, setHello } from "../../services/helloService.js";

Deno.test("Function getHello returns 'Hello service world!'", () => {
  assertEquals(getHello(), "Hello service world!");
});

Deno.test("Message set using setHello returned from getHello", () => {
  setHello("hello world!");
  assertEquals(getHello(), "hello world!");
});

Deno.test("Secrets returned with 'secrets'", () => {
  setHello("secrets");
  assertEquals(getHello(), "Here are the secrets");
});

Now, when we run the tests and check for coverage again, we observe that the test coverage for the app.js file is better. Note that we do have to remove the old cov folder before doing this.

deno test --coverage=test_cov
running 3 tests from ./tests/services/helloService_test.js
Function getHello returns 'Hello service world!' ... ok (0ms)
Message set using setHello returned from getHello ... ok (0ms)
Secrets returned with 'secrets' ... ok (0ms)

ok | 3 passed | 0 failed (134ms)

deno coverage test_cov
cover file:///path-to-project/services/helloService.js ... 100.000% (15/15)

Interpreting test coverage

What do these results mean and does high test coverage imply good tests? In practice, Deno checks which paths in code are executed when running tests and tries to look for paths that are not executed. High test coverage does not imply that the functionality in those paths would be appropriately tested, but only that those paths have been covered. As an example, the following tests also have a high test coverage, even though no assertions are used.

import { assertEquals } from "https://deno.land/std@0.222.1/testing/asserts.ts";
import { getHello, setHello } from "../../services/helloService.js";

Deno.test("Function getHello returns 'Hello service world!'", () => {
  getHello()
});

Deno.test("Message set using setHello returned from getHello", () => {
  setHello("hello world!");
  getHello()
});

Deno.test("Secrets returned with 'secrets'", () => {
  setHello("secrets");
  getHello()
});

deno test --coverage=test_cov
running 3 tests from ./tests/services/helloService_test.js
Function getHello returns 'Hello service world!' ... ok (0ms)
Message set using setHello returned from getHello ... ok (0ms)
Secrets returned with 'secrets' ... ok (0ms)

ok | 3 passed | 0 failed (121ms)

deno coverage test_cov
cover file:///path-to-project/services/helloService.js ... 100.000% (15/15)

In the above case, if the functionality exposed by helloService.js would be broken, the above tests would not tell us anything about it, even though the tests pass and the coverage is at 100%. The above tests are in practice completely useless and misleading. The above example also illustrates the need for responsibility when writing tests -- when writing tests, you are responsible for writing such tests that really test the functionality of the program.