Tutorial: How to Write Integration Tests for REST APIs with node.js

In this tutorial we're going to look at how to create a resilient integration testing solution for REST APIs.

This tutorial is not about simply installing mocha + chai and writing a few tests. We will go step-by-step (not leaving out a single detail) and create a setup that allows you to seeminglessly write integration tests for your web API. This setup will automatically start a server to run tests against and connect to a testing database that is reset between tests. We will basically be creating a very thin framework on top of mocha, fine-tuned to our testing needs.

TL;DR and Quick Links

If you don't want to follow along with the tutorial and want to instead just see the setup in action, you can check out the code on GitHub directly!

• Part 1 on GitHub
• Part 2 on GitHub
• Part 3 on GitHub

A Brief Introduction to Integration Testing

What is Integration Testing?

Integration testing basically means having automatic tests that run against your entire app. So you're not simply testing individual functions or classes (as you would in unit tests) but (in the context of a REST API) your tests make HTTP requests against a running web server, look at the response to your request and sometimes look inside the database to see if everything is in the state it's supposed to be in.

Why are Integration Tests Awesome?

The great thing about integration tests is that they are very simple to write and they're highly maintainable because you are testing the "most public interface" of your code – the REST API. That means you could do the most brutal refactoring or even rewrite your app in a completely different programming language and your integration tests can remain unchanged, as long as the REST API it exposes does not change its behaviour.

What Are the Downsides of Intergation Tests?

Although integration tests are very easy to write, it can be tricky to get the initial testing setup going in a way that it allows you to easily start up testing servers and make requests against them. No worries though, this is exactly what you will learn in this tutorial!

Another downside of integration tests is that they are quite a bit slower than unit tests. A single test can run somewhere between 20ms and several seconds. This is a small price to pay though and can be easily mitigated significantly by parallelizing tests.

What You Will Learn

Part 1: Creating a Minimal Backend for a Todolist App: "todoman"

First, we need something to write tests against, so we will create a very simple REST API for a todolist app. To keep things simple, we'll use node.js with express and MongoDB for this. The app will be very minimal but we will write it in a way so that it stays maintainable for when we add features later on. If you're a somewhat experienced node.js developer, you can probably skip this part and dive into part 2 directly.

Part 2: Creating the Testing Setup and Writing our First Tests

In the second part of this article, we will get our hands dirty and write the actual code for our testing setup. We will use the battle-tested mocha/chai-combo as our libraries of choice but the testing setup will work in a very similar way for any other test runner or assertion library. This will be the most important part of the series because getting the testing setup right is probably the trickiest part. But if we do it right, it will be a breeze to add on additional tests for new endpoints later on.

Part 3: Adopting a Test-driven Workflow When Working on REST APIs

In part three, we will see how we can adopt a test-driven workflow for when we add endpoints to our CRUD app. While I don't believe in "TDD for everything", I think a test-driven approach works very well for writing endpoints. If done correctly, you will never really need to do any manual testing for your endpoints which saves a ton of time already during, but especially after, development.

1. Creating a Minimal Backend for a Todolist App: "todoman"

1.1. Initial Planning and Getting Things to Run

OK, let's get right into it. We will now create a little node.js backend for an imaginary todo app called "todoman". It will be a simple REST API that, for the start, exposes two routes:

• POST /todos - To create a new todo
• GET /todos - To list all todos
• (We'll add a PATCH /todos/:todoId for updating todos later in part 3)

A todo will have the following structure:

Todo {
  _id: ObjectId;
  title: string;
  completed: boolean;
  createdAt: Date;
  updatedAt: Date;
}

To get started, let's create a new folder that will contain our code:

$ mkdir ~/code/todoman-backend
$ cd ~/code/todoman-backend

As with almost any node.js project, we'll want to start out by installing some packages. We won't need much:

• express - To run our web server
• mongodb - To talk to our database
• body-parser - To parse request bodies in JSON format for us

We'll install these dependencies with yarn:

$ yarn add express mongodb body-parser

Now let's write some code! I think the directory structure is very important for the maintainability of any project, so let's think about that for a second. We'll keep all our application code in a folder called src/. The entry point will be src/main.js. All code that deals with todos will live in src/todos/.

In the end we will end up with something like this:

src/
  errors/
    ValidationError.js
  todos/
    routes.js
      # POST /todos
      # GET  /todos
    TodosService.js
      # createTodo({ name: string })
      # listTodos()
  main.js
    # Connect to database
    # Bootstrap express-app
    # Start http server
package.json
yarn.lock

Maybe you can already tell from the layout that we will separate our transport layer (HTTP, living in src/todos/routes.js) from our business logic (living in src/todos/TodosService.js). This is something that might be a little bit overkill for a small project like this but I really find that it helps with separating concerns and it will make our business logic highly reusable in the context of maybe a command-line application or when writing test setups.

Let's go ahead and create our entry point. This is nothing fancy, it's very typical for an express/MongoDB app:

src/main.js
const express = require('express')
const { MongoClient } = require('mongodb')
const PORT = process.env.PORT || 3000
const MONGODB_URI = process.env.MONGODB_URI || 'mongodb://localhost:27017'
const MONGODB_DB = process.env.MONGODB_DB || 'todoman'
main()
async function main() {
    const mongoClient = await MongoClient.connect(MONGODB_URI, {
        useNewUrlParser: true
    })
    const db = mongoClient.db(MONGODB_DB)
    const app = express()
    app.use(function attachDb(req, res, next) {
        req.db = db
        next()
    })
    app.get('/', function(req, res, next) {
        res.status(200).json({ name: 'todoman-backend' })
    })
    app.listen(PORT, err => {
        if (err) {
            throw err
        }
        // eslint-disable-next-line no-console
        console.log(`api-server listening on port ${PORT}`)
    })
}

Let's give our code a test-run. But before we do, let's go ahead and add nodemon so that we can automatically have our server restart when we make changes:

$ yarn add nodemon --dev

We'll also add a dev script to our package.json so that we can start our app in dev-mode by running yarn dev:

package.json
{
    "scripts": {
        "dev": "nodemon src/main"
    },
    "dependencies": {
        "body-parser": "^1.18.3",
        "express": "^4.16.3",
        "mongodb": "^3.1.1"
    },
    "devDependencies": {
        "nodemon": "^1.18.3"
    },
    "prettier": {
        "singleQuote": true,
        "semi": false
    }
}

Let's run our server:

$ yarn dev
yarn run v1.6.0
warning package.json: No license field
$ nodemon src/main
[nodemon] 1.18.3
[nodemon] to restart at any time, enter `rs`
[nodemon] watching: *.*
[nodemon] starting `node src/main.js`
api-server listening on port 3000

Now if you view http://localhost:3000 in your browser, you should see the following JSON output:

{
  "name": "todoman-backend"
}

Awesome, it's working! :)

Now let's add our todo-specific code!

1.2. Adding Our First Endpoints

We'll start with the TodosService. This is a class that contains the business logic for all our todo-endpoints.

src/todos/TodosService.js
const ValidationError = require('../errors/ValidationError')
module.exports = class TodosService {
    constructor(db) {
        this.db = db
    }
    async listTodos() {
        const todos = await this.db
            .collection('todos')
            .find({})
            .sort({ _id: 1 })
            .toArray()
        return todos
    }
    async createTodo(todoData) {
        if (!todoData.title) {
            throw new ValidationError({ title: 'required' })
        }
        if (typeof todoData.title !== 'string') {
            throw new ValidationError({ title: 'must be a string' })
        }
        todoData.title = todoData.title.trim()
        const result = await this.db.collection('todos').insert({
            title: todoData.title,
            completed: false,
            createdAt: new Date(),
            updatedAt: new Date()
        })
        const todo = result.ops[0]
        return todo
    }
}

Take a look at the file – it should be fairly straight-forward, although there might be some MongoDB-specific code pieces that could look foreign if you're not used to working with it.

Inside TodosService#createTodo() we validate user input and, in case it doesn't pass validation, throw an error. Make sure to create that error:

src/errors/ValidationError.js
module.exports = class ValidationError extends Error {
    constructor(invalidFields) {
        super()
        this.name = 'ValidationError'
        this.invalidFields = invalidFields
    }
}

#listTodos() will be called when hitting GET /todos and #createTodo(todoData) will be called from POST /todos. Let's get that hooked up:

src/todos/routes.js
const express = require('express')
const TodosService = require('./TodosService')
const router = express.Router()
router.get('/todos', function(req, res, next) {
    const { db } = req
    const todosService = new TodosService(db)
    todosService
        .listTodos()
        .then(todos => res.status(200).json({ todos }))
        .catch(next)
})
router.post('/todos', function(req, res, next) {
    const { db, body } = req
    const todosService = new TodosService(db)
    todosService
        .createTodo(body)
        .then(todo => res.status(200).json({ todo }))
        .catch(next)
})
module.exports = router

We'll need to make a few changes to src/main.js to hook up our newly added todo-specific code:

1. We'll need to include the routes we just created. The cool thing about express.Router() is that we can use it from a separate express app like any middleware. So from src/main.js we can simply do app.use(require('./todos/routes'))
2. We need to make sure we're parsing JSON request bodies correctly. Express doesn't take care of this out-of-the-box but it's very easy to add this functionality. For this we will use the body-parser package that we installed earlier.
3. We need to handle any potential ValidationErrors that might be thrown during a request and create a proper response for the client. For this, we will register an error handling middleware as our last middleware of our app.

src/main.js
const express = require('express')
const { MongoClient } = require('mongodb')
const bodyParser = require('body-parser')
const ValidationError = require('./errors/ValidationError')
const PORT = process.env.PORT || 3000
const MONGODB_URI = process.env.MONGODB_URI || 'mongodb://localhost:27017'
const MONGODB_DB = process.env.MONGODB_DB || 'todoman'
main()
async function main() {
    const mongoClient = await MongoClient.connect(MONGODB_URI, {
        useNewUrlParser: true
    })
    const db = mongoClient.db(MONGODB_DB)
    const app = express()
    app.use(bodyParser.json())
    app.use(function attachDb(req, res, next) {
        req.db = db
        next()
    })
    app.get('/', function(req, res, next) {
        res.status(200).json({ name: 'todoman-backend' })
    })
    app.use(require('./todos/routes'))
    app.use(function handleErrors(err, req, res, next) {
        if (err instanceof ValidationError) {
            return res.status(400).json({ error: err })
        }
        next(err)
    })
    app.listen(PORT, err => {
        if (err) {
            throw err
        }
        // eslint-disable-next-line no-console
        console.log(`api-server listening on port ${PORT}`)
    })
}

With that, all the basic functionality of our todoman backend is in place! Now let's see that everything works.

1.3. Testing Our Endpoints

First, make sure that your development process (yarn dev) is still running. Now, hit GET /todos with something like cURL:

$ curl http://localhost:3000/todos
{ "todos": [] }

As expected, there are no todos yet. Let's create a few at POST /todos:

$ curl -H "Content-type: application/json" -X POST -d '{ "title": "Todo 1" }' http://localhost:3000/todos
{
  "todo": {
    "title": "Todo 1",
    "completed": false,
    "createdAt": "2018-08-13T17:45:11.020Z",
    "updatedAt": "2018-08-13T17:45:11.020Z",
    "_id": "5b71c3a702b5d752675eb4e8"
  }
}
$ curl -H "Content-type: application/json" -X POST -d '{ "title": "Todo 2" }' http://localhost:3000/todos
{
  "todo": {
    "title": "Todo 2",
    "completed": false,
    "createdAt": "2018-08-13T17:46:58.330Z",
    "updatedAt": "2018-08-13T17:46:58.330Z",
    "_id": "5b71c41202b5d752675eb4e9"
  }
}
$ curl -H "Content-type: application/json" -X POST -d '{ "title": "Todo 3" }' http://localhost:3000/todos
{
  "todo": {
    "title": "Todo 3",
    "completed": false,
    "createdAt": "2018-08-13T17:47:20.205Z",
    "updatedAt": "2018-08-13T17:47:20.205Z",
    "_id": "5b71c42802b5d752675eb4ea"
  }
}

If we check back to our GET /todos route, we can see that those todos are now returned in the order they were created:

$ curl http://localhost:3000/todos
{
  "todos": [
    {
      "_id": "5b71c3a702b5d752675eb4e8",
      "title": "Todo 1",
      "completed": false,
      "createdAt": "2018-08-13T17:45:11.020Z",
      "updatedAt": "2018-08-13T17:45:11.020Z"
    },
    {
      "_id": "5b71c41202b5d752675eb4e9",
      "title": "Todo 2",
      "completed": false,
      "createdAt": "2018-08-13T17:46:58.330Z",
      "updatedAt": "2018-08-13T17:46:58.330Z"
    },
    {
      "_id": "5b71c42802b5d752675eb4ea",
      "title": "Todo 3",
      "completed": false,
      "createdAt": "2018-08-13T17:47:20.205Z",
      "updatedAt": "2018-08-13T17:47:20.205Z"
    }
  ]
}

Let's now test our input validation at POST /todos.

First, we'll try an empty request body:

$ curl -H "Content-type: application/json" -X POST http://localhost:3000/todos
{
  "error": {
    "name": "ValidationError",
    "invalidFields": { "title": "required" }
  }
}

Looks good! Now, what if we pass an empty string?

$ curl -H "Content-type: application/json" -X POST -d '{ "title": "" }' http://localhost:3000/todos
{
  "error": {
    "name": "ValidationError",
    "invalidFields": { "title": "required" }
  }
}

Nice, that works, too!

The final test is to send something other than a string for the title:

$ curl -H "Content-type: application/json" -X POST -d '{ "title": 123 }' http://localhost:3000/todos
{
  "error": {
    "name": "ValidationError",
    "invalidFields": { "title": "must be a string" }
  }
}

Cool! :) One more thing that we might want to test is our little bit of input sanitization that trims the title for us:

$ curl -H "Content-type: application/json" -X POST -d '{ "title": "  Todo with Spaces " }' http://localhost:3000/todos
{
  "todo": {
    "title": "Todo with Spaces",
    "completed": false,
    "createdAt": "2018-08-13T18:02:40.300Z",
    "updatedAt": "2018-08-13T18:02:40.300Z",
    "_id": "5b71c7c058e7445392c86209"
  }
}

Awesome, this is looking pretty solid. All these manual tests we just went through to verify that our API works as expected are what we want to automate in the next part. Our app is working, so let's think about how we can write those automated integration tests for it now!

2. Creating the Testing Setup and Writing our First Tests

Before we dive in and start coding, let's take a little step back and think about what we need so that we can painlessly write our integration tests.

2.1. How Our Integration Tests Will Work

Our integration tests will make actual HTTP requests and hit the database. That means that we need an actual server running and we also need to connect to a database. The server should be started automatically as part of our testing setup and shut down again once all tests have been run. Similarly, we want to connect to the database before our tests run and disconnect afterwards. Also, we want to reset the database completely between tests so that tests don't affect one another.

To summarize:

1. Automatically start server before tests and stop afterwards
2. Automatically connect to database before tests and disconnect afterwards
3. Reset database between tests

I imagine our tests to look something like this:

describe('GET /todos', function() {
    testServer.useInTest() // Provide a test server
    testDb.useInTest() // Connect to database and reset it between tests
    it('responds with 200 { todos }', async function() {
        const api = this.api // An axios client with its baseURL set to the test server
        const db = this.db // A mongodb database client
    })
})

So here is what we want:

1. A testServer utility that provides a testing server for us to make requests against and exposes an axios client as this.api in every test
2. A testDb utility that provides a connection to a test database, resets that database between tests and exposes a mongodb client in every test as this.db

That would provide us with everything we need to seemlessly write integration tests against any route.

So how can we achieve this? Let's go step-by-step, starting with the testDb utility because it's a bit easier to implement than testServer.

But first – where do we put our files? Where do the actual tests go and where do our test helpers go?

2.2. Organizing our Test Files

I like putting test files as close to the implementation as possible. This makes it very easy to tell which endpoints are covered by tests and which ones are not. I also like creating one test file per route because the tests for a single route can get quite big over time.

So let's put our test files right inside src/todos/ like this:

src/
  errors/
    ValidationError.js
  todos/
    __http-tests__
      create-todo.test.js # Like TodosService#createTodo(todoData)
      list-todos.test.js  # Like TodosService#listTodos()
    routes.js
    TodosService.js
  main.js
package.json
yarn.lock

Our test helpers should go into a folder at the root of the workspace so that they can easily be required from anywhere. Maybe even from a different package/repository:

src/
  errors/
    ValidationError.js
  todos/
    __http-tests__
      create-todo.test.js
      list-todos.test.js
    routes.js
    TodosService.js
  main.js
test-helpers/
  test-db.js
  test-server.js
package.json
yarn.lock

Ok, I think we're ready to start writing our testDb utility:

2.3. Writing the testDb Utility

test-helpers/test-db.js
/* eslint-env mocha */
const { MongoClient } = require('mongodb')
exports.useInTest = function() {
    before(async function connectToTestDB() {
        const mongoClient = await MongoClient.connect(
            'mongodb://localhost:27017',
            { useNewUrlParser: true }
        )
        const db = mongoClient.db('todoman-test')
        this.mongoClient = mongoClient
        this.db = db
    })
    beforeEach(function dropTestDB() {
        return this.db.dropDatabase()
    })
    after(function disconnectTestDB() {
        return this.mongoClient.close()
    })
}

this allows us expose the mongo client's db object to our tests. Apart from that the code should be quite straight-forward.

If we call testDb.useInTest(), we will:

1. Before all tests, connect to the test database and expose this.db
2. Before each test, reset the database
3. After all tests, disconnect from the database

Great, now we can use test-helpers/test-db.js from within any test file to gain access to a test database!

Now let's get the same thing working for the server!

2.4. Writing the testServer Utility

We will need to install two packages to help us with our testServer utility:

1. axios which is a great promise-based http client that we will setup and expose to our tests as this.api once the server is running.
2. get-port which allows us to find an available port on our machine. Our test server needs to listen on some port. We don't want to hard-code the port otherwise our tests will fail if something else is already listening on that port. So we will simply find any available port at runtime to listen to!

$ yarn add axios get-port --dev

Alright! Now let's code up our testServer:

test-helpers/test-server.js
/* eslint-env mocha */
const axios = require('axios')
const spawn = require('child_process').spawn
const getPort = require('get-port')
// TEST_ENV contains the environment variables for our test server
const TEST_ENV = {
    PORT: undefined, // Assigned at runtime
    MONGODB_URI: 'mongodb://localhost:27017',
    MONGODB_DB: 'todoman-test'
}
// spawnServer is a utility function that starts a test server with a given set
// of environment variables and returns a promise that resolves to the
// ChildProcess of the server once it is reachable at its base URL
function spawnServer(env) {
    return new Promise((resolve, reject) => {
        const server = spawn('node', ['src/main'], { env })
        // Pipe the test server's stdout and stderr to our main process so that we
        // can see console.logs and errors of our server when running our tests
        server.stdout.pipe(process.stdout)
        server.stderr.pipe(process.stderr)
        server.on('error', reject)
        // Wait for the server to be reachable
        return waitForURLReachable(`http://localhost:${env.PORT}`)
            .then(() => resolve(server))
            .catch(reject)
    })
}
// waitForURLReachable is a utility function that tries to GET a URL until it
// succeeds. It will throw an error if it cannot reach the URL within the
// provided `opts.timeout` (default: 1000ms)
async function waitForURLReachable(url, { timeout = 1000 } = {}) {
    const timeoutThreshold = Date.now() + timeout
    while (true) {
        try {
            await axios.get(url)
            return true
        } catch (err) {
            if (Date.now() > timeoutThreshold) {
                throw new Error(`URL ${url} not reachable after ${timeout}ms`)
            }
            await new Promise(resolve => setTimeout(resolve, 100))
        }
    }
}
exports.useInTest = function() {
    before(async function startTestServer() {
        // The test server's environment variables should be set to TEST_ENV as
        // declared at the top of the file, but:
        // 1. Assign PATH to the user's PATH so that the `node` binary can be found
        // 2. Assign PORT to a random free port
        const env = Object.assign({}, TEST_ENV, {
            PATH: process.env.PATH,
            PORT: await getPort()
        })
        // Use our utility function that we created above to spawn the test server
        const testServer = await spawnServer(env)
        // Create an axios instance that is configured to use the test server as its
        // base URL and expose it as `this.api`. This allows us to easily make
        // requests like `this.api.get('/todos')` from within our test files
        const api = axios.create({ baseURL: `http://localhost:${env.PORT}` })
        this.testServer = testServer
        this.api = api
    })
    after(function stopTestServer() {
        // After all tests, stop the test server...
        this.testServer.kill()
        // ...and wait for it to shut down
        return new Promise(resolve =>
            this.testServer.on('close', () => resolve())
        )
    })
}

We can now start writing our first test and put our little testing utilities into action.

2.5. Writing Our First Test for GET /todos

Before we get into the actual code, we need to once again install some dependencies:

1. mocha which will be our test runner of choice. It's a bit older but works very well. The this context as introduced earlier is brilliant. The test output is very pretty and it just works pretty much without any configuration.
2. chai is the traditional sidekick to mocha and will be our assertion library. Some people dislike its chainable interface (mostly for ideological reasons if you ask me) but it suits this assertion library very well and lets you easily compose multiple assertions in a way that reads like plain english: expect(todo).to.have.property('title').that.is.a('string') – beautiful!

$ yarn add mocha chai --dev

Let's first test GET /todos because we only need to write a single test for that route.

src/todos/__http-tests__/list-todos.test.js
/* eslint-env mocha */
const { expect } = require('chai')
const testServer = require('../../../test-helpers/test-server')
const testDb = require('../../../test-helpers/test-db')
describe('GET /todos', function() {
    testServer.useInTest()
    testDb.useInTest()
    it('responds with 200 { todos }', async function() {
        const api = this.api
        // Create three todos
        await api.post('/todos', { title: 'Todo 1' })
        await api.post('/todos', { title: 'Todo 2' })
        await api.post('/todos', { title: 'Todo 3' })
        // Make the actual request to GET /todos
        const response = await api.get('/todos')
        // Assert status code 200
        expect(response).to.have.property('status', 200)
        // Assert that all three todos are included
        expect(response)
            .to.have.nested.property('data.todos')
            .that.is.an('array')
            .with.lengthOf(3)
        const todos = response.data.todos
        // Assert that every todo contains all desired fields
        todos.forEach(todo => {
            expect(todo)
                .to.have.property('title')
                .that.is.a('string')
            expect(todo).to.have.property('completed', false)
            expect(todo)
                .to.have.property('createdAt')
                .that.is.a('string')
            expect(todo)
                .to.have.property('updatedAt')
                .that.is.a('string')
        })
        // Assert that todos are listed in order of creation
        expect(todos.map(todo => todo.title)).to.deep.equal([
            'Todo 1',
            'Todo 2',
            'Todo 3'
        ])
    })
})

Note how we are using testServer.useInTest() and testDb.useInTest() at the beginning of the describe block. This is the piece of magic that gets our testing server and testing database ready for us.

We only wrote a single test but it is pretty thorough (which is good!). We are testing that GET /todos:

1. Returns the correct status code of 200
2. Returns all todos as an array
3. Returns all todos as a property todos of the response body
4. Returns every todo with all desired fields
5. Returns all todos in the correct order

That seems pretty solid!

To run our tests, we will first add a new script test to our package.json:

package.json
{
    "name": "todoman-backend",
    "scripts": {
        "dev": "nodemon src/main",
        "test": "mocha  \"src/**/*.test.js\""
    },
    "dependencies": {
        "body-parser": "^1.18.3",
        "express": "^4.16.3",
        "mongodb": "^3.1.1"
    },
    "devDependencies": {
        "axios": "^0.18.0",
        "chai": "^4.1.2",
        "get-port": "^4.0.0",
        "mocha": "^5.2.0",
        "nodemon": "^1.18.3"
    },
    "prettier": {
        "singleQuote": true,
        "semi": false
    }
}

Now you can run yarn test to run all tests:

$ yarn test
yarn run v1.6.0
warning package.json: No license field
$ mocha  "src/**/*.test.js"
  GET /todos
api-server listening on port 53477
    ✓ responds with 200 { todos } (94ms)
  1 passing (788ms)
✨  Done in 1.48s.

Great, our test is passing! :)

Now you should go into your implementation code and try to break every single one of the assertions to verify that the test actually works. E.g. return a different status code and make sure that the test now fails. This is super-important because, although test code is usually much more straight-forward than implementation codee, there is always the possibility of bugs creeping in and your tests passing even though they shouldn't be.

You will be able to skip this step once we adopt a test-driven workflow in part 3 but always try break your tests if you wrote them after your implementation code!

Ok, let's write some tests for our second route: POST /todos.

2.5. Testing the POST /todos Endpoint

This will be a bit more involved so let's take another small step back first and think about all the things we want to test. We should test that POST /todos:

• Responds with 400 ValidationError if title is missing
• Responds with 400 ValidationError if title is empty
• Responds with 400 ValidationError if title is not a string
• Responds with 200 { todo }
• Saves todo to the database
• Trims title from input

As you can see we have 3 tests planned for input validation. Honestly this might be a little bit much for an integration test. If our validation was complex enough to warrant a separate file, we would instead write only one general-purpose integration test for the input validation and test our validator separately with unit tests. But this is an optimization we can take care of at a later point if it really bothers us and in general I don't think it's too bad to test validation in integration tests. The tests run a bit slower this way but we get to move a bit faster!

There is however something else I would like to do about our input validation tests. I want to create a little utility that makes it easier to write them. Because not only do we want to assert that our API responds with the correct status code (400) and response body (ValidationError) but we also want to make sure that the correct error message is being delivered for any given failed validation (e.g. "title": "required" if title is missing). I imagine an API something like this:

it('responds with 400 ValidationError if title is missing', async function() {
    const api = this.api
    const request = api.post('/todos', {})
    return assertResponse.isValidationError(request, { title: 'required' })
})

That would be pretty neat, ey? It will make it a breeze to test any endpoints for validation errors in the future! :)

Let's create a new test helper for this. For it, we will want to use a little chai plugin called chai-as-promised that will allows us to make assertions about promises:

$ yarn add chai-as-promised --dev

Now let's write our helper:

test-helpers/assert-response.js
const { expect } = require('chai').use(require('chai-as-promised'))
exports.isValidationError = async function isValidationError(
    request,
    invalidFields
) {
    // The `.eventually` is provided by chai-as-promised. We are doing two things
    // here:
    // - Expecting the axios request to be rejected
    // - Awaiting the assertion resolves to the error of the rejected promise
    const error = await expect(request).to.eventually.be.rejected
    // Axios errors contain the response as `error.response`
    const response = error.response
    // Assert that we respond with status code 400
    expect(response).to.have.property('status', 400)
    // Assert that our response body is a validation error
    expect(response).to.have.nested.property(
        'data.error.name',
        'ValidationError'
    )
    // Assert that the provided invalidFields are included in the response
    Object.entries(invalidFields).forEach(([fieldName, expectedFieldError]) => {
        expect(response).to.have.nested.property(
            `data.error.invalidFields.${fieldName}`,
            expectedFieldError
        )
    })
}

When writing tests there is always a fine line between keeping your tests extremely simple (don't be afraid to copy-paste) but also easy-to-write-and-read and maintainable. If you ever find yourself writing many assertions over and over that are a bit hard to read, do consider extracting them into a utility function that you can reuse – like this one (isValidationError).

Ok, now we have everything we need to create our next tests:

src/todos/__http-tests__/create-todo.test.js
/* eslint-env mocha */
const { expect } = require('chai')
const testServer = require('../../../test-helpers/test-server')
const testDb = require('../../../test-helpers/test-db')
const assertResponse = require('../../../test-helpers/assert-response')
describe('POST /todos', function() {
    testServer.useInTest()
    testDb.useInTest()
    it('responds with 400 ValidationError if title is missing', async function() {
        const api = this.api
        const request = api.post('/todos', {})
        return assertResponse.isValidationError(request, { title: 'required' })
    })
    it('responds with 400 ValidationError if title is empty', async function() {
        const api = this.api
        const request = api.post('/todos', { title: '' })
        return assertResponse.isValidationError(request, { title: 'required' })
    })
    it('responds with 400 ValidationError if title is not a string', async function() {
        const api = this.api
        const request = api.post('/todos', { title: 123 })
        return assertResponse.isValidationError(request, {
            title: 'must be a string'
        })
    })
    it('responds with 200 { todo }', async function() {
        const api = this.api
        const response = await api.post('/todos', { title: 'My Test Todo' })
        expect(response).to.have.property('status', 200)
        expect(response.data)
            .to.have.property('todo')
            .that.is.an('object')
        expect(response.data.todo)
            .to.have.property('_id')
            .that.is.a('string')
        expect(response.data.todo).to.have.property('title', 'My Test Todo')
        expect(response.data.todo).to.have.property('completed', false)
        expect(response.data.todo)
            .to.have.property('createdAt')
            .that.is.a('string')
        expect(new Date(response.data.todo.createdAt).valueOf()).to.be.closeTo(
            Date.now(),
            1000
        )
        expect(response.data.todo)
            .to.have.property('updatedAt')
            .that.is.a('string')
        expect(response.data.todo.updatedAt).to.equal(
            response.data.todo.createdAt
        )
    })
    it('saves todo to the database', async function() {
        const api = this.api
        const db = this.db
        await api.post('/todos', { title: 'My Test Todo' })
        const [latestTodo] = await db
            .collection('todos')
            .find({})
            .sort({ _id: -1 })
            .toArray()
        expect(latestTodo).to.be.ok
        expect(latestTodo).to.have.property('title', 'My Test Todo')
        expect(latestTodo).to.have.property('completed', false)
        expect(latestTodo)
            .to.have.property('createdAt')
            .that.is.an.instanceOf(Date)
        expect(latestTodo.createdAt.valueOf()).to.be.closeTo(Date.now(), 1000)
        expect(latestTodo)
            .to.have.property('updatedAt')
            .that.is.an.instanceOf(Date)
        expect(latestTodo.updatedAt).to.deep.equal(latestTodo.createdAt)
    })
    it('trims title from input', async function() {
        const api = this.api
        const response = await api.post('/todos', { title: '  My Test Todo ' })
        expect(response).to.have.property('status', 200)
        expect(response.data.todo).to.have.property('title', 'My Test Todo')
    })
})

And that's it! Let's run all our tests:

$ yarn test
yarn run v1.6.0
warning package.json: No license field
$ mocha "src/**/*.test.js"
  POST /todos
api-server listening on port 53843
    ✓ responds with 400 ValidationError if title is missing
    ✓ responds with 400 ValidationError if title is empty
    ✓ responds with 400 ValidationError if title is not a string
    ✓ responds with 200 { todo } (40ms)
    ✓ saves todo to the database (40ms)
    ✓ trims title from input
  GET /todos
api-server listening on port 53862
    ✓ responds with 200 { todos } (66ms)
  7 passing (2s)
✨  Done in 2.42s.

Again, now you should go back and test the tests by altering your implementation code and making sure that the tests fail. After you've done that, you're done with part 2!

You now have a little todo app with integration tests that cover pretty much your entire codebase! Nice! :)

3. Adopting a Test-driven Workflow When Working on REST APIs

So far our todoman app doesn't support checking off todos yet. Damn! In this section we will add this functionality in a test-driven manner. As mentioned in the intro, test-driven workflows work very well for creating endpoints. In this section we will code up this endpoint but write our tests first, step-by-step.

3.1. Why Write Tests First?

Writing tests first has the awesome benefit that you know your tests work because they will pass as soon as the implementation works and fail as long as they don't. Also it allows you to "think first" about all the requirements a specific endpoint should meet. When you're doing the actual implementation you can relax your mind a little bit because your specs are in place and you just need to get your tests to pass one-by-one.

3.2. Requirements for PATCH /todos/:todoId

First, let's again think about the requirements our route should meet. It will be available at PATCH /todos/:todoId and we should make sure that it:

• Responds with 404 ResourceNotFoundError { Todo } if no todo exists with given id
• Responds with 400 ValidationError if completed is not a boolean
• Responds with 400 ValidationError if title is not a string
• Responds with 400 ValidationError if title is an empty string
• Responds with 200 { todo }
• Only updates fields that are sent in the request body
• Updates todo in the database
• Trims title from input

First thing to notice here is that we're going to introduce a new error. It will look something like this:

ResourceNotFoundError {
  name: 'ResourceNotFoundError';
  resource: string;
}

resource will refer to the name of the resource that couldn't be found. In our case Todo. It could also be User or BlogPost or TodoList or something like that. It would also be great if we could make assertions against requests that we expect to reject with ResourceNotFoundErrors like this:

it('responds with 404 ResourceNotFoundError { Todo } if no todo exists with given id', function() {
    const api = this.api
    const request = api.patch('/todos/1', {})
    return assertResponse.isResourceNotFoundError(request, 'Todo')
})

No problem! Let's add another assertion helper for the ResourceNotFoundError:

test-helpers/assert-response.js
const { expect } = require('chai').use(require('chai-as-promised'))
exports.isValidationError = async function isValidationError(
    request,
    invalidFields
) {
    const error = await expect(request).to.eventually.be.rejected
    const response = error.response
    expect(response).to.have.property('status', 400)
    expect(response).to.have.nested.property(
        'data.error.name',
        'ValidationError'
    )
    Object.entries(invalidFields).forEach(([fieldName, expectedFieldError]) => {
        expect(response).to.have.nested.property(
            `data.error.invalidFields.${fieldName}`,
            expectedFieldError
        )
    })
}
exports.isResourceNotFoundError = async function isResourceNotFoundError(
    request,
    resource
) {
    const error = await expect(request).to.eventually.be.rejected
    const response = error.response
    expect(response).to.have.property('status', 404)
    expect(response).to.have.nested.property(
        'data.error.name',
        'ResourceNotFoundError'
    )
    expect(response).to.have.nested.property('data.error.resource', resource)
}

That should save us some time when we need to assert for ResourceNotFoundErrors in the future.

Now let's write out our tests. :)

3.3. Writing the Tests for PATCH /todos/:todoId

src/todos/__http-tests__/patch-todo.test.js
/* eslint-env mocha */
const { expect } = require('chai')
const { ObjectId } = require('mongodb')
const testServer = require('../../../test-helpers/test-server')
const testDb = require('../../../test-helpers/test-db')
const assertResponse = require('../../../test-helpers/assert-response')
// createTodo is a utility function that lets us create a todo via HTTP request
// and returns the todo as it was saved to the database
async function createTodo(mochaContext, todoData = {}) {
    const { api, db } = mochaContext
    todoData = Object.assign({}, todoData, { title: 'My Todo' })
    const response = await api.post('/todos', todoData)
    const todoId = new ObjectId(response.data.todo._id)
    return db.collection('todos').findOne({ _id: todoId })
}
describe('PATCH /todos/:todoId', function() {
    testServer.useInTest()
    testDb.useInTest()
    it('responds with 404 ResourceNotFoundError { Todo } if no todo exists with given id', async function() {
        const api = this.api
        // Test with non-Object ID
        const request1 = api.patch('/todos/1', { completed: true })
        await assertResponse.isResourceNotFoundError(request1, 'Todo')
        // Test with non-existent Object ID
        const request2 = api.patch('/todos/5b72ecfdbf16f1384b053639', {
            completed: true
        })
        await assertResponse.isResourceNotFoundError(request2, 'Todo')
    })
    it('responds with 400 ValidationError if completed is not a boolean', async function() {
        const api = this.api
        const todo = await createTodo(this)
        const request = api.patch(`/todos/${todo._id}`, { completed: 'true' })
        return assertResponse.isValidationError(request, {
            completed: 'must be a boolean'
        })
    })
    it('responds with 400 ValidationError if title is not a string', async function() {
        const api = this.api
        const todo = await createTodo(this)
        const request = api.patch(`/todos/${todo._id}`, { title: 123 })
        return assertResponse.isValidationError(request, {
            title: 'must be a string'
        })
    })
    it('responds with 400 ValidationError if title is an empty string', async function() {
        const api = this.api
        const todo = await createTodo(this)
        const request = api.patch(`/todos/${todo._id}`, { title: '' })
        return assertResponse.isValidationError(request, {
            title: 'cannot be an empty string'
        })
    })
    it('responds with 200 { todo }', async function() {
        const api = this.api
        const todo = await createTodo(this)
        const response = await api.patch(`/todos/${todo._id}`, {
            completed: true,
            title: 'My Updated Todo'
        })
        expect(response).to.have.property('status', 200)
        expect(response.data)
            .to.have.property('todo')
            .that.is.an('object')
        expect(response.data.todo)
            .to.have.property('_id')
            .that.is.a('string')
        expect(response.data.todo).to.have.property('title', 'My Updated Todo')
        expect(response.data.todo).to.have.property('completed', true)
        expect(response.data.todo)
            .to.have.property('createdAt')
            .that.is.a('string')
        expect(response.data.todo)
            .to.have.property('updatedAt')
            .that.is.a('string')
        expect(new Date(response.data.todo.updatedAt).valueOf()).to.be.closeTo(
            Date.now(),
            1000
        )
        expect(response.data.todo.updatedAt).to.not.equal(
            response.data.todo.createdAt
        )
    })
    it('only updates fields that are sent in the request body', async function() {
        const api = this.api
        const db = this.db
        const todo = await createTodo(this, { title: 'My Todo' })
        await api.patch(`/todos/${todo._id}`, { completed: true })
        const todoV2 = await db.collection('todos').findOne({ _id: todo._id })
        expect(todoV2).to.have.property('completed', true)
        expect(todoV2).to.have.property('title', 'My Todo')
        await api.patch(`/todos/${todo._id}`, { title: 'Updated Title' })
        const todoV3 = await db.collection('todos').findOne({ _id: todo._id })
        expect(todoV3).to.have.property('completed', true)
        expect(todoV3).to.have.property('title', 'Updated Title')
    })
    it('updates todo in the database', async function() {
        const api = this.api
        const db = this.db
        const todo = await createTodo(this)
        await api.patch(`/todos/${todo._id}`, {
            completed: true,
            title: 'My Updated Todo'
        })
        const updatedTodo = await db
            .collection('todos')
            .findOne({ _id: todo._id })
        expect(updatedTodo).to.have.property('completed', true)
        expect(updatedTodo).to.have.property('title', 'My Updated Todo')
        expect(updatedTodo.updatedAt.valueOf()).to.be.closeTo(Date.now(), 1000)
        expect(updatedTodo.updatedAt).to.not.deep.equal(updatedTodo.createdAt)
    })
    it('trims title from input', async function() {
        const api = this.api
        const todo = await createTodo(this)
        const response = await api.patch(`/todos/${todo._id}`, {
            completed: true,
            title: '  My Updated Todo '
        })
        expect(response).to.have.property('status', 200)
        expect(response.data.todo).to.have.property('title', 'My Updated Todo')
    })
})

Ok. If you try to run them now, all tests will fail:

$ yarn test -g "PATCH /todos/:todoId"
yarn run v1.6.0
warning package.json: No license field
$ mocha  "src/**/*.test.js" -g 'PATCH /todos/:todoId'
  PATCH /todos/:todoId
api-server listening on port 57508
    1) responds with 404 ResourceNotFoundError { Todo } if no todo exists with given id
    2) responds with 400 ValidationError if completed is not a boolean
    3) responds with 400 ValidationError if title is not a string
    4) responds with 400 ValidationError if title is an empty string
    5) responds with 200 { todo }
    6) only updates fields that are sent in the request body
    7) updates todo in the database
    8) trims title from input
  0 passing (1s)
  8 failing
  1) PATCH /todos/:todoId
       responds with 404 ResourceNotFoundError { Todo } if no todo exists with given id:
     AssertionError: expected { Object (status, statusText, ...) } to have nested property 'data.error.name'
      at Object.isResourceNotFoundError (test-helpers/assert-response.js:33:35)
      at process._tickCallback (internal/process/next_tick.js:68:7)
  2) PATCH /todos/:todoId
       responds with 400 ValidationError if completed is not a boolean:
      AssertionError: expected { Object (status, statusText, ...) } to have property 'status' of 400, but got 404
      + expected - actual
      -404
      +400
      at Object.isValidationError (test-helpers/assert-response.js:11:28)
      at process._tickCallback (internal/process/next_tick.js:68:7)
  3) PATCH /todos/:todoId
       responds with 400 ValidationError if title is not a string:
      AssertionError: expected { Object (status, statusText, ...) } to have property 'status' of 400, but got 404
      + expected - actual
      -404
      +400
      at Object.isValidationError (test-helpers/assert-response.js:11:28)
      at process._tickCallback (internal/process/next_tick.js:68:7)
  4) PATCH /todos/:todoId
       responds with 400 ValidationError if title is an empty string:
      AssertionError: expected { Object (status, statusText, ...) } to have property 'status' of 400, but got 404
      + expected - actual
      -404
      +400
      at Object.isValidationError (test-helpers/assert-response.js:11:28)
      at process._tickCallback (internal/process/next_tick.js:68:7)
  5) PATCH /todos/:todoId
       responds with 200 { todo }:
     Error: Request failed with status code 404
      at createError (node_modules/axios/lib/core/createError.js:16:15)
      at settle (node_modules/axios/lib/core/settle.js:18:12)
      at IncomingMessage.handleStreamEnd (node_modules/axios/lib/adapters/http.js:201:11)
      at endReadableNT (_stream_readable.js:1081:12)
      at process._tickCallback (internal/process/next_tick.js:63:19)
  6) PATCH /todos/:todoId
       only updates fields that are sent in the request body:
     Error: Request failed with status code 404
      at createError (node_modules/axios/lib/core/createError.js:16:15)
      at settle (node_modules/axios/lib/core/settle.js:18:12)
      at IncomingMessage.handleStreamEnd (node_modules/axios/lib/adapters/http.js:201:11)
      at endReadableNT (_stream_readable.js:1081:12)
      at process._tickCallback (internal/process/next_tick.js:63:19)
  7) PATCH /todos/:todoId
       updates todo in the database:
     Error: Request failed with status code 404
      at createError (node_modules/axios/lib/core/createError.js:16:15)
      at settle (node_modules/axios/lib/core/settle.js:18:12)
      at IncomingMessage.handleStreamEnd (node_modules/axios/lib/adapters/http.js:201:11)
      at endReadableNT (_stream_readable.js:1081:12)
      at process._tickCallback (internal/process/next_tick.js:63:19)
  8) PATCH /todos/:todoId
       trims title from input:
     Error: Request failed with status code 404
      at createError (node_modules/axios/lib/core/createError.js:16:15)
      at settle (node_modules/axios/lib/core/settle.js:18:12)
      at IncomingMessage.handleStreamEnd (node_modules/axios/lib/adapters/http.js:201:11)
      at endReadableNT (_stream_readable.js:1081:12)
      at process._tickCallback (internal/process/next_tick.js:63:19)
error Command failed with exit code 8.

Now it's time to get these tests to pass!

3.4. Getting the Tests to Pass

We'll begin by adding our new error to src/errors/ and handling it in our error handler:

src/errors/ResourceNotFoundError.js
module.exports = class ResourceNotFoundError extends Error {
    constructor(resource) {
        super()
        this.name = 'ResourceNotFoundError'
        this.resource = resource
    }
}

src/main.js
const express = require('express')
const { MongoClient } = require('mongodb')
const bodyParser = require('body-parser')
const ValidationError = require('./errors/ValidationError')
const ResourceNotFoundError = require('./errors/ResourceNotFoundError')
const PORT = process.env.PORT || 3000
const MONGODB_URI = process.env.MONGODB_URI || 'mongodb://localhost:27017'
const MONGODB_DB = process.env.MONGODB_DB || 'todoman'
main()
async function main() {
    const mongoClient = await MongoClient.connect(MONGODB_URI, {
        useNewUrlParser: true
    })
    const db = mongoClient.db(MONGODB_DB)
    const app = express()
    app.use(bodyParser.json())
    app.use(function attachDb(req, res, next) {
        req.db = db
        next()
    })
    app.get('/', function(req, res, next) {
        res.status(200).json({ name: 'todoman-backend' })
    })
    app.use(require('./todos/routes'))
    app.use(function handleErrors(err, req, res, next) {
        if (err instanceof ValidationError) {
            return res.status(400).json({ error: err })
        }
        if (err instanceof ResourceNotFoundError) {
            return res.status(404).json({ error: err })
        }
        next(err)
    })
    app.listen(PORT, err => {
        if (err) {
            throw err
        }
        // eslint-disable-next-line no-console
        console.log(`api-server listening on port ${PORT}`)
    })
}