HomeAboutContactBlog
ProjectsHire Us

Integration Testing In Go Using Testcontainers

Shaibu Shaibu
Software Engineer
Published: Jan 30, 2024

Photo by Frank Mckenna on unsplash

To accomplish a particular task, every application usually would need to interact with other applications and services. Whether it’s a database (SQL and NoSQL) to store and retrieve data, payment services to accept payments in the application or a queuing service like AWS SQS, it could even be another service provided by a different team in the same organization, and many more. As software engineers, we need to write automated tests (unit tests, integration tests, end-to-end tests).

Since applications depend on one or more of these services to function, it is important to test that the services will/still work as expected and that the application is using them as it should. The purpose of integration tests is to test that the “integration” between applications and these services works as expected and that we can confidently ship our application.

In this post, you will:

  • Write a simple CRUD program that uses a repository to store and retrieve notes in a PostgreSQL database
  • Use Redis to cache notes to reduce the amount of hits to our Postgres database.
  • Write integration tests in our Golang projects using Testcontainers.

If you’d like to follow through with the implementation of the repository and how we write the integration tests, keep reading, and if you’d prefer to jump straight to the test you can get the full source code here, go through the app.go to understand the program we want to test, and then, skip to the integration test section of this tutorial.

Prerequisites

To complete this tutorial, you’ll need the following:

  • A good understanding of the Go programming language. If you are new to Go, a good source to learn Go would be https://go.dev/learn/ and https://gobyexample.com/
  • Some understanding of automated tests in Go
  • Golang (my Go version is 1.21.3 for this but perhaps you can use a higher version)
  • Docker (my docker version is 24.0.6)
  • A code editor like VScode or Goland
  • Understanding of SQL basics

Optionally:

  • Some understanding of Gorm or any ORM at all
  • Some understanding of Redis

Step 1 - Setup your project structure

First, create a directory for your project:

$ mkdir integration_testing_with_test_cotainers_go

Change into the new directory:

$ cd integration_testing_with_test_cotainers_go

Initialize a Go module in your directory:

$ go mod init [modulename]

You can replace modulename with any name - preferably a Github url to the repository hosting this module. In my case, modulename is github.com/Shaibujnr/integration_testing_with_test_containers_go. Next we need to install dependencies in our module.

$ go get -u gorm.io/gorm
$ go get -u gorm.io/driver/postgres
$ go get -u github.com/redis/go-redis/v9
$ go get -u github.com/stretchr/testify
$ go get -u github.com/testcontainers/testcontainers-go
$ go get -u github.com/testcontainers/testcontainers-go/modules/postgres
$ go get -u github.com/testcontainers/testcontainers-go/modules/redis
$ go get -u github.com/DATA-DOG/go-sqlmock

Create the app package and files:

$ mkdir app
$ cd app
$ touch app.go
$ touch app_test.go

now open the project in your preferred editor.

Step 2 - Define our database model

Open the app.go file, declare the package, and import the dependencies at the top of the file. Some IDEs automatically import and remove unused imports.

package app

import (
	"context"
	"errors"
	"fmt"
	"github.com/redis/go-redis/v9"
	"gorm.io/gorm"
	"log/slog"
	"strconv"
	"time"
)

Let’s recap what we are trying to build:

  • A program that uses a repository to save and retrieve notes
  • The repository saves our notes in a Postgres database
  • The program uses a read-through caching strategy which means, whenever we want to get a particular note using the repository, the repository first checks the cache (Redis) for the note, if it finds it, it loads the note from the cache and returns it to the caller (without hitting our Postgres database) and if it doesn’t find the note, it loads it from Postgres and stores it in the cache before returning it to the caller.
  • Updating the note invalidates the cache (deletes the note from the cache) so that retrieving the note will result in caching the latest version of the note.

Now let’s model our note:

type Note struct {
	gorm.Model
	// Title is the title of the note.
	Title string `gorm:"column:title;not null;unique"`
	// Content is the content of the note.
	Content string `gorm:"column:content;not null"`
}

Our note is a struct with a title and content. Using tags, we specify that both columns should not be nullable and that every note title must be unique. The embedded gorm.Model will simply add four (4) additional columns for us:

  • ID integer primary key
  • CreatedAt datetime column
  • UpdatedAt datetime column
  • DeletedAt datetime column

Step 3 - Define and Implement our Repository

Our repository will have 4 methods

  • SaveNote to save the note in the database. This method will create or update the note
  • GetNoteById to retrieve a note by its id
  • GetNoteByTitle to retrieve a note by it’s title
  • DeleteNote to delete a note by it’s id.

Define the repository struct:

type NoteRepository struct {
	db    *gorm.DB
	redis *redis.Client
}

We defined a NoteRepository struct with two properties db and redis. The db is an instance of gorm.DB and it provides us a connection to the postgres database and we can use this to query the database. The redis, similar to the db provides us a connection to the redis server.

Now, add the following factory function we can use to create a new repository.

func NewNoteRepository(db *gorm.DB, rd *redis.Client) *NoteRepository {
	return &NoteRepository{
		db:    db,
		redis: rd,
	}
}

Now, before we go ahead to define the main methods for our repository, let’s define some helper methods just for a clean implementation.

func (repo *NoteRepository) convertMapToNote(noteMap map[string]string) (Note, error) {
	// convert the id from string to integer
	noteID, err := strconv.Atoi(noteMap["id"])
	if err != nil {
		return Note{}, err
	}
	// parse the created_at time string
	createdAt, err := time.Parse(time.RFC3339Nano, noteMap["created_at"])
	if err != nil {
		return Note{}, err
	}
	// parse the updated_at time string
	updatedAt, err := time.Parse(time.RFC3339Nano, noteMap["updated_at"])
	if err != nil {
		return Note{}, err
	}

	return Note{
		Model: gorm.Model{
			ID:        uint(noteID),
			CreatedAt: createdAt,
			UpdatedAt: updatedAt,
		},
		Title:   noteMap["title"],
		Content: noteMap["content"],
	}, nil
}

When we fetch data from Redis using HGetAll as you would see, what we get is a map with both key and value as strings. convertMapToNote is a utility method that converts the map into a note object also converting from string to int and to Time where necessary. It returns any error that arises from converting and parsing this data to the caller.

Since we have two ways of retrieving notes from the store (by id and by title ), we need to be able to retrieve a note by id and by title on both Postgres and Redis. On postgres we can easily do that with SQL , however, on redis, the easiest method is to store the same data under two different keys. So if we have a note object where the id is 4, the title is “Hello”, the content is “World”, we would store it in redis as a hash first under the key notes:4 and notes:Hello. When we need to retrieve the note by its id, we use the notes:4 key and when we need to retrieve the note by its title, we use the notes:Hello key.

The next utility method is a function to cache a note under these two keys:

func (repo *NoteRepository) cacheNote(note Note) error {
	idHashKey := fmt.Sprintf("notes:%d", note.ID)
	titleHashKey := fmt.Sprintf("notes:%s", note.Title)
	noteMap := map[string]any{
		"id":         note.ID,
		"title":      note.Title,
		"content":    note.Content,
		"created_at": note.CreatedAt,
		"updated_at": note.UpdatedAt,
	}
	for key, val := range noteMap {
		err := repo.redis.HSet(context.Background(), idHashKey, key, val).Err()
		if err != nil {
			return err
		}
		err = repo.redis.HSet(context.Background(), titleHashKey, key, val).Err()
		if err != nil {
			return err
		}
	}
	return nil
}

We construct our two keys idHashKey and titleHashKey and then, we construct our noteMap which is the note data we wish to store under those two keys in Redis. Then we loop through the map and set each key pair as a field and value under both keys using Redis’ HSet command.

The next utility method is for getting a note from the cache by its id:

 func (repo *NoteRepository) getNoteFromCache(id int) *Note {
	result := repo.redis.HGetAll(context.Background(), fmt.Sprintf("notes:%d", id)).Val()
	if len(result) == 0 {
		return nil
	}
	note, err := repo.convertMapToNote(result)
	if err != nil {
		panic(err)
	}
	return &note
}

Here, we construct the id key and we perform a HGetAll to get all the fields and values set under that key. If we have a note under that key, then result will be a map of strings and we can use our utility method convertMapToNote to convert the map to a note and return it. If the key doesn’t exist in the cache, then result will be an empty map and we return nil.

func (repo *NoteRepository) getNoteByTitleFromCache(title string) *Note {
	result := repo.redis.HGetAll(context.Background(), fmt.Sprintf("notes:%s", title)).Val()
	if len(result) == 0 {
		return nil
	}
	note, err := repo.convertMapToNote(result)
	if err != nil {
		panic(err)
	}
	return &note
}

Same as getNoteFromCache but this time, using the title of the note.

And our final utility method:

func (repo *NoteRepository) deleteFromCache(note Note) error {
	keysToDelete := make([]string, 0)
	if note.ID > 0 {
		keysToDelete = append(keysToDelete, fmt.Sprintf("notes:%d", note.ID))
	}
	if note.Title != "" {
		keysToDelete = append(keysToDelete, fmt.Sprintf("notes:%s", note.Title))
	}
	return repo.redis.Del(context.Background(), keysToDelete...).Err()
}

This will delete the note from the cache by deleting both the id key and the title key from Redis.

We can now move on to the actual implementation of our repository.

func (repo *NoteRepository) SaveNote(note *Note) error {
	err := repo.deleteFromCache(*note)
	if err != nil {
		return err
	}
	result := repo.db.Save(note)
	if result.Error != nil {
		return result.Error
	}
	return nil
}

SaveNote method takes a note, invalidates the cache by deleting the note from the cache, and then saves the note in the postgres database. Gorm’s Save method will create or update the note if it exists.

func (repo *NoteRepository) GetNoteById(id int) *Note {
	cachedNote := repo.getNoteFromCache(id)
	if cachedNote != nil {
		return cachedNote
	}
	note := Note{Model: gorm.Model{ID: uint(id)}}
	result := repo.db.First(&note)
	if result.Error != nil {
		if errors.Is(result.Error, gorm.ErrRecordNotFound) {
			return nil
		}
		panic(result.Error)
	}
	err := repo.cacheNote(note)
	if err != nil {
		panic(err)
	}
	return &note
}

GetNoteById will first try to get the note from the cache, if it’s not in the cache, it tries to get it from the database, if the note does not exist, it returns nil and if it does exist, it first saves the note in the cache so that subsequent calls would use the cache, and then it returns the note to the caller.

func (repo *NoteRepository) GetNoteByTitle(title string) *Note {
	cachedNote := repo.getNoteByTitleFromCache(title)
	if cachedNote != nil {
		return cachedNote
	}
	note := Note{Title: title}
	result := repo.db.First(&note)
	if result.Error != nil {
		if errors.Is(result.Error, gorm.ErrRecordNotFound) {
			return nil
		}
		panic(result.Error)
	}
	err := repo.cacheNote(note)
	if err != nil {
		panic(err)
	}
	return &note
}

GetNoteByTitle is very similar to GetNoteById except it retrieves the note by its title.

Finally:

func (repo *NoteRepository) DeleteNote(id int) error {
	cachedNote := repo.getNoteFromCache(id)
	if cachedNote != nil {
		err := repo.deleteFromCache(*cachedNote)
		if err != nil {
			return err
		}
	}
	result := repo.db.Delete(&Note{}, id)
	return result.Error
}

DeleteNote will delete the note from the cache and then. delete the note from the postgres database.

We have a complete implementation of our repository. Our repository has methods to store and retrieve notes by their ids and titles and it also caches notes to speed up retrieval and prevent hits to our database. How are we sure that these really work though? How are we sure that our implementation does prevent hits to our database and that our caching works as expected? How are we sure that the Redis commands store and return the data we need in the format we expect it to?

Enter Integration Tests……

Step 4 - Write Integration Tests For Our Note Repository

Before we proceed to write tests, let’s briefly talk about the tools and the approach. In Step 1, we installed a couple of modules: testcontainers-go, testcontainers-go/modules/postgres , testcontainers-go/modules/redis, sqlmock and testify.

Testify is a toolkit built on Go’s existing test framework that provides packages to make testing easier, providing functionalities like mocks, assertions and suites. For this test, we would be using testify’s suite package. Suite allows us to group and run tests together with fixtures that can run before/after each test and even before and after the entire suite of tests.

sqlmock is a mock toolkit to mock sql databases. We would be using this tool to test and ensure that querying a cached note doesn’t actually query our postgres database.

Testcontainers is a tool that allows us to spin up docker containers and test against them. testcontainers-go allows us to use testcontainers with Golang. testcontainers-go/modules/redis and testcontainers-go/modules/postgres are Go modules that provide functions to easily spin up Redis and Postgres docker containers respectively.

Approach

We will define a suite of tests for our NoteRepository. Testify suites allow us to define fixtures that run before the suite starts, after all the tests in the suite are done executing, and before/after every test in the suite, here is what we would like to do:

  • Before the suite starts, we would like to spin up our containers (postgres container and redis container)
  • After the suite is finished, we would like to destroy the containers (always a good practice to close resources you are done using).
  • Before each test, we would create our notes database table in postgres
  • After each test, we would flush (empty) our cache and delete our notes table from the postgres database. This is so that results from the test do not interfere with the next test that would be executed in the suite.
  • Before the update note test, we would seed the cache with note data to ensure that the cache is invalidated when the note is updated

Step 4.1 - Define our Test Suite and Fixtures

Let’s declare our package and import our dependencies:

package app

import (
	"context"
	"fmt"
	"github.com/DATA-DOG/go-sqlmock"
	rd "github.com/redis/go-redis/v9"
	"github.com/stretchr/testify/suite"
	"github.com/testcontainers/testcontainers-go"
	"github.com/testcontainers/testcontainers-go/modules/postgres"
	"github.com/testcontainers/testcontainers-go/modules/redis"
	"github.com/testcontainers/testcontainers-go/wait"
	pg "gorm.io/driver/postgres"
	"gorm.io/gorm"
	"strconv"
	"testing"
	"time"
)

Add this code to define our NoteRepoTestSuite

type NoteRepoTestSuite struct {
	suite.Suite
	ctx                context.Context
	db                 *gorm.DB
	pgContainer        *postgres.PostgresContainer
	pgConnectionString string
	rdContainer        *redis.RedisContainer
	rdConnectionString string
	rdClient           *rd.Client
}

Our NoteRepoTestSuite embeds testify’s suite and define some extra properties. ctx will be the test execution context shared by all tests. db will hold the postgres database connection. pgContainer will hold a reference to the created postgres docker container, pgConnectionString is the connection string we will use to establish a connection to the postgres database in the container. rdContainer is the reference to the created Redis container.

rdConnectionString is the string we will use to establish a connection to the Redis server in the container, and rdClient is the client we will use to communicate with our Redis server.

func (suite *NoteRepoTestSuite) SetupSuite() {
	suite.ctx = context.Background()
	pgContainer, err := postgres.RunContainer(
		suite.ctx,
		testcontainers.WithImage("postgres:15.3-alpine"),
		postgres.WithDatabase("notesdb"),
		postgres.WithUsername("postgres"),
		postgres.WithPassword("postgres"),
		testcontainers.WithWaitStrategy(
			wait.ForLog("database system is ready to accept connections").
				WithOccurrence(2).WithStartupTimeout(5*time.Second)),
	)
	suite.NoError(err)

	connStr, err := pgContainer.ConnectionString(suite.ctx, "sslmode=disable")
	suite.NoError(err)

	db, err := gorm.Open(pg.Open(connStr), &gorm.Config{})
	suite.NoError(err)

	suite.pgContainer = pgContainer
	suite.pgConnectionString = connStr
	suite.db = db

	sqlDB, err := suite.db.DB()
	suite.NoError(err)

	err = sqlDB.Ping()
	suite.NoError(err)

	redisContainer, err := redis.RunContainer(suite.ctx, testcontainers.WithImage("redis:6"))
	suite.NoError(err)
	rdConnStr, err := redisContainer.ConnectionString(suite.ctx)
	suite.NoError(err)

	rdConnOptions, err := rd.ParseURL(rdConnStr)
	suite.NoError(err)

	rdClient := rd.NewClient(rdConnOptions)

	suite.rdContainer = redisContainer
	suite.rdConnectionString = connStr
	suite.rdClient = rdClient

	err = suite.rdClient.Ping(suite.ctx).Err()
	suite.NoError(err)

}

The SetupSuite method will be executed before any test in the suite. Key things to note here are:

  • We use the testcontainers-go/modules/posgres postgres module’s function RunContainer to spin up a new container.
    • We tell it to use docker image postgres:15.3-alpine - it will try to pull this image from docker hub or you can manually pull this image using the command docker pull postgres:15.3-alpine.
    • We set the name of the database in the container to notesdb, the username to postgres and the password to postgres too.
    • WithWaitStrategy and wait.ForLog tell the function not to return and wait until the container outputs the log "database system is ready to accept connections". This is how we know that the postgres database is ready.
  • Once the postgres container is ready, we get the connection string from the container and use it to create a connection with gorm DB. Then, we set them on the suite and proceeded to test our connection to the db by pinging it.
  • We do the same thing with Redis. We run the container using the module with docker image redis:6 , we get the connection string, establish a connection, set the properties of the suite, and ping the redis container to ensure we do have a connection.
func (suite *NoteRepoTestSuite) TearDownSuite() {
	err := suite.pgContainer.Terminate(suite.ctx)
	suite.NoError(err)

	err = suite.rdContainer.Terminate(suite.ctx)
	suite.NoError(err)
}

We added a TearDownSuite method that would be called after all the tests in the suite are executed before the suite exits. We simply terminate the containers we have created in the setup stage.

func (suite *NoteRepoTestSuite) SetupTest() {
	err := suite.db.AutoMigrate(&Note{})
	suite.NoError(err)
}

SetupTest method will be executed before every test in the suite. Here, we use Gorm’s Automigrate to create our notes table in the postgres database.

func (suite *NoteRepoTestSuite) TearDownTest() {
	suite.db.Exec("DROP TABLE IF EXISTS notes CASCADE;")
	suite.rdClient.FlushAll(suite.ctx)
}

TearDownTest method will drop our notes table and flush our cache after every test in the suite.

func (suite *NoteRepoTestSuite) BeforeTest(_ string, testName string) {
	if testName == "TestSaveUpdatedNote" || testName == "TestDeleteNote" {
		note := Note{Title: "Test Update", Content: "This note will be inserted now"}
		result := suite.db.Save(&note)
		suite.NoError(result.Error)

		idKey := fmt.Sprintf("notes:%d", note.ID)
		titleKey := fmt.Sprintf("notes:%s", note.Title)
		err := suite.rdClient.HSet(suite.ctx, idKey, "id", note.ID).Err()
		suite.NoError(err)
		err = suite.rdClient.HSet(suite.ctx, idKey, "title", note.Title).Err()
		suite.NoError(err)
		err = suite.rdClient.HSet(suite.ctx, idKey, "content", note.Content).Err()
		suite.NoError(err)
		err = suite.rdClient.HSet(suite.ctx, idKey, "created_at", note.CreatedAt).Err()
		suite.NoError(err)
		err = suite.rdClient.HSet(suite.ctx, idKey, "updated_at", note.UpdatedAt).Err()
		suite.NoError(err)
		err = suite.rdClient.HSet(suite.ctx, titleKey, "id", note.ID).Err()
		suite.NoError(err)
		err = suite.rdClient.HSet(suite.ctx, titleKey, "title", note.Title).Err()
		suite.NoError(err)
		err = suite.rdClient.HSet(suite.ctx, titleKey, "content", note.Content).Err()
		suite.NoError(err)
		err = suite.rdClient.HSet(suite.ctx, titleKey, "created_at", note.CreatedAt).Err()
		suite.NoError(err)
		err = suite.rdClient.HSet(suite.ctx, titleKey, "updated_at", note.UpdatedAt).Err()
		suite.NoError(err)
	}
}

BeforeTest method is similar to SetupTest and it runs immediately after the SetupTest function just before the actual test is executed. The name of the suite and the name of the test are passed to the method. For our update and delete tests, we want to seed a note in the database as well as in the cache. We can actually do this seeding in the specific tests, however, this is a nice way to have it done in one place and to also show how cool testify suites are.

Also, in this method, instead of using our repository methods to seed the db and cache, we directly use the clients. This is so that our test doesn’t depend on the code it is testing. Our tests should catch the bugs in our code, not use them.

Step 4.2 - Integration Tests

To add a test to our test suite, we need to define it as a method of the test suite and the name of the method must begin with Test. Let’s test our save note method.

func (suite *NoteRepoTestSuite) TestSaveNewNote() {
	// ensure that the cache is empty
	keys, err := suite.rdClient.Keys(suite.ctx, "*").Result()
	suite.NoError(err)
	suite.Empty(keys)

	// ensure that the postgres database is empty
	var notes []Note
	result := suite.db.Find(&notes)
	suite.NoError(result.Error)
	suite.Empty(notes)

	// create repository and save new note
	repo := NewNoteRepository(suite.db, suite.rdClient)
	newNote := Note{Title: "Testing 123", Content: "This note was just inserted"}
	err = repo.SaveNote(&newNote)
	suite.NoError(err)

	// ensure the cache is still empty
	keys, err = suite.rdClient.Keys(suite.ctx, "*").Result()
	suite.NoError(err)
	suite.Empty(keys)

	// ensure that we have a new note in the database
	result = suite.db.Find(&notes)
	suite.NoError(result.Error)
	suite.Equal(1, len(notes))
	suite.Equal(newNote.ID, notes[0].ID)
	suite.Equal(newNote.Title, notes[0].Title)
	suite.Equal(newNote.Content, notes[0].Content)

}

Here, we first ensure that the cache is empty by ensuring that there are no keys defined in Redis. Then, we also query the notes table in postgres database and ensure it’s empty as well. We then instantiate our NoteRepository using the suite’s postgres client db and Redis client rdClient, create a note and use the repo’s SaveNote method to save the note. We check our cache again to ensure it’s empty since write operations should invalidate cache. And lastly, we check our postgres database for the note to ensure it’s there.

func (suite *NoteRepoTestSuite) TestSaveUpdatedNote() {
	// ensure that we have a note in the database
	var note Note
	result := suite.db.First(&note)
	suite.NoError(result.Error)
	suite.NotZero(note)

	idKey := fmt.Sprintf("notes:%d", note.ID)
	titleKey := fmt.Sprintf("notes:%s", note.Title)

	// ensure that we have the note cached under its id
	res, err := suite.rdClient.Exists(suite.ctx, idKey).Result()
	suite.NoError(err)
	suite.Greater(res, int64(0))

	// ensure that we have the note cached under its title
	res, err = suite.rdClient.Exists(suite.ctx, titleKey).Result()
	suite.NoError(err)
	suite.Greater(res, int64(0))

	// update the note and save it
	repo := NewNoteRepository(suite.db, suite.rdClient)
	note.Content = "This is the updated note"
	err = repo.SaveNote(&note)
	suite.NoError(err)

	// ensure the cache is invalidated
	res, err = suite.rdClient.Exists(suite.ctx, idKey).Result()
	suite.NoError(err)
	suite.Equal(int64(0), res)
	res, err = suite.rdClient.Exists(suite.ctx, titleKey).Result()
	suite.NoError(err)
	suite.Equal(int64(0), res)

	// ensure the note has been updated in the database.
	var notes []Note
	result = suite.db.Find(&notes)
	suite.NoError(result.Error)
	suite.Equal(1, len(notes))
	suite.Equal(note.ID, notes[0].ID)
	suite.Equal(note.Title, notes[0].Title)
	suite.Equal(note.Content, notes[0].Content)
}

Remember, in the BeforeTest fixture, we seeded a note in the db and cache for this TestSaveUpdatedNote, so first, we check to ensure that we do have a note in Postgres and in Redis. We get the note from Postgres, create an instance of our NoteRepository, update the content of the note and then save the note using our repo. To be sure that our repository works correctly, we check that the cache is invalidated (the note is deleted from the cache) and we check that the note is still in the Postgres database by querying it again and ensuring the content was updated.

func (suite *NoteRepoTestSuite) TestDeleteNote() {
	// ensure we have a note in the database
	var note Note
	result := suite.db.First(&note)
	suite.NoError(result.Error)
	suite.NotZero(note)

	idKey := fmt.Sprintf("notes:%d", note.ID)
	titleKey := fmt.Sprintf("notes:%s", note.Title)

	// ensure we have the note cached
	res, err := suite.rdClient.Exists(suite.ctx, idKey).Result()
	suite.NoError(err)
	suite.Greater(res, int64(0))
	res, err = suite.rdClient.Exists(suite.ctx, titleKey).Result()
	suite.NoError(err)
	suite.Greater(res, int64(0))

	// delete the note
	repo := NewNoteRepository(suite.db, suite.rdClient)
	err = repo.DeleteNote(int(note.ID))
	suite.NoError(err)

	// ensure that the cache has been cleared
	res, err = suite.rdClient.Exists(suite.ctx, idKey).Result()
	suite.NoError(err)
	suite.Equal(int64(0), res)
	res, err = suite.rdClient.Exists(suite.ctx, titleKey).Result()
	suite.NoError(err)
	suite.Equal(int64(0), res)

	// ensure that the note has been deleted in  postgres
	var notes []Note
	result = suite.db.Find(&notes)
	suite.NoError(result.Error)
	suite.Empty(notes)
}

For the delete note, we also seeded the cache and database using the BeforeTest fixture. We check to ensure that we do have a note in the database and in the cache, then we create an instance of our repository and delete the note by its id. We then check to ensure that the note is no longer in the cache or in the database.

Next, let’s test getting the note. This final test has 4 subtests i.e. testing getting the note by id for the first time, testing it when it has been cached to ensure the database wasn’t hit and doing the same two tests but by title. To keep this tutorial short (I guess), I’d only be demonstrating the get by id. You can check out the remaining tests here.

For the first subtest, just like with the TearDownTest feature, for each subtest, we have to clear the cache and database after the subtest is done to ensure the results don’t interfere with other subtests. We do that by using go’s testing Cleanup function

suite.T().Cleanup(func() {
	suite.db.Exec("DELETE FROM notes;")
	suite.rdClient.FlushAll(suite.ctx)
})

Then, we insert a note into the database so that we have a note to get:

dbNote := Note{
	Title:   "Testing 123",
	Content: "This is a test content",
}
result := suite.db.Save(&dbNote)
suite.NoError(result.Error)

Now, we ensure that the cache is empty

res, err := suite.rdClient.Exists(suite.ctx, fmt.Sprintf("notes:%d", dbNote.ID)).Result()
suite.NoError(err)
suite.Equal(int64(0), res)
res, err = suite.rdClient.Exists(suite.ctx, "notes:Testing 123").Result()
suite.NoError(err)
suite.Equal(int64(0), res)

Then, we construct our repository and get the note by id and ensure that the note was actually returned

repo := NewNoteRepository(suite.db, suite.rdClient)
note := repo.GetNoteById(int(dbNote.ID))
suite.NotNil(note)

Next, we check the cache to ensure that the note has been cached under its id and its title

res, err = suite.rdClient.Exists(suite.ctx, fmt.Sprintf("notes:%d", dbNote.ID)).Result()
suite.NoError(err)
suite.Greater(res, int64(0))

res, err = suite.rdClient.Exists(suite.ctx, "notes:Testing 123").Result()
suite.NoError(err)
suite.Greater(res, int64(0))

noteMap, err := suite.rdClient.HGetAll(suite.ctx, fmt.Sprintf("notes:%d", dbNote.ID)).Result()
suite.NoError(err)
suite.Equal(strconv.Itoa(int(dbNote.ID)), noteMap["id"])
suite.Equal("Testing 123", noteMap["title"])
suite.Equal("This is a test content", noteMap["content"])

noteMap, err = suite.rdClient.HGetAll(suite.ctx, "notes:Testing 123").Result()
suite.NoError(err)
suite.Equal(strconv.Itoa(int(dbNote.ID)), noteMap["id"])
suite.Equal("Testing 123", noteMap["title"])
suite.Equal("This is a test content", noteMap["content"])

The entire subtest looks like this.

suite.Run("Get note when note does not exist in cache", func() {
	// empty the notes table and flush the cache
	suite.T().Cleanup(func() {
		suite.db.Exec("DELETE FROM notes;")
		suite.rdClient.FlushAll(suite.ctx)
	})

	// insert a new note in the database
	dbNote := Note{
		Title:   "Testing 123",
		Content: "This is a test content",
	}
	result := suite.db.Save(&dbNote)
	suite.NoError(result.Error)

	// ensure that the cache is empty
	res, err := suite.rdClient.Exists(suite.ctx, fmt.Sprintf("notes:%d", dbNote.ID)).Result()
	suite.NoError(err)
	suite.Equal(int64(0), res)
	res, err = suite.rdClient.Exists(suite.ctx, "notes:Testing 123").Result()
	suite.NoError(err)
	suite.Equal(int64(0), res)

	// get a note by its id
	repo := NewNoteRepository(suite.db, suite.rdClient)
	note := repo.GetNoteById(int(dbNote.ID))
	suite.NotNil(note)

	// ensure that the note is now cached
	res, err = suite.rdClient.Exists(suite.ctx, fmt.Sprintf("notes:%d", dbNote.ID)).Result()
	suite.NoError(err)
	suite.Greater(res, int64(0))

	res, err = suite.rdClient.Exists(suite.ctx, "notes:Testing 123").Result()
	suite.NoError(err)
	suite.Greater(res, int64(0))

	noteMap, err := suite.rdClient.HGetAll(suite.ctx, fmt.Sprintf("notes:%d", dbNote.ID)).Result()
	suite.NoError(err)
	suite.Equal(strconv.Itoa(int(dbNote.ID)), noteMap["id"])
	suite.Equal("Testing 123", noteMap["title"])
	suite.Equal("This is a test content", noteMap["content"])

	noteMap, err = suite.rdClient.HGetAll(suite.ctx, "notes:Testing 123").Result()
	suite.NoError(err)
	suite.Equal(strconv.Itoa(int(dbNote.ID)), noteMap["id"])
	suite.Equal("Testing 123", noteMap["title"])
	suite.Equal("This is a test content", noteMap["content"])
})

We do the same thing for the other subtest. This subtest is to test that when we have a note in the cache, a call to get the note will return the cached note and not hit the database.

First, we set the cleanup function to clear the cache and the database after the subtest is done and then, we seed the database and cache with a note:

suite.T().Cleanup(func() {
	suite.db.Exec("DELETE FROM notes;")
	suite.rdClient.FlushAll(suite.ctx)
})

// insert note in database
dbNote := Note{
	Title:   "Testing 123",
	Content: "This is a test content",
}
result := suite.db.Save(&dbNote)
suite.NoError(result.Error)

idKey := fmt.Sprintf("notes:%d", dbNote.ID)
titleKey := fmt.Sprintf("notes:%s", dbNote.Title)

// cache the note
suite.rdClient.HSet(suite.ctx, idKey, "id", dbNote.ID)
suite.rdClient.HSet(suite.ctx, idKey, "title", dbNote.Title)
suite.rdClient.HSet(suite.ctx, idKey, "content", dbNote.Content)
suite.rdClient.HSet(suite.ctx, idKey, "created_at", dbNote.CreatedAt)
suite.rdClient.HSet(suite.ctx, idKey, "updated_at", dbNote.UpdatedAt)
suite.rdClient.HSet(suite.ctx, titleKey, "id", dbNote.ID)
suite.rdClient.HSet(suite.ctx, titleKey, "title", dbNote.Title)
suite.rdClient.HSet(suite.ctx, titleKey, "content", dbNote.Content)
suite.rdClient.HSet(suite.ctx, titleKey, "created_at", dbNote.CreatedAt)
suite.rdClient.HSet(suite.ctx, titleKey, "updated_at", dbNote.UpdatedAt)

Since we need a way to check if the database was actually hit or not, we cannot use the db connection we have on the suite, instead, we will create a mock db using sqlmock. With this mock, we can check if a query was attempted or not.

mockDb, mock, err := sqlmock.New()
suite.NoError(err)
suite.T().Cleanup(func() {
	mockDb.Close()
})

dialector := pg.New(pg.Config{
	Conn:       mockDb,
	DriverName: "postgres",
})
db, err := gorm.Open(dialector, &gorm.Config{})
suite.NoError(err)

Let’s go over how sqlmock works and how we would use it to test that our database is not queried. With sqlmock, if we want to test that a function will query the database, we will set sqlmock to expect the query (using one of it’s Expect functions) before executing the target function we would like to test. After the target function is executed, we can then ensure that query (it was expecting) was truly made by calling the ExpectationsWereMet() function. The ExpectationsWereMet() function will fail if it was expecting a query and the query was never made, it will also fail if a query was made and it wasn’t expecting that query.

In our case, since we don’t expect any queries to the database, we would not set any expectations on sqlmock and therefore, calling ExpectationsWereMet() on the object should fail if our repo attempts to query the database.

// get the note by id and ensure the note was successfully retrieved
repo := NewNoteRepository(db, suite.rdClient)
note := repo.GetNoteById(int(dbNote.ID))
suite.NotNil(note)
suite.Equal(dbNote.ID, note.ID)
suite.Equal(dbNote.Title, note.Title)
suite.Equal(dbNote.Content, note.Content)

// ensure all query expectations were met
// since we didn't set any expectations
// this would only pass if no query was executed on the database
// indicating that our repository didn't query the database to get
// the note.
err = mock.ExpectationsWereMet()
suite.NoError(err)

The complete code for our second subtest looks like this:

suite.Run("Get note when note exists in cache", func() {
	// empty the notes table and flush the cache
	suite.T().Cleanup(func() {
		suite.db.Exec("DELETE FROM notes;")
		suite.rdClient.FlushAll(suite.ctx)
	})

	// insert note in database
	dbNote := Note{
		Title:   "Testing 123",
		Content: "This is a test content",
	}
	result := suite.db.Save(&dbNote)
	suite.NoError(result.Error)

	idKey := fmt.Sprintf("notes:%d", dbNote.ID)
	titleKey := fmt.Sprintf("notes:%s", dbNote.Title)

	// cache the note
	suite.rdClient.HSet(suite.ctx, idKey, "id", dbNote.ID)
	suite.rdClient.HSet(suite.ctx, idKey, "title", dbNote.Title)
	suite.rdClient.HSet(suite.ctx, idKey, "content", dbNote.Content)
	suite.rdClient.HSet(suite.ctx, idKey, "created_at", dbNote.CreatedAt)
	suite.rdClient.HSet(suite.ctx, idKey, "updated_at", dbNote.UpdatedAt)
	suite.rdClient.HSet(suite.ctx, titleKey, "id", dbNote.ID)
	suite.rdClient.HSet(suite.ctx, titleKey, "title", dbNote.Title)
	suite.rdClient.HSet(suite.ctx, titleKey, "content", dbNote.Content)
	suite.rdClient.HSet(suite.ctx, titleKey, "created_at", dbNote.CreatedAt)
	suite.rdClient.HSet(suite.ctx, titleKey, "updated_at", dbNote.UpdatedAt)

	// construct mock SQL database
	mockDb, mock, err := sqlmock.New()
	suite.NoError(err)
	suite.T().Cleanup(func() {
		mockDb.Close()
	})

	dialector := pg.New(pg.Config{
		Conn:       mockDb,
		DriverName: "postgres",
	})
	db, err := gorm.Open(dialector, &gorm.Config{})
	suite.NoError(err)

	// get the note by id and ensure the note was successfully retrieved
	repo := NewNoteRepository(db, suite.rdClient)
	note := repo.GetNoteById(int(dbNote.ID))
	suite.NotNil(note)
	suite.Equal(dbNote.ID, note.ID)
	suite.Equal(dbNote.Title, note.Title)
	suite.Equal(dbNote.Content, note.Content)

	// ensure all query expectations were met
	// since we didn't set any expectations
	// this would only pass if no query was executed on the database
	// indicating that our repository didn't query the database to get
	// the note.
	err = mock.ExpectationsWereMet()
	suite.NoError(err)

})

and then we need to put both subtests into the TestGetNote test. like this

func (suite *NoteRepoTestSuite) TestGetNote() {
	//first subtest
	suite.Run("Get note when note does not exist in cache", func() {...})
	// second subtest
	suite.Run("Get note when note exists in cache", func() {...})
}

Lastly, we need to run our test suite when we run go test

func TestNoteRepository(t *testing.T) {
	suite.Run(t, new(NoteRepoTestSuite))
}

We can now run our integration tests by running:

$ go test -v ./...

We should see the logs of our containers getting spun up, our tests running successfully and the containers getting terminated after all the tests have been executed.

Conclusion

We have successfully written a Go program that uses a repository to save and retrieve notes in a Postgres database and Redis. We also wrote integration tests to test that our notes were indeed stored correctly in the Postgres database, cached in Redis when they ought to be. We ensured that our cache was invalidated when we expected it to be and that our database was not queried when we cached data. You can find the full source code for this tutorial here