Using Go modules

This post is part 1 in a series.

Part 1 — Using Go Modules (this post)
Part 2 — Migrating To Go Modules
Part 3 — Publishing Go Modules
Part 4 — Go Modules: v2 and Beyond

Go 1.11 and 1.12 include preliminary [[https://golang.org/doc/go1.11#modules][support for modules]], Go’s [[https://blog.golang.org/versioning-proposal][new dependency management system]] that makes dependency version information explicit and easier to manage. This blog post is an introduction to the basic operations needed to get started using modules.

A module is a collection of [[https://golang.org/ref/spec#Packages][Go packages]] stored in a file tree with a go.mod file at its root. The go.mod file defines the module’s module_path, which is also the import path used for the root directory, and its dependency_requirements, which are the other modules needed for a successful build. Each dependency requirement is written as a module path and a specific [[http://semver.org/][semantic version]].

As of Go 1.11, the go command enables the use of modules when the current directory or any parent directory has a go.mod, provided the directory is outside $GOPATH/src. (Inside $GOPATH/src, for compatibility, the go command still runs in the old GOPATH mode, even if a go.mod is found. See the [[https://golang.org/cmd/go/#hdr-Preliminary_module_support][go command documentation]] for details.) Starting in Go 1.13, module mode will be the default for all development.

This post walks through a sequence of common operations that arise when developing Go code with modules:

Creating a new module.
Adding a dependency.
Upgrading dependencies.
Adding a dependency on a new major version.
Upgrading a dependency to a new major version.
Removing unused dependencies.

Creating a new module

Let’s create a new module.

Create a new, empty directory somewhere outside $GOPATH/src, cd into that directory, and then create a new source file, hello.go:

package hello

func Hello() string {
    return "Ahoy, world!"
}

cd $GOPATH/src

echo 'package hello

func Hello() string {
    return "Ahoy, world!"
}' > hello.go

Let’s write a test, too, in hello_test.go:

package hello

import "testing"

func TestHello(t *testing.T) {
        want := "Ahoy, world!"
        if got := Hello(); got != want {
                t.Errorf("Hello() = %q, want %q", got, want)
        }
}

cd $GOPATH/src

echo 'package hello

import "testing"

func TestHello(t *testing.T) {
        want := "Ahoy, world!"
        if got := Hello(); got != want {
                t.Errorf("Hello() = %q, want %q", got, want)
        }
}' > hello_test.go

At this point, the directory contains a package, but not a module, because there is no go.mod file. If we were working in /home/gopher/hello and ran gotest` now, we’d see:

$ go test
PASS
ok  	_/home/gopher/hello	0.020s
$

go test

The last line summarizes the overall package test. Because we are working outside $GOPATH and also outside any module, the go command knows no import path for the current directory and makes up a fake one based on the directory name: _/home/gopher/hello.

Let’s make the current directory the root of a module by using go mod init and then try go test again:

$ go mod init example.com/hello
go: creating new go.mod: module example.com/hello
$ go test
PASS
ok  	example.com/hello	0.020s
$

go mod init example.com/hello

go test

Congratulations! You’ve written and tested your first module.

The go mod init command wrote a go.mod file:

$ cat go.mod
module example.com/hello

go 1.13
$

cat go.mod

The go.mod file only appears in the root of the module. Packages in subdirectories have import paths consisting of the module path plus the path to the subdirectory. For example, if we created a subdirectory world, we would not need to (nor want to) run gomodinit there. The package would automatically be recognized as part of the example.com/hello module, with import path example.com/hello/world.

Adding a dependency

The primary motivation for Go modules was to improve the experience of using (that is, adding a dependency on) code written by other developers.

Let’s update our hello.go to import rsc.io/quote and use it to implement Hello:

package hello

import "rsc.io/quote"

func Hello() string {
        return quote.Hello()
}

Now let’s run the test again:

$ go test
go: finding rsc.io/quote v1.5.2
go: downloading rsc.io/quote v1.5.2
go: extracting rsc.io/quote v1.5.2
go: finding rsc.io/sampler v1.3.0
go: finding golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c
go: downloading rsc.io/sampler v1.3.0
go: extracting rsc.io/sampler v1.3.0
go: downloading golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c
go: extracting golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c
PASS
ok  	example.com/hello	0.023s
$

cd $GOPATH/src

echo 'package hello

import "rsc.io/quote"

func Hello() string {
    return quote.Hello()
}' > hello.go

go test

The go command resolves imports by using the specific dependency module versions listed in go.mod. When it encounters an import of a package not provided by any module in go.mod, the go command automatically looks up the module containing that package and adds it to go.mod, using the latest version. (“Latest” is defined as the latest tagged stable (non-[[https://semver.org/#spec-item-9][prerelease]]) version, or else the latest tagged prerelease version, or else the latest untagged version.) In our example, gotest resolved the new import rsc.io/quoteto the modulersc.io/quotev1.5.2. It also downloaded two dependencies used by rsc.io/quote, namely rsc.io/sampler and golang.org/x/text. Only direct dependencies are recorded in the go.mod file:

$ cat go.mod
module example.com/hello

go 1.13

require rsc.io/quote v1.5.2
$

cat go.mod

A second gotestcommand will not repeat this work, since thego.modis now up-to-date and the downloaded modules are cached locally (in$GOPATH/pkg/mod`):

$ go test
PASS
ok  	example.com/hello	0.020s
$

go test

Note that while the go command makes adding a new dependency quick and easy, it is not without cost. Your module now literally depends on the new dependency in critical areas such as correctness, security, and proper licensing, just to name a few. For more considerations, see Russ Cox’s blog post, “[[https://research.swtch.com/deps][Our Software Dependency Problem]].”

As we saw above, adding one direct dependency often brings in other indirect dependencies too. The command golist-mall` lists the current module and all its dependencies:

$ go list -m all
example.com/hello
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c
rsc.io/quote v1.5.2
rsc.io/sampler v1.3.0
$

go list -m all

In the golist` output, the current module, also known as the main_module, is always the first line, followed by dependencies sorted by module path.

The golang.org/x/text version v0.0.0-20170915032832-14c0d48ead0c is an example of a [[https://golang.org/cmd/go/#hdr-Pseudo_versions][pseudo-version]], which is the go command’s version syntax for a specific untagged commit.

In addition to go.mod, the go command maintains a file named go.sum containing the expected [[https://golang.org/cmd/go/#hdr-Module_downloading_and_verification][cryptographic hashes]] of the content of specific module versions:

$ cat go.sum
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c h1:qgOY6WgZO...
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:Nq...
rsc.io/quote v1.5.2 h1:w5fcysjrx7yqtD/aO+QwRjYZOKnaM9Uh2b40tElTs3...
rsc.io/quote v1.5.2/go.mod h1:LzX7hefJvL54yjefDEDHNONDjII0t9xZLPX...
rsc.io/sampler v1.3.0 h1:7uVkIFmeBqHfdjD+gZwtXXI+RODJ2Wc4O7MPEh/Q...
rsc.io/sampler v1.3.0/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9...
$

cat go.sum

The go command uses the go.sum file to ensure that future downloads of these modules retrieve the same bits as the first download, to ensure the modules your project depends on do not change unexpectedly, whether for malicious, accidental, or other reasons. Both go.mod and go.sum should be checked into version control.

Upgrading dependencies

With Go modules, versions are referenced with semantic version tags. A semantic version has three parts: major, minor, and patch. For example, for v0.1.2, the major version is 0, the minor version is 1, and the patch version is 2. Let’s walk through a couple minor version upgrades. In the next section, we’ll consider a major version upgrade.

From the output of golist-mall, we can see we're using an untagged version ofgolang.org/x/text`. Let’s upgrade to the latest tagged version and test that everything still works:

$ go get golang.org/x/text
go: finding golang.org/x/text v0.3.0
go: downloading golang.org/x/text v0.3.0
go: extracting golang.org/x/text v0.3.0
$ go test
PASS
ok  	example.com/hello	0.013s
$

go get golang.org/x/text

go test

Woohoo! Everything passes. Let’s take another look at golist-mall and the go.mod` file:

$ go list -m all
example.com/hello
golang.org/x/text v0.3.0
rsc.io/quote v1.5.2
rsc.io/sampler v1.3.0
$ cat go.mod
module example.com/hello

go 1.13

require (
	golang.org/x/text v0.3.0 // indirect
	rsc.io/quote v1.5.2
)
$

go list -m all

cat go.mod

The golang.org/x/text package has been upgraded to the latest tagged version (v0.3.0). The go.mod file has been updated to specify v0.3.0 too. The indirect comment indicates a dependency is not used directly by this module, only indirectly by other module dependencies. See gohelpmodules for details.

Now let’s try upgrading the rsc.io/sampler minor version. Start the same way, by running goget` and running tests:

$ go get rsc.io/sampler
go: finding rsc.io/sampler v1.99.99
go: downloading rsc.io/sampler v1.99.99
go: extracting rsc.io/sampler v1.99.99
$ go test
--- FAIL: TestHello (0.00s)
    hello_test.go:8: Hello() = "99 bottles of beer on the wall, 99 bottles of beer, ...", want "Ahoy, world!"
FAIL
exit status 1
FAIL	example.com/hello	0.014s
$

go get rsc.io/sampler

go test

Uh, oh! The test failure shows that the latest version of rsc.io/sampler is incompatible with our usage. Let’s list the available tagged versions of that module:

$ go list -m -versions rsc.io/sampler
rsc.io/sampler v1.0.0 v1.2.0 v1.2.1 v1.3.0 v1.3.1 v1.99.99
$

go list -m -versions rsc.io/sampler

We had been using v1.3.0; v1.99.99 is clearly no good. Maybe we can try using v1.3.1 instead:

$ go get rsc.io/sampler@v1.3.1
go: finding rsc.io/sampler v1.3.1
go: downloading rsc.io/sampler v1.3.1
go: extracting rsc.io/sampler v1.3.1
$ go test
PASS
ok  	example.com/hello	0.022s
$

go get rsc.io/sampler@v1.3.1

go test

Note the explicit @v1.3.1 in the gogetargument. In general each argument passed togoget can take an explicit version; the default is @latest, which resolves to the latest version as defined earlier.

Adding a dependency on a new major version

Let’s add a new function to our package: funcProverbreturns a Go concurrency proverb, by callingquote.Concurrency, which is provided by the modulersc.io/quote/v3. First we updatehello.go` to add the new function:

package hello

import (
        "rsc.io/quote"
        quoteV3 "rsc.io/quote/v3"
)

func Hello() string {
        return quote.Hello()
}

func Proverb() string {
        return quoteV3.Concurrency()
}

echo 'package hello

import (
        "rsc.io/quote"
        quoteV3 "rsc.io/quote/v3"
)

func Hello() string {
        return quote.Hello()
}

func Proverb() string {
        return quoteV3.Concurrency()
}' > hello.go

Then we add a test to hello_test.go:

func TestProverb(t *testing.T) {
        want := "Concurrency is not parallelism."
        if got := Proverb(); got != want {
                t.Errorf("Proverb() = %q, want %q", got, want)
        }
}

echo 'func TestProverb(t *testing.T) {
        want := "Concurrency is not parallelism."
        if got := Proverb(); got != want {
                t.Errorf("Proverb() = %q, want %q", got, want)
        }
}' >> hello.go

Then we can test our code:

$ go test
go: finding rsc.io/quote/v3 v3.1.0
go: downloading rsc.io/quote/v3 v3.1.0
go: extracting rsc.io/quote/v3 v3.1.0
PASS
ok  	example.com/hello	0.024s
$

go test

Note that our module now depends on both rsc.io/quote and rsc.io/quote/v3:

$ go list -m rsc.io/q...
rsc.io/quote v1.5.2
rsc.io/quote/v3 v3.1.0
$

go list -m rsc.io/q...

Each different major version (v1, v2, and so on) of a Go module uses a different module path: starting at v2, the path must end in the major version. In the example, v3 of rsc.io/quote is no longer rsc.io/quote: instead, it is identified by the module path rsc.io/quote/v3. This convention is called [[https://research.swtch.com/vgo-import][semantic import versioning]], and it gives incompatible packages (those with different major versions) different names. In contrast, v1.6.0 of rsc.io/quote should be backwards-compatible with v1.5.2, so it reuses the name rsc.io/quote. (In the previous section, rsc.io/sampler v1.99.99 should have been backwards-compatible with rsc.io/sampler v1.3.0, but bugs or incorrect client assumptions about module behavior can both happen.)

The go command allows a build to include at most one version of any particular module path, meaning at most one of each major version: one rsc.io/quote, one rsc.io/quote/v2, one rsc.io/quote/v3, and so on. This gives module authors a clear rule about possible duplication of a single module path: it is impossible for a program to build with both rsc.io/quotev1.5.2 and rsc.io/quotev1.6.0. At the same time, allowing different major versions of a module (because they have different paths) gives module consumers the ability to upgrade to a new major version incrementally. In this example, we wanted to use quote.Concurrency from rsc/quote/v3v3.1.0but are not yet ready to migrate our uses ofrsc.io/quotev1.5.2. The ability to migrate incrementally is especially important in a large program or codebase.

Upgrading a dependency to a new major version

Let’s complete our conversion from using rsc.io/quote to using only rsc.io/quote/v3. Because of the major version change, we should expect that some APIs may have been removed, renamed, or otherwise changed in incompatible ways. Reading the docs, we can see that Hello has become HelloV3:

$ go doc rsc.io/quote/v3
package quote // import "rsc.io/quote"

Package quote collects pithy sayings.

func Concurrency() string
func GlassV3() string
func GoV3() string
func HelloV3() string
func OptV3() string
$

go doc rsc.io/quote/v3

(There is also a [[https://golang.org/issue/30778][known bug]] in the output; the displayed import path has incorrectly dropped the /v3.)

We can update our use of quote.Hello() in hello.go to use quoteV3.HelloV3():

package hello

import quoteV3 "rsc.io/quote/v3"

func Hello() string {
        return quoteV3.HelloV3()
}

func Proverb() string {
        return quoteV3.Concurrency()
}

And then at this point, there’s no need for the renamed import anymore, so we can undo that:

package hello

import "rsc.io/quote/v3"

func Hello() string {
        return quote.HelloV3()
}

func Proverb() string {
        return quote.Concurrency()
}

echo 'package hello

import "rsc.io/quote/v3"

func Hello() string {
        return quote.HelloV3()
}

func Proverb() string {
        return quote.Concurrency()
}' > hello.go

Let’s re-run the tests to make sure everything is working:

$ go test
PASS
ok      example.com/hello       0.014s

go test

Removing unused dependencies

We’ve removed all our uses of rsc.io/quote, but it still shows up in golist-mall and in our go.mod` file:

$ go list -m all
example.com/hello
golang.org/x/text v0.3.0
rsc.io/quote v1.5.2
rsc.io/quote/v3 v3.1.0
rsc.io/sampler v1.3.1
$ cat go.mod
module example.com/hello

go 1.13

require (
	golang.org/x/text v0.3.0 // indirect
	rsc.io/quote v1.5.2
	rsc.io/quote/v3 v3.0.0
	rsc.io/sampler v1.3.1 // indirect
)
$

go list -m all

Why? Because building a single package, like with gobuild or gotest, can easily tell when something is missing and needs to be added, but not when something can safely be removed. Removing a dependency can only be done after checking all packages in a module, and all possible build tag combinations for those packages. An ordinary build command does not load this information, and so it cannot safely remove dependencies.

The gomodtidy command cleans up these unused dependencies:

$ go mod tidy
$ go list -m all
example.com/hello
golang.org/x/text v0.3.0
rsc.io/quote/v3 v3.1.0
rsc.io/sampler v1.3.1
$ cat go.mod
module example.com/hello

go 1.13

require (
	golang.org/x/text v0.3.0 // indirect
	rsc.io/quote/v3 v3.1.0
	rsc.io/sampler v1.3.1 // indirect
)

$ go test
PASS
ok  	example.com/hello	0.020s
$

go mod tidy
go mod list -m all

cat go.mod

go test

#* Conclusion

Go modules are the future of dependency management in Go. Module functionality is now available in all supported Go versions (that is, in Go 1.11 and Go 1.12).

This post introduced these workflows using Go modules:

gomodinit creates a new module, initializing the go.mod file that describes it.
gobuild, gotest, and other package-building commands add new dependencies to go.mod as needed.
golist-mall` prints the current module’s dependencies.
goget` changes the required version of a dependency (or adds a new dependency).
gomodtidy removes unused dependencies.

We encourage you to start using modules in your local development and to add go.mod and go.sum files to your projects. To provide feedback and help shape the future of dependency management in Go, please send us [[https://golang.org/issue/new][bug reports]] or [[https://golang.org/wiki/ExperienceReports][experience reports]].

Thanks for all your feedback and help improving modules.