Support for tools requiring GOPATH during the build

Introduction

There are tools written in Go that require full valid GOPATH to operate. Some examples include:

• stringer - is a tool to automate the creation of methods that satisfy the fmt.Stringer interface.
• mockgen - tool that generates mocks given a set of inputs.
• linters - perform various static checks such as finding unused functions and variables.

Setting up GOPATH is now possible with commands like buck project but it won’t work if build targets depend on code generation done by these tools and we would run into chicken and egg problem where code must exist in order to setup a workspace and workspace setup is needed to generate code.

Possible solutions

We can approach the problem and allow using these tools during the build in two different ways:

• Decouple compilation and execution by refactoring each and every tool and introducing two-step process. On the first step tool generates a wrapper that does the job and that may have external dependencies. On the second step run the tool given compiled binary.
• Symlink a valid GOPATH for each tool. This way all tools would be happy without any changes to the source code.

Main downside of approach number one is that it would require us changing each tool and likely maintain forks because some of those changes would never be upstreamed.

Note that it’s not possible to implement second solution in Skylark since it needs to get a full transitive closure of provided dependencies and symlink them after the build which means that it requires tapping into build phase.

Proposal

We suggest adding a new rule to buck named go_genrule that in addition to normal arguments that genrule takes also accepts deps which can be a list of go_library targets.

Example:

go_genrule(
    name="stringer",
    srcs=glob(
            include = ["*.go"],
            exclude = ["*_test.go"],
        ),
    out=".",
    cmd="stringer ...",
    deps=["//src/code/foo/bar:lib"],

It then will setup transitive closure of deps into a GOPATH-like structure in the working dir (or TMP dir) and will point GOPATH to a right location before running actual binary.

Alternatives

One interesting direction may come from using go modules that may provide ability to run things without having them directly on GOPATH. However this approach requires more exploration as it’s not immediately clear how that would work end to end.

Please provide your feedback on it and let’s discuss what would be preferred path in this case.

[mkaczanowski]

I would be very hesitant to add another flavor of genrule. This is hard to maintain in long term.

Internally we use the mockgen with standard genrule and that works well, since it produces files that just need to be added as sources. Doesn’t that work for you as well?

Example

// wrapper around genrule & mockgen tool
go_mock(
    name = "file_system_mock",
    src = "example/file_system.go",
    out = "file_system_mock.go",
    external = False,
    mock_package = "provider",
    package = "",
)

go_test(
   ...
   srcs = [":file_system_mock"]
)

The external tools usually generates files based on the sources, once passed through the srcs argument, the dependency tree will resolve it correctly. I can imagine the stringer would work in similar fashion.

[mkaczanowski]

We solved gomock (https://github.com/golang/mock) issue by set of genrules and reflection mode.

I don’t have any great solution for stringer though. I think the extra rule is not great but on the other hand I don’t see any better way.

@linzhp @vitarb feel free to come up with PR