Remove the 'ignore' build constraint from integration_test.go and
arrange things such that the integration test runs on longtest
(but not longtest-race, for now anyway) builders. The existing
scripts for running the invocation test locally are preserved
and can be used as before.
This change relies on the builders being configured to place pre-built
protoc and conformance_test_runner binaries in $PATH (CL 547116); thanks
to Michael for making them available via a CIPD package.
Apply a few improvements to downloadFile and downloadArchive that came
about from earlier prototypes that used them and uncovered some issues.
Fixesgolang/go#64066.
Change-Id: I48b163e3ea9c441b748071da340d3e37282cc22b
Co-authored-by: Michael Stapelberg <stapelberg@golang.org>
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/541123
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Drop -mod=vendor because it's not needed by now. There are multiple
reasons for this: 1) integration_test.go has no external dependencies,
2) there's no checked-in vendor directory, 3) CL 544855 deleted the
last 'go mod vendor' invocation in code, and 4) now that the go.mod
has a go directive at 1.14+ (CL 547337), automatic vendoring¹ is on.
For golang/go#64066.
¹ https://go.dev/doc/go1.14#vendor, https://go.dev/ref/mod#vendoring
Change-Id: If8f3ea85d560b2796a2ccdb4ae7dbc38f0760982
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/548115
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Rather than waiting for all tests for all Go versions to finish
only to find that there was some minor breakage,
fail fast so that we stop wasting time.
Change-Id: Ie255ceb5ac2cbdc598a074c6da281f5e49eb1326
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189019
Reviewed-by: Damien Neil <dneil@google.com>
Running "go build ./..." does not descend into testdata directories.
However, the testdata in this repository is source code that is
intended to build properly. We could rename the directory, but that does
not test whether the generated packages can initialize properly.
Thus, we generate a trivial test that simply blank imports all packages.
Doing this reveals that some of the generated files have incorrect imports,
leading to registration conflicts.
To avoid introducing a dependency on gRPC from our go.mod file, we put
the testdata directories in their own module. Also, we avoid running
internal/testprotos through the grpc plugin because the servie and method
definitions in that directory are more for testing proto file initialization
rather than testing grpc generation.
Change-Id: Iaa6a06449787a085200e31bc7606e3ac904d3180
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/164917
Reviewed-by: Damien Neil <dneil@google.com>
Changes made:
* Call autogen.sh before building protobuf.
It's unclear how this worked before, but the README instructions for
protobuf does state to run autogen.sh prior to building and installing.
* Fix downloadArchive to take in a prefix path rather than the number of
prefix directories to skip. The reason for this change is due to a bug
in the Go build system where unexpected directories were being packed.
See https://golang.org/issue/29906
* Explicitly set Travis dist to be "xenial", which comes with Go1.11.
We require Go1.11 for two reasons:
* The use of t.Helper in integration_test.go
* Proper understanding of the -mod=vendor flag
(even if all that flag does is disable modules).
* Add a hack to integration_test.go to periodically output the timestamp
to work around a restriction in Travis where it auto-kills the test
after 10 minutes of no stdout activity.
Change-Id: I114fe2855faeed091c34d79df3d97068be7eccd8
Reviewed-on: https://go-review.googlesource.com/c/164919
Reviewed-by: Herbie Ong <herbie@google.com>
This CL switches the integration test to be written in Go instead of bash.
The benefits are:
* The logic for setting up the dependencies is more robust and
handles better a situation where the dependency failed to initialize
(due to network trouble, FS permission problems, etc).
* The logic does a better job at cleaning up stale dependencies.
For example, my .cache folder has go1.9.4, go1.9.5, go1.10.5, and go1.10.6
folders even though we don't use them anymore.
* Being able to run only a subset of the integration test since you can
pass "-run" to the script.
* A signifcant amount of complexity in the test.bash script was running
the tests in parallel. This is trivial to do in Go.
The major detriment is that Go is more verbose as a "scripting" language.
Change-Id: Id7e5b303fb305fdbc0368bdf809dbf29fca1d983
Reviewed-on: https://go-review.googlesource.com/c/164861
Reviewed-by: Herbie Ong <herbie@google.com>
Create a single binary for handling generation of protos.
This replaces previous logic spread throughout the repo in:
* regenerate.bash
* cmd/protoc-gen-go/golden_test.go
* cmd/protoc-gen-go-grpc/golden_test.go
* (indirectly) internal/protogen/goldentest
One of the problems with the former approaches is that they relied on
a version of protoc that was specific to a developer's workstation.
This meant that the result of generation was not hermetic.
To address this, we rely on the hard-coded version of protobuf specified
in the test.bash script.
A summary of changes in this CL are:
* The internal_gengo.GenerateFile and internal_gengogrpc.GenerateFile
functions are unified to have consistent signatures. It seems that the
former accepted a *protogen.GeneratedFile to support v1 where gRPC code
was generated into the same file as the base .pb.go file. However, the
same functionality can be achieved by having the function return
the generated file object.
* The test.bash script patches the protobuf toolchain to have properly
specified go_package options in each proto source file.
* The test.bash script accepts a "-regenerate" argument.
* Add generation for the well-known types. Contrary to how these were
laid out in the v1 repo, all the well-known types are placed in the
same Go package.
* Add generation for the conformance proto.
* Remove regenerate.bash
* Remove internal/protogen
* Remove cmd/protoc-gen-go/golden_test.go
* Remove cmd/protoc-gen-go-grpc/golden_test.go
* Add cmd/protoc-gen-go/annotation_test.go
Change-Id: I4a1a97ae6f66e2fabcf4e4d292c95ab2a2db0248
Reviewed-on: https://go-review.googlesource.com/c/164477
Reviewed-by: Damien Neil <dneil@google.com>
The golden tests are sensitive to the exact version used:
* Protobuf toolchain since the generated Descriptor is dependendent on protoc
(for example, default json_names were auto-populated in later versions of protoc)
* Go toolchain since the generated .pb.go files is dependent on the exact output
of gofmt and the gzip compression used.
There are other areas where it depends on unstable output, but the above are the
major causes.
Since test.bash ensures that the golden tests are run with exact versions of the
protobuf and Go toolchains, we can ensure that the tests are reproducible across
different developer workstations.
Thus, we add a "golden" build tag to disable them for normal invocations of
"go test ./..." and only run them if test.bash is run.
If test.bash is ever updated with newer versions, the golden testdata can be
updated at the same time.
Change-Id: Ia2b7b039aad2ddaef7652e332215bf9403a6d856
Reviewed-on: https://go-review.googlesource.com/c/142458
Reviewed-by: Damien Neil <dneil@google.com>
This is a straight translation of the v1 API gRPC "plugin" to protogen.
Add a protoc-gen-go-grpc command. The preferred way to generate gRPC
services is to invoke both plugins separately:
protoc --go_out=. --go-grpc_out=. foo.proto
When invoked in this fashion, the generators will produce separate
foo.pb.go and foo_grpc.pb.go files.
Change-Id: Ie180385dab3da7063db96f7c2f9de3abbd749f63
Reviewed-on: https://go-review.googlesource.com/137037
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The "go list" command is used only to get the module name.
However, just invoking the command results in a series of network IO
fetching package information about dependencies.
That information is then placed in the default GOPATH,
which previously was the system's default.
This is not generally a problem, but results in additional time spent
in Travis since the system's GOPATH is not cached between test runs.
By setting the GOPATH beforehand to be within TEST_DIR, we can ensure
that the result is cached and preserved between runs.
Change-Id: Ib750a9a31b9891b354fe9a70b3f592d949186ba3
Reviewed-on: https://go-review.googlesource.com/136738
Reviewed-by: Damien Neil <dneil@google.com>
The minimum supported version is currently Go1.10 since we use strings.Builder
for a fairly significant optimization when constructing all of the descriptor
full names.
The strings.Builder implementation is not particularly complicated,
so just fork it into our code. The golang/protobuf and the golang/go
projects share the same authors and copyright license.
Change-Id: Ibb9519dbe756327a07369f10f80c15761002b5e7
Reviewed-on: https://go-review.googlesource.com/136735
Reviewed-by: Damien Neil <dneil@google.com>
Use a local ".cache" directory in the repository root for the cache
instead of a directory in /tmp. On many operating systems, files are arbitrarily
purged as part of the OS' cleanup procedures, leading to obscure failures.
Since the build cache is in the repo root, we adjust the invocation of gofmt
to only act upon the set of Go files tracked by git.
Change-Id: I10cd0ea0de3370ea25fcd3870451fb41c4204a4b
Reviewed-on: https://go-review.googlesource.com/133875
Reviewed-by: Damien Neil <dneil@google.com>
Changes made:
* create a /bin directory for symlinks and prepend it to the PATH
* include the "go" prefix in the Go toolchain versions
* move setting up GOPATH before calling "go mod vendor" since modules dump their
results inside a special directory within the GOPATH
* setup a per-version Go cache to 1) work around a bug in the toolchain and
2) to allow Travis to also cache this state
* factor out common logic into a check function that ensures stdout is empty
Change-Id: I9400452b85a66f019cb616aaded33610f2c1eb2f
Reviewed-on: https://go-review.googlesource.com/129402
Reviewed-by: Herbie Ong <herbie@google.com>
The prototype package provides constructors to create protobuf types that
implement the interfaces defined in the protoreflect package.
High-level API:
func NewFile(t *File) (protoreflect.FileDescriptor, error)
type File struct{ ... }
type Message struct{ ... }
type Field struct{ ... }
type Oneof struct{ ... }
type Enum struct{ ... }
type EnumValue struct{ ... }
type Extension struct{ ... }
type Service struct{ ... }
type Method struct{ ... }
func NewEnum(t *StandaloneEnum) (protoreflect.EnumDescriptor, error)
func NewMessage(t *StandaloneMessage) (protoreflect.MessageDescriptor, error)
func NewExtension(t *StandaloneExtension) (protoreflect.ExtensionDescriptor, error)
type StandaloneEnum struct{ ... }
type StandaloneMessage struct{ ... }
type StandaloneExtension struct{ ... }
func PlaceholderFile(path string, pkg protoreflect.FullName) protoreflect.FileDescriptor
func PlaceholderEnum(name protoreflect.FullName) protoreflect.EnumDescriptor
func PlaceholderMessage(name protoreflect.FullName) protoreflect.MessageDescriptor
This CL is missing some features that are to be added later:
* The stringer methods are not implemented, providing no way to print the
descriptors in a humanly readable manner.
* There is no support for proto options or retrieving the raw descriptor.
* There are constructors for Go specific types (e.g., protoreflect.MessageType).
We drop go1.9 support since we use strings.Builder.
We switch to go.11rc1 to obtain some bug fixes for modules.
Change-Id: Ieac9a2530afc81e5a5bb9ab5816804372f652b18
Reviewed-on: https://go-review.googlesource.com/129057
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Herbie Ong <herbie@google.com>
Package protoreflect provides APIs for programatically interacting with messages
according to the protobuf type system.
Change-Id: I11fa3874e4b3f94771514c69efb2ae8bb812d7fa
Reviewed-on: https://go-review.googlesource.com/127823
Reviewed-by: Damien Neil <dneil@google.com>
Modify the test harness to run Go tests concurrently.
This vastly speeds up execution time on machines with many cores.
Other changes:
* Some stylistic changes to make the output colored and more readable.
* Add protoc and conformance-test-runner to the $PATH
* Check that Go source files were properly formatted
Change-Id: I5d7e14defeef32c587e6f8e2f66367143413a41f
Reviewed-on: https://go-review.googlesource.com/128415
Reviewed-by: Herbie Ong <herbie@google.com>
When Travis-CI runs the test.bash, it unnecessarily pollutes the environment
with a GOROOT environment variable, which confuses each of the custom invocations
of different Go toolchains. Rather than setting the GOROOT prior to each
invocation of the Go toolchain, just clear the variable once, and let each
toolchain figure out the root for themselves (which all recent versions of Go
know how to do so correctly).
Change-Id: I45ae9f9f98f5573ff42502b7244c90bf1569f66e
Reviewed-on: https://go-review.googlesource.com/128363
Reviewed-by: Herbie Ong <herbie@google.com>
Use module support in Go1.11 to download the exact version of dependencies
as specified in the go.mod file, this is contrary to "go get -u", which grabs
the latest version, making reproducible builds and tests difficult.
In order for Go1.9 and Go1.10 to work, we also emit a vendor directory for
pre-module support.
Lastly, check whether the go.mod or go.sum files changed
(by shelling to "git diff"). This provides protection in case
a new dependency was added and the go.mod file was not updated.
Change-Id: Iac4e9b224ca9188dc9e63be720f188bfb5ee56ef
Reviewed-on: https://go-review.googlesource.com/127916
Reviewed-by: Bryan C. Mills <bcmills@google.com>
This script is not currently executed by Try-Bots since they do not have the
necessary infrastructure, yet. Instead, this script is intended to be ran as
a pre-commit hook on each contributor's workstation.
The test.bash script works on both Darwin and Linux and tests the repository
across a variety of Go toolchains. The toolchains are downloaded and built
into their own versioned directories and cached. Subsequent runs of the script
only need to run the tests.
Change-Id: Id87841ed42da422ab2e966ea073255ba866c80f7
Reviewed-on: https://go-review.googlesource.com/127826
Reviewed-by: Herbie Ong <herbie@google.com>
