Changes made:
* Call autogen.sh before building protobuf.
It's unclear how this worked before, but the README instructions for
protobuf does state to run autogen.sh prior to building and installing.
* Fix downloadArchive to take in a prefix path rather than the number of
prefix directories to skip. The reason for this change is due to a bug
in the Go build system where unexpected directories were being packed.
See https://golang.org/issue/29906
* Explicitly set Travis dist to be "xenial", which comes with Go1.11.
We require Go1.11 for two reasons:
* The use of t.Helper in integration_test.go
* Proper understanding of the -mod=vendor flag
(even if all that flag does is disable modules).
* Add a hack to integration_test.go to periodically output the timestamp
to work around a restriction in Travis where it auto-kills the test
after 10 minutes of no stdout activity.
Change-Id: I114fe2855faeed091c34d79df3d97068be7eccd8
Reviewed-on: https://go-review.googlesource.com/c/164919
Reviewed-by: Herbie Ong <herbie@google.com>
Generate a list of descriptor fields from the descriptor.proto itself
as an internal package. Use these fields for the internal implementation.
Change-Id: Ib1ab0c5c6deb332ba6c8018ef55136b7e5974944
Reviewed-on: https://go-review.googlesource.com/c/164864
Reviewed-by: Herbie Ong <herbie@google.com>
We're rewriting internal/encoding/json. So, make a copy of it first in
order not to break encoding/jsonpb package.
Change-Id: I8b63c468d3f432102d2af4db22a7549998ce3876
Reviewed-on: https://go-review.googlesource.com/c/164642
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
This CL switches the integration test to be written in Go instead of bash.
The benefits are:
* The logic for setting up the dependencies is more robust and
handles better a situation where the dependency failed to initialize
(due to network trouble, FS permission problems, etc).
* The logic does a better job at cleaning up stale dependencies.
For example, my .cache folder has go1.9.4, go1.9.5, go1.10.5, and go1.10.6
folders even though we don't use them anymore.
* Being able to run only a subset of the integration test since you can
pass "-run" to the script.
* A signifcant amount of complexity in the test.bash script was running
the tests in parallel. This is trivial to do in Go.
The major detriment is that Go is more verbose as a "scripting" language.
Change-Id: Id7e5b303fb305fdbc0368bdf809dbf29fca1d983
Reviewed-on: https://go-review.googlesource.com/c/164861
Reviewed-by: Herbie Ong <herbie@google.com>
Using arrays in the generated reflection information adds unnecessary
eq and hash functions being added to the package. Change to slices
to reduce bloat.
Change-Id: I1a4f6d59021644d93dd6c24679b9233141e89a75
Reviewed-on: https://go-review.googlesource.com/c/164640
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Create a single binary for handling generation of protos.
This replaces previous logic spread throughout the repo in:
* regenerate.bash
* cmd/protoc-gen-go/golden_test.go
* cmd/protoc-gen-go-grpc/golden_test.go
* (indirectly) internal/protogen/goldentest
One of the problems with the former approaches is that they relied on
a version of protoc that was specific to a developer's workstation.
This meant that the result of generation was not hermetic.
To address this, we rely on the hard-coded version of protobuf specified
in the test.bash script.
A summary of changes in this CL are:
* The internal_gengo.GenerateFile and internal_gengogrpc.GenerateFile
functions are unified to have consistent signatures. It seems that the
former accepted a *protogen.GeneratedFile to support v1 where gRPC code
was generated into the same file as the base .pb.go file. However, the
same functionality can be achieved by having the function return
the generated file object.
* The test.bash script patches the protobuf toolchain to have properly
specified go_package options in each proto source file.
* The test.bash script accepts a "-regenerate" argument.
* Add generation for the well-known types. Contrary to how these were
laid out in the v1 repo, all the well-known types are placed in the
same Go package.
* Add generation for the conformance proto.
* Remove regenerate.bash
* Remove internal/protogen
* Remove cmd/protoc-gen-go/golden_test.go
* Remove cmd/protoc-gen-go-grpc/golden_test.go
* Add cmd/protoc-gen-go/annotation_test.go
Change-Id: I4a1a97ae6f66e2fabcf4e4d292c95ab2a2db0248
Reviewed-on: https://go-review.googlesource.com/c/164477
Reviewed-by: Damien Neil <dneil@google.com>
Use the full path (including the extension) for the generation of
the per-file variable name. Several reasons for this:
* The current logic is buggy in the case where pathType == pathTypeImport
since the prefix variable will be mangled with the Go import path.
* The extension is technically part of the path.
Thus, "path/to/foo.proto" and "path/to/foo.protodevel" are two
distinctly different imports.
* Style-wise, it subjectively looks better. Rather than being a mixture
of camelCase and snake_case, it is all snake_case for the common case:
before: ProtoFile_google_protobuf_any
after: File_google_protobuf_any_proto
* Since the extension is almost always ".proto", this results in a
suffix of "_proto", which provides an additional layer of protection
against possible name conflicts. The previous approach could possibly
have a conflict between "Foo.proto" and a message named ProtoFile
with a sub-message called Foo.
Also, use the per-file variable name for the raw descriptor variables
instead of the hashed version.
Change-Id: Ic91e326b7593e5985cee6ececc60539c27fe32fe
Reviewed-on: https://go-review.googlesource.com/c/164379
Reviewed-by: Damien Neil <dneil@google.com>
Calling a helper function directly should reduce binary bloat slightly.
Change-Id: I6068dc4cd00c8d90d2e6e6d99633b81388bc8781
Reviewed-on: https://go-review.googlesource.com/c/164679
Reviewed-by: Damien Neil <dneil@google.com>
For a file "foo/bar.proto", put the FileDescriptor in "ProtoFile_foo_bar"
rather than "Bar_fileDescriptor".
Avoid name clashes when a package contains "a/foo.proto" and "b/foo.proto".
Don't camelcase the filename: These vars weren't fully camelcased to begin
with, and leaving the filename relatively unchanged is clearer and more
predictable.
Move "ProtoFile" from the end of the var name to the start, so that vars
will sort better in packages with multiple descriptors.
These changes do add a chance of name collision when the input filename
begins with an uppercase letter: Foo.proto becomes "ProtoFile_Foo", which
could be the result of camelcasing "proto_file.foo". The readability
benefits seem worth it.
Change-Id: If27d3a0d7b5bf3535aa1607a8579eb057c74d2dc
Reviewed-on: https://go-review.googlesource.com/c/163199
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Reviewed-by: Herbie Ong <herbie@google.com>
This does not handle extensions, messagesets, and well-known types yet.
Change-Id: I2786c429f490fe8c57f3f85cd25058d936b58bf7
Reviewed-on: https://go-review.googlesource.com/c/162637
Reviewed-by: Damien Neil <dneil@google.com>
These test proto definitions will be reused for encoding/jsonpb package
and hence move these one directory up.
Also, add and simplify some tests.
Change-Id: I5297546fd9b853a7fd3e72dfab2fdc7237332c9c
Reviewed-on: https://go-review.googlesource.com/c/162537
Reviewed-by: Damien Neil <dneil@google.com>
This test examines the result of converting math.NaN() to a fixed byte
string. Change it to use a specific NaN value instead, since the value
returned by math.NaN is specified only as being a NaN, not a specific
one.
Use specific float32 and float64 NaN values, since the result of
converting a float64 NaN to a float32 can and does vary.
Fixes test failure on ARM.
Change-Id: Ia1517fdba768cdd88e5ee5f5af37f0b481e651b4
Reviewed-on: https://go-review.googlesource.com/c/162117
Reviewed-by: Herbie Ong <herbie@google.com>
Updates golang/protobuf#798 by adding testcases to show the intention.
Also, slightly move code blocks in encode.go w/o affecting logic to make
it cleaner.
Change-Id: I14575f6e7139a0908483bd318b599339c2daf8ad
Reviewed-on: https://go-review.googlesource.com/c/161717
Reviewed-by: Damien Neil <dneil@google.com>
This only handles compliant MessageSet extension fields where the field
name has to be message_set_extension.
Current C++ lib allows for different message field names, which is a
possible bug as it makes marshal output possibly contain duplicate names
when more than one field extends the same MessageSet, and makes
unmarshaling confusing as to which field to populate.
Change-Id: Ifda828ba794fe7e058ee6004f03001b1031f6d1e
Reviewed-on: https://go-review.googlesource.com/c/156758
Reviewed-by: Damien Neil <dneil@google.com>
These methods are no longer used. Drop them.
Change-Id: Ie5595c1a56dd76183782019384d16ea97d5f7e9e
Reviewed-on: https://go-review.googlesource.com/c/160357
Reviewed-by: Herbie Ong <herbie@google.com>
Test the fileinit and protodesc packages by verifying that passing a
FileDescriptorProto through a round trip of these packages leaves it
unchanged.
Change-Id: I6bfb894d95f1736f9908adee1ab63e9653b3f1be
Reviewed-on: https://go-review.googlesource.com/c/159762
Reviewed-by: Herbie Ong <herbie@google.com>
Add a collection of functions which take a reflect.*Descriptor and
return the corresponding google.protobuf.*DescriptorProto.
Change-Id: Ic186c412c8d3b7bc582c31bb8d8274459ce51a20
Reviewed-on: https://go-review.googlesource.com/c/159761
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
This CL takes a significantly different approach to generating support
for protobuf reflection. The previous approach involved generating a
large number of Go literals to represent the reflection information.
While that approach was correct, it resulted in too much binary bloat.
The approach taken here initializes the reflection information from
the raw descriptor proto, which is a relatively dense representation
of the protobuf reflection information. In order to keep initialization
cost low, several measures were taken:
* At program init, the bare minimum is parsed in order to initialize
naming information for enums, messages, extensions, and services declared
in the file. This is done because those top-level declarations are often
relevant for registration.
* Only upon first are most of the other data structures for protobuf
reflection actually initialized.
* Instead of using proto.Unmarshal, a hand-written unmarshaler is used.
This allows us to avoid a dependendency on the descriptor proto and also
because the API for the descriptor proto is fundamentally non-performant
since it requires an allocation for every primitive field.
At a high-level, the new implementation lives in internal/fileinit.
Several changes were made to other parts of the repository:
* cmd/protoc-gen-go:
* Stop compressing the raw descriptors. While compression does reduce
the size of the descriptors by approximately 2x, it is a pre-mature
optimization since the descriptors themselves are around 1% of the total
binary bloat that is due to generated protobufs.
* Seeding protobuf reflection from the raw descriptor significantly
simplifies the generator implementation since it is no longer responsible
for constructing a tree of Go literals to represent the same information.
* We remove the generation of the shadow types and instead call
protoimpl.MessageType.MessageOf. Unfortunately, this incurs an allocation
for every call to ProtoReflect since we need to allocate a tuple that wraps
a pointer to the message value, and a pointer to message type.
* internal/impl:
* We add a MessageType.GoType field and make it required that it is
set prior to first use. This is done so that we can avoid calling
MessageType.init except for when it is actually needed. The allows code
to call (*FooMessage)(nil).ProtoReflect().Type() without fearing that the
init code will run, possibly triggering a recursive deadlock (where the
init code depends on getting the Type of some dependency which may be
declared within the same file).
* internal/cmd/generate-types:
* The code to generate reflect/prototype/protofile_list_gen.go was copied
and altered to generated internal/fileinit.desc_list_gen.go.
At a high-level this CL adds significant technical complexity.
However, this is offset by several possible future changes:
* The prototype package can be drastically simplified. We can probably
reimplement internal/legacy to use internal/fileinit instead, allowing us
to drop another dependency on the prototype package. As a result, we can
probably delete most of the constructor types in that package.
* With the prototype package significantly pruned, and the fact that generated
code no longer depend on depends on that package, we can consider merging
what's left of prototype into protodesc.
Change-Id: I6090f023f2e1b6afaf62bd3ae883566242e30715
Reviewed-on: https://go-review.googlesource.com/c/158539
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
When encoding/textpb marshals out float32 values, it was previously
formatting it as float64 bitsize since both float types are stored as
float64 and internal/encoding/text only has one Float type. A
consequence of this is that the output may display a different value
than expected, e.g. 1.02 becomes 1.0199999809265137.
This CL splits Float type into Float32 and Float64 to keep track of
which bitsize to use when formatting. Values of both types are still
stored as float64 to keep the logic simple.
Decoding will always use Float64, but users can ask for a float32 value
from it.
Change-Id: Iea5b14b283fec2236a0c3946fac34d4d79b95274
Reviewed-on: https://go-review.googlesource.com/c/158497
Reviewed-by: Damien Neil <dneil@google.com>
In https://golang.org/cl/157821, I attempted to fix handling of group
field names but forgot about repeated group fields as I forgot to
properly update the marshaling tests as well for it.
Unmarshal logic for both repeated and non-repeated was already fixed in
that CL.
Change-Id: Icb4a00d8b169709ca12dfee272b2bd73e7585e6e
Reviewed-on: https://go-review.googlesource.com/c/157857
Reviewed-by: Damien Neil <dneil@google.com>
Also fix marshaling Any in expanded form to contain the correct type_url
value.
Change-Id: I4b467e74bb1d73255effd9cc4cfff9cf4558940f
Reviewed-on: https://go-review.googlesource.com/c/156342
Reviewed-by: Damien Neil <dneil@google.com>
Group field name in textproto should be the type name. Its field name
is derived from lowercasing its type name.
Change-Id: Ia12aafe934d3a59f3e07d09fe7939cfa6595a7b8
Reviewed-on: https://go-review.googlesource.com/c/157821
Reviewed-by: Damien Neil <dneil@google.com>
A "_ProtoFile" suffix can potentially conflict with a sub-message named
"ProtoFile" nested within a message that matches the camel-cased
form of the basename of the .proto source file.
Avoid unlikely conflicts and rename this to use a "_protoFile" suffix,
which can never conflict except with an enum value that is also named
"protoFile" (which is a violation of the style guide).
Change-Id: Ie9d22f9f741a63021b8f76906b20c6c2f599885b
Reviewed-on: https://go-review.googlesource.com/c/157218
Reviewed-by: Damien Neil <dneil@google.com>
Rather than maintaining our own list of builtin Go identifiers, use the
types.Universe API in the standard library instead.
It is okay if this list of names changes over time since we use this only
to derive a locally-used package name in the generated file.
Change-Id: Ib1688abc47d5c97b557f6e1f4d60c78e0951e65b
Reviewed-on: https://go-review.googlesource.com/c/157818
Reviewed-by: Damien Neil <dneil@google.com>
This change preserves consistency with CL/157077,
where New returns a value closer to the reflective type.
Change-Id: I85bfdae24e1ce1a10c3c7b939420fa1043bff743
Reviewed-on: https://go-review.googlesource.com/c/157078
Reviewed-by: Damien Neil <dneil@google.com>
Rephrase the documentation in terms of usage of this package: Lead with
the description of what this package contains and provide guidance on where
to get a descriptor or value interface.
Change-Id: I40a43cd59d1fbca6c60eff1c3afe50ff4a15b82f
Reviewed-on: https://go-review.googlesource.com/c/157217
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Most usages of New actually prefer to interact with the reflective view
rather than the native Go type. Thus, change New to return that instead.
This parallels reflect.New, which returns the reflective view
(i.e., reflect.Value) instead of native type (i.e., interface{}).
We make the equivalent change to KnownFields.NewMessage, List.NewMessage,
and Map.NewMessage for consistency.
Since this is a subtle change where the type system will not always
catch the changed type, this change was made by both changing the type
and renaming the function to NewXXX and manually looking at every usage
of the the function to ensure that the usage correctly operates
on either the native Go type or the reflective view of the type.
After the entire codebase was cleaned up, a rename was performed to convert
NewXXX back to New.
Change-Id: I153fef627b4bf0a427e4039ce0aaec52e20c7950
Reviewed-on: https://go-review.googlesource.com/c/157077
Reviewed-by: Damien Neil <dneil@google.com>
The generator currently uses an unintuitive and stateful algorithm
for name generation where it "fixes" name conflicts by appending "_"
to the end of the new name.
PR#657 refactored the generator code and noticed that the above
algorithm was not properly taking into account that a Get method is
generated for parent oneofs, fixing it in the same PR. While this is
more correct, this breaks users (see #780) since it means that the
generation of names can change.
This PR changes the name mangling logic to be as it was previously.
This does mean that some new proto files may be unbuildable,
but that is arguably better than breaking existing proto files
Change-Id: I2e354f4bb5d9c2b562fa2faa9149e949e2d86a0f
Reviewed-on: https://go-review.googlesource.com/c/156877
Reviewed-by: Damien Neil <dneil@google.com>
Drop the protoreflect.ProtoEnum type (containing a single method
returning a protoreflect.Enum) and make generated enum types
directly implement protoreflect.Enum instead.
Messages have a two-level type split (ProtoMessage and Message) to
minimize conflicts between reflection methods and field names. Enums
need no such split, since enums do not have fields and therefore have
no source of conflicts.
Change-Id: I2b6222e9404253e6bfef2217859e1b760ffcd29b
Reviewed-on: https://go-review.googlesource.com/c/156902
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Reviewed-by: Damien Neil <dneil@google.com>
Consider the case:
a.proto publicly imports b.proto
b.proto publicly imports c.proto
Should a.pb.go include symbols defined in c.pb.go?
Historically, it has not. As of #155677, it does. Regardless of which behavior
is preferable, #155677 produces broken code in some common situations: If
a.proto also publicly imports c.proto, we now generate two copies of the
forwarding decls for that file.
Restore the pre-#155677 behavior to avoid this breakage.
Change-Id: I283600b3be19eac2c3b3c14233bb69fa64661581
Reviewed-on: https://go-review.googlesource.com/c/156348
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
New test case demonstrating an import public bug:
a.proto publicly imports b.proto and c.proto
b.proto publicly imports c.proto
a.pb.go includes two copies of symbols in c.pb.go
The problem is that the new implementation of public imports, which
parses the generated code for the import to determine what symbols
are present, doesn't distinguish between symbols defined in a file
and symbols imported from a public import of some other file. This
can then lead to duplicate definitions in cases like the above.
Change-Id: Ia86e9f188d7bae8d9d4afbd2b5db9b64071425c3
Reviewed-on: https://go-review.googlesource.com/c/156347
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
File is out of date, regenerate it. (This didn't show up as a test failure
because we don't test the non-grpc golden files under the grpc directory.)
Change-Id: Ied485d28184c45bbca5b52138eb9cf300d813a57
Reviewed-on: https://go-review.googlesource.com/c/156345
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Change the go_package option to match the actual import path of this package.
Change-Id: Ie8630878ce75e34ca76d97c6e2922254cf964801
Reviewed-on: https://go-review.googlesource.com/c/156344
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
If there are any kind of errors in trying to expand the Any message,
always fallback to marshaling it as regular message. This makes it
consistent with V1 and C++ libs.
Change-Id: I007414c1767e682623c45d4dd8c82b9998f61781
Reviewed-on: https://go-review.googlesource.com/c/156257
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Marshal well-known type Any in expanded form by default, else fallback
to marshaling it as a regular message.
Change-Id: Ic7e9e37b47042a163941f8849dc366ffe48103ca
Reviewed-on: https://go-review.googlesource.com/c/156097
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Rather than explicitly enumerating the set of symbols to import,
just parse the imported file and extract every exported symbol.
This is possibly a bit more code, but adapts much better to future
expansion.
Change-Id: I4429664f4c068a2a55949d46aefc19865b008a77
Reviewed-on: https://go-review.googlesource.com/c/155677
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Always generate the init func containing the proto.RegisterFile call.
This call used to be in a different init func which was unconditionally
generated. Consolidating the init funcs means that we need to always
generate the single remaining func.
Change-Id: Icbf7d14d018d693dab2824f114c57e4e36384568
Reviewed-on: https://go-review.googlesource.com/c/156197
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The first commit of protoregistry only added a registry for files.
However, a separate type of registry is needed to provide a mapping between
protobuf names and actual Go types representing those names.
Additional high-level API:
var GlobalTypes = new(Types)
type Type interface{ ... }
type Types struct{ ... }
func NewTypes(...Type) *Types
func (*Types) Register(...Type) error
func (*Types) FindEnumByName(pref.FullName) (pref.EnumType, error)
func (*Types) FindMessageByName(pref.FullName) (pref.MessageType, error)
func (*Types) FindMessageByURL(string) (pref.MessageType, error)
func (*Types) FindExtensionByName(pref.FullName) (pref.ExtensionType, error)
func (*Types) FindExtensionByNumber(pref.FullName, pref.FieldNumber) (pref.ExtensionType, error)
func (*Types) RangeEnums(func(pref.EnumType) bool)
func (*Types) RangeMessages(func(pref.MessageType) bool)
func (*Types) RangeExtensions(func(pref.ExtensionType) bool)
func (*Types) RangeExtensionsByMessage(pref.FullName, func(pref.ExtensionType) bool)
Change-Id: I0d07705801684a1eb5853bcd05fcce12598a0047
Reviewed-on: https://go-review.googlesource.com/c/131345
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The purego tag (see https://golang.org/issue/23172) is a community agreed
upon signal that a given build environment does not support unsafe.
The appengine environment is supposed to respect this tag, but does not
properly do so. Add this tag back in until they fix their environment.
Change-Id: I9a70062be4339c2e1a93cac31d387698c561b8aa
Reviewed-on: https://go-review.googlesource.com/c/154743
Reviewed-by: Damien Neil <dneil@google.com>
Introduce a protoreflect.OptionsMessage interface type to represent
options messages. The protoreflect package can't deal with concrete
options types, to avoid dependency cycles; OptionsMessage serves both
as documentation and a single point to define the constraints we apply
to options.
Change the constraints on options from ProtoMessage to interface{} to
permit use of option message types which only implement the v1
proto.Message interface.
This still leaves a requirement in the internal/legacy package that
options implement protoreflect.ProtoMessage, since that package now uses
the v2 Unmarshal.
Change-Id: I547518ab2c3b90c3911ef641b05b169d50a4b33a
Reviewed-on: https://go-review.googlesource.com/c/154877
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
