118 Commits

Author SHA1 Message Date
Damien Neil
3b46adeb85 proto: add proto3 encode/decode test cases
Change-Id: I5f859fbd34cbd3e95a6c39a5581791dedc619e6b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169477
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-26 21:54:17 +00:00
Joe Tsai
35ec98fdcb cmd/protoc-gen-go: generate for v2-only dependencies
This removes yet another set of dependencies of v2 on v1.
The only remaining dependency are in the _test.go files,
primarily for proto.Equal.

Changes made:
* cmd/protoc-gen-go no longer generates any functionality that depends
on the v1 package, and instead only depends on v2.
* internal/fileinit.FileBuilder.MessageOutputTypes is switched from
protoreflect.MessageType to protoimpl.MessageType since the
implementation must be fully inialized before registration occurs.
* The test for internal/legacy/file_test.go is switched to a legacy_test
package to avoid a cyclic dependency.
This requires Load{Enum,Message,File}Desc to be exported.

Change-Id: I43e2fe64cff4eea204258ce11e791aca5eb6e569
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169298
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-26 17:03:31 +00:00
Herbie Ong
a3421952ac internal/encoding/json: improve decoding of JSON numbers for floats
Per Joe's suggestion, remove producing numberParts when parsing a JSON
number to produce corresponding Value. This saves having to store it
inside Value as well. Only produce numberParts for calls to
Value.{Int,Uint} call.

numberParts is only used for producing integers and removing the logic to
produce numberParts improves overall decoding speed for floats, and shows no
change for integers.

name     old time/op  new time/op  delta
Float-4   559ns ± 0%   288ns ± 0%   ~     (p=1.000 n=1+1)
Int-4     471ns ± 0%   466ns ± 0%   ~     (p=1.000 n=1+1)

Change-Id: I21bf304ca67dda8d41a4ea0022dcbefd51058c1c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168781
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-23 22:02:55 +00:00
Joe Tsai
4fddebafc0 all: move v1 types over to the v2 repository
As a goal, v2 should not depend on v1. As another step towards that end,
we move all the types that used to be in the v1 protoapi package over to v2.

For now, we place MessageV1, ExtensionRangeV1, and ExtensionDescV1
in runtime/protoiface since these are types that generated messages will
probably have to reference forever. An alternative location could be
reflect/protoreflect, but it seems unfortunate to have to dirty the
namespace of that package with these types.

We move ExtensionFieldV1, ExtensionFieldsV1, and ExtensionFieldsOf
to internal/impl, since these are related to the implementation of a
generated message.

Since moving these types from v1 to v2 implies that the v1 protoapi
package is useless, we update all usages of v1 protoapi in the v2
repository to point to the relevant v2 type or functionality.

CL/168538 is the corresponding change to alter v1.
There will be a temporary build failure as it is not possible
to submit CL/168519 and CL/168538 atomically.

Change-Id: Ide4025c1b6af5b7f0696f4b65b988b4d10a50f0b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168519
Reviewed-by: Herbie Ong <herbie@google.com>
2019-03-22 20:01:07 +00:00
Joe Tsai
4532dd7969 internal/fileinit: prevent map entry descriptors from implementing MessageType
The protobuf type system hacks the representation of map entries into that
of a pseudo-message descriptor.

Previously, we made all message descriptors implement MessageType
where type descriptors had a GoType method that simply returned nil.
Unfortunately, this violates a nice property in the Go type system
where being able to assert to a MessageType guarantees that Go type
information is truly associated with that descriptor.

This CL makes it such that message descriptors for map entries
do not implement MessageType.

Change-Id: I23873cb71fe0ab3c0befd8052830ea6e53c97ca9
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168399
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-21 17:47:49 +00:00
Joe Tsai
ca46d8c924 internal/fieldnum: generate field numbers for the google.protobuf package
Generate field numbers for the well-known types,
so that encoding/jsonpb can benefit from them as well.

This CL fixes internal/cmd/generate-protos, which was silently failing
because the modulePath was not properly initialized. We fix this by
moving it to the start of the init function.

Change-Id: I87637176f29218cffa512b4baa49f39dae924061
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168497
Reviewed-by: Herbie Ong <herbie@google.com>
2019-03-21 00:57:40 +00:00
Herbie Ong
3a385917c0 encoding/jsonpb: change MarshalOptions.Compact option to Indent
This makes it consistent with jsonpb.MarshalOptions. This does change
the default to be in compact form.

Change-Id: I1b07f06f282c019b30f3f1cbb43f6c8cba18f385
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168405
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-20 22:46:14 +00:00
Joe Tsai
1321a0e05b cmd/protoc-gen-go: generate descriptor proto using v2 textpb
Changes:
* Modify protoc-gen-go to use a helper from internal/impl
to print the message as text.
* Add a helper function to internal/impl that calls v2 textpb.
* Modify encoding/textpb to avoid depending on descriptor proto,
which would cause an import cycle.
* Modify internal/fileinit to populate a pseudo-internal
method on MessageDescriptor to check whether the message should
use the message set wire format. We avoid adding this to the
main MessageDescriptor interface since message sets are a
deprecated proto1 feature that we should avoid promoting.

Change-Id: Ibaf79a563af695756f11ddc4db69b38e25a8f1a7
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168439
Reviewed-by: Herbie Ong <herbie@google.com>
2019-03-20 20:58:24 +00:00
Joe Tsai
559d47f1da cmd/protoc-gen-go: fix init order for v1 registration
The v1 registration leaks the message types out to the proto package.
When doing that, it must ensure that the reflection data structures
for those types are properly initialized first. We achieve that by
doing v1 registration at the end of the reflection init function.

Change-Id: If6df18df59d05bad50ff39c2eff6beb19e7466cc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168348
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-20 04:16:33 +00:00
Joe Tsai
b936504483 internal/legacy: guarantee one-to-one generation of wrapper types
When concurrent requests to wrap a type occurs, it is possible that
two different descriptors are created for the same Go type.
Add sufficient synchronization to ensure that one descriptor is ever
returned for one Go type.

Change-Id: Idbbca4c1877a70317b39900ae83bfc3085d4a9c5
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168398
Reviewed-by: Herbie Ong <herbie@google.com>
2019-03-19 23:48:09 +00:00
Joe Tsai
08cd8842f1 cmd/protoc-gen-go: register with the v2 registry for descriptor
By passing the global registries to fileinit.Builder, the Build
method can register the newly constructed descriptors on our behalf.
As a result, we can stop registering with the v1 registries as well.

We only do this for descriptor proto to remove another dependency on v1.
We deliberately keep the v1 registration logic for now to make it easy
to patch away this new behavior.

Change-Id: Ic6aa8ffba3d2d0abe08a61fc5e1c9ca7668e0988
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168000
Reviewed-by: Herbie Ong <herbie@google.com>
2019-03-19 05:04:43 +00:00
Joe Tsai
ce07f39688 internal/cmd/generate-protos: disable detrand
The .pb.go.meta file is checked into the repository as testdata.
The deliberate instability of outputs breaks golden tests within
our own repository. It is reasonable for us to depend on stability
since we control the output.

Change-Id: I1f73027a08a0757732d46610f334d40840cc4cfd
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168001
Reviewed-by: Herbie Ong <herbie@google.com>
2019-03-19 05:04:31 +00:00
Joe Tsai
8e506a8704 cmd/protoc-gen-go: rely on protoimpl for basic helpers
The EnumName, UnmarshalJSONEnum, and CompressGZIP helpers currently live
in v1 protoapi, which would cause all generated messages to depend on v1.
In an effort to break the dependency of v2 on v1, we move these helper
functions to v2 (and re-written to take advantage of protobuf reflection).

These helpers are unfortunate, but we cannot eliminate the functionality
that they implement since they are exposed in the publicly generated API.

Since EnumName does not rely on the enum maps, it removes another dependency
on those variables. Eventually, we can get to the point where these variables
(though declared) are not linked into the binary if the user does not use them.

Also, we rely on the v1 proto package for registration instead of v1 protoapi.
This may re-introduce a cyclic dependency on descriptor proto again in the
future, but the better approach is to just start registering with v2.

Change-Id: Id755585a7a1df14e4a6a2dfa650df221a3c153fb
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/167921
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-18 18:50:16 +00:00
Herbie Ong
c96a79da29 encoding/jsonpb: add support for basic unmarshaling
Unmarshaling of scalar, messages, repeated, and maps.

Need to further improve on error messages for consistency, some error
messages contain the position info while some currently do not.  There
are cases where position info is wrong as well when a value is decoded
in another pass, e.g. numbers in string value, or map keys.

Change-Id: I6f9e903c499b5e87fb258dbdada7434389fc7522
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/166338
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-15 18:53:18 +00:00
Joe Tsai
afb455eaf8 cmd/protoc-gen-go: correlate v1 ExtensionDesc with v2 ExtensionType
Unfortunately a good amount of code uses pointer comparisons on the
v1 ExtensionDesc to determine exactly which extension field is set,
rather than checking whether the extension descriptor semantically
describes the field that they are interested in.

To preserve this behavior in v1, we need a 1:1 mapping between
a v2 ExtensionType and a specific v1 ExtensionDesc.

Change-Id: I852b3cefb4585bd656e48e5adad6cc28795d02df
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/167759
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-15 01:24:42 +00:00
Joe Tsai
69996ba400 internal/legacy: use proto.Unmarshal from v2
Change-Id: Ic1e757a11f233657d6c3a840719ebef34d81e4f5
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/167716
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-14 21:46:02 +00:00
Damien Neil
99f24c33b0 proto: wire encoding support
Add proto.Marshal.

Change-Id: If7254bb4c4cbbee782a2a163762f9fcf98e7ab08
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/167388
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-14 19:12:27 +00:00
Damien Neil
4866b95a73 internal/impl: try to work with ExtensionDescs when legacy is absent
Don't rely on legacyWrapper.ExtensionTypeFromDesc when desc.Type
is populated. Avoids panics when working with extensions when
internal/legacy has not been imported.

Change-Id: Id41ffa3c60b6e5503d0a3e29ef48e4f523fad588
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/167465
Reviewed-by: Joe Tsai <joetsai@google.com>
2019-03-14 01:46:22 +00:00
Joe Tsai
990b9f5919 internal/prototype: move from reflect/prototype
The prototype package was initially used by generated reflection support,
but has now been replaced by internal/fileinit.
Eventually, this functionality should be deleted and re-written in terms
of other components in the repo.

Usages that prototype currently provides (but should be moved) are:
* Constructing standalone messages and enums, which is behavior we should
provide in reflect/protodesc. The google.protobuf.{Enum,Type} are well-known
proto messages designed for this purpose.
* Constructing placeholder files, enums, and messages.
* Consructing protoreflect.{Message,Enum,Extension}Types, which are protobuf
descriptors with associated Go type information.

Change-Id: Id7dbefff952682781b439aa555508c59b2629f9e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/167383
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-13 20:17:00 +00:00
Herbie Ong
250c6eaf92 internal/encoding/text: change Value.Float{32,64} to Value.Float
Collapse Value.Float32 and Value.Float64 into single API to keep it
consistent with Value.{Int,Uint}.

Change-Id: I07737e72715fe3cc3f6bcad579cf5d6cfe3757d5
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/167317
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-13 04:35:13 +00:00
Joe Tsai
4989810018 cmd/protoc-gen-go: always generate support for reflection
Now that binary bloat concerns have been addressed, remove the flag
to control whether to generate support for reflection.

Change-Id: Ia0d11183707572caaf91d2f01dfa77e3aac0a417
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/167140
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-12 23:04:05 +00:00
Joe Tsai
bb786ea59e test.bash: update to protobuf 3.7.0
Change-Id: Id1237eaf92d6a07c57b279a59fb6bd794af0f231
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/167041
Reviewed-by: Herbie Ong <herbie@google.com>
2019-03-12 08:34:56 +00:00
Herbie Ong
87608a75eb encoding/jsonpb: switch MarshalOptions to use new JSON encoder
Delete temporary copy of old JSON encoder/decoder internal/encoding/jsonx.

Change-Id: I8b23d7907370d069d0930c360979a2d8b62adc93
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/165778
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-11 21:54:53 +00:00
Herbie Ong
d3f8f2d412 internal/encoding/json: rewrite to a token-based encoder and decoder
Previous decoder decodes a JSON number into a float64, which lacks
64-bit integer precision.

I attempted to retrofit it with storing the raw bytes and parsed out
number parts, see golang.org/cl/164377.  While that is possible, the
encoding logic for Value is not symmetrical with the decoding logic and
can be confusing since both utilizes the same Value struct.

Joe and I decided that it would be better to rewrite the JSON encoder
and decoder to be token-based instead, removing the need for sharing a
model type plus making it more efficient.

Change-Id: Ic0601428a824be4e20141623409ab4d92b6167c7
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/165677
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-11 21:53:21 +00:00
Joe Tsai
22b1ebd068 internal/impl: split messageWrapper into different types
The previous implementation of messageWrapper implemented both
proto.Message and protoreflect.Message. This made it possible for users
to accidentally rely on this fact when it was actually an internal
implementation detail.

To avoid this, we split the wrapper type into two, ensuring that each
only implements one or the other. Doing so also revealed bugs in our
own code where we accidentally relied on this fact.

Change-Id: I0ff521b5c806b7dcb0b86942bd97e8319d8e8657
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/166938
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-11 21:23:31 +00:00
Joe Tsai
d56458e71b internal/cmd/generate-protos: generate test for testdata
Running "go build ./..." does not descend into testdata directories.
However, the testdata in this repository is source code that is
intended to build properly. We could rename the directory, but that does
not test whether the generated packages can initialize properly.

Thus, we generate a trivial test that simply blank imports all packages.

Doing this reveals that some of the generated files have incorrect imports,
leading to registration conflicts.

To avoid introducing a dependency on gRPC from our go.mod file, we put
the testdata directories in their own module. Also, we avoid running
internal/testprotos through the grpc plugin because the servie and method
definitions in that directory are more for testing proto file initialization
rather than testing grpc generation.

Change-Id: Iaa6a06449787a085200e31bc7606e3ac904d3180
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/164917
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-11 19:40:53 +00:00
Damien Neil
0fc224513b cmd/protoc-gen-go: enforce init order within packages
Ensure that the init funcs for files within a Go package run in the
dependency order of the source .proto files. That is, if a.proto and b.proto
are part of the same Go package, and a.proto imports b.proto, then b.pb.go's
init funcs must run before a.pb.go's.

Change-Id: I0e86ff22e5c4cab9df7a73fe4805390fadd34b0d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/166419
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Reviewed-by: Herbie Ong <herbie@google.com>
2019-03-10 22:19:14 +00:00
Joe Tsai
1af1de0c15 internal/cmd/generate-protos: generate internal descfield package
Generate a list of descriptor fields from the descriptor.proto itself
as an internal package. Use these fields for the internal implementation.

Change-Id: Ib1ab0c5c6deb332ba6c8018ef55136b7e5974944
Reviewed-on: https://go-review.googlesource.com/c/164864
Reviewed-by: Herbie Ong <herbie@google.com>
2019-03-02 23:16:45 +00:00
Herbie Ong
1e1e78b71b internal/encoding/jsonx: copy internal/encoding/json
We're rewriting internal/encoding/json. So, make a copy of it first in
order not to break encoding/jsonpb package.

Change-Id: I8b63c468d3f432102d2af4db22a7549998ce3876
Reviewed-on: https://go-review.googlesource.com/c/164642
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-02 00:23:28 +00:00
Damien Neil
6bb8dec7f6 cmd/protoc-gen-go: change some arrays to slices to save bytes
Using arrays in the generated reflection information adds unnecessary
eq and hash functions being added to the package. Change to slices
to reduce bloat.

Change-Id: I1a4f6d59021644d93dd6c24679b9233141e89a75
Reviewed-on: https://go-review.googlesource.com/c/164640
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-01 23:53:03 +00:00
Joe Tsai
19058431cd internal/cmd/generate-protos: initial commit
Create a single binary for handling generation of protos.
This replaces previous logic spread throughout the repo in:
* regenerate.bash
* cmd/protoc-gen-go/golden_test.go
* cmd/protoc-gen-go-grpc/golden_test.go
* (indirectly) internal/protogen/goldentest

One of the problems with the former approaches is that they relied on
a version of protoc that was specific to a developer's workstation.
This meant that the result of generation was not hermetic.
To address this, we rely on the hard-coded version of protobuf specified
in the test.bash script.

A summary of changes in this CL are:
* The internal_gengo.GenerateFile and internal_gengogrpc.GenerateFile
functions are unified to have consistent signatures. It seems that the
former accepted a *protogen.GeneratedFile to support v1 where gRPC code
was generated into the same file as the base .pb.go file. However, the
same functionality can be achieved by having the function return
the generated file object.
* The test.bash script patches the protobuf toolchain to have properly
specified go_package options in each proto source file.
* The test.bash script accepts a "-regenerate" argument.
* Add generation for the well-known types. Contrary to how these were
laid out in the v1 repo, all the well-known types are placed in the
same Go package.
* Add generation for the conformance proto.
* Remove regenerate.bash
* Remove internal/protogen
* Remove cmd/protoc-gen-go/golden_test.go
* Remove cmd/protoc-gen-go-grpc/golden_test.go
* Add cmd/protoc-gen-go/annotation_test.go

Change-Id: I4a1a97ae6f66e2fabcf4e4d292c95ab2a2db0248
Reviewed-on: https://go-review.googlesource.com/c/164477
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-01 20:47:52 +00:00
Joe Tsai
4069211bcd protogen: use full path for generated file variable name
Use the full path (including the extension) for the generation of
the per-file variable name. Several reasons for this:

* The current logic is buggy in the case where pathType == pathTypeImport
since the prefix variable will be mangled with the Go import path.

* The extension is technically part of the path.
Thus, "path/to/foo.proto" and "path/to/foo.protodevel" are two
distinctly different imports.

* Style-wise, it subjectively looks better. Rather than being a mixture
of camelCase and snake_case, it is all snake_case for the common case:
	before: ProtoFile_google_protobuf_any
	after:  File_google_protobuf_any_proto

* Since the extension is almost always ".proto", this results in a
suffix of "_proto", which provides an additional layer of protection
against possible name conflicts. The previous approach could possibly
have a conflict between "Foo.proto" and a message named ProtoFile
with a sub-message called Foo.

Also, use the per-file variable name for the raw descriptor variables
instead of the hashed version.

Change-Id: Ic91e326b7593e5985cee6ececc60539c27fe32fe
Reviewed-on: https://go-review.googlesource.com/c/164379
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-01 00:13:31 +00:00
Joe Tsai
cf81e67b53 cmd/protoc-gen-go: use protoapi.CompressGZIP
Calling a helper function directly should reduce binary bloat slightly.

Change-Id: I6068dc4cd00c8d90d2e6e6d99633b81388bc8781
Reviewed-on: https://go-review.googlesource.com/c/164679
Reviewed-by: Damien Neil <dneil@google.com>
2019-02-28 22:28:32 +00:00
Herbie Ong
e09920e4f8 internal/errors: fix New in eliding prefix
Change-Id: I59c4c03f4115bfe6c4377924fadfd664245e878f
Reviewed-on: https://go-review.googlesource.com/c/164277
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-02-27 22:36:14 +00:00
Damien Neil
71c6603a26 protogen: use full filename in per-file vars
For a file "foo/bar.proto", put the FileDescriptor in "ProtoFile_foo_bar"
rather than "Bar_fileDescriptor".

Avoid name clashes when a package contains "a/foo.proto" and "b/foo.proto".

Don't camelcase the filename: These vars weren't fully camelcased to begin
with, and leaving the filename relatively unchanged is clearer and more
predictable.

Move "ProtoFile" from the end of the var name to the start, so that vars
will sort better in packages with multiple descriptors.

These changes do add a chance of name collision when the input filename
begins with an uppercase letter: Foo.proto becomes "ProtoFile_Foo", which
could be the result of camelcasing "proto_file.foo". The readability
benefits seem worth it.

Change-Id: If27d3a0d7b5bf3535aa1607a8579eb057c74d2dc
Reviewed-on: https://go-review.googlesource.com/c/163199
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Reviewed-by: Herbie Ong <herbie@google.com>
2019-02-21 22:37:58 +00:00
Damien Neil
3fa6d3f003 internal/encoding/pack: don't depend on exact math.NaN bits
This test examines the result of converting math.NaN() to a fixed byte
string. Change it to use a specific NaN value instead, since the value
returned by math.NaN is specified only as being a NaN, not a specific
one.

Use specific float32 and float64 NaN values, since the result of
converting a float64 NaN to a float32 can and does vary.

Fixes test failure on ARM.

Change-Id: Ia1517fdba768cdd88e5ee5f5af37f0b481e651b4
Reviewed-on: https://go-review.googlesource.com/c/162117
Reviewed-by: Herbie Ong <herbie@google.com>
2019-02-12 21:38:35 +00:00
Damien Neil
374cdb81f7 internal/impl: drop MessageType.(Unk|K)nownFieldsOf
These methods are no longer used. Drop them.

Change-Id: Ie5595c1a56dd76183782019384d16ea97d5f7e9e
Reviewed-on: https://go-review.googlesource.com/c/160357
Reviewed-by: Herbie Ong <herbie@google.com>
2019-01-30 01:34:45 +00:00
Damien Neil
e475eaa7ad internal/fileinit: add tests for fileinit and protodesc
Test the fileinit and protodesc packages by verifying that passing a
FileDescriptorProto through a round trip of these packages leaves it
unchanged.

Change-Id: I6bfb894d95f1736f9908adee1ab63e9653b3f1be
Reviewed-on: https://go-review.googlesource.com/c/159762
Reviewed-by: Herbie Ong <herbie@google.com>
2019-01-30 01:34:30 +00:00
Damien Neil
8012b444ee internal/fileinit: generate reflect data structures from raw descriptors
This CL takes a significantly different approach to generating support
for protobuf reflection. The previous approach involved generating a
large number of Go literals to represent the reflection information.
While that approach was correct, it resulted in too much binary bloat.

The approach taken here initializes the reflection information from
the raw descriptor proto, which is a relatively dense representation
of the protobuf reflection information. In order to keep initialization
cost low, several measures were taken:
* At program init, the bare minimum is parsed in order to initialize
naming information for enums, messages, extensions, and services declared
in the file. This is done because those top-level declarations are often
relevant for registration.
* Only upon first are most of the other data structures for protobuf
reflection actually initialized.
* Instead of using proto.Unmarshal, a hand-written unmarshaler is used.
This allows us to avoid a dependendency on the descriptor proto and also
because the API for the descriptor proto is fundamentally non-performant
since it requires an allocation for every primitive field.

At a high-level, the new implementation lives in internal/fileinit.

Several changes were made to other parts of the repository:
* cmd/protoc-gen-go:
  * Stop compressing the raw descriptors. While compression does reduce
the size of the descriptors by approximately 2x, it is a pre-mature
optimization since the descriptors themselves are around 1% of the total
binary bloat that is due to generated protobufs.
  * Seeding protobuf reflection from the raw descriptor significantly
simplifies the generator implementation since it is no longer responsible
for constructing a tree of Go literals to represent the same information.
  * We remove the generation of the shadow types and instead call
protoimpl.MessageType.MessageOf. Unfortunately, this incurs an allocation
for every call to ProtoReflect since we need to allocate a tuple that wraps
a pointer to the message value, and a pointer to message type.
* internal/impl:
  * We add a MessageType.GoType field and make it required that it is
set prior to first use. This is done so that we can avoid calling
MessageType.init except for when it is actually needed. The allows code
to call (*FooMessage)(nil).ProtoReflect().Type() without fearing that the
init code will run, possibly triggering a recursive deadlock (where the
init code depends on getting the Type of some dependency which may be
declared within the same file).
* internal/cmd/generate-types:
  * The code to generate reflect/prototype/protofile_list_gen.go was copied
and altered to generated internal/fileinit.desc_list_gen.go.

At a high-level this CL adds significant technical complexity.
However, this is offset by several possible future changes:
* The prototype package can be drastically simplified. We can probably
reimplement internal/legacy to use internal/fileinit instead, allowing us
to drop another dependency on the prototype package. As a result, we can
probably delete most of the constructor types in that package.
* With the prototype package significantly pruned, and the fact that generated
code no longer depend on depends on that package, we can consider merging
what's left of prototype into protodesc.

Change-Id: I6090f023f2e1b6afaf62bd3ae883566242e30715
Reviewed-on: https://go-review.googlesource.com/c/158539
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-01-30 01:33:46 +00:00
Herbie Ong
84f0960b04 internal/encoding/text: format using 32 bitsize when encoding float32
When encoding/textpb marshals out float32 values, it was previously
formatting it as float64 bitsize since both float types are stored as
float64 and internal/encoding/text only has one Float type.  A
consequence of this is that the output may display a different value
than expected, e.g.  1.02 becomes 1.0199999809265137.

This CL splits Float type into Float32 and Float64 to keep track of
which bitsize to use when formatting.  Values of both types are still
stored as float64 to keep the logic simple.

Decoding will always use Float64, but users can ask for a float32 value
from it.

Change-Id: Iea5b14b283fec2236a0c3946fac34d4d79b95274
Reviewed-on: https://go-review.googlesource.com/c/158497
Reviewed-by: Damien Neil <dneil@google.com>
2019-01-18 17:54:23 +00:00
Joe Tsai
5681bb2587 protogen: use _protoFile suffix for file descriptor variable
A "_ProtoFile" suffix can potentially conflict with a sub-message named
"ProtoFile" nested within a message that matches the camel-cased
form of the basename of the .proto source file.

Avoid unlikely conflicts and rename this to use a "_protoFile" suffix,
which can never conflict except with an enum value that is also named
"protoFile" (which is a violation of the style guide).

Change-Id: Ie9d22f9f741a63021b8f76906b20c6c2f599885b
Reviewed-on: https://go-review.googlesource.com/c/157218
Reviewed-by: Damien Neil <dneil@google.com>
2019-01-14 20:23:59 +00:00
Joe Tsai
d18bd31838 reflect: switch ExtensionType.New to return Value
This change preserves consistency with CL/157077,
where New returns a value closer to the reflective type.

Change-Id: I85bfdae24e1ce1a10c3c7b939420fa1043bff743
Reviewed-on: https://go-review.googlesource.com/c/157078
Reviewed-by: Damien Neil <dneil@google.com>
2019-01-09 21:05:08 +00:00
Joe Tsai
3bc7d6f5cd reflect: switch MessageType.New to return Message
Most usages of New actually prefer to interact with the reflective view
rather than the native Go type. Thus, change New to return that instead.
This parallels reflect.New, which returns the reflective view
(i.e., reflect.Value) instead of native type (i.e., interface{}).
We make the equivalent change to KnownFields.NewMessage, List.NewMessage,
and Map.NewMessage for consistency.

Since this is a subtle change where the type system will not always
catch the changed type, this change was made by both changing the type
and renaming the function to NewXXX and manually looking at every usage
of the the function to ensure that the usage correctly operates
on either the native Go type or the reflective view of the type.
After the entire codebase was cleaned up, a rename was performed to convert
NewXXX back to New.

Change-Id: I153fef627b4bf0a427e4039ce0aaec52e20c7950
Reviewed-on: https://go-review.googlesource.com/c/157077
Reviewed-by: Damien Neil <dneil@google.com>
2019-01-09 20:29:29 +00:00
Damien Neil
a8593bae57 reflect/protoreflect: drop the ProtoEnum type
Drop the protoreflect.ProtoEnum type (containing a single method
returning a protoreflect.Enum) and make generated enum types
directly implement protoreflect.Enum instead.

Messages have a two-level type split (ProtoMessage and Message) to
minimize conflicts between reflection methods and field names. Enums
need no such split, since enums do not have fields and therefore have
no source of conflicts.

Change-Id: I2b6222e9404253e6bfef2217859e1b760ffcd29b
Reviewed-on: https://go-review.googlesource.com/c/156902
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Reviewed-by: Damien Neil <dneil@google.com>
2019-01-09 00:40:35 +00:00
Damien Neil
c31bc2da9e internal/testprotos: fix import path of test.proto
Change the go_package option to match the actual import path of this package.

Change-Id: Ie8630878ce75e34ca76d97c6e2922254cf964801
Reviewed-on: https://go-review.googlesource.com/c/156344
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-01-05 00:45:35 +00:00
Herbie Ong
2b0bba88b8 internal/legacy: fix ExtensionTypeFromDesc
Change-Id: I1da7fac27ae38e8556cc6bf360d71a38b16c22cd
Reviewed-on: https://go-review.googlesource.com/c/155457
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-27 19:33:04 +00:00
Joe Tsai
bc534a98a5 all: add appengine build tag
The purego tag (see https://golang.org/issue/23172) is a community agreed
upon signal that a given build environment does not support unsafe.
The appengine environment is supposed to respect this tag, but does not
properly do so. Add this tag back in until they fix their environment.

Change-Id: I9a70062be4339c2e1a93cac31d387698c561b8aa
Reviewed-on: https://go-review.googlesource.com/c/154743
Reviewed-by: Damien Neil <dneil@google.com>
2018-12-19 00:36:57 +00:00
Damien Neil
4be2fb4ada internal/legacy: use v2 proto.Unmarshal
The v2 decoder isn't 100% complete, but it's good enough.
Delete the vendored copy of the v1 Unmarshal implementation.

Change-Id: Ibeabbb2e9109a1ec3df57e71f98b7aa4a583fc5b
Reviewed-on: https://go-review.googlesource.com/c/154577
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-17 22:15:00 +00:00
Damien Neil
ba23aa55b2 proto: wire decoding support
Add proto.Unmarshal.

Test cases all produce identical results to the v1 unmarshaller.

Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d
Reviewed-on: https://go-review.googlesource.com/c/153918
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-17 18:48:00 +00:00
Damien Neil
9ecf75b72e internal/value: bugfix for v2 message conversion
PBValueOf returns a protoreflect.Message, not a
protoreflect.ProtoMessage.

Change-Id: I88ed55f52bada6fc2b29ffd63e30de09e1febe8c
Reviewed-on: https://go-review.googlesource.com/c/153917
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-13 01:11:07 +00:00