Commit Graph

92 Commits

Author SHA1 Message Date
Damien Neil
e89e6244e0 all: change module to google.golang.org/protobuf
Temporarily remove go.mod, since we can't generate an accurate one until
the corresponding v1 change is submitted.

Change-Id: I1e1ad97f2b455e33f61ffaeb8676289795e47e72
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/177000
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-05-14 17:28:29 +00:00
Joe Tsai
ac31a352ca reflect/protoreflect: add helper methods to FieldDescriptor
Added API:
	FieldDescriptor.IsExtension
	FieldDescriptor.IsList
	FieldDescriptor.MapKey
	FieldDescriptor.MapValue
	FieldDescriptor.ContainingOneof
	FieldDescriptor.ContainingMessage

Deprecated API (to be removed in subsequent CL):
	FieldDescriptor.Oneof
	FieldDescriptor.Extendee

These methods help cleanup several common usage patterns.

Change-Id: I9a3ffabc2edb2173c536509b22f330f98bba7cf3
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/176977
Reviewed-by: Damien Neil <dneil@google.com>
2019-05-14 17:14:05 +00:00
Joe Tsai
67c1d9b2c1 reflect/protoreflect: add Descriptor.ParentFile
Querying for the parent file that contains a descriptor declaration
is a common enough operation to warrant its own first-class method.

Change-Id: I2f41e5126a5b465df23897904a6513dd3ed8dd92
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/176777
Reviewed-by: Damien Neil <dneil@google.com>
2019-05-13 20:21:18 +00:00
Joe Tsai
0fc49f8225 reflect/protoreflect: add Descriptor specific methods
Added methods:
	Enum.Descriptor
	Message.Descriptor
	EnumType.Descriptor
	MessageType.Descriptor
	ExtensionType.Descriptor
	Message.New

All functionality is switched over to use those methods instead of
implicitly relying on the fact that {Enum,Message}Type implicitly
implement the associated descriptor interface.

This CL does not yet remove {Enum,Message}.Type or prevent
{Enum,Message,Extension}Type from implementating a descriptor.
That is a subsequent CL.

The Message.New method is also added to replace functionality
that will be lost when the Type methods are removed.

Change-Id: I7fefde1673bbd40bfdac489aca05cec9a6c98eb1
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/174918
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Herbie Ong <herbie@google.com>
2019-05-13 19:34:41 +00:00
John Wright
bac4cd44d4 reflect/protodesc: check file imports by name
Weak imports are added to Imports as placeholders even if they can be
found in the Files registry, so we have to look at the name rather than
the actual FileDescriptor.

Change-Id: I28f62e945f233119014e5e8cb1bcbde7dca831a7
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/175897
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-05-08 01:49:39 +00:00
John Wright
9a824c9a9e reflect/protodesc: add validation for NewFile
This covers most of the TODO around validation. I left open the ones
that we didn't have clear consensus on yet.

Change-Id: I336c53173ee8d7447558b1e3a0c1ef945e986cd5
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/175140
Reviewed-by: Joe Tsai <joetsai@google.com>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-05-08 00:41:07 +00:00
Joe Tsai
c90b6ada23 reflect/protoreflect: remove methods with Type suffix
CL/172238 added equivalent methods without the Type suffix,
while keeping the old methods.
CL/172582 updates the v1 codebase to use the new methods.
This CL removes the methods with the Type suffix.

Change-Id: Iaaaa4fff11cac1694735657db2e5fd7cadc90afe
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/173138
Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-20 08:14:13 +00:00
Joe Tsai
d24bc72368 reflect/protoreflect: rename methods with Type suffix
The protobuf type system uses the word "descriptor" instead of "type".
We should avoid the "type" verbage when we aren't talking about Go types.
The old names are temporarily kept around for compatibility reasons.

Change-Id: Icc99c913528ead011f7a74aa8399d9c5ec6dc56e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/172238
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
2019-04-20 06:35:24 +00:00
Joe Tsai
5e71dc95f2 cmd/protoc-gen-go: used named types for special fields
Using a named fields gives us the flexibility to change the underlying
representation of special fields without needing to regenerate user code.

We add a named type for ExtensionFields, UnknownFields, and SizeCache.

Change-Id: I107cf82899850ea76665310ce79def60f0f7ab97
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/172402
Reviewed-by: Damien Neil <dneil@google.com>
2019-04-16 22:04:16 +00:00
Joe Tsai
7ca7098113 cmd/protoc-gen-go: drop xxx_ prefix for package-level variables
Lower-casing the FileDescriptor variable provides for shorter and
more readable global variable names. This has a very minor benefit
that the read-only data section of a binary is slightly smaller
since function names do end up in the final binary.

Change-Id: I077364905e5c9adea69873b3ea580fddc68b9eb8
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/172119
Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-16 16:31:54 +00:00
Damien Neil
2300c18725 reflect/protoreflect, reflect/protoregistry: move descriptor lookup to registry
Drop the protoreflect.FileDescriptor.DescriptorByName method.
Descriptor lookup will always happen through a protoregistry.Files, which
is more generally useful (it's rare that you want to find a descriptor in a
specific file, as opposed to a package which may be composed of multiple files).

Split protoregistry.Files descriptor lookup into individual per-type functions
(enum, message, extension, service), matching the preg.Types API.

Drop the ability to look up enum values, message fields, and service methods
for now. This can be easily added later if needed, and is trivial to implement
in user code. (e.g., look up the service and then consult sd.Methods.ByName().)

Change-Id: I2b3d8ef888921a8464ba1434eddab20c7d3a458e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/172118
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-04-16 06:03:04 +00:00
Joe Tsai
a2dd228004 internal/fileinit: make fileinit responsible for registering concrete options types
Changes:
* Remove protoreflect.OptionsMessage and use protoreflect.ProtoMessage
now that the generated options natively implement reflection.
* Make registration of concrete option types the responsibility of
internal/fileinit since the init logic of descriptor.pb.go uses that.
* Remove equivalent logic in protoc-gen-go to special-case descriptor.

Change-Id: Id814651fafa4d888ff4532d59c6a4f9b68145157
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171465
Reviewed-by: Damien Neil <dneil@google.com>
2019-04-15 20:37:10 +00:00
Joe Tsai
5b9fa1f023 all: remove direct dependence on internal/legacy
Change-Id: Ic547a5edab4a1e7799c421b59ea78eca98534e96
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171699
Reviewed-by: Damien Neil <dneil@google.com>
2019-04-15 20:23:52 +00:00
Joe Tsai
09b5b46567 cmd/protoc-gen-go: revert Enum simplication
Go1.12 on GoLLVM with LLVM 9.0.0 has a bug where the simpler but
equivalent expression results in memory corruption.
Revert to the more complex expression to avoid breaking GoLLVM and
it is not worth waiting for a GoLLVM fix.

Change-Id: I8b42965ca9bd333deb8693021e7b22f966e13521
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171463
Reviewed-by: Damien Neil <dneil@google.com>
2019-04-10 22:39:46 +00:00
Joe Tsai
4ec39c7663 reflect/protoreflect: add KnownFields.WhichOneof
Add a method that provides efficiently querying for which member field
in a oneof is actually set. This is useful when dealing with oneofs
with many member fields.

Change-Id: I918b566c432f8bdd24dcecbb5501d231ffefef29
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170580
Reviewed-by: Damien Neil <dneil@google.com>
2019-04-09 21:18:45 +00:00
Damien Neil
0d3e8cc096 proto, runtime/protoiface: add support for fast-path marshaling
Allow message implementations to provide optimized versions of standard
operations. Generated messages now include a ProtoReflectMethods method,
returning a protoiface.Methods struct containing pointers to assorted
optional functions.

The Methods struct also includes a Flags field indicating support for
optional features such as deterministic marshaling.

Implementation of the fast paths (and tests) will come in later CLs.

Change-Id: Idd1beed0ecf43ec5e5e7b8da2ee1e08d3ce32213
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170340
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-04-02 21:23:04 +00:00
Joe Tsai
61968ce130 cmd/protoc-gen-go: perform some code cleanup
Minor changes:
* Use x as the receiver since "e" and "m" are meaningless in the presence
of user-defined enum and message names.
* Consistently keep enum methods together, rather awkwardly split apart
by the value maps.

Change-Id: I68e5666efb56ac7a4d062fb223b9f826dc72aba9
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170357
Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-01 20:44:00 +00:00
Joe Tsai
5d72cc2d37 cmd/protoc-gen-go: lazily GZIP-encode the raw descriptor
This reduces the init-time cost slightly since the GZIP'd
raw descriptor is constructed lazily on demand.

Change-Id: I482c6a2201b8786e425d7dee5612fdfd60ab1500
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169917
Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-01 20:24:54 +00:00
Joe Tsai
35ec98fdcb cmd/protoc-gen-go: generate for v2-only dependencies
This removes yet another set of dependencies of v2 on v1.
The only remaining dependency are in the _test.go files,
primarily for proto.Equal.

Changes made:
* cmd/protoc-gen-go no longer generates any functionality that depends
on the v1 package, and instead only depends on v2.
* internal/fileinit.FileBuilder.MessageOutputTypes is switched from
protoreflect.MessageType to protoimpl.MessageType since the
implementation must be fully inialized before registration occurs.
* The test for internal/legacy/file_test.go is switched to a legacy_test
package to avoid a cyclic dependency.
This requires Load{Enum,Message,File}Desc to be exported.

Change-Id: I43e2fe64cff4eea204258ce11e791aca5eb6e569
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169298
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-26 17:03:31 +00:00
Joe Tsai
4fddebafc0 all: move v1 types over to the v2 repository
As a goal, v2 should not depend on v1. As another step towards that end,
we move all the types that used to be in the v1 protoapi package over to v2.

For now, we place MessageV1, ExtensionRangeV1, and ExtensionDescV1
in runtime/protoiface since these are types that generated messages will
probably have to reference forever. An alternative location could be
reflect/protoreflect, but it seems unfortunate to have to dirty the
namespace of that package with these types.

We move ExtensionFieldV1, ExtensionFieldsV1, and ExtensionFieldsOf
to internal/impl, since these are related to the implementation of a
generated message.

Since moving these types from v1 to v2 implies that the v1 protoapi
package is useless, we update all usages of v1 protoapi in the v2
repository to point to the relevant v2 type or functionality.

CL/168538 is the corresponding change to alter v1.
There will be a temporary build failure as it is not possible
to submit CL/168519 and CL/168538 atomically.

Change-Id: Ide4025c1b6af5b7f0696f4b65b988b4d10a50f0b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168519
Reviewed-by: Herbie Ong <herbie@google.com>
2019-03-22 20:01:07 +00:00
Joe Tsai
559d47f1da cmd/protoc-gen-go: fix init order for v1 registration
The v1 registration leaks the message types out to the proto package.
When doing that, it must ensure that the reflection data structures
for those types are properly initialized first. We achieve that by
doing v1 registration at the end of the reflection init function.

Change-Id: If6df18df59d05bad50ff39c2eff6beb19e7466cc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168348
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-20 04:16:33 +00:00
Joe Tsai
8e506a8704 cmd/protoc-gen-go: rely on protoimpl for basic helpers
The EnumName, UnmarshalJSONEnum, and CompressGZIP helpers currently live
in v1 protoapi, which would cause all generated messages to depend on v1.
In an effort to break the dependency of v2 on v1, we move these helper
functions to v2 (and re-written to take advantage of protobuf reflection).

These helpers are unfortunate, but we cannot eliminate the functionality
that they implement since they are exposed in the publicly generated API.

Since EnumName does not rely on the enum maps, it removes another dependency
on those variables. Eventually, we can get to the point where these variables
(though declared) are not linked into the binary if the user does not use them.

Also, we rely on the v1 proto package for registration instead of v1 protoapi.
This may re-introduce a cyclic dependency on descriptor proto again in the
future, but the better approach is to just start registering with v2.

Change-Id: Id755585a7a1df14e4a6a2dfa650df221a3c153fb
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/167921
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-18 18:50:16 +00:00
Joe Tsai
afb455eaf8 cmd/protoc-gen-go: correlate v1 ExtensionDesc with v2 ExtensionType
Unfortunately a good amount of code uses pointer comparisons on the
v1 ExtensionDesc to determine exactly which extension field is set,
rather than checking whether the extension descriptor semantically
describes the field that they are interested in.

To preserve this behavior in v1, we need a 1:1 mapping between
a v2 ExtensionType and a specific v1 ExtensionDesc.

Change-Id: I852b3cefb4585bd656e48e5adad6cc28795d02df
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/167759
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-15 01:24:42 +00:00
Joe Tsai
990b9f5919 internal/prototype: move from reflect/prototype
The prototype package was initially used by generated reflection support,
but has now been replaced by internal/fileinit.
Eventually, this functionality should be deleted and re-written in terms
of other components in the repo.

Usages that prototype currently provides (but should be moved) are:
* Constructing standalone messages and enums, which is behavior we should
provide in reflect/protodesc. The google.protobuf.{Enum,Type} are well-known
proto messages designed for this purpose.
* Constructing placeholder files, enums, and messages.
* Consructing protoreflect.{Message,Enum,Extension}Types, which are protobuf
descriptors with associated Go type information.

Change-Id: Id7dbefff952682781b439aa555508c59b2629f9e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/167383
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-13 20:17:00 +00:00
Damien Neil
0fc224513b cmd/protoc-gen-go: enforce init order within packages
Ensure that the init funcs for files within a Go package run in the
dependency order of the source .proto files. That is, if a.proto and b.proto
are part of the same Go package, and a.proto imports b.proto, then b.pb.go's
init funcs must run before a.pb.go's.

Change-Id: I0e86ff22e5c4cab9df7a73fe4805390fadd34b0d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/166419
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Reviewed-by: Herbie Ong <herbie@google.com>
2019-03-10 22:19:14 +00:00
Damien Neil
6bb8dec7f6 cmd/protoc-gen-go: change some arrays to slices to save bytes
Using arrays in the generated reflection information adds unnecessary
eq and hash functions being added to the package. Change to slices
to reduce bloat.

Change-Id: I1a4f6d59021644d93dd6c24679b9233141e89a75
Reviewed-on: https://go-review.googlesource.com/c/164640
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-01 23:53:03 +00:00
Joe Tsai
19058431cd internal/cmd/generate-protos: initial commit
Create a single binary for handling generation of protos.
This replaces previous logic spread throughout the repo in:
* regenerate.bash
* cmd/protoc-gen-go/golden_test.go
* cmd/protoc-gen-go-grpc/golden_test.go
* (indirectly) internal/protogen/goldentest

One of the problems with the former approaches is that they relied on
a version of protoc that was specific to a developer's workstation.
This meant that the result of generation was not hermetic.
To address this, we rely on the hard-coded version of protobuf specified
in the test.bash script.

A summary of changes in this CL are:
* The internal_gengo.GenerateFile and internal_gengogrpc.GenerateFile
functions are unified to have consistent signatures. It seems that the
former accepted a *protogen.GeneratedFile to support v1 where gRPC code
was generated into the same file as the base .pb.go file. However, the
same functionality can be achieved by having the function return
the generated file object.
* The test.bash script patches the protobuf toolchain to have properly
specified go_package options in each proto source file.
* The test.bash script accepts a "-regenerate" argument.
* Add generation for the well-known types. Contrary to how these were
laid out in the v1 repo, all the well-known types are placed in the
same Go package.
* Add generation for the conformance proto.
* Remove regenerate.bash
* Remove internal/protogen
* Remove cmd/protoc-gen-go/golden_test.go
* Remove cmd/protoc-gen-go-grpc/golden_test.go
* Add cmd/protoc-gen-go/annotation_test.go

Change-Id: I4a1a97ae6f66e2fabcf4e4d292c95ab2a2db0248
Reviewed-on: https://go-review.googlesource.com/c/164477
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-01 20:47:52 +00:00
Joe Tsai
4069211bcd protogen: use full path for generated file variable name
Use the full path (including the extension) for the generation of
the per-file variable name. Several reasons for this:

* The current logic is buggy in the case where pathType == pathTypeImport
since the prefix variable will be mangled with the Go import path.

* The extension is technically part of the path.
Thus, "path/to/foo.proto" and "path/to/foo.protodevel" are two
distinctly different imports.

* Style-wise, it subjectively looks better. Rather than being a mixture
of camelCase and snake_case, it is all snake_case for the common case:
	before: ProtoFile_google_protobuf_any
	after:  File_google_protobuf_any_proto

* Since the extension is almost always ".proto", this results in a
suffix of "_proto", which provides an additional layer of protection
against possible name conflicts. The previous approach could possibly
have a conflict between "Foo.proto" and a message named ProtoFile
with a sub-message called Foo.

Also, use the per-file variable name for the raw descriptor variables
instead of the hashed version.

Change-Id: Ic91e326b7593e5985cee6ececc60539c27fe32fe
Reviewed-on: https://go-review.googlesource.com/c/164379
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-01 00:13:31 +00:00
Joe Tsai
cf81e67b53 cmd/protoc-gen-go: use protoapi.CompressGZIP
Calling a helper function directly should reduce binary bloat slightly.

Change-Id: I6068dc4cd00c8d90d2e6e6d99633b81388bc8781
Reviewed-on: https://go-review.googlesource.com/c/164679
Reviewed-by: Damien Neil <dneil@google.com>
2019-02-28 22:28:32 +00:00
Damien Neil
71c6603a26 protogen: use full filename in per-file vars
For a file "foo/bar.proto", put the FileDescriptor in "ProtoFile_foo_bar"
rather than "Bar_fileDescriptor".

Avoid name clashes when a package contains "a/foo.proto" and "b/foo.proto".

Don't camelcase the filename: These vars weren't fully camelcased to begin
with, and leaving the filename relatively unchanged is clearer and more
predictable.

Move "ProtoFile" from the end of the var name to the start, so that vars
will sort better in packages with multiple descriptors.

These changes do add a chance of name collision when the input filename
begins with an uppercase letter: Foo.proto becomes "ProtoFile_Foo", which
could be the result of camelcasing "proto_file.foo". The readability
benefits seem worth it.

Change-Id: If27d3a0d7b5bf3535aa1607a8579eb057c74d2dc
Reviewed-on: https://go-review.googlesource.com/c/163199
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Reviewed-by: Herbie Ong <herbie@google.com>
2019-02-21 22:37:58 +00:00
Damien Neil
f14380bd8a reflect/protodesc: add functions to convert descs to protos
Add a collection of functions which take a reflect.*Descriptor and
return the corresponding google.protobuf.*DescriptorProto.

Change-Id: Ic186c412c8d3b7bc582c31bb8d8274459ce51a20
Reviewed-on: https://go-review.googlesource.com/c/159761
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-01-30 01:34:14 +00:00
Damien Neil
8012b444ee internal/fileinit: generate reflect data structures from raw descriptors
This CL takes a significantly different approach to generating support
for protobuf reflection. The previous approach involved generating a
large number of Go literals to represent the reflection information.
While that approach was correct, it resulted in too much binary bloat.

The approach taken here initializes the reflection information from
the raw descriptor proto, which is a relatively dense representation
of the protobuf reflection information. In order to keep initialization
cost low, several measures were taken:
* At program init, the bare minimum is parsed in order to initialize
naming information for enums, messages, extensions, and services declared
in the file. This is done because those top-level declarations are often
relevant for registration.
* Only upon first are most of the other data structures for protobuf
reflection actually initialized.
* Instead of using proto.Unmarshal, a hand-written unmarshaler is used.
This allows us to avoid a dependendency on the descriptor proto and also
because the API for the descriptor proto is fundamentally non-performant
since it requires an allocation for every primitive field.

At a high-level, the new implementation lives in internal/fileinit.

Several changes were made to other parts of the repository:
* cmd/protoc-gen-go:
  * Stop compressing the raw descriptors. While compression does reduce
the size of the descriptors by approximately 2x, it is a pre-mature
optimization since the descriptors themselves are around 1% of the total
binary bloat that is due to generated protobufs.
  * Seeding protobuf reflection from the raw descriptor significantly
simplifies the generator implementation since it is no longer responsible
for constructing a tree of Go literals to represent the same information.
  * We remove the generation of the shadow types and instead call
protoimpl.MessageType.MessageOf. Unfortunately, this incurs an allocation
for every call to ProtoReflect since we need to allocate a tuple that wraps
a pointer to the message value, and a pointer to message type.
* internal/impl:
  * We add a MessageType.GoType field and make it required that it is
set prior to first use. This is done so that we can avoid calling
MessageType.init except for when it is actually needed. The allows code
to call (*FooMessage)(nil).ProtoReflect().Type() without fearing that the
init code will run, possibly triggering a recursive deadlock (where the
init code depends on getting the Type of some dependency which may be
declared within the same file).
* internal/cmd/generate-types:
  * The code to generate reflect/prototype/protofile_list_gen.go was copied
and altered to generated internal/fileinit.desc_list_gen.go.

At a high-level this CL adds significant technical complexity.
However, this is offset by several possible future changes:
* The prototype package can be drastically simplified. We can probably
reimplement internal/legacy to use internal/fileinit instead, allowing us
to drop another dependency on the prototype package. As a result, we can
probably delete most of the constructor types in that package.
* With the prototype package significantly pruned, and the fact that generated
code no longer depend on depends on that package, we can consider merging
what's left of prototype into protodesc.

Change-Id: I6090f023f2e1b6afaf62bd3ae883566242e30715
Reviewed-on: https://go-review.googlesource.com/c/158539
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-01-30 01:33:46 +00:00
Joe Tsai
5681bb2587 protogen: use _protoFile suffix for file descriptor variable
A "_ProtoFile" suffix can potentially conflict with a sub-message named
"ProtoFile" nested within a message that matches the camel-cased
form of the basename of the .proto source file.

Avoid unlikely conflicts and rename this to use a "_protoFile" suffix,
which can never conflict except with an enum value that is also named
"protoFile" (which is a violation of the style guide).

Change-Id: Ie9d22f9f741a63021b8f76906b20c6c2f599885b
Reviewed-on: https://go-review.googlesource.com/c/157218
Reviewed-by: Damien Neil <dneil@google.com>
2019-01-14 20:23:59 +00:00
Joe Tsai
d18bd31838 reflect: switch ExtensionType.New to return Value
This change preserves consistency with CL/157077,
where New returns a value closer to the reflective type.

Change-Id: I85bfdae24e1ce1a10c3c7b939420fa1043bff743
Reviewed-on: https://go-review.googlesource.com/c/157078
Reviewed-by: Damien Neil <dneil@google.com>
2019-01-09 21:05:08 +00:00
Damien Neil
e3f854b22f reflect/protoreflect: clarify documentation
Rephrase the documentation in terms of usage of this package: Lead with
the description of what this package contains and provide guidance on where
to get a descriptor or value interface.

Change-Id: I40a43cd59d1fbca6c60eff1c3afe50ff4a15b82f
Reviewed-on: https://go-review.googlesource.com/c/157217
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-01-09 21:05:02 +00:00
Joe Tsai
3bc7d6f5cd reflect: switch MessageType.New to return Message
Most usages of New actually prefer to interact with the reflective view
rather than the native Go type. Thus, change New to return that instead.
This parallels reflect.New, which returns the reflective view
(i.e., reflect.Value) instead of native type (i.e., interface{}).
We make the equivalent change to KnownFields.NewMessage, List.NewMessage,
and Map.NewMessage for consistency.

Since this is a subtle change where the type system will not always
catch the changed type, this change was made by both changing the type
and renaming the function to NewXXX and manually looking at every usage
of the the function to ensure that the usage correctly operates
on either the native Go type or the reflective view of the type.
After the entire codebase was cleaned up, a rename was performed to convert
NewXXX back to New.

Change-Id: I153fef627b4bf0a427e4039ce0aaec52e20c7950
Reviewed-on: https://go-review.googlesource.com/c/157077
Reviewed-by: Damien Neil <dneil@google.com>
2019-01-09 20:29:29 +00:00
Damien Neil
a8593bae57 reflect/protoreflect: drop the ProtoEnum type
Drop the protoreflect.ProtoEnum type (containing a single method
returning a protoreflect.Enum) and make generated enum types
directly implement protoreflect.Enum instead.

Messages have a two-level type split (ProtoMessage and Message) to
minimize conflicts between reflection methods and field names. Enums
need no such split, since enums do not have fields and therefore have
no source of conflicts.

Change-Id: I2b6222e9404253e6bfef2217859e1b760ffcd29b
Reviewed-on: https://go-review.googlesource.com/c/156902
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Reviewed-by: Damien Neil <dneil@google.com>
2019-01-09 00:40:35 +00:00
Herbie Ong
6e67a1d614 reflect/protoregistry: add Types registry
The first commit of protoregistry only added a registry for files.
However, a separate type of registry is needed to provide a mapping between
protobuf names and actual Go types representing those names.

Additional high-level API:
    var GlobalTypes = new(Types)
    type Type interface{ ... }
    type Types struct{ ... }
        func NewTypes(...Type) *Types
        func (*Types) Register(...Type) error
        func (*Types) FindEnumByName(pref.FullName) (pref.EnumType, error)
        func (*Types) FindMessageByName(pref.FullName) (pref.MessageType, error)
        func (*Types) FindMessageByURL(string) (pref.MessageType, error)
        func (*Types) FindExtensionByName(pref.FullName) (pref.ExtensionType, error)
        func (*Types) FindExtensionByNumber(pref.FullName, pref.FieldNumber) (pref.ExtensionType, error)
        func (*Types) RangeEnums(func(pref.EnumType) bool)
        func (*Types) RangeMessages(func(pref.MessageType) bool)
        func (*Types) RangeExtensions(func(pref.ExtensionType) bool)
        func (*Types) RangeExtensionsByMessage(pref.FullName, func(pref.ExtensionType) bool)

Change-Id: I0d07705801684a1eb5853bcd05fcce12598a0047
Reviewed-on: https://go-review.googlesource.com/c/131345
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-28 00:06:31 +00:00
Damien Neil
eb6b5877e6 reflect/prototype: change registered option types to interface{}
Followup to https://go-review.googlesource.com/c/154877, changing some
additional references to the XXXOptions messages to interface{}.

Change-Id: I907ae5bede296ca95622eaaf9fdfb0c94b2f2a5f
Reviewed-on: https://go-review.googlesource.com/c/155458
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-20 23:20:27 +00:00
Joe Tsai
bc534a98a5 all: add appengine build tag
The purego tag (see https://golang.org/issue/23172) is a community agreed
upon signal that a given build environment does not support unsafe.
The appengine environment is supposed to respect this tag, but does not
properly do so. Add this tag back in until they fix their environment.

Change-Id: I9a70062be4339c2e1a93cac31d387698c561b8aa
Reviewed-on: https://go-review.googlesource.com/c/154743
Reviewed-by: Damien Neil <dneil@google.com>
2018-12-19 00:36:57 +00:00
Damien Neil
c05538c6a8 reflect/protoreflect: change Options back to an interface{}
Introduce a protoreflect.OptionsMessage interface type to represent
options messages. The protoreflect package can't deal with concrete
options types, to avoid dependency cycles; OptionsMessage serves both
as documentation and a single point to define the constraints we apply
to options.

Change the constraints on options from ProtoMessage to interface{} to
permit use of option message types which only implement the v1
proto.Message interface.

This still leaves a requirement in the internal/legacy package that
options implement protoreflect.ProtoMessage, since that package now uses
the v2 Unmarshal.

Change-Id: I547518ab2c3b90c3911ef641b05b169d50a4b33a
Reviewed-on: https://go-review.googlesource.com/c/154877
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-18 23:38:22 +00:00
Herbie Ong
7bf88d1377 reflect/prototype: fix mapping of Go types for fixed{32,64} kinds
Change-Id: I700e61c819a72d31a3b77b7de7fab7799aa22db3
Reviewed-on: https://go-review.googlesource.com/c/154582
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-17 22:20:00 +00:00
Damien Neil
232ea15589 reflect/prototype: hoist semantic options into builders
Add fields to the Message and Field builder structs which hold the value
of MessageOptions.map_entry, FieldOptions.packed, and FieldOptions.weak
options. Remove all access to the contents of options messages from the
prototype package.

Change IsPacked to always return false for unpackable field types,
which is consistent with the equivalent C++ API.

This change helps avoid dependency cycles between prototype and the
options messages. (Previously this was resolved by accessing options
with reflection, but just breaking the dependency from prototype to the
options message is cleaner and simpler.)

Change-Id: I756aefe2e04cfa8fea31eaaaa0b5a99d4ac9e851
Reviewed-on: https://go-review.googlesource.com/c/153517
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-11 20:25:45 +00:00
Joe Tsai
2653420232 reflect/protoreflect: remove DescriptorProto method
The DescriptorProto method was originally intended to return a raw descriptor.
However, that functionality will be responsibility of the protodesc package
in the near future.

Also document that any callers of Options must import the descriptor package.
This is reasonable since any usage of that method will almost always be of
the form:
	fd.Options().(*descriptorpb.FieldOptions)

Change-Id: I1085f02305faef8dd724e0cb3ad45ac12bd5df0a
Reviewed-on: https://go-review.googlesource.com/c/153497
Reviewed-by: Damien Neil <dneil@google.com>
2018-12-10 21:53:06 +00:00
Damien Neil
97e7f57dbb reflect/protoreflect: replace Mutable with NewMessage
Remove the Mutable methods from KnownFields, List, and Map, replacing
them with methods which return a new, empty message value without adding
that value to the collection.

The new API is simpler, since it clearly applies only to message values,
and more orthogonal, since it provides a way to create a value without
mutating the collection. This latter point is particularly useful in
map deserialization, where the key may be unknown at the time the value
is deserialized.

Drop the Mutable interface, since it is no longer necessary.

Change-Id: Ic5f3d06a2aa331a5d5cd2b4e670a3dba4a74f77c
Reviewed-on: https://go-review.googlesource.com/c/153278
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-10 21:17:16 +00:00
Joe Tsai
c9899da4d5 internal/encoding/defval: unify logic for handling default values
Logic for serializing the default value in textual form exists in
multiple places in trivially similar forms. Centralize that logic.

Change-Id: I4408ddfeef2c0dfa5c7468e01a4d4df5654ae57f
Reviewed-on: https://go-review.googlesource.com/c/153022
Reviewed-by: Herbie Ong <herbie@google.com>
2018-12-07 22:40:20 +00:00
Joe Tsai
bce82b8e0d reflect/protoreflect: add HasJSONName, ReservedRanges, and ReservedNames
These properties of descriptors are currently missing and makes it impossible
to convert a FileDescriptorProto into one of the structured Go representations
and convert it back to a proto message without loss of information.

Furthermore, ReservedRanges and ReservedNames has semantic importance
to text serialization.

Change-Id: Ic33c30020ad51912b143156b95f47a4fb8da3503
Reviewed-on: https://go-review.googlesource.com/c/153019
Reviewed-by: Damien Neil <dneil@google.com>
2018-12-07 20:10:15 +00:00
Joe Tsai
381f04c102 reflect/protoreflect: add MessageDescriptor.ExtensionRangeOptions
Considerable thought was given to whether a seperate ExtensionRanges interface
should be made that encapsulates FieldNumbers with an Options method.
However, I decided against this design for the following reasons:
* Extension ranges share syntax with reserved numbers and fields.
Why is extension ranges so special that it can have options, while the other
two currently do not? How do we know that those other two won't grow options
in the future? If they do, then those APIs can be expanded in the same way as
how extension range options is being expanded here today.
* Extension range options stand out like a sore thumb compared to the other
eight options. The other options correspond with a named declaration and have
a full protobuf name that they are associated with. Extension range options
is the only options that is not correlated with a full name.
* Extension range options are an extremely rarely used feature and
it seems unfortunate complicating the common case with additional structure.

Change-Id: Ib284a0b798c57dc264febe304692eee5b9c8e91b
Reviewed-on: https://go-review.googlesource.com/c/153018
Reviewed-by: Damien Neil <dneil@google.com>
2018-12-07 19:42:29 +00:00
Joe Tsai
a4cbffe4bc reflect/prototype: add registration hooks for options
Add pseudo-hidden functions to register the concrete Go type used for
the optional types. Also, augment protoc-gen-go to specially generate the
descriptor proto to register these types.

This change does not add validation logic yet to ensure that the correct option
types are passed to the API.

Change-Id: I5decc897e14b4bf570a61cf17b57a066a2a0f9d7
Reviewed-on: https://go-review.googlesource.com/c/153017
Reviewed-by: Damien Neil <dneil@google.com>
2018-12-07 19:38:57 +00:00
Joe Tsai
05d60cda2c reflect/prototype: disable detrand for test
Change-Id: I62cadc7f02afb9a6ee84c5ae2d7857be4b405360
Reviewed-on: https://go-review.googlesource.com/c/153021
Reviewed-by: Herbie Ong <herbie@google.com>
2018-12-07 00:53:04 +00:00