54 Commits

Author SHA1 Message Date
Joe Tsai
d24bc72368 reflect/protoreflect: rename methods with Type suffix
The protobuf type system uses the word "descriptor" instead of "type".
We should avoid the "type" verbage when we aren't talking about Go types.
The old names are temporarily kept around for compatibility reasons.

Change-Id: Icc99c913528ead011f7a74aa8399d9c5ec6dc56e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/172238
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
2019-04-20 06:35:24 +00:00
Joe Tsai
ab61d41ec9 cmd/protoc-gen-go: add constants to control generation of deprecated features
These deprecated features add a non-trivial amount of binary bloat.
Protect these under a constant to make it easy to patch them out.

Change-Id: I12cc33613c19ef60ceec4c4449d0cf3692835170
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/172407
Reviewed-by: Damien Neil <dneil@google.com>
2019-04-16 22:29:53 +00:00
Joe Tsai
5e71dc95f2 cmd/protoc-gen-go: used named types for special fields
Using a named fields gives us the flexibility to change the underlying
representation of special fields without needing to regenerate user code.

We add a named type for ExtensionFields, UnknownFields, and SizeCache.

Change-Id: I107cf82899850ea76665310ce79def60f0f7ab97
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/172402
Reviewed-by: Damien Neil <dneil@google.com>
2019-04-16 22:04:16 +00:00
Joe Tsai
a2dd228004 internal/fileinit: make fileinit responsible for registering concrete options types
Changes:
* Remove protoreflect.OptionsMessage and use protoreflect.ProtoMessage
now that the generated options natively implement reflection.
* Make registration of concrete option types the responsibility of
internal/fileinit since the init logic of descriptor.pb.go uses that.
* Remove equivalent logic in protoc-gen-go to special-case descriptor.

Change-Id: Id814651fafa4d888ff4532d59c6a4f9b68145157
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171465
Reviewed-by: Damien Neil <dneil@google.com>
2019-04-15 20:37:10 +00:00
Joe Tsai
09b5b46567 cmd/protoc-gen-go: revert Enum simplication
Go1.12 on GoLLVM with LLVM 9.0.0 has a bug where the simpler but
equivalent expression results in memory corruption.
Revert to the more complex expression to avoid breaking GoLLVM and
it is not worth waiting for a GoLLVM fix.

Change-Id: I8b42965ca9bd333deb8693021e7b22f966e13521
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171463
Reviewed-by: Damien Neil <dneil@google.com>
2019-04-10 22:39:46 +00:00
Joe Tsai
872b50047a cmd/protoc-gen-go: generate oneof types seperately
Seperate out the generation of the oneof wrapper types from the
code block that is about getter methods.

Change-Id: Ief44ef953d0b5ad8c998a8542c830ca70468a3bf
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171029
Reviewed-by: Damien Neil <dneil@google.com>
2019-04-08 21:41:25 +00:00
Joe Tsai
61968ce130 cmd/protoc-gen-go: perform some code cleanup
Minor changes:
* Use x as the receiver since "e" and "m" are meaningless in the presence
of user-defined enum and message names.
* Consistently keep enum methods together, rather awkwardly split apart
by the value maps.

Change-Id: I68e5666efb56ac7a4d062fb223b9f826dc72aba9
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170357
Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-01 20:44:00 +00:00
Joe Tsai
5d72cc2d37 cmd/protoc-gen-go: lazily GZIP-encode the raw descriptor
This reduces the init-time cost slightly since the GZIP'd
raw descriptor is constructed lazily on demand.

Change-Id: I482c6a2201b8786e425d7dee5612fdfd60ab1500
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169917
Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-01 20:24:54 +00:00
Joe Tsai
35ec98fdcb cmd/protoc-gen-go: generate for v2-only dependencies
This removes yet another set of dependencies of v2 on v1.
The only remaining dependency are in the _test.go files,
primarily for proto.Equal.

Changes made:
* cmd/protoc-gen-go no longer generates any functionality that depends
on the v1 package, and instead only depends on v2.
* internal/fileinit.FileBuilder.MessageOutputTypes is switched from
protoreflect.MessageType to protoimpl.MessageType since the
implementation must be fully inialized before registration occurs.
* The test for internal/legacy/file_test.go is switched to a legacy_test
package to avoid a cyclic dependency.
This requires Load{Enum,Message,File}Desc to be exported.

Change-Id: I43e2fe64cff4eea204258ce11e791aca5eb6e569
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169298
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-26 17:03:31 +00:00
Joe Tsai
4fddebafc0 all: move v1 types over to the v2 repository
As a goal, v2 should not depend on v1. As another step towards that end,
we move all the types that used to be in the v1 protoapi package over to v2.

For now, we place MessageV1, ExtensionRangeV1, and ExtensionDescV1
in runtime/protoiface since these are types that generated messages will
probably have to reference forever. An alternative location could be
reflect/protoreflect, but it seems unfortunate to have to dirty the
namespace of that package with these types.

We move ExtensionFieldV1, ExtensionFieldsV1, and ExtensionFieldsOf
to internal/impl, since these are related to the implementation of a
generated message.

Since moving these types from v1 to v2 implies that the v1 protoapi
package is useless, we update all usages of v1 protoapi in the v2
repository to point to the relevant v2 type or functionality.

CL/168538 is the corresponding change to alter v1.
There will be a temporary build failure as it is not possible
to submit CL/168519 and CL/168538 atomically.

Change-Id: Ide4025c1b6af5b7f0696f4b65b988b4d10a50f0b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168519
Reviewed-by: Herbie Ong <herbie@google.com>
2019-03-22 20:01:07 +00:00
Joe Tsai
ca46d8c924 internal/fieldnum: generate field numbers for the google.protobuf package
Generate field numbers for the well-known types,
so that encoding/jsonpb can benefit from them as well.

This CL fixes internal/cmd/generate-protos, which was silently failing
because the modulePath was not properly initialized. We fix this by
moving it to the start of the init function.

Change-Id: I87637176f29218cffa512b4baa49f39dae924061
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168497
Reviewed-by: Herbie Ong <herbie@google.com>
2019-03-21 00:57:40 +00:00
Joe Tsai
1321a0e05b cmd/protoc-gen-go: generate descriptor proto using v2 textpb
Changes:
* Modify protoc-gen-go to use a helper from internal/impl
to print the message as text.
* Add a helper function to internal/impl that calls v2 textpb.
* Modify encoding/textpb to avoid depending on descriptor proto,
which would cause an import cycle.
* Modify internal/fileinit to populate a pseudo-internal
method on MessageDescriptor to check whether the message should
use the message set wire format. We avoid adding this to the
main MessageDescriptor interface since message sets are a
deprecated proto1 feature that we should avoid promoting.

Change-Id: Ibaf79a563af695756f11ddc4db69b38e25a8f1a7
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168439
Reviewed-by: Herbie Ong <herbie@google.com>
2019-03-20 20:58:24 +00:00
Joe Tsai
559d47f1da cmd/protoc-gen-go: fix init order for v1 registration
The v1 registration leaks the message types out to the proto package.
When doing that, it must ensure that the reflection data structures
for those types are properly initialized first. We achieve that by
doing v1 registration at the end of the reflection init function.

Change-Id: If6df18df59d05bad50ff39c2eff6beb19e7466cc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168348
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-20 04:16:33 +00:00
Joe Tsai
08cd8842f1 cmd/protoc-gen-go: register with the v2 registry for descriptor
By passing the global registries to fileinit.Builder, the Build
method can register the newly constructed descriptors on our behalf.
As a result, we can stop registering with the v1 registries as well.

We only do this for descriptor proto to remove another dependency on v1.
We deliberately keep the v1 registration logic for now to make it easy
to patch away this new behavior.

Change-Id: Ic6aa8ffba3d2d0abe08a61fc5e1c9ca7668e0988
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168000
Reviewed-by: Herbie Ong <herbie@google.com>
2019-03-19 05:04:43 +00:00
Joe Tsai
f31bf2612e cmd/protoc-gen-go: only depend on v2 proto package
This CL breaks another dependency of v2 on v1.
A missing feature in v2 is proto.Clone, but we can use a
Marshal/Unmarshal roundtrip to achieve the same effect
as a temporary stop-gap.

Change-Id: I9d0432fd3efe9fc8b34db6c5a1eabbef0c36277c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168217
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-18 23:11:12 +00:00
Joe Tsai
8e506a8704 cmd/protoc-gen-go: rely on protoimpl for basic helpers
The EnumName, UnmarshalJSONEnum, and CompressGZIP helpers currently live
in v1 protoapi, which would cause all generated messages to depend on v1.
In an effort to break the dependency of v2 on v1, we move these helper
functions to v2 (and re-written to take advantage of protobuf reflection).

These helpers are unfortunate, but we cannot eliminate the functionality
that they implement since they are exposed in the publicly generated API.

Since EnumName does not rely on the enum maps, it removes another dependency
on those variables. Eventually, we can get to the point where these variables
(though declared) are not linked into the binary if the user does not use them.

Also, we rely on the v1 proto package for registration instead of v1 protoapi.
This may re-introduce a cyclic dependency on descriptor proto again in the
future, but the better approach is to just start registering with v2.

Change-Id: Id755585a7a1df14e4a6a2dfa650df221a3c153fb
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/167921
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-18 18:50:16 +00:00
Joe Tsai
afb455eaf8 cmd/protoc-gen-go: correlate v1 ExtensionDesc with v2 ExtensionType
Unfortunately a good amount of code uses pointer comparisons on the
v1 ExtensionDesc to determine exactly which extension field is set,
rather than checking whether the extension descriptor semantically
describes the field that they are interested in.

To preserve this behavior in v1, we need a 1:1 mapping between
a v2 ExtensionType and a specific v1 ExtensionDesc.

Change-Id: I852b3cefb4585bd656e48e5adad6cc28795d02df
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/167759
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-15 01:24:42 +00:00
Joe Tsai
1af1de0c15 internal/cmd/generate-protos: generate internal descfield package
Generate a list of descriptor fields from the descriptor.proto itself
as an internal package. Use these fields for the internal implementation.

Change-Id: Ib1ab0c5c6deb332ba6c8018ef55136b7e5974944
Reviewed-on: https://go-review.googlesource.com/c/164864
Reviewed-by: Herbie Ong <herbie@google.com>
2019-03-02 23:16:45 +00:00
Joe Tsai
19058431cd internal/cmd/generate-protos: initial commit
Create a single binary for handling generation of protos.
This replaces previous logic spread throughout the repo in:
* regenerate.bash
* cmd/protoc-gen-go/golden_test.go
* cmd/protoc-gen-go-grpc/golden_test.go
* (indirectly) internal/protogen/goldentest

One of the problems with the former approaches is that they relied on
a version of protoc that was specific to a developer's workstation.
This meant that the result of generation was not hermetic.
To address this, we rely on the hard-coded version of protobuf specified
in the test.bash script.

A summary of changes in this CL are:
* The internal_gengo.GenerateFile and internal_gengogrpc.GenerateFile
functions are unified to have consistent signatures. It seems that the
former accepted a *protogen.GeneratedFile to support v1 where gRPC code
was generated into the same file as the base .pb.go file. However, the
same functionality can be achieved by having the function return
the generated file object.
* The test.bash script patches the protobuf toolchain to have properly
specified go_package options in each proto source file.
* The test.bash script accepts a "-regenerate" argument.
* Add generation for the well-known types. Contrary to how these were
laid out in the v1 repo, all the well-known types are placed in the
same Go package.
* Add generation for the conformance proto.
* Remove regenerate.bash
* Remove internal/protogen
* Remove cmd/protoc-gen-go/golden_test.go
* Remove cmd/protoc-gen-go-grpc/golden_test.go
* Add cmd/protoc-gen-go/annotation_test.go

Change-Id: I4a1a97ae6f66e2fabcf4e4d292c95ab2a2db0248
Reviewed-on: https://go-review.googlesource.com/c/164477
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-01 20:47:52 +00:00
Joe Tsai
4069211bcd protogen: use full path for generated file variable name
Use the full path (including the extension) for the generation of
the per-file variable name. Several reasons for this:

* The current logic is buggy in the case where pathType == pathTypeImport
since the prefix variable will be mangled with the Go import path.

* The extension is technically part of the path.
Thus, "path/to/foo.proto" and "path/to/foo.protodevel" are two
distinctly different imports.

* Style-wise, it subjectively looks better. Rather than being a mixture
of camelCase and snake_case, it is all snake_case for the common case:
	before: ProtoFile_google_protobuf_any
	after:  File_google_protobuf_any_proto

* Since the extension is almost always ".proto", this results in a
suffix of "_proto", which provides an additional layer of protection
against possible name conflicts. The previous approach could possibly
have a conflict between "Foo.proto" and a message named ProtoFile
with a sub-message called Foo.

Also, use the per-file variable name for the raw descriptor variables
instead of the hashed version.

Change-Id: Ic91e326b7593e5985cee6ececc60539c27fe32fe
Reviewed-on: https://go-review.googlesource.com/c/164379
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-01 00:13:31 +00:00
Joe Tsai
cf81e67b53 cmd/protoc-gen-go: use protoapi.CompressGZIP
Calling a helper function directly should reduce binary bloat slightly.

Change-Id: I6068dc4cd00c8d90d2e6e6d99633b81388bc8781
Reviewed-on: https://go-review.googlesource.com/c/164679
Reviewed-by: Damien Neil <dneil@google.com>
2019-02-28 22:28:32 +00:00
Damien Neil
8012b444ee internal/fileinit: generate reflect data structures from raw descriptors
This CL takes a significantly different approach to generating support
for protobuf reflection. The previous approach involved generating a
large number of Go literals to represent the reflection information.
While that approach was correct, it resulted in too much binary bloat.

The approach taken here initializes the reflection information from
the raw descriptor proto, which is a relatively dense representation
of the protobuf reflection information. In order to keep initialization
cost low, several measures were taken:
* At program init, the bare minimum is parsed in order to initialize
naming information for enums, messages, extensions, and services declared
in the file. This is done because those top-level declarations are often
relevant for registration.
* Only upon first are most of the other data structures for protobuf
reflection actually initialized.
* Instead of using proto.Unmarshal, a hand-written unmarshaler is used.
This allows us to avoid a dependendency on the descriptor proto and also
because the API for the descriptor proto is fundamentally non-performant
since it requires an allocation for every primitive field.

At a high-level, the new implementation lives in internal/fileinit.

Several changes were made to other parts of the repository:
* cmd/protoc-gen-go:
  * Stop compressing the raw descriptors. While compression does reduce
the size of the descriptors by approximately 2x, it is a pre-mature
optimization since the descriptors themselves are around 1% of the total
binary bloat that is due to generated protobufs.
  * Seeding protobuf reflection from the raw descriptor significantly
simplifies the generator implementation since it is no longer responsible
for constructing a tree of Go literals to represent the same information.
  * We remove the generation of the shadow types and instead call
protoimpl.MessageType.MessageOf. Unfortunately, this incurs an allocation
for every call to ProtoReflect since we need to allocate a tuple that wraps
a pointer to the message value, and a pointer to message type.
* internal/impl:
  * We add a MessageType.GoType field and make it required that it is
set prior to first use. This is done so that we can avoid calling
MessageType.init except for when it is actually needed. The allows code
to call (*FooMessage)(nil).ProtoReflect().Type() without fearing that the
init code will run, possibly triggering a recursive deadlock (where the
init code depends on getting the Type of some dependency which may be
declared within the same file).
* internal/cmd/generate-types:
  * The code to generate reflect/prototype/protofile_list_gen.go was copied
and altered to generated internal/fileinit.desc_list_gen.go.

At a high-level this CL adds significant technical complexity.
However, this is offset by several possible future changes:
* The prototype package can be drastically simplified. We can probably
reimplement internal/legacy to use internal/fileinit instead, allowing us
to drop another dependency on the prototype package. As a result, we can
probably delete most of the constructor types in that package.
* With the prototype package significantly pruned, and the fact that generated
code no longer depend on depends on that package, we can consider merging
what's left of prototype into protodesc.

Change-Id: I6090f023f2e1b6afaf62bd3ae883566242e30715
Reviewed-on: https://go-review.googlesource.com/c/158539
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-01-30 01:33:46 +00:00
Damien Neil
a7cbd06b9b cmd/protoc-gen-go: don't chain public imports
Consider the case:

	a.proto publicly imports b.proto
	b.proto publicly imports c.proto

Should a.pb.go include symbols defined in c.pb.go?

Historically, it has not. As of #155677, it does. Regardless of which behavior
is preferable, #155677 produces broken code in some common situations: If
a.proto also publicly imports c.proto, we now generate two copies of the
forwarding decls for that file.

Restore the pre-#155677 behavior to avoid this breakage.

Change-Id: I283600b3be19eac2c3b3c14233bb69fa64661581
Reviewed-on: https://go-review.googlesource.com/c/156348
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-01-07 23:17:30 +00:00
Damien Neil
7bf3ce2145 cmd/protoc-gen-go: generate public imports by parsing the imported .pb.go
Rather than explicitly enumerating the set of symbols to import,
just parse the imported file and extract every exported symbol.

This is possibly a bit more code, but adapts much better to future
expansion.

Change-Id: I4429664f4c068a2a55949d46aefc19865b008a77
Reviewed-on: https://go-review.googlesource.com/c/155677
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-01-03 21:46:42 +00:00
Damien Neil
60e9f879a3 cmd/protoc-gen-go: remove stale comment
Change-Id: I88c5536c29602206657cbeab81347c872732da56
Reviewed-on: https://go-review.googlesource.com/c/156198
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-01-03 19:29:30 +00:00
Damien Neil
0dbce8360e cmd/protoc-gen-go: always register file descriptor
Always generate the init func containing the proto.RegisterFile call.
This call used to be in a different init func which was unconditionally
generated. Consolidating the init funcs means that we need to always
generate the single remaining func.

Change-Id: Icbf7d14d018d693dab2824f114c57e4e36384568
Reviewed-on: https://go-review.googlesource.com/c/156197
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-01-03 17:58:25 +00:00
Joe Tsai
7390346c05 cmd/protoc-gen-go: invert conditional check for proto syntax
To avoid accidentally emitting the UnmarshalMethod in the unlikely
(but possible) event that a proto4 syntax was added, swich the conditional
to only trigger on the proto2 syntax.

Change-Id: I0c201eace56e9ecc92cd791e45ca5d3b99e2ba86
Reviewed-on: https://go-review.googlesource.com/c/154317
Reviewed-by: Damien Neil <dneil@google.com>
2018-12-17 22:19:10 +00:00
Joe Tsai
9667c4816d cmd/protoc-gen-go: reduce technical debt
The following TODOs were addressed:
* Consistently collect all enums, messages, and extensions in a breadth-first order.
The practical affect of this is that the declaration order in a Go file may change.
This simplifies reflection generation, which relies on consistent ordering.
* Removal of placeholder declarations (e.g., "var _ = proto.Marshal") since
protogen is intelligent about including imports as necessary.
* Always generate a default variable or constant for explicit empty strings.
The practical effect of this is the addition of new declarations in some cases.
However, it simplifies our logic such that it matches the protobuf data model.
* Generate the registration statements in a consistent order.

Change-Id: I627bb72589432bb65d53b50965ea88e5f7983977
Reviewed-on: https://go-review.googlesource.com/c/152778
Reviewed-by: Damien Neil <dneil@google.com>
2018-12-07 03:13:48 +00:00
Joe Tsai
24ceb2b095 cmd/protoc-gen-go: generate descriptor and plugin with reflection
In CL/152020, we checked in pre-generated versions of descriptor and plugin.
This CL makes it such that they are generated by protoc-gen-go.

We modify protoc-gen-go to avoid reflection support by default
since the binary size increase is still an issue to investigate.
Reflection support is temporarily enabled by setting a special
PROTOC_GEN_GO_ENABLE_REFLECT environment variable.

Reflection support is always enabled for descriptor and plugin.
Furthermore, we change descriptor to depend on the protoapi package
instead of the proto package. The reason we do not switch to protoapi
for all generated protos is because we still depend on v1 proto
for the table-driven InternalMessageInfo type. Dropping it from descriptor
is semantically correct, but does incur slight performance cost.
It does not seem appropriate to drop it for all generated messages.

We could move InternalMessageInfo to protoapi, but the logic behind that
is significant.

Change-Id: I5c3fff7f6eab1a5a2399049d42fa6bf42d4c93f9
Reviewed-on: https://go-review.googlesource.com/c/152547
Reviewed-by: Damien Neil <dneil@google.com>
2018-12-06 17:47:27 +00:00
Joe Tsai
b6405bd782 cmd/protoc-gen-go: add support for protobuf reflection
Implement support in protoc-gen-go for generating messages and enums
that satisfy the v2 protobuf reflection interfaces. Specifically, the following
are added:
* top-level variable representing the file descriptor
* ProtoReflect method on enums (to implement protoV2.Enum)
* ProtoReflect method on messages (to implement protoV2.Message)

The following are not supported yet:
* resolving transitive dependencies for file imports
* Extension descriptors
* Service descriptors

The implementation approach creates a single array for all the message and enum
declarations and references sections of that array to complete dependencies.
Since protobuf declarations can form a graph (a message may depend on itself),
it is difficult to construct a graph as a single literal. One way is to use
placeholder descriptors, but that is not efficient as it requires encoding
the full name of each dependent enum and message and then later resolving it;
thus, both expanding the binary size and also increasing initialization cost.
Instead, we add a prototype.{Enum,Message}.Reference method to obtain a
descriptor reference for the purposes for satisfying dependencies.
As such, nested declarations and dependencies are populated in an init function.

Other changes to support the implementation:
* Added a protoimpl package to expose the MessageType type and also the
MessageTypeOf and EnumTypeOf helper functions.
* Added a protogen.File.GoIdent field to provide a suggested variable name
for the file descriptor.
* Added prototype.{Enum,Message}.Reference that provides a descriptor reference
for the purposes for satisfying cyclic dependencies.
* Added protoreflect.{Syntax,Cardinality,Kind}.GoString to obtain a Go source
identifier that represents the given constant.

Change-Id: I9455764882dee6ad10f251901e7d419091e8bf1d
Reviewed-on: https://go-review.googlesource.com/c/150074
Reviewed-by: Damien Neil <dneil@google.com>
2018-12-05 19:32:12 +00:00
Joe Tsai
e1f8d50e17 reflect/protodesc: split descriptor related functionality from prototype
In order to generate descriptor.proto, the generated code would want to depend
on the prototype package to construct the reflection data structures.
However, this is a problem since descriptor itself is one of the dependencies
for prototype. To break this dependency, we do the following:
* Avoid using concrete *descriptorpb.XOptions messages in the public API, and
instead just use protoreflect.ProtoMessage. We do lose some type safety here
as a result.
* Use protobuf reflection to interpret the Options message.
* Split out NewFileFromDescriptorProto into a separate protodesc package since
constructing protobuf reflection from the descriptor proto obviously depends
on the descriptor protos themselves.

As part of this CL, we check in a pre-generated version of descriptor and plugin
that supports protobuf reflection natively and switchover all usages of those
protos to the new definitions. These files were generated by protoc-gen-go
from CL/150074, but hand-modified to remove dependencies on the v1 proto runtime.

Change-Id: I81e03c42eeab480b03764e2fcbe1aae0e058fc57
Reviewed-on: https://go-review.googlesource.com/c/152020
Reviewed-by: Damien Neil <dneil@google.com>
2018-12-05 00:38:30 +00:00
Damien Neil
7e5c64778d cmd/protoc-gen-go: fix forwarding decls for inf and NaN defaults
Floating-point defaults of -inf, +inf, and NaN must be generated as
vars rather than consts. Fix the forwarding declaration for public
imports of these vars. (Was "const", is now "var".)

Change-Id: Ic6dc90ab7f88378ba477bff39e047de5f5193c35
Reviewed-on: https://go-review.googlesource.com/c/151757
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-11-29 23:43:35 +00:00
Joe Tsai
f757390b52 cmd/protoc-gen-go: remove generation of RegisterMessageSetType
PR#741 in the v1 repository deprecated this behavior.

Change-Id: Ife48f1d586f178d875b9b3002a88b3336a2cd3b4
Reviewed-on: https://go-review.googlesource.com/c/151401
Reviewed-by: Damien Neil <dneil@google.com>
2018-11-28 00:40:08 +00:00
Joe Tsai
b6dbf23033 cmd/protoc-gen-go: remove generation of {Unmarshal,Marshal}JSON for MessageSets
PR#741 in the v1 repository deprecated this behavior.

Change-Id: Idffa3884d43f9cc5528ce8d9f303676e0e501b67
Reviewed-on: https://go-review.googlesource.com/c/151400
Reviewed-by: Damien Neil <dneil@google.com>
2018-11-27 23:47:23 +00:00
Joe Tsai
d7e97bc71b cmd/protoc-gen-go: generate XXX_OneofWrappers instead of XXX_OneofFuncs
The marshaler, unmarshaler, and sizer functions are unused ever since
the underlying implementation was switched to be table-driven.
Change the function to only return the wrapper structs.

This change:
* enables generated protos to drop dependencies on certain proto types
* reduces the size of generated protos
* simplifies the implementation of oneofs in protoc-gen-go

Updates #708

Change-Id: I845c9009bc0236d1b51d34b014dc3e184303c0f2
Reviewed-on: https://go-review.googlesource.com/c/151357
Reviewed-by: Damien Neil <dneil@google.com>
2018-11-27 19:36:27 +00:00
Joe Tsai
c1c17aa013 protogen: add GoImportPath.Ident helper
The GoImportPath.Ident helper creates a GoIdent using the receiver
as the GoImportPath in the GoIdent. This helper helps with the construction
of qualified identifiers.

Example usage:
	const protoPackage = protogen.GoImportPath("github.com/golang/protobuf/proto")
	protoPackage.Ident("ExtensionRange") // produces "proto.ExtensionRange"

The advantage of this helper is that usage of it looks similar to how
the identifier will eventually be rendered.

This is significantly more readable than the current approach:
	protogen.GoIdent{
		GoImportPath: protoPackage,
		GoName: "ExtensionRange",
	}

Change-Id: If7ecd7e60fad12bc491eee0dcb05f8fdebc9c94e
Reviewed-on: https://go-review.googlesource.com/c/150058
Reviewed-by: Damien Neil <dneil@google.com>
2018-11-16 19:34:14 +00:00
Joe Tsai
05828dba44 internal/encoding/tag: centralize logic for protobuf struct tag serialization
The bespoke text-serialization of field descriptors in protoc-gen-go is also
used in the legacy implementation of protobuf reflection to derive a
protoreflect.FieldDescriptor from legacy messages and also to convert to/from
protoreflect.ExtensionDescriptor and protoV1.ExtensionDesc.

Centralize this logic in a single place:
* to avoid reimplementing the same logic in internal/impl
* to keep the marshal and unmarshal logic co-located

Change-Id: I634c5afbb9dc6eda91d6cb6b0e68dbd724cb1ccb
Reviewed-on: https://go-review.googlesource.com/c/146758
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
2018-11-05 20:12:56 +00:00
Damien Neil
6238696abc cmd/protoc-gen-go: replicate v1 generator behavior for MessageSet extensions
Given:

  package foo
  extend proto2.bridge.MessageSet {
    optional Message message_set_extension = 100;
  }

Register the extension as a message set extension and give it the name
"foo.".

We really shouldn't do this in this case; the special-case treatment of
extensions to MessageSet is only for extensions nested in a parent
message. However, this is consistent with the behavior of the v1 generator.
Match that for now.

Change-Id: I919c409605a197904fd3227efc920192d484f431
Reviewed-on: https://go-review.googlesource.com/c/145957
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-10-30 18:46:20 +00:00
Damien Neil
6b54131a89 cmd/protoc-gen-go: generate forwarding decls for ExtensionDescs
When publicly importing a package with extension definitions, generate
forwarding declarations for the "E_..." ExtensionDesc var:

    var E_ExtensionField = publicimport.E_ExtensionField

Change-Id: Ifd57c487c3a44f303c2c098a42ea249b219b734f
Reviewed-on: https://go-review.googlesource.com/c/145498
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-10-29 17:01:50 +00:00
Damien Neil
40a08052f9 cmd/protoc-gen-go: don't import _ weak imports
Don't generate a blank import for unused weak imports.

Full support for weak imports would involve not importing the package at
all. This just avoids generating an import when we don't need one.

Change-Id: I7e8491f415dc8333a2837db5225256b959921be2
Reviewed-on: https://go-review.googlesource.com/c/145497
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-10-29 16:16:32 +00:00
Damien Neil
a485fbd353 reflect/protoreflect: add FieldDescriptor.DefaultEnumValue method
Default values for enums are specified by name, not number. An enum
may contain multiple values with different names but the same number.
Representing the default as a protoreflect.Value containing an EnumNumber
can discard information.

Add a method returning the EnumValueDescriptor.

Change-Id: If8beee3f81d41c4f9af45423252603b86949c7a5
Reviewed-on: https://go-review.googlesource.com/c/145158
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-10-29 15:59:23 +00:00
Damien Neil
44000a1bea cmd/protoc-gen-go: treat extensions fields as proto2 in some places
The v1 generator doesn't include a "proto3" tag on extension fields,
even when the field is defined in a proto3 file. Match that behavior for
consistency. We can probably change this later if we want to; it's
unlikely anyone is depending on this behavior.

The v1 generator uses pointer types for extension fields, even when the
field is defined in a proto3 file. (e.g., *FooEnum instead of FooEnum.)
Match this behavior. We can't change this without breaking compatibility
in the generated code.

Change-Id: I4072f3dd1c915bf9ab89f1d5198e0144cb4de20f
Reviewed-on: https://go-review.googlesource.com/c/144282
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-10-24 19:46:45 +00:00
Damien Neil
204f1c0ad8 reflect/protoreflect: add Descriptor.Options method
Add a method to fetch descriptor options. Since options are proto
messages (e.g., google.protobuf.FieldOptions), and proto message
packages depend on the protoreflect package, returning the actual option
type would cause a dependency cycle. Instead, we return an interface
value which can be type asserted to the appropriate concrete type.

Add options support to the prototype package.

Some of the prototype constructors included fields (such as
Field.IsPacked) which represent information from the options
(such as google.protobuf.FieldOptions.packed). To avoid confusion about
the canonical source of information, drop these fields in favor of the
options.

Drop the unimplemented Descriptor.DescriptorOptionsProto.

Change-Id: I66579b6a7d10d99eb6977402a247306a78913e74
Reviewed-on: https://go-review.googlesource.com/c/144277
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-10-23 23:44:11 +00:00
Joe Tsai
bda671fa93 cmd/protoc-gen-go: re-escape default byte values
Historically, protoc-gen-go outputted the escaped form of bytes as provided by
protoc verbatim. This behavior is buggy, but nothing really uses this tag
since default values are properties of getters instead of serialization.
Rather than fixing it, just preserve prior behavior. Otherwise, logic depending
on the old legacy behavior will not be able to distinguish between the unescaped
or the escaped forms.

Furthermore, since protoc-gen-go historically copied the protoc output verbatim,
we will need to escape the default bytes in a way that is identical to the
CEscape function from strutil.cc of the protoc source code.

Change-Id: I0ab55e220ae430dd123ad050406e285788f6cb40
Reviewed-on: https://go-review.googlesource.com/c/143543
Reviewed-by: Damien Neil <dneil@google.com>
2018-10-22 17:14:19 +00:00
Damien Neil
ba1159f426 protogen: move comment generation into protogen
Most plugins need to copy comments from .proto source files into the
generated code. Move this functionality into protogen to avoid
duplicating it everywhere.

Change-Id: I48a96ba794192e7ddc00281342afd4805ef6fe0f
Reviewed-on: https://go-review.googlesource.com/c/142890
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-10-17 21:03:49 +00:00
Damien Neil
162c12703c protogen: generate .meta file with code annotations
When the generator parameter 'annotate_code' is provided, generate a .meta
file containing a GeneratedCodeInfo message describing the generated code's
relation to the source .proto file.

Annotations are added with (*protogen.GeneratedFile).Annotate, which takes the
name of a Go identifier (e.g., "SomeMessage" or "SomeMessage.GetField") and an
associated source location. The generator examines the generated AST to
determine source offsets for the symbols.

Change the []int32 "Path" in protogen types to a "Location", which also captures
the source file name.

Change-Id: Icd2340875831f40a1f91d495e3bd7ea381475c77
Reviewed-on: https://go-review.googlesource.com/c/139759
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-10-16 17:22:41 +00:00
Damien Neil
2193e8df3b cmd/protoc-gen-go: import public: forward default const/vars
Generate forwarders for default value const/vars defined in public
imports:

    const Default_Message_Field = pubimport.Default_Message_Field

Change-Id: Ife09e38ae6a674b4460dd6613a8264e23f30b277
Reviewed-on: https://go-review.googlesource.com/c/140897
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-10-16 16:53:07 +00:00
Damien Neil
daa4fad986 cmd/protoc-gen-go: remove spurious . from enums with no package
Enums, for historical reasons, are registered with the proto package
under the name "<proto_package>.<go_type_name>". Don't include the dot
if there is no package statement in the .proto source file.

Change-Id: I6fb57d0803506668f60123a29fa06ae87fec523b
Reviewed-on: https://go-review.googlesource.com/c/140657
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-10-16 16:04:35 +00:00
Damien Neil
ccf3fa6cad cmd/protoc-gen-go: don't generate default values for [default=""].
The previous generator considers a default value of "" (valid only for
string and bytes fields) to be unset, and does not generate a const or
var to hold the default value.

e.g.,

  message M {
    optional F string = 1 [default=""];
  }

does not generate this constant, even though the field has a default
value:

  const Default_M_F string = ""

Maintain consistent output.

Change-Id: Ib172b02d59c15c05e19a7056d05ce1c619a2fa40
Reviewed-on: https://go-review.googlesource.com/138518
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-09-28 21:56:28 +00:00
Damien Neil
ea7baf45f8 cmd/protoc-gen-go: annotate all well-known-types
Change-Id: I3559bb47cb8e93aadeea6224857d65e84ca4b27a
Reviewed-on: https://go-review.googlesource.com/138517
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-09-28 21:56:15 +00:00