Consider the case:
a.proto publicly imports b.proto
b.proto publicly imports c.proto
Should a.pb.go include symbols defined in c.pb.go?
Historically, it has not. As of #155677, it does. Regardless of which behavior
is preferable, #155677 produces broken code in some common situations: If
a.proto also publicly imports c.proto, we now generate two copies of the
forwarding decls for that file.
Restore the pre-#155677 behavior to avoid this breakage.
Change-Id: I283600b3be19eac2c3b3c14233bb69fa64661581
Reviewed-on: https://go-review.googlesource.com/c/156348
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
New test case demonstrating an import public bug:
a.proto publicly imports b.proto and c.proto
b.proto publicly imports c.proto
a.pb.go includes two copies of symbols in c.pb.go
The problem is that the new implementation of public imports, which
parses the generated code for the import to determine what symbols
are present, doesn't distinguish between symbols defined in a file
and symbols imported from a public import of some other file. This
can then lead to duplicate definitions in cases like the above.
Change-Id: Ia86e9f188d7bae8d9d4afbd2b5db9b64071425c3
Reviewed-on: https://go-review.googlesource.com/c/156347
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
File is out of date, regenerate it. (This didn't show up as a test failure
because we don't test the non-grpc golden files under the grpc directory.)
Change-Id: Ied485d28184c45bbca5b52138eb9cf300d813a57
Reviewed-on: https://go-review.googlesource.com/c/156345
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Change the go_package option to match the actual import path of this package.
Change-Id: Ie8630878ce75e34ca76d97c6e2922254cf964801
Reviewed-on: https://go-review.googlesource.com/c/156344
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
If there are any kind of errors in trying to expand the Any message,
always fallback to marshaling it as regular message. This makes it
consistent with V1 and C++ libs.
Change-Id: I007414c1767e682623c45d4dd8c82b9998f61781
Reviewed-on: https://go-review.googlesource.com/c/156257
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Marshal well-known type Any in expanded form by default, else fallback
to marshaling it as a regular message.
Change-Id: Ic7e9e37b47042a163941f8849dc366ffe48103ca
Reviewed-on: https://go-review.googlesource.com/c/156097
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Rather than explicitly enumerating the set of symbols to import,
just parse the imported file and extract every exported symbol.
This is possibly a bit more code, but adapts much better to future
expansion.
Change-Id: I4429664f4c068a2a55949d46aefc19865b008a77
Reviewed-on: https://go-review.googlesource.com/c/155677
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Always generate the init func containing the proto.RegisterFile call.
This call used to be in a different init func which was unconditionally
generated. Consolidating the init funcs means that we need to always
generate the single remaining func.
Change-Id: Icbf7d14d018d693dab2824f114c57e4e36384568
Reviewed-on: https://go-review.googlesource.com/c/156197
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The first commit of protoregistry only added a registry for files.
However, a separate type of registry is needed to provide a mapping between
protobuf names and actual Go types representing those names.
Additional high-level API:
var GlobalTypes = new(Types)
type Type interface{ ... }
type Types struct{ ... }
func NewTypes(...Type) *Types
func (*Types) Register(...Type) error
func (*Types) FindEnumByName(pref.FullName) (pref.EnumType, error)
func (*Types) FindMessageByName(pref.FullName) (pref.MessageType, error)
func (*Types) FindMessageByURL(string) (pref.MessageType, error)
func (*Types) FindExtensionByName(pref.FullName) (pref.ExtensionType, error)
func (*Types) FindExtensionByNumber(pref.FullName, pref.FieldNumber) (pref.ExtensionType, error)
func (*Types) RangeEnums(func(pref.EnumType) bool)
func (*Types) RangeMessages(func(pref.MessageType) bool)
func (*Types) RangeExtensions(func(pref.ExtensionType) bool)
func (*Types) RangeExtensionsByMessage(pref.FullName, func(pref.ExtensionType) bool)
Change-Id: I0d07705801684a1eb5853bcd05fcce12598a0047
Reviewed-on: https://go-review.googlesource.com/c/131345
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The purego tag (see https://golang.org/issue/23172) is a community agreed
upon signal that a given build environment does not support unsafe.
The appengine environment is supposed to respect this tag, but does not
properly do so. Add this tag back in until they fix their environment.
Change-Id: I9a70062be4339c2e1a93cac31d387698c561b8aa
Reviewed-on: https://go-review.googlesource.com/c/154743
Reviewed-by: Damien Neil <dneil@google.com>
Introduce a protoreflect.OptionsMessage interface type to represent
options messages. The protoreflect package can't deal with concrete
options types, to avoid dependency cycles; OptionsMessage serves both
as documentation and a single point to define the constraints we apply
to options.
Change the constraints on options from ProtoMessage to interface{} to
permit use of option message types which only implement the v1
proto.Message interface.
This still leaves a requirement in the internal/legacy package that
options implement protoreflect.ProtoMessage, since that package now uses
the v2 Unmarshal.
Change-Id: I547518ab2c3b90c3911ef641b05b169d50a4b33a
Reviewed-on: https://go-review.googlesource.com/c/154877
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
To avoid accidentally emitting the UnmarshalMethod in the unlikely
(but possible) event that a proto4 syntax was added, swich the conditional
to only trigger on the proto2 syntax.
Change-Id: I0c201eace56e9ecc92cd791e45ca5d3b99e2ba86
Reviewed-on: https://go-review.googlesource.com/c/154317
Reviewed-by: Damien Neil <dneil@google.com>
If a field has an ExtensionType defined, use it in decoding.
Change-Id: I85f3da0f52a11578500cf28e4611fa4eb31f0623
Reviewed-on: https://go-review.googlesource.com/c/154581
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The v2 decoder isn't 100% complete, but it's good enough.
Delete the vendored copy of the v1 Unmarshal implementation.
Change-Id: Ibeabbb2e9109a1ec3df57e71f98b7aa4a583fc5b
Reviewed-on: https://go-review.googlesource.com/c/154577
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Add proto.Unmarshal.
Test cases all produce identical results to the v1 unmarshaller.
Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d
Reviewed-on: https://go-review.googlesource.com/c/153918
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
We want to codify the naming logic to be understandable in a specification.
Clean up the camelCase function to be a little more readable.
Cleanup and fix the cleanGoName function when mustExport is true.
It is okay to change this behavior since this is new logic in v2 that has not
yet been exposed to the universe.
In the mustExport code-path, we do not need to check for conflicts with keywords
since Go keywords are never uppercase, so the uppercasing of the first letter
is sufficient to break a conflict. Also, we fix the logic for uppercasing
the first character since not every lowercase character has an uppercase form.
Change-Id: If14422d773bb89ed7038d874135e3dcd12683101
Reviewed-on: https://go-review.googlesource.com/c/154180
Reviewed-by: Damien Neil <dneil@google.com>
Do not treat reserved field names as unknown. Skip over these instead
of returning error for unknown field.
Change-Id: Iba9371fd901a0690e5b7329bccf261570352b55d
Reviewed-on: https://go-review.googlesource.com/c/154178
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
This initial textproto unmarshaling implementation covers messages
without the use of extensions, Any expansion, nor weak.
Updated encoding tests. Split some testcases to keep each simpler.
Added TestRoundTrip for example messages like the well-known types.
Change-Id: Icffab02834aa004fa8409a9da70624f687f604fb
Reviewed-on: https://go-review.googlesource.com/c/153020
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
PBValueOf returns a protoreflect.Message, not a
protoreflect.ProtoMessage.
Change-Id: I88ed55f52bada6fc2b29ffd63e30de09e1febe8c
Reviewed-on: https://go-review.googlesource.com/c/153917
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Record annotations on methods in interface types (e.g., gRPC
server/client methods).
Generate annotations in golden tests, even when not checking the
content, to catch annotations without a corresponding symbol.
Change-Id: I44ae6caf66f709dc7f4686e931be04b8b6fa843d
Reviewed-on: https://go-review.googlesource.com/c/153877
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Was using a pref.Message where we want a pref.ProtoMessage.
Change-Id: I61d986a43eaf8f945a1378a7a10120474aa89d6f
Reviewed-on: https://go-review.googlesource.com/c/153697
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Add fields to the Message and Field builder structs which hold the value
of MessageOptions.map_entry, FieldOptions.packed, and FieldOptions.weak
options. Remove all access to the contents of options messages from the
prototype package.
Change IsPacked to always return false for unpackable field types,
which is consistent with the equivalent C++ API.
This change helps avoid dependency cycles between prototype and the
options messages. (Previously this was resolved by accessing options
with reflection, but just breaking the dependency from prototype to the
options message is cleaner and simpler.)
Change-Id: I756aefe2e04cfa8fea31eaaaa0b5a99d4ac9e851
Reviewed-on: https://go-review.googlesource.com/c/153517
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Fix a few points of divergence between the v1 and v2 generators around
when to apply camel-casing to service and method names.
Change-Id: I862f89c0995c540e4862013316d7af772e1ab0d8
Reviewed-on: https://go-review.googlesource.com/c/153658
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Change the service and method names in grpc.proto to not use camel-case,
to make it explicit where rewriting occurs in the generated code.
Change-Id: I9e4a851097b0ee14817a589f5f959adcc5a14fe3
Reviewed-on: https://go-review.googlesource.com/c/153657
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The DescriptorProto method was originally intended to return a raw descriptor.
However, that functionality will be responsibility of the protodesc package
in the near future.
Also document that any callers of Options must import the descriptor package.
This is reasonable since any usage of that method will almost always be of
the form:
fd.Options().(*descriptorpb.FieldOptions)
Change-Id: I1085f02305faef8dd724e0cb3ad45ac12bd5df0a
Reviewed-on: https://go-review.googlesource.com/c/153497
Reviewed-by: Damien Neil <dneil@google.com>
Remove the Mutable methods from KnownFields, List, and Map, replacing
them with methods which return a new, empty message value without adding
that value to the collection.
The new API is simpler, since it clearly applies only to message values,
and more orthogonal, since it provides a way to create a value without
mutating the collection. This latter point is particularly useful in
map deserialization, where the key may be unknown at the time the value
is deserialized.
Drop the Mutable interface, since it is no longer necessary.
Change-Id: Ic5f3d06a2aa331a5d5cd2b4e670a3dba4a74f77c
Reviewed-on: https://go-review.googlesource.com/c/153278
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Logic for serializing the default value in textual form exists in
multiple places in trivially similar forms. Centralize that logic.
Change-Id: I4408ddfeef2c0dfa5c7468e01a4d4df5654ae57f
Reviewed-on: https://go-review.googlesource.com/c/153022
Reviewed-by: Herbie Ong <herbie@google.com>
These properties of descriptors are currently missing and makes it impossible
to convert a FileDescriptorProto into one of the structured Go representations
and convert it back to a proto message without loss of information.
Furthermore, ReservedRanges and ReservedNames has semantic importance
to text serialization.
Change-Id: Ic33c30020ad51912b143156b95f47a4fb8da3503
Reviewed-on: https://go-review.googlesource.com/c/153019
Reviewed-by: Damien Neil <dneil@google.com>
Considerable thought was given to whether a seperate ExtensionRanges interface
should be made that encapsulates FieldNumbers with an Options method.
However, I decided against this design for the following reasons:
* Extension ranges share syntax with reserved numbers and fields.
Why is extension ranges so special that it can have options, while the other
two currently do not? How do we know that those other two won't grow options
in the future? If they do, then those APIs can be expanded in the same way as
how extension range options is being expanded here today.
* Extension range options stand out like a sore thumb compared to the other
eight options. The other options correspond with a named declaration and have
a full protobuf name that they are associated with. Extension range options
is the only options that is not correlated with a full name.
* Extension range options are an extremely rarely used feature and
it seems unfortunate complicating the common case with additional structure.
Change-Id: Ib284a0b798c57dc264febe304692eee5b9c8e91b
Reviewed-on: https://go-review.googlesource.com/c/153018
Reviewed-by: Damien Neil <dneil@google.com>
Add pseudo-hidden functions to register the concrete Go type used for
the optional types. Also, augment protoc-gen-go to specially generate the
descriptor proto to register these types.
This change does not add validation logic yet to ensure that the correct option
types are passed to the API.
Change-Id: I5decc897e14b4bf570a61cf17b57a066a2a0f9d7
Reviewed-on: https://go-review.googlesource.com/c/153017
Reviewed-by: Damien Neil <dneil@google.com>
The following TODOs were addressed:
* Consistently collect all enums, messages, and extensions in a breadth-first order.
The practical affect of this is that the declaration order in a Go file may change.
This simplifies reflection generation, which relies on consistent ordering.
* Removal of placeholder declarations (e.g., "var _ = proto.Marshal") since
protogen is intelligent about including imports as necessary.
* Always generate a default variable or constant for explicit empty strings.
The practical effect of this is the addition of new declarations in some cases.
However, it simplifies our logic such that it matches the protobuf data model.
* Generate the registration statements in a consistent order.
Change-Id: I627bb72589432bb65d53b50965ea88e5f7983977
Reviewed-on: https://go-review.googlesource.com/c/152778
Reviewed-by: Damien Neil <dneil@google.com>
In CL/152020, we checked in pre-generated versions of descriptor and plugin.
This CL makes it such that they are generated by protoc-gen-go.
We modify protoc-gen-go to avoid reflection support by default
since the binary size increase is still an issue to investigate.
Reflection support is temporarily enabled by setting a special
PROTOC_GEN_GO_ENABLE_REFLECT environment variable.
Reflection support is always enabled for descriptor and plugin.
Furthermore, we change descriptor to depend on the protoapi package
instead of the proto package. The reason we do not switch to protoapi
for all generated protos is because we still depend on v1 proto
for the table-driven InternalMessageInfo type. Dropping it from descriptor
is semantically correct, but does incur slight performance cost.
It does not seem appropriate to drop it for all generated messages.
We could move InternalMessageInfo to protoapi, but the logic behind that
is significant.
Change-Id: I5c3fff7f6eab1a5a2399049d42fa6bf42d4c93f9
Reviewed-on: https://go-review.googlesource.com/c/152547
Reviewed-by: Damien Neil <dneil@google.com>
Custom descriptor types would want to benefit from descriptor formatting.
As such, move the logic out from prototype into an internal package for
the benefit of usages outside the prototype package.
Change-Id: I4bb2144221e656aa36909d33a77189fe084f700b
Reviewed-on: https://go-review.googlesource.com/c/152777
Reviewed-by: Herbie Ong <herbie@google.com>
Implement support in protoc-gen-go for generating messages and enums
that satisfy the v2 protobuf reflection interfaces. Specifically, the following
are added:
* top-level variable representing the file descriptor
* ProtoReflect method on enums (to implement protoV2.Enum)
* ProtoReflect method on messages (to implement protoV2.Message)
The following are not supported yet:
* resolving transitive dependencies for file imports
* Extension descriptors
* Service descriptors
The implementation approach creates a single array for all the message and enum
declarations and references sections of that array to complete dependencies.
Since protobuf declarations can form a graph (a message may depend on itself),
it is difficult to construct a graph as a single literal. One way is to use
placeholder descriptors, but that is not efficient as it requires encoding
the full name of each dependent enum and message and then later resolving it;
thus, both expanding the binary size and also increasing initialization cost.
Instead, we add a prototype.{Enum,Message}.Reference method to obtain a
descriptor reference for the purposes for satisfying dependencies.
As such, nested declarations and dependencies are populated in an init function.
Other changes to support the implementation:
* Added a protoimpl package to expose the MessageType type and also the
MessageTypeOf and EnumTypeOf helper functions.
* Added a protogen.File.GoIdent field to provide a suggested variable name
for the file descriptor.
* Added prototype.{Enum,Message}.Reference that provides a descriptor reference
for the purposes for satisfying cyclic dependencies.
* Added protoreflect.{Syntax,Cardinality,Kind}.GoString to obtain a Go source
identifier that represents the given constant.
Change-Id: I9455764882dee6ad10f251901e7d419091e8bf1d
Reviewed-on: https://go-review.googlesource.com/c/150074
Reviewed-by: Damien Neil <dneil@google.com>
In order to transition more of v1 proto package to use the v2 API,
we need the v2 API to stop depending on v1 proto. The legacy package currently
depends on v1 proto because it needs to unmarshal the descriptor protos.
Ideally, we would switch this to use the v2 implementation of wire unmarshaling.
However, that is not available yet. So, instead, we vendor a minified version
of the v1 proto package that only supports unmarshaling.
The only changes to the vendored v1 code are:
* Delete code not needed to implement proto.Unmarshal
* Drop support for message sets
* Drop support for reporting the full field name for required not set errors
The unused tool was used to delete unrelated code:
https://github.com/dominikh/go-tools/tree/master/cmd/unused
To verify that the dependency was dropped:
$ cd internal/legacy
$ go list -f "{{join .Deps \"\n\"}}" | sort | uniq | grep protobuf
github.com/golang/protobuf/protoapi
github.com/golang/protobuf/v2/internal/detrand
github.com/golang/protobuf/v2/internal/encoding/tag
github.com/golang/protobuf/v2/internal/encoding/text
github.com/golang/protobuf/v2/internal/encoding/wire
github.com/golang/protobuf/v2/internal/errors
github.com/golang/protobuf/v2/internal/flags
github.com/golang/protobuf/v2/internal/impl
github.com/golang/protobuf/v2/internal/legacy/protoV1
github.com/golang/protobuf/v2/internal/pragma
github.com/golang/protobuf/v2/internal/scalar
github.com/golang/protobuf/v2/internal/set
github.com/golang/protobuf/v2/internal/value
github.com/golang/protobuf/v2/reflect/protoreflect
github.com/golang/protobuf/v2/reflect/prototype
github.com/golang/protobuf/v2/runtime/protoimpl
github.com/golang/protobuf/v2/types/descriptor
Change-Id: I470865f1a987203574339fefc7d83843a12af966
Reviewed-on: https://go-review.googlesource.com/c/152545
Reviewed-by: Damien Neil <dneil@google.com>
In order to generate descriptor.proto, the generated code would want to depend
on the prototype package to construct the reflection data structures.
However, this is a problem since descriptor itself is one of the dependencies
for prototype. To break this dependency, we do the following:
* Avoid using concrete *descriptorpb.XOptions messages in the public API, and
instead just use protoreflect.ProtoMessage. We do lose some type safety here
as a result.
* Use protobuf reflection to interpret the Options message.
* Split out NewFileFromDescriptorProto into a separate protodesc package since
constructing protobuf reflection from the descriptor proto obviously depends
on the descriptor protos themselves.
As part of this CL, we check in a pre-generated version of descriptor and plugin
that supports protobuf reflection natively and switchover all usages of those
protos to the new definitions. These files were generated by protoc-gen-go
from CL/150074, but hand-modified to remove dependencies on the v1 proto runtime.
Change-Id: I81e03c42eeab480b03764e2fcbe1aae0e058fc57
Reviewed-on: https://go-review.googlesource.com/c/152020
Reviewed-by: Damien Neil <dneil@google.com>
The legacy prefix made sense when this functionality was part of impl.
Now that it is in its own package called legacy, the legacy prefix is silly.
Change-Id: I9e6ddb6185ce1f701e02768b505e6a05f3986f77
Reviewed-on: https://go-review.googlesource.com/c/152543
Reviewed-by: Herbie Ong <herbie@google.com>
Converting to/from v1/v2 extension descriptor types is a common operation
for v1 and v2 interoperability. Optimize these operations with a cache.
Change-Id: I5feca810f60376847c791654982acd3b6a37a5db
Reviewed-on: https://go-review.googlesource.com/c/152542
Reviewed-by: Herbie Ong <herbie@google.com>
Also replace the ident function with the handy GoImportPath.Ident method.
Change-Id: Ie4e820556fb83e659ab7e7af98f27fc24cdcd760
Reviewed-on: https://go-review.googlesource.com/c/152177
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Similar to how generated messages allow you to call Get methods on a
nil pointer, we permit similar functionality when protobuf reflection
is used on a nil pointer.
Change-Id: Ie2f596d39105c191073b42d7d689525c3b715240
Reviewed-on: https://go-review.googlesource.com/c/152021
Reviewed-by: Damien Neil <dneil@google.com>
