Test the fileinit and protodesc packages by verifying that passing a
FileDescriptorProto through a round trip of these packages leaves it
unchanged.
Change-Id: I6bfb894d95f1736f9908adee1ab63e9653b3f1be
Reviewed-on: https://go-review.googlesource.com/c/159762
Reviewed-by: Herbie Ong <herbie@google.com>
Add a collection of functions which take a reflect.*Descriptor and
return the corresponding google.protobuf.*DescriptorProto.
Change-Id: Ic186c412c8d3b7bc582c31bb8d8274459ce51a20
Reviewed-on: https://go-review.googlesource.com/c/159761
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
This CL takes a significantly different approach to generating support
for protobuf reflection. The previous approach involved generating a
large number of Go literals to represent the reflection information.
While that approach was correct, it resulted in too much binary bloat.
The approach taken here initializes the reflection information from
the raw descriptor proto, which is a relatively dense representation
of the protobuf reflection information. In order to keep initialization
cost low, several measures were taken:
* At program init, the bare minimum is parsed in order to initialize
naming information for enums, messages, extensions, and services declared
in the file. This is done because those top-level declarations are often
relevant for registration.
* Only upon first are most of the other data structures for protobuf
reflection actually initialized.
* Instead of using proto.Unmarshal, a hand-written unmarshaler is used.
This allows us to avoid a dependendency on the descriptor proto and also
because the API for the descriptor proto is fundamentally non-performant
since it requires an allocation for every primitive field.
At a high-level, the new implementation lives in internal/fileinit.
Several changes were made to other parts of the repository:
* cmd/protoc-gen-go:
* Stop compressing the raw descriptors. While compression does reduce
the size of the descriptors by approximately 2x, it is a pre-mature
optimization since the descriptors themselves are around 1% of the total
binary bloat that is due to generated protobufs.
* Seeding protobuf reflection from the raw descriptor significantly
simplifies the generator implementation since it is no longer responsible
for constructing a tree of Go literals to represent the same information.
* We remove the generation of the shadow types and instead call
protoimpl.MessageType.MessageOf. Unfortunately, this incurs an allocation
for every call to ProtoReflect since we need to allocate a tuple that wraps
a pointer to the message value, and a pointer to message type.
* internal/impl:
* We add a MessageType.GoType field and make it required that it is
set prior to first use. This is done so that we can avoid calling
MessageType.init except for when it is actually needed. The allows code
to call (*FooMessage)(nil).ProtoReflect().Type() without fearing that the
init code will run, possibly triggering a recursive deadlock (where the
init code depends on getting the Type of some dependency which may be
declared within the same file).
* internal/cmd/generate-types:
* The code to generate reflect/prototype/protofile_list_gen.go was copied
and altered to generated internal/fileinit.desc_list_gen.go.
At a high-level this CL adds significant technical complexity.
However, this is offset by several possible future changes:
* The prototype package can be drastically simplified. We can probably
reimplement internal/legacy to use internal/fileinit instead, allowing us
to drop another dependency on the prototype package. As a result, we can
probably delete most of the constructor types in that package.
* With the prototype package significantly pruned, and the fact that generated
code no longer depend on depends on that package, we can consider merging
what's left of prototype into protodesc.
Change-Id: I6090f023f2e1b6afaf62bd3ae883566242e30715
Reviewed-on: https://go-review.googlesource.com/c/158539
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
When encoding/textpb marshals out float32 values, it was previously
formatting it as float64 bitsize since both float types are stored as
float64 and internal/encoding/text only has one Float type. A
consequence of this is that the output may display a different value
than expected, e.g. 1.02 becomes 1.0199999809265137.
This CL splits Float type into Float32 and Float64 to keep track of
which bitsize to use when formatting. Values of both types are still
stored as float64 to keep the logic simple.
Decoding will always use Float64, but users can ask for a float32 value
from it.
Change-Id: Iea5b14b283fec2236a0c3946fac34d4d79b95274
Reviewed-on: https://go-review.googlesource.com/c/158497
Reviewed-by: Damien Neil <dneil@google.com>
In https://golang.org/cl/157821, I attempted to fix handling of group
field names but forgot about repeated group fields as I forgot to
properly update the marshaling tests as well for it.
Unmarshal logic for both repeated and non-repeated was already fixed in
that CL.
Change-Id: Icb4a00d8b169709ca12dfee272b2bd73e7585e6e
Reviewed-on: https://go-review.googlesource.com/c/157857
Reviewed-by: Damien Neil <dneil@google.com>
Also fix marshaling Any in expanded form to contain the correct type_url
value.
Change-Id: I4b467e74bb1d73255effd9cc4cfff9cf4558940f
Reviewed-on: https://go-review.googlesource.com/c/156342
Reviewed-by: Damien Neil <dneil@google.com>
Group field name in textproto should be the type name. Its field name
is derived from lowercasing its type name.
Change-Id: Ia12aafe934d3a59f3e07d09fe7939cfa6595a7b8
Reviewed-on: https://go-review.googlesource.com/c/157821
Reviewed-by: Damien Neil <dneil@google.com>
A "_ProtoFile" suffix can potentially conflict with a sub-message named
"ProtoFile" nested within a message that matches the camel-cased
form of the basename of the .proto source file.
Avoid unlikely conflicts and rename this to use a "_protoFile" suffix,
which can never conflict except with an enum value that is also named
"protoFile" (which is a violation of the style guide).
Change-Id: Ie9d22f9f741a63021b8f76906b20c6c2f599885b
Reviewed-on: https://go-review.googlesource.com/c/157218
Reviewed-by: Damien Neil <dneil@google.com>
Rather than maintaining our own list of builtin Go identifiers, use the
types.Universe API in the standard library instead.
It is okay if this list of names changes over time since we use this only
to derive a locally-used package name in the generated file.
Change-Id: Ib1688abc47d5c97b557f6e1f4d60c78e0951e65b
Reviewed-on: https://go-review.googlesource.com/c/157818
Reviewed-by: Damien Neil <dneil@google.com>
This change preserves consistency with CL/157077,
where New returns a value closer to the reflective type.
Change-Id: I85bfdae24e1ce1a10c3c7b939420fa1043bff743
Reviewed-on: https://go-review.googlesource.com/c/157078
Reviewed-by: Damien Neil <dneil@google.com>
Rephrase the documentation in terms of usage of this package: Lead with
the description of what this package contains and provide guidance on where
to get a descriptor or value interface.
Change-Id: I40a43cd59d1fbca6c60eff1c3afe50ff4a15b82f
Reviewed-on: https://go-review.googlesource.com/c/157217
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Most usages of New actually prefer to interact with the reflective view
rather than the native Go type. Thus, change New to return that instead.
This parallels reflect.New, which returns the reflective view
(i.e., reflect.Value) instead of native type (i.e., interface{}).
We make the equivalent change to KnownFields.NewMessage, List.NewMessage,
and Map.NewMessage for consistency.
Since this is a subtle change where the type system will not always
catch the changed type, this change was made by both changing the type
and renaming the function to NewXXX and manually looking at every usage
of the the function to ensure that the usage correctly operates
on either the native Go type or the reflective view of the type.
After the entire codebase was cleaned up, a rename was performed to convert
NewXXX back to New.
Change-Id: I153fef627b4bf0a427e4039ce0aaec52e20c7950
Reviewed-on: https://go-review.googlesource.com/c/157077
Reviewed-by: Damien Neil <dneil@google.com>
The generator currently uses an unintuitive and stateful algorithm
for name generation where it "fixes" name conflicts by appending "_"
to the end of the new name.
PR#657 refactored the generator code and noticed that the above
algorithm was not properly taking into account that a Get method is
generated for parent oneofs, fixing it in the same PR. While this is
more correct, this breaks users (see #780) since it means that the
generation of names can change.
This PR changes the name mangling logic to be as it was previously.
This does mean that some new proto files may be unbuildable,
but that is arguably better than breaking existing proto files
Change-Id: I2e354f4bb5d9c2b562fa2faa9149e949e2d86a0f
Reviewed-on: https://go-review.googlesource.com/c/156877
Reviewed-by: Damien Neil <dneil@google.com>
Drop the protoreflect.ProtoEnum type (containing a single method
returning a protoreflect.Enum) and make generated enum types
directly implement protoreflect.Enum instead.
Messages have a two-level type split (ProtoMessage and Message) to
minimize conflicts between reflection methods and field names. Enums
need no such split, since enums do not have fields and therefore have
no source of conflicts.
Change-Id: I2b6222e9404253e6bfef2217859e1b760ffcd29b
Reviewed-on: https://go-review.googlesource.com/c/156902
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Reviewed-by: Damien Neil <dneil@google.com>
Consider the case:
a.proto publicly imports b.proto
b.proto publicly imports c.proto
Should a.pb.go include symbols defined in c.pb.go?
Historically, it has not. As of #155677, it does. Regardless of which behavior
is preferable, #155677 produces broken code in some common situations: If
a.proto also publicly imports c.proto, we now generate two copies of the
forwarding decls for that file.
Restore the pre-#155677 behavior to avoid this breakage.
Change-Id: I283600b3be19eac2c3b3c14233bb69fa64661581
Reviewed-on: https://go-review.googlesource.com/c/156348
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
New test case demonstrating an import public bug:
a.proto publicly imports b.proto and c.proto
b.proto publicly imports c.proto
a.pb.go includes two copies of symbols in c.pb.go
The problem is that the new implementation of public imports, which
parses the generated code for the import to determine what symbols
are present, doesn't distinguish between symbols defined in a file
and symbols imported from a public import of some other file. This
can then lead to duplicate definitions in cases like the above.
Change-Id: Ia86e9f188d7bae8d9d4afbd2b5db9b64071425c3
Reviewed-on: https://go-review.googlesource.com/c/156347
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
File is out of date, regenerate it. (This didn't show up as a test failure
because we don't test the non-grpc golden files under the grpc directory.)
Change-Id: Ied485d28184c45bbca5b52138eb9cf300d813a57
Reviewed-on: https://go-review.googlesource.com/c/156345
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Change the go_package option to match the actual import path of this package.
Change-Id: Ie8630878ce75e34ca76d97c6e2922254cf964801
Reviewed-on: https://go-review.googlesource.com/c/156344
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
If there are any kind of errors in trying to expand the Any message,
always fallback to marshaling it as regular message. This makes it
consistent with V1 and C++ libs.
Change-Id: I007414c1767e682623c45d4dd8c82b9998f61781
Reviewed-on: https://go-review.googlesource.com/c/156257
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Marshal well-known type Any in expanded form by default, else fallback
to marshaling it as a regular message.
Change-Id: Ic7e9e37b47042a163941f8849dc366ffe48103ca
Reviewed-on: https://go-review.googlesource.com/c/156097
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Rather than explicitly enumerating the set of symbols to import,
just parse the imported file and extract every exported symbol.
This is possibly a bit more code, but adapts much better to future
expansion.
Change-Id: I4429664f4c068a2a55949d46aefc19865b008a77
Reviewed-on: https://go-review.googlesource.com/c/155677
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Always generate the init func containing the proto.RegisterFile call.
This call used to be in a different init func which was unconditionally
generated. Consolidating the init funcs means that we need to always
generate the single remaining func.
Change-Id: Icbf7d14d018d693dab2824f114c57e4e36384568
Reviewed-on: https://go-review.googlesource.com/c/156197
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The first commit of protoregistry only added a registry for files.
However, a separate type of registry is needed to provide a mapping between
protobuf names and actual Go types representing those names.
Additional high-level API:
var GlobalTypes = new(Types)
type Type interface{ ... }
type Types struct{ ... }
func NewTypes(...Type) *Types
func (*Types) Register(...Type) error
func (*Types) FindEnumByName(pref.FullName) (pref.EnumType, error)
func (*Types) FindMessageByName(pref.FullName) (pref.MessageType, error)
func (*Types) FindMessageByURL(string) (pref.MessageType, error)
func (*Types) FindExtensionByName(pref.FullName) (pref.ExtensionType, error)
func (*Types) FindExtensionByNumber(pref.FullName, pref.FieldNumber) (pref.ExtensionType, error)
func (*Types) RangeEnums(func(pref.EnumType) bool)
func (*Types) RangeMessages(func(pref.MessageType) bool)
func (*Types) RangeExtensions(func(pref.ExtensionType) bool)
func (*Types) RangeExtensionsByMessage(pref.FullName, func(pref.ExtensionType) bool)
Change-Id: I0d07705801684a1eb5853bcd05fcce12598a0047
Reviewed-on: https://go-review.googlesource.com/c/131345
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The purego tag (see https://golang.org/issue/23172) is a community agreed
upon signal that a given build environment does not support unsafe.
The appengine environment is supposed to respect this tag, but does not
properly do so. Add this tag back in until they fix their environment.
Change-Id: I9a70062be4339c2e1a93cac31d387698c561b8aa
Reviewed-on: https://go-review.googlesource.com/c/154743
Reviewed-by: Damien Neil <dneil@google.com>
Introduce a protoreflect.OptionsMessage interface type to represent
options messages. The protoreflect package can't deal with concrete
options types, to avoid dependency cycles; OptionsMessage serves both
as documentation and a single point to define the constraints we apply
to options.
Change the constraints on options from ProtoMessage to interface{} to
permit use of option message types which only implement the v1
proto.Message interface.
This still leaves a requirement in the internal/legacy package that
options implement protoreflect.ProtoMessage, since that package now uses
the v2 Unmarshal.
Change-Id: I547518ab2c3b90c3911ef641b05b169d50a4b33a
Reviewed-on: https://go-review.googlesource.com/c/154877
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
To avoid accidentally emitting the UnmarshalMethod in the unlikely
(but possible) event that a proto4 syntax was added, swich the conditional
to only trigger on the proto2 syntax.
Change-Id: I0c201eace56e9ecc92cd791e45ca5d3b99e2ba86
Reviewed-on: https://go-review.googlesource.com/c/154317
Reviewed-by: Damien Neil <dneil@google.com>
If a field has an ExtensionType defined, use it in decoding.
Change-Id: I85f3da0f52a11578500cf28e4611fa4eb31f0623
Reviewed-on: https://go-review.googlesource.com/c/154581
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The v2 decoder isn't 100% complete, but it's good enough.
Delete the vendored copy of the v1 Unmarshal implementation.
Change-Id: Ibeabbb2e9109a1ec3df57e71f98b7aa4a583fc5b
Reviewed-on: https://go-review.googlesource.com/c/154577
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Add proto.Unmarshal.
Test cases all produce identical results to the v1 unmarshaller.
Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d
Reviewed-on: https://go-review.googlesource.com/c/153918
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
We want to codify the naming logic to be understandable in a specification.
Clean up the camelCase function to be a little more readable.
Cleanup and fix the cleanGoName function when mustExport is true.
It is okay to change this behavior since this is new logic in v2 that has not
yet been exposed to the universe.
In the mustExport code-path, we do not need to check for conflicts with keywords
since Go keywords are never uppercase, so the uppercasing of the first letter
is sufficient to break a conflict. Also, we fix the logic for uppercasing
the first character since not every lowercase character has an uppercase form.
Change-Id: If14422d773bb89ed7038d874135e3dcd12683101
Reviewed-on: https://go-review.googlesource.com/c/154180
Reviewed-by: Damien Neil <dneil@google.com>
Do not treat reserved field names as unknown. Skip over these instead
of returning error for unknown field.
Change-Id: Iba9371fd901a0690e5b7329bccf261570352b55d
Reviewed-on: https://go-review.googlesource.com/c/154178
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
This initial textproto unmarshaling implementation covers messages
without the use of extensions, Any expansion, nor weak.
Updated encoding tests. Split some testcases to keep each simpler.
Added TestRoundTrip for example messages like the well-known types.
Change-Id: Icffab02834aa004fa8409a9da70624f687f604fb
Reviewed-on: https://go-review.googlesource.com/c/153020
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
PBValueOf returns a protoreflect.Message, not a
protoreflect.ProtoMessage.
Change-Id: I88ed55f52bada6fc2b29ffd63e30de09e1febe8c
Reviewed-on: https://go-review.googlesource.com/c/153917
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Record annotations on methods in interface types (e.g., gRPC
server/client methods).
Generate annotations in golden tests, even when not checking the
content, to catch annotations without a corresponding symbol.
Change-Id: I44ae6caf66f709dc7f4686e931be04b8b6fa843d
Reviewed-on: https://go-review.googlesource.com/c/153877
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Was using a pref.Message where we want a pref.ProtoMessage.
Change-Id: I61d986a43eaf8f945a1378a7a10120474aa89d6f
Reviewed-on: https://go-review.googlesource.com/c/153697
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Add fields to the Message and Field builder structs which hold the value
of MessageOptions.map_entry, FieldOptions.packed, and FieldOptions.weak
options. Remove all access to the contents of options messages from the
prototype package.
Change IsPacked to always return false for unpackable field types,
which is consistent with the equivalent C++ API.
This change helps avoid dependency cycles between prototype and the
options messages. (Previously this was resolved by accessing options
with reflection, but just breaking the dependency from prototype to the
options message is cleaner and simpler.)
Change-Id: I756aefe2e04cfa8fea31eaaaa0b5a99d4ac9e851
Reviewed-on: https://go-review.googlesource.com/c/153517
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Fix a few points of divergence between the v1 and v2 generators around
when to apply camel-casing to service and method names.
Change-Id: I862f89c0995c540e4862013316d7af772e1ab0d8
Reviewed-on: https://go-review.googlesource.com/c/153658
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Change the service and method names in grpc.proto to not use camel-case,
to make it explicit where rewriting occurs in the generated code.
Change-Id: I9e4a851097b0ee14817a589f5f959adcc5a14fe3
Reviewed-on: https://go-review.googlesource.com/c/153657
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The DescriptorProto method was originally intended to return a raw descriptor.
However, that functionality will be responsibility of the protodesc package
in the near future.
Also document that any callers of Options must import the descriptor package.
This is reasonable since any usage of that method will almost always be of
the form:
fd.Options().(*descriptorpb.FieldOptions)
Change-Id: I1085f02305faef8dd724e0cb3ad45ac12bd5df0a
Reviewed-on: https://go-review.googlesource.com/c/153497
Reviewed-by: Damien Neil <dneil@google.com>
Remove the Mutable methods from KnownFields, List, and Map, replacing
them with methods which return a new, empty message value without adding
that value to the collection.
The new API is simpler, since it clearly applies only to message values,
and more orthogonal, since it provides a way to create a value without
mutating the collection. This latter point is particularly useful in
map deserialization, where the key may be unknown at the time the value
is deserialized.
Drop the Mutable interface, since it is no longer necessary.
Change-Id: Ic5f3d06a2aa331a5d5cd2b4e670a3dba4a74f77c
Reviewed-on: https://go-review.googlesource.com/c/153278
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
