This CL takes a significantly different approach to generating support
for protobuf reflection. The previous approach involved generating a
large number of Go literals to represent the reflection information.
While that approach was correct, it resulted in too much binary bloat.
The approach taken here initializes the reflection information from
the raw descriptor proto, which is a relatively dense representation
of the protobuf reflection information. In order to keep initialization
cost low, several measures were taken:
* At program init, the bare minimum is parsed in order to initialize
naming information for enums, messages, extensions, and services declared
in the file. This is done because those top-level declarations are often
relevant for registration.
* Only upon first are most of the other data structures for protobuf
reflection actually initialized.
* Instead of using proto.Unmarshal, a hand-written unmarshaler is used.
This allows us to avoid a dependendency on the descriptor proto and also
because the API for the descriptor proto is fundamentally non-performant
since it requires an allocation for every primitive field.
At a high-level, the new implementation lives in internal/fileinit.
Several changes were made to other parts of the repository:
* cmd/protoc-gen-go:
* Stop compressing the raw descriptors. While compression does reduce
the size of the descriptors by approximately 2x, it is a pre-mature
optimization since the descriptors themselves are around 1% of the total
binary bloat that is due to generated protobufs.
* Seeding protobuf reflection from the raw descriptor significantly
simplifies the generator implementation since it is no longer responsible
for constructing a tree of Go literals to represent the same information.
* We remove the generation of the shadow types and instead call
protoimpl.MessageType.MessageOf. Unfortunately, this incurs an allocation
for every call to ProtoReflect since we need to allocate a tuple that wraps
a pointer to the message value, and a pointer to message type.
* internal/impl:
* We add a MessageType.GoType field and make it required that it is
set prior to first use. This is done so that we can avoid calling
MessageType.init except for when it is actually needed. The allows code
to call (*FooMessage)(nil).ProtoReflect().Type() without fearing that the
init code will run, possibly triggering a recursive deadlock (where the
init code depends on getting the Type of some dependency which may be
declared within the same file).
* internal/cmd/generate-types:
* The code to generate reflect/prototype/protofile_list_gen.go was copied
and altered to generated internal/fileinit.desc_list_gen.go.
At a high-level this CL adds significant technical complexity.
However, this is offset by several possible future changes:
* The prototype package can be drastically simplified. We can probably
reimplement internal/legacy to use internal/fileinit instead, allowing us
to drop another dependency on the prototype package. As a result, we can
probably delete most of the constructor types in that package.
* With the prototype package significantly pruned, and the fact that generated
code no longer depend on depends on that package, we can consider merging
what's left of prototype into protodesc.
Change-Id: I6090f023f2e1b6afaf62bd3ae883566242e30715
Reviewed-on: https://go-review.googlesource.com/c/158539
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
When encoding/textpb marshals out float32 values, it was previously
formatting it as float64 bitsize since both float types are stored as
float64 and internal/encoding/text only has one Float type. A
consequence of this is that the output may display a different value
than expected, e.g. 1.02 becomes 1.0199999809265137.
This CL splits Float type into Float32 and Float64 to keep track of
which bitsize to use when formatting. Values of both types are still
stored as float64 to keep the logic simple.
Decoding will always use Float64, but users can ask for a float32 value
from it.
Change-Id: Iea5b14b283fec2236a0c3946fac34d4d79b95274
Reviewed-on: https://go-review.googlesource.com/c/158497
Reviewed-by: Damien Neil <dneil@google.com>
In https://golang.org/cl/157821, I attempted to fix handling of group
field names but forgot about repeated group fields as I forgot to
properly update the marshaling tests as well for it.
Unmarshal logic for both repeated and non-repeated was already fixed in
that CL.
Change-Id: Icb4a00d8b169709ca12dfee272b2bd73e7585e6e
Reviewed-on: https://go-review.googlesource.com/c/157857
Reviewed-by: Damien Neil <dneil@google.com>
Also fix marshaling Any in expanded form to contain the correct type_url
value.
Change-Id: I4b467e74bb1d73255effd9cc4cfff9cf4558940f
Reviewed-on: https://go-review.googlesource.com/c/156342
Reviewed-by: Damien Neil <dneil@google.com>
Group field name in textproto should be the type name. Its field name
is derived from lowercasing its type name.
Change-Id: Ia12aafe934d3a59f3e07d09fe7939cfa6595a7b8
Reviewed-on: https://go-review.googlesource.com/c/157821
Reviewed-by: Damien Neil <dneil@google.com>
A "_ProtoFile" suffix can potentially conflict with a sub-message named
"ProtoFile" nested within a message that matches the camel-cased
form of the basename of the .proto source file.
Avoid unlikely conflicts and rename this to use a "_protoFile" suffix,
which can never conflict except with an enum value that is also named
"protoFile" (which is a violation of the style guide).
Change-Id: Ie9d22f9f741a63021b8f76906b20c6c2f599885b
Reviewed-on: https://go-review.googlesource.com/c/157218
Reviewed-by: Damien Neil <dneil@google.com>
Most usages of New actually prefer to interact with the reflective view
rather than the native Go type. Thus, change New to return that instead.
This parallels reflect.New, which returns the reflective view
(i.e., reflect.Value) instead of native type (i.e., interface{}).
We make the equivalent change to KnownFields.NewMessage, List.NewMessage,
and Map.NewMessage for consistency.
Since this is a subtle change where the type system will not always
catch the changed type, this change was made by both changing the type
and renaming the function to NewXXX and manually looking at every usage
of the the function to ensure that the usage correctly operates
on either the native Go type or the reflective view of the type.
After the entire codebase was cleaned up, a rename was performed to convert
NewXXX back to New.
Change-Id: I153fef627b4bf0a427e4039ce0aaec52e20c7950
Reviewed-on: https://go-review.googlesource.com/c/157077
Reviewed-by: Damien Neil <dneil@google.com>
Drop the protoreflect.ProtoEnum type (containing a single method
returning a protoreflect.Enum) and make generated enum types
directly implement protoreflect.Enum instead.
Messages have a two-level type split (ProtoMessage and Message) to
minimize conflicts between reflection methods and field names. Enums
need no such split, since enums do not have fields and therefore have
no source of conflicts.
Change-Id: I2b6222e9404253e6bfef2217859e1b760ffcd29b
Reviewed-on: https://go-review.googlesource.com/c/156902
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Reviewed-by: Damien Neil <dneil@google.com>
If there are any kind of errors in trying to expand the Any message,
always fallback to marshaling it as regular message. This makes it
consistent with V1 and C++ libs.
Change-Id: I007414c1767e682623c45d4dd8c82b9998f61781
Reviewed-on: https://go-review.googlesource.com/c/156257
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Marshal well-known type Any in expanded form by default, else fallback
to marshaling it as a regular message.
Change-Id: Ic7e9e37b47042a163941f8849dc366ffe48103ca
Reviewed-on: https://go-review.googlesource.com/c/156097
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Do not treat reserved field names as unknown. Skip over these instead
of returning error for unknown field.
Change-Id: Iba9371fd901a0690e5b7329bccf261570352b55d
Reviewed-on: https://go-review.googlesource.com/c/154178
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
This initial textproto unmarshaling implementation covers messages
without the use of extensions, Any expansion, nor weak.
Updated encoding tests. Split some testcases to keep each simpler.
Added TestRoundTrip for example messages like the well-known types.
Change-Id: Icffab02834aa004fa8409a9da70624f687f604fb
Reviewed-on: https://go-review.googlesource.com/c/153020
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The impl package currently supports wrapping legacy v1 enums and messages
so that they implement the v2 reflective APIs. This functionality is necessary
for v1 and v2 to interoperate. However, the existence of this functionality
presents several problems:
* A significant portion of the complexity in impl is for legacy wrapping.
* This complexity is linked into a Go binary even if all the other messages
in the binary natively support v2 reflection.
* It presents a cyclic dependency when trying to generate descriptor proto.
Suppose you are generating descriptor.proto. The generated code would want to
depend on the impl package because impl is the runtime implementation for
protobuf messages. However, impl currently depends depends on descriptor in
order to wrap legacy enum and messages since it needs the ability to dynamically
create new protobuf descriptor types. In the case of descriptor.proto, it would
presumably be generated with native reflection support, so the legacy wrapping
logic is unneccessary.
To break the dependency of impl on descriptor, we move the legacy support logic
to a different package and instead add hooks in impl so that legacy support could
be dynamically registered at runtime. This is dependency injection.
Change-Id: I01a582908ed5629993f6699e9bf2f4bee93857a4
Reviewed-on: https://go-review.googlesource.com/c/151877
Reviewed-by: Herbie Ong <herbie@google.com>
This initial implementation covers marshaling Message without use
of extensions, Any expansion, weak yet.
Change-Id: Ic787939c1d2a4e70e40c3a1654c6e7073052b7d3
Reviewed-on: https://go-review.googlesource.com/c/151677
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
