Historically, extensions have been placed in the unknown fields section
of the unmarshaled message and decoded lazily on demand. The current
unmarshal implementation decodes extensions eagerly at unmarshal time,
permitting errors to be immediately reported and correctly detecting
unset required fields in extension values.
Add support for validated lazy extension decoding, where the extension
value is fully validated at initial unmarshal time but the fully
unmarshaled message is only created lazily.
Make this behavior conditional on the protolegacy flag for now.
Change-Id: I9d742496a4bd4dafea83fca8619cd6e8d7e65bc3
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/216764
Reviewed-by: Joe Tsai <joetsai@google.com>
Change the representation of option flags in protoiface from bools to a
bitfield. This brings the representation of options in protoiface in
sync with that in internal/impl.
This change has several benefits:
1. We will probably find that we need to add more option flags over time.
Converting to the more efficient representation of these flags as high
in the call stack as possible minimizes the performance implication of
the struct growing.
2. On a similar note, this avoids the need to convert from the compact
representation to the larger one when passing from internal/impl to
proto, since the {Marshal,Unmarshal}State methods take the compact form.
3. This removes unused options from protoiface. Instead of documenting
that AllowPartial is always set, we can just not include an AllowPartial
flag in the protoiface options.
4. Conversely, this provides a way to add option flags to protoiface
that we don't want to expose in the proto package.
name old time/op new time/op delta
EmptyMessage/Wire/Marshal-12 11.1ns ± 7% 10.1ns ± 1% -9.35% (p=0.000 n=8+8)
EmptyMessage/Wire/Unmarshal-12 7.07ns ± 0% 6.74ns ± 1% -4.58% (p=0.000 n=8+8)
EmptyMessage/Wire/Validate-12 4.30ns ± 1% 3.80ns ± 8% -11.45% (p=0.000 n=7+8)
RepeatedInt32/Wire/Marshal-12 1.17µs ± 1% 1.21µs ± 7% +4.09% (p=0.000 n=8+8)
RepeatedInt32/Wire/Unmarshal-12 938ns ± 0% 942ns ± 3% ~ (p=0.178 n=7+8)
RepeatedInt32/Wire/Validate-12 521ns ± 4% 543ns ± 7% ~ (p=0.157 n=7+8)
Required/Wire/Marshal-12 97.2ns ± 1% 95.3ns ± 1% -1.98% (p=0.001 n=7+7)
Required/Wire/Unmarshal-12 41.0ns ± 9% 38.6ns ± 3% -5.73% (p=0.048 n=8+8)
Required/Wire/Validate-12 25.4ns ±11% 21.4ns ± 3% -15.62% (p=0.000 n=8+7)
Change-Id: I3ac1b00ab36cfdf61316ec087a5dd20d9248e4f6
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/216760
Reviewed-by: Joe Tsai <joetsai@google.com>
Add a fast check for required fields to the fast path unmarshal.
This is best-effort and will fail to detect some initialized
messages: Messages with more than 64 required fields, messages
split across multiple tags, possibly other cases.
In the cases where it works (which is most of them in practice),
this permits us to skip the IsInitialized check.
Change-Id: I6b70953a333033a5e64fb7ca37a59786cb0f75a0
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/215878
Reviewed-by: Joe Tsai <joetsai@google.com>
Add functions to the proto package which plumb through the fast-path state.
As a sample use case: A followup CL adds an Initialized field to
protoiface.UnmarshalOutput, permitting the unmarshaller to report back
when it can confirm that a message is fully initialized. We want to
preserve that information when an unmarshal operation threads through
the proto package (such as when unmarshaling extensions).
To allow these functions to be added as methods of MarshalOptions and
UnmarshalOptions rather than top-level functions, separate the options
from the input structs.
Also update options passed to fast-path methods to set AllowPartial and
Merge to reflect the expected behavior of those methods. (Always allow
partial, never merge.)
Change-Id: I482477b0c9340793be533e75a86d0bb88708716a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/215877
Reviewed-by: Joe Tsai <joetsai@google.com>
We may want to make changes to the inputs and outputs of the fast-path
functions in the future. For example, we likely want to add the ability
for the fast-path unmarshal to report back whether the unmarshaled
message is known to be initialized.
Change the signatures of these functions to take in and return struct
types which can be extended with whatever fields we want in the future.
Change-Id: Idead360785df730283a4630ea405265b72482e62
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/215719
Reviewed-by: Joe Tsai <joetsai@google.com>
Promote the fast-path magic ProtoMethods method to first-class citizen
of the protoreflect.Message interface.
To avoid polluting the protoreflect package with the various types
required by this method, make the necessary protoiface types unnamed and
duplicate them in protoreflect.
Updates golang/protobuf#1022.
Change-Id: I9595bae40b3bc7536d727fb6f99b3bce8f73da87
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/215718
Reviewed-by: Joe Tsai <joetsai@google.com>
Implementations of the legacy Marshaler type have no way to efficiently
compute the size of the message. Rather than generating an inefficient
fast-path Size method which marshals the message and examines the
length of the result, don't generate a fast-path at all.
Drop the requirement that a fast-path MarshalAppend requires a
corresponding Size.
Avoids O(N^2) behavior when marshaling a legacy Marshaler that
recursively calls proto.Marshal.
Change-Id: I4793cf32275d08f29c8e1a1a44a193d9a5724058
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/213443
Reviewed-by: Joe Tsai <joetsai@google.com>
We change Unmarshal to reset a message by default.
* We add a Merge option to UnmarshalOptions for explicit merging.
* We speed up Reset by checking for the Reset method.
* Remove TODOs in prototext and protojson about reset behavior.
Fixesgolang/protobuf#890
Change-Id: Ibd8963c741053f564acf061fbdb846699942109c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/195457
Reviewed-by: Damien Neil <dneil@google.com>
Change protoV1.ExtensionDesc to directly implement ExtensionType
rather than delegating to one.
Unify the previous types protoiface.ExtensionDescV1 and
filetype.Extension in impl.ExtensionInfo. The protoV1.ExtensionDesc
type becomes an alias to ExtensionInfo.
This gives us:
- Just one implementation of ExtensionType.
- Generated foopb.E_Ext vars are canonical ExtensionTypes.
- Generated foopb.E_Ext vars are also v1.ExtensionDescs for backwards
compatibility.
- Conversion between legacy and modern representations happens
transparently when lazily initializing an ExtensionInfo.
Overall, a simplification for users of generated code, since they can
mostly ignore the ExtensionDesc/ExtentionType distinction and use the
same value in either the old or new API.
This is change 3/5 in a series of commits changing protoV1.ExtensionDesc
to directly implement protoreflect.ExtensionType.
1. [v2] Add protoimpl.ExtensionInfo as an alias for
protoiface.ExtensionDescV1.
2. [v1] Update references to protoimpl.ExtensionInfo to use
protoiface.ExtensionInfo.
3. [v2] Create protoimpl.ExtensionInfo (an alias to a new type in
the impl package) and remove protoiface.ExtensionDescV1.
4. [v1] Remove unneeded explicit conversions between ExtensionDesc and
ExtensionType (since the former now directly implements the latter).
5. [v2] Remove stub conversion functions.
Change-Id: I96ee890541ec11b2412e1a72c9d7b96e4d7f66b4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189563
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
This is change 1/5 in a series of commits changing protoV1.ExtensionDesc
to directly implement protoreflect.ExtensionType.
1. [v2] Add protoimpl.ExtensionInfo as an alias for
protoiface.ExtensionDescV1.
2. [v1] Update references to protoimpl.ExtensionInfo to use
protoiface.ExtensionInfo.
3. [v2] Create protoimpl.ExtensionInfo (an alias to a new type in
the impl package) and remove protoiface.ExtensionDescV1.
4. [v1] Remove unneeded explicit conversions between ExtensionDesc and
ExtensionType (since the former now directly implements the latter).
5. [v2] Remove stub conversion functions.
Change-Id: If6c7fd5f55364613387a05e6f8e9aa38cbfcc5b5
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189562
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Drop the dependency from generated files on prototype.Enum: Generated
code should only depend on runtime/proto{iface,impl}.
Drop the Enums, Messages, and Extensions returns form
filetype.Builder.Build. Of these, only Enums was used by generated code.
Change the generated init function to pass the builder a slice of values
to fill in (as is done for messages and extensions).
Remove the filetype dependency on prototype in preparation for
eventually dropping the prototype package entirely.
Change-Id: I28a3420f5dfcc13fed531a64ef07b9afddfd9d55
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189200
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Change protoiface.ExtensionDescV1 to implement protoreflect.ExtensionType.
ExtensionDescV1's Name field conflicts with the Descriptor Name method,
so change the protoreflect.{Message,Enum,Extension}Type types to no
longer implement the corresponding Descriptor interface. This also leads
to a clearer distinction between the two types.
Introduce a protoreflect.ExtensionTypeDescriptor type which bridges
between ExtensionType and ExtensionDescriptor.
Add extension accessor functions to the proto package:
proto.{Has,Clear,Get,Set}Extension. These functions take a
protoreflect.ExtensionType parameter, which allows writing the
same function call using either the old or new API:
proto.GetExtension(message, somepb.E_ExtensionFoo)
Fixesgolang/protobuf#908
Change-Id: Ibc65d12a46666297849114fd3aefbc4a597d9f08
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189199
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
In order for protoc-gen-go to output the current version,
it needs to know what version it is currently running as.
However, we cannot rely on the git tags since the tags are not
made until *after* the commit has been submitted.
Instead, we manually encode the version into the code and
make sure that git tags match up with the version in the code.
The version.go file in runtime/protoimpl contains instructions
for how to make a release. Essentially:
* Every non-release commit has a version string with "devel" in it.
* Every release commit must not have "devel" in it and must be unique.
* The "release process" involves submitting two CLs.
The first CL creates a version string without "devel",
which is the commit that a git tag will actually reference.
The second CL follows immediately and re-introduces "devel"
into the version string.
The following example shows a possible sequence of VersionStrings
for git commits in time-ascending order:
v1.19.0-devel (this CL)
v1.19.0-devel
v1.19.0-devel
v1.19.0-devel
v1.20.0-rc.1 <- tagged
v1.20.0-rc.1.devel
v1.20.0-rc.1.devel
v1.20.0-rc.1.devel
v1.20.0-rc.2 <- tagged
v1.20.0-rc.2.devel
v1.20.0 <- tagged (future public release)
v1.20.0-devel
v1.20.0-devel
v1.20.0-devel
v1.20.0-devel
v1.20.1 <- tagged
v1.20.1-devel
v1.20.1-devel
v1.21.0 <- tagged
v1.21.0-devel
Note that we start today with v1.19.0-devel, which means that our initial
release will be v1.20.0. This number was intentionally chosen since
1) the number 20 has some correlation to the fact that we keep calling
the new implementation the "v2" implementation, and
2) the set of tagged versions for github.com/golang/protobuf
and google.golang.org/protobuf are unlikely to ever overlap.
This way, the version of protoc-gen-go is never ambiguous which module
it was built from.
Now that we have version information, we add support for generating .pb.go
files with the version information recorded. However, we do not emit
these for .pb.go files in our own repository since they are always guaranteed
to be at the right version (enforced by integration_test.go).
Updates golang/protobuf#524
Change-Id: I25495a45042c2aa39a39cb7e7738ae8e831a9d26
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186117
Reviewed-by: Damien Neil <dneil@google.com>
Reduce the stutter in the name since the type of Builder
is obvious from the package it is from.
Change-Id: I0046a5122717536cc6bb5ebdb32b67a1560cfc23
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189020
Reviewed-by: Damien Neil <dneil@google.com>
This implements generation of and reflection support for weak fields.
Weak fields are a proto1 feature where the "weak" option can be specified
on a singular message field. A weak reference results in generated code
that does not directly link in the dependency containing the weak message.
Weak field support is not added to any of the serialization logic.
Change-Id: I08ccfa72bc80b2ffb6af527a1677a0a81dcf33fb
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185399
Reviewed-by: Damien Neil <dneil@google.com>
The following adjustments were made:
* The pragma.NoUnkeyedLiterals is moved to be the first field.
This is done to keep the options struct smaller. Even if the last
field is zero-length, Go GC implementation details forces the struct
to be padded at the end.
* Methods are documented as always treating AllowPartial as true.
* Added a support flag for UnmarshalOptions.DiscardUnknown.
Change-Id: I1f75d226542ab2bb0123d9cea143c7060df226d8
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185998
Reviewed-by: Damien Neil <dneil@google.com>
We define MessageState, which is essentially an atomically set *MessageInfo.
By nesting this as the first field in every generated message, we can
implement the reflective methods on a *MessageState when obtained by
unsafe casting a concrete message pointer as a *MessageState.
The MessageInfo held by MessageState provides additional Go type information
to interpret the memory that comes after the contents of the MessageState.
Since we are nesting a MessageState in every message,
the memory use of every message instance grows by 8B.
On average, the body of ProtoReflect grows from 133B to 202B (+50%).
However, this is offset by XXX_Methods, which is 108B and
will be removed in a future CL. Taking into account the eventual removal
of XXX_Methods, this is a net reduction of 25%.
name old time/op new time/op delta
Name/Value-4 70.3ns ± 2% 17.5ns ± 6% -75.08% (p=0.000 n=10+10)
Name/Nil-4 70.6ns ± 3% 33.4ns ± 2% -52.66% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
Name/Value-4 16.0B ± 0% 0.0B -100.00% (p=0.000 n=10+10)
Name/Nil-4 16.0B ± 0% 0.0B -100.00% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
Name/Value-4 1.00 ± 0% 0.00 -100.00% (p=0.000 n=10+10)
Name/Nil-4 1.00 ± 0% 0.00 -100.00% (p=0.000 n=10+10)
Change-Id: I92bd58dc681c57c92612fd5ba7fc066aea34e95a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185460
Reviewed-by: Damien Neil <dneil@google.com>
We modify protoc-gen-go to stop generating exported XXX fields.
The unsafe implementation is unaffected by this change since unsafe
can access fields regardless of visibility. However, for the purego
implementation, we need to respect Go visibility rules as enforced
by the reflect package.
We work around this by generating a exporter function that given
a reference to the message and the field to export, returns a reference
to the unexported field value. This exporter function is protected by
a constant such that it is not linked into the final binary in non-purego
build environment.
Updates golang/protobuf#276
Change-Id: Idf5c1f158973fa1c61187ff41440acb21c5dac94
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185141
Reviewed-by: Damien Neil <dneil@google.com>
The internal/fileinit package is split apart into two packages:
* internal/filedesc constructs descriptors from the raw proto.
It is very similar to the previous internal/fileinit package.
* internal/filetype wraps descriptors with Go type information
Overview:
* The internal/fileinit package will be deleted in a future CL.
It is kept around since the v1 repo currently depends on it.
* The internal/prototype package is deleted. All former usages of it
are now using internal/filedesc instead. Most significantly,
the reflect/protodesc package was almost entirely re-written.
* The internal/impl package drops support for messages that do not
have a Descriptor method (pre-2016). This removes a significant amount
of technical debt.
filedesc.Builder to parse raw descriptors.
* The internal/encoding/defval package now handles enum values by name.
Change-Id: I3957bcc8588a70470fd6c7de1122216b80615ab7
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/182360
Reviewed-by: Damien Neil <dneil@google.com>
Added API:
Message.Len
Message.Range
Message.Has
Message.Clear
Message.Get
Message.Set
Message.Mutable
Message.NewMessage
Message.WhichOneof
Message.GetUnknown
Message.SetUnknown
Deprecated API (to be removed in subsequent CL):
Message.KnownFields
Message.UnknownFields
The primary difference with the new API is that the top-level
Message methods are keyed by FieldDescriptor rather than FieldNumber
with the following semantics:
* For known fields, the FieldDescriptor must exactly match the
field descriptor known by the message.
* For extension fields, the FieldDescriptor must implement ExtensionType,
where ContainingMessage.FullName matches the message name, and
the field number is within the message's extension range.
When setting an extension field, it automatically stores
the extension type information.
* Extension fields are always considered nullable,
implying that repeated extension fields are nullable.
That is, you can distinguish between a unpopulated list and an empty list.
* Message.Get always returns a valid Value even if unpopulated.
The behavior is already well-defined for scalars, but for unpopulated
composite types, it now returns an empty read-only version of it.
Change-Id: Ia120630b4db221aeaaf743d0f64160e1a61a0f61
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/175458
Reviewed-by: Damien Neil <dneil@google.com>
Add ExtensionField.{SetType,GetType} to hide the fact that the underlying
descriptor is actually an ExtensionDescV1.
Change-Id: I1d0595484ced0a88d2df0852a732fdf0fe9aa232
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/180538
Reviewed-by: Damien Neil <dneil@google.com>
Generate the needed infrastructure to ensure that we can statically
enforce minimum and maximum versions. This enables us to have a policy
when we release v2 where it fails to build for:
* new generated code with really old runtimes
* new runtimes with really old generated code
Change-Id: Ib699ad62c06dff8f9285806394a741c18db00288
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/178546
Reviewed-by: Damien Neil <dneil@google.com>
Contrary to the WARNING, this package should be stable.
However, it still should not be imported by the end-user.
Change-Id: I05b4b9ebb1e0d28ab626f8c51fbe827b5acf237e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/178545
Reviewed-by: Damien Neil <dneil@google.com>
The internal/legacy package was originally separated out from internal/impl
to avoid a cyclic dependency on descriptor proto. However, the dependency
that legacy has on descriptor has long been dropped such that we can
now merge the two packages together again.
All legacy related logic are in a file with a legacy prefix.
Change-Id: I2424fc0f50721696ad06fa7cebb9bdd0babea13c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/178542
Reviewed-by: Damien Neil <dneil@google.com>
Instead of accepting a concrete protoregistry.Types type,
accept an interface that provides the necessary functionality
to perform the serialization.
The advantages of this approach:
* There is no need for complex logic to allow a Parent or custom
Resolver on the protoregistry.Types type.
* Users can pass their own custom resolver implementations directly
to the serialization functions.
* This is a more principled approach to plumbing custom resolvers
than the previous approach of overloading behavior on the concrete
Types type.
The disadvantages of this approach:
* A pointer to a concrete type is 8B, while an interface is 16B.
However, the expansion of the {Marshal,Unmarshal}Options structs
should be a concern solved separately from how to plumb custom resolvers.
* The resolver interfaces as defined today may be insufficient to
provide functionality needed in the future if protobuf expands its
feature set. For example, let's suppose the Any message permits
directly representing a enum by name. This would require the ability
to lookup an enum by name. To support that hypothetical need,
we can document that the serializers type-assert the provided Resolver
to a EnumTypeResolver and use that if possible. There is some loss
of type safety with this approach, but provides a clear path forward.
Change-Id: I81ca80e59335d36be6b43d57ec8e17abfdfa3bad
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/177044
Reviewed-by: Damien Neil <dneil@google.com>
The name MessageType is easily confused with protoreflect.MessageType.
Rename it as MessageInfo, which follows the pattern set by v1,
where the equivalent data structure is called InternalMessageInfo.
Change-Id: I535956e1f7c6e9b07e9585e889d5e93388d0d2ce
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/178478
Reviewed-by: Damien Neil <dneil@google.com>
Temporarily remove go.mod, since we can't generate an accurate one until
the corresponding v1 change is submitted.
Change-Id: I1e1ad97f2b455e33f61ffaeb8676289795e47e72
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/177000
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Added methods:
Enum.Descriptor
Message.Descriptor
EnumType.Descriptor
MessageType.Descriptor
ExtensionType.Descriptor
Message.New
All functionality is switched over to use those methods instead of
implicitly relying on the fact that {Enum,Message}Type implicitly
implement the associated descriptor interface.
This CL does not yet remove {Enum,Message}.Type or prevent
{Enum,Message,Extension}Type from implementating a descriptor.
That is a subsequent CL.
The Message.New method is also added to replace functionality
that will be lost when the Type methods are removed.
Change-Id: I7fefde1673bbd40bfdac489aca05cec9a6c98eb1
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/174918
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Herbie Ong <herbie@google.com>
CL/172399 switches the v1 code to eagerly unmarshal extensions.
This CL does the equivalent for v2.
For the test, we simply switch from protoV1.Equal to protoV2.Equal,
since the v2 equal does not magically unmarshal raw extensions.
Change-Id: I6f64455b0a75bbc9a9a82108558641a29bd2b982
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/175838
Reviewed-by: Damien Neil <dneil@google.com>
Using a named fields gives us the flexibility to change the underlying
representation of special fields without needing to regenerate user code.
We add a named type for ExtensionFields, UnknownFields, and SizeCache.
Change-Id: I107cf82899850ea76665310ce79def60f0f7ab97
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/172402
Reviewed-by: Damien Neil <dneil@google.com>
Using an option instead of a separate method has several useful properties:
It makes it explicit whether the fast-path AppendMarshal is expected to use
cached sizes or not.
It properly plumbs the decision to use cached sizes through the call stack.
Consider the case where message A includes B includes C: If A and C support
cached sizes but B does not, we would like to use the size cache in all
messages which support it. Placing this decision in the options allows this
to work properly with no additional effort.
Placing this option in MarshalOptions permits users to request use of
existing cached sizes. This is a two-edged sword: There are places where
this ability can permit substantial efficiencies, but this is also an
exceedingly sharp-edged API. I believe that on balance the benefits
outweigh the risks, especially since the prerequisites for using
cached sizes are intuitively obvious. (You must have called Size, and
you must not have changed the message.)
This CL adds a Size method to MarshalOptions, rather than adding a SizeOptions
type. Future additions to MarshalOptions may affect the size of the encoded
output (e.g., an option to skip encoding unknown fields) and using the same
options for both Marshal and Size makes it easier to use a consistent
configuration for each.
Change-Id: I6adbb55b717dd03d39f067e1d0b7381945000976
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171119
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Remove the Reflection field from MarshalOptions and UnmarshalOptions.
Disable the fast path and use the reflection-based implementation when
the 'protoreflect' build tag is set.
Change-Id: Ic674e3af67501de27fb03ec2712fbed40eae7fef
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170896
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Move all checks for required fields into a proto.IsInitialized function.
Initial testing makes me confident that we can provide a fast-path
implementation of IsInitialized which will perform more than
acceptably. (In the degenerate-but-common case where a message
transitively contains no required fields, this check can be nearly
zero cost.)
Unifying checks into a single function provides consistent behavior
between the wire, text, and json codecs.
Performing the check after decoding eliminates the wire decoder bug
where a split message is incorrectly seen as missing required fields.
Performing the check after decoding also provides consistent and
arguably more correct behavior when the target message was partially
prepopulated.
Change-Id: I9478b7bebb263af00c0d9f66a1f26e31ff553522
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170787
Reviewed-by: Herbie Ong <herbie@google.com>
Allow message implementations to provide optimized versions of standard
operations. Generated messages now include a ProtoReflectMethods method,
returning a protoiface.Methods struct containing pointers to assorted
optional functions.
The Methods struct also includes a Flags field indicating support for
optional features such as deterministic marshaling.
Implementation of the fast paths (and tests) will come in later CLs.
Change-Id: Idd1beed0ecf43ec5e5e7b8da2ee1e08d3ce32213
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170340
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
As a goal, v2 should not depend on v1. As another step towards that end,
we move all the types that used to be in the v1 protoapi package over to v2.
For now, we place MessageV1, ExtensionRangeV1, and ExtensionDescV1
in runtime/protoiface since these are types that generated messages will
probably have to reference forever. An alternative location could be
reflect/protoreflect, but it seems unfortunate to have to dirty the
namespace of that package with these types.
We move ExtensionFieldV1, ExtensionFieldsV1, and ExtensionFieldsOf
to internal/impl, since these are related to the implementation of a
generated message.
Since moving these types from v1 to v2 implies that the v1 protoapi
package is useless, we update all usages of v1 protoapi in the v2
repository to point to the relevant v2 type or functionality.
CL/168538 is the corresponding change to alter v1.
There will be a temporary build failure as it is not possible
to submit CL/168519 and CL/168538 atomically.
Change-Id: Ide4025c1b6af5b7f0696f4b65b988b4d10a50f0b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168519
Reviewed-by: Herbie Ong <herbie@google.com>
This CL takes a significantly different approach to generating support
for protobuf reflection. The previous approach involved generating a
large number of Go literals to represent the reflection information.
While that approach was correct, it resulted in too much binary bloat.
The approach taken here initializes the reflection information from
the raw descriptor proto, which is a relatively dense representation
of the protobuf reflection information. In order to keep initialization
cost low, several measures were taken:
* At program init, the bare minimum is parsed in order to initialize
naming information for enums, messages, extensions, and services declared
in the file. This is done because those top-level declarations are often
relevant for registration.
* Only upon first are most of the other data structures for protobuf
reflection actually initialized.
* Instead of using proto.Unmarshal, a hand-written unmarshaler is used.
This allows us to avoid a dependendency on the descriptor proto and also
because the API for the descriptor proto is fundamentally non-performant
since it requires an allocation for every primitive field.
At a high-level, the new implementation lives in internal/fileinit.
Several changes were made to other parts of the repository:
* cmd/protoc-gen-go:
* Stop compressing the raw descriptors. While compression does reduce
the size of the descriptors by approximately 2x, it is a pre-mature
optimization since the descriptors themselves are around 1% of the total
binary bloat that is due to generated protobufs.
* Seeding protobuf reflection from the raw descriptor significantly
simplifies the generator implementation since it is no longer responsible
for constructing a tree of Go literals to represent the same information.
* We remove the generation of the shadow types and instead call
protoimpl.MessageType.MessageOf. Unfortunately, this incurs an allocation
for every call to ProtoReflect since we need to allocate a tuple that wraps
a pointer to the message value, and a pointer to message type.
* internal/impl:
* We add a MessageType.GoType field and make it required that it is
set prior to first use. This is done so that we can avoid calling
MessageType.init except for when it is actually needed. The allows code
to call (*FooMessage)(nil).ProtoReflect().Type() without fearing that the
init code will run, possibly triggering a recursive deadlock (where the
init code depends on getting the Type of some dependency which may be
declared within the same file).
* internal/cmd/generate-types:
* The code to generate reflect/prototype/protofile_list_gen.go was copied
and altered to generated internal/fileinit.desc_list_gen.go.
At a high-level this CL adds significant technical complexity.
However, this is offset by several possible future changes:
* The prototype package can be drastically simplified. We can probably
reimplement internal/legacy to use internal/fileinit instead, allowing us
to drop another dependency on the prototype package. As a result, we can
probably delete most of the constructor types in that package.
* With the prototype package significantly pruned, and the fact that generated
code no longer depend on depends on that package, we can consider merging
what's left of prototype into protodesc.
Change-Id: I6090f023f2e1b6afaf62bd3ae883566242e30715
Reviewed-on: https://go-review.googlesource.com/c/158539
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
In order to generate descriptor.proto, the generated code would want to depend
on the prototype package to construct the reflection data structures.
However, this is a problem since descriptor itself is one of the dependencies
for prototype. To break this dependency, we do the following:
* Avoid using concrete *descriptorpb.XOptions messages in the public API, and
instead just use protoreflect.ProtoMessage. We do lose some type safety here
as a result.
* Use protobuf reflection to interpret the Options message.
* Split out NewFileFromDescriptorProto into a separate protodesc package since
constructing protobuf reflection from the descriptor proto obviously depends
on the descriptor protos themselves.
As part of this CL, we check in a pre-generated version of descriptor and plugin
that supports protobuf reflection natively and switchover all usages of those
protos to the new definitions. These files were generated by protoc-gen-go
from CL/150074, but hand-modified to remove dependencies on the v1 proto runtime.
Change-Id: I81e03c42eeab480b03764e2fcbe1aae0e058fc57
Reviewed-on: https://go-review.googlesource.com/c/152020
Reviewed-by: Damien Neil <dneil@google.com>
The internal/impl and internal/legacy packages are only visible from packages
within the v2 repository. However, selected parts of these packages need to
be visible from the v1 repository or from generated code.
Change-Id: I1d87227dacc38f5b1ac464e9b9dd43bd34559cda
Reviewed-on: https://go-review.googlesource.com/c/151858
Reviewed-by: Herbie Ong <herbie@google.com>
