Move all fast-path inputs and outputs into the Input/Output structs.
Collapse all booleans into bitfields.
Change-Id: I79ebfbac9cd1d8ef5ec17c4f955311db007391ca
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219505
Reviewed-by: Joe Tsai <joetsai@google.com>
Refactor the fast-path size, marshal, unmarshal, and isinit functions to
take the *coderFieldInfo for the field as input.
This replaces a number of closures capturing field-specific information
with functions taking that information as an explicit parameter.
Change-Id: I8cb39701265edb7b673f6f04a0152d5f4dbb4d5d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/218937
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Add a sentinel proto.Error error which matches all errors returned by
packages in this module.
Document that protoregistry.NotFound is an exact sentinel value for
performance reasons.
Add a Wrap function to the internal/errors package and use it to wrap
errors from outside sources (resolvers). Wrapped errors match
proto.Error.
Fixesgolang/protobuf#1021.
Change-Id: I45567df3fd6c8dc9a5caafdb55654827f6fb1941
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/215338
Reviewed-by: Joe Tsai <joetsai@google.com>
Add a test to confirm that extensions are lazily decoded when we expect.
Drop the UnmarshalDefaultResolver flag. I added it thinking for some
reason that internal/impl couldn't depend on protoregistry; since it can
(and does), it's simpler to just test if the resolver is the expected
value.
Use a default set of options when lazily unmarshaling extensions.
Change-Id: Ied7666ffdc3bf90630260a80c9568d9a945048bc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/218038
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Bit of a weird case in why this wasn't caught by tests: When validating
extension groups, we were validating an empty buffer rather than the
message content. For groups, this validation always fails due to a lack
of a group end tag. We'd then skip lazy decoding of the extension field
and proceed with eager decoding, which would behave correctly.
Change extension validation to report an error immediately on an invalid
result from the validator, which is both safe (assuming we trust the
validator) and would have caught this problem (by failing to decode the
extension field, rather than silently failing to eager decoding).
Change-Id: Id6c2d21fb687062bc74d9eb93760a1c24a6fe883
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/217767
Reviewed-by: Joe Tsai <joetsai@google.com>
Return the size of the field read from the validator, permitting us to
avoid an extra parse when skipping over groups.
Return an UnmarshalOutput from the validator, since it already combines
two of the validator outputs: bytes read and initialization status.
Remove initialization status from the ValidationStatus enum, since it's
covered by the UnmarshalOutput.
Change-Id: I3e684c45d15aa1992d8dc3bde0f608880d34a94b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/217763
Reviewed-by: Joe Tsai <joetsai@google.com>
Unknown extensions are initialized.
Valid extensions with no isInit func are initialized.
Change-Id: I2975c7ef85d2b777eca467d3b1861d20de8e24fc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/216960
Reviewed-by: Joe Tsai <joetsai@google.com>
Historically, extensions have been placed in the unknown fields section
of the unmarshaled message and decoded lazily on demand. The current
unmarshal implementation decodes extensions eagerly at unmarshal time,
permitting errors to be immediately reported and correctly detecting
unset required fields in extension values.
Add support for validated lazy extension decoding, where the extension
value is fully validated at initial unmarshal time but the fully
unmarshaled message is only created lazily.
Make this behavior conditional on the protolegacy flag for now.
Change-Id: I9d742496a4bd4dafea83fca8619cd6e8d7e65bc3
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/216764
Reviewed-by: Joe Tsai <joetsai@google.com>
Change the representation of option flags in protoiface from bools to a
bitfield. This brings the representation of options in protoiface in
sync with that in internal/impl.
This change has several benefits:
1. We will probably find that we need to add more option flags over time.
Converting to the more efficient representation of these flags as high
in the call stack as possible minimizes the performance implication of
the struct growing.
2. On a similar note, this avoids the need to convert from the compact
representation to the larger one when passing from internal/impl to
proto, since the {Marshal,Unmarshal}State methods take the compact form.
3. This removes unused options from protoiface. Instead of documenting
that AllowPartial is always set, we can just not include an AllowPartial
flag in the protoiface options.
4. Conversely, this provides a way to add option flags to protoiface
that we don't want to expose in the proto package.
name old time/op new time/op delta
EmptyMessage/Wire/Marshal-12 11.1ns ± 7% 10.1ns ± 1% -9.35% (p=0.000 n=8+8)
EmptyMessage/Wire/Unmarshal-12 7.07ns ± 0% 6.74ns ± 1% -4.58% (p=0.000 n=8+8)
EmptyMessage/Wire/Validate-12 4.30ns ± 1% 3.80ns ± 8% -11.45% (p=0.000 n=7+8)
RepeatedInt32/Wire/Marshal-12 1.17µs ± 1% 1.21µs ± 7% +4.09% (p=0.000 n=8+8)
RepeatedInt32/Wire/Unmarshal-12 938ns ± 0% 942ns ± 3% ~ (p=0.178 n=7+8)
RepeatedInt32/Wire/Validate-12 521ns ± 4% 543ns ± 7% ~ (p=0.157 n=7+8)
Required/Wire/Marshal-12 97.2ns ± 1% 95.3ns ± 1% -1.98% (p=0.001 n=7+7)
Required/Wire/Unmarshal-12 41.0ns ± 9% 38.6ns ± 3% -5.73% (p=0.048 n=8+8)
Required/Wire/Validate-12 25.4ns ±11% 21.4ns ± 3% -15.62% (p=0.000 n=8+7)
Change-Id: I3ac1b00ab36cfdf61316ec087a5dd20d9248e4f6
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/216760
Reviewed-by: Joe Tsai <joetsai@google.com>
UnmarshalOptions.DiscardUnknown was simply not working. Oops. Fix it.
Add a test.
Change-Id: I76888eae1221d99a007f0e9cdb711d292e6856b1
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/216762
Reviewed-by: Joe Tsai <joetsai@google.com>
Required field validation populates a bitmask of observed required
fields. Store a uint64 containing the bit to set in the validationInfo
rather than the index of the bit. Provides a noticeable speed increase
in validation.
name old time/op new time/op delta
EmptyMessage/Wire/Unmarshal 40.2ns ± 1% 40.2ns ± 2% ~ (p=0.860 n=35+37)
EmptyMessage/Wire/Unmarshal-12 7.13ns ± 5% 7.12ns ± 1% ~ (p=0.112 n=37+37)
RepeatedInt32/Wire/Unmarshal 6.57µs ± 1% 6.46µs ± 1% -1.56% (p=0.000 n=39+35)
RepeatedInt32/Wire/Unmarshal-12 1.05µs ± 2% 1.05µs ± 2% ~ (p=0.659 n=37+33)
Required/Wire/Unmarshal 258ns ± 1% 251ns ± 1% -2.87% (p=0.000 n=32+38)
Required/Wire/Unmarshal-12 44.3ns ± 2% 42.4ns ± 1% -4.36% (p=0.000 n=36+37)
Change-Id: Ib1cb74d3e348355a6a2f66aecf8fdc4b58cd84d4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/216420
Reviewed-by: Joe Tsai <joetsai@google.com>
Add a fast check for required fields to the fast path unmarshal.
This is best-effort and will fail to detect some initialized
messages: Messages with more than 64 required fields, messages
split across multiple tags, possibly other cases.
In the cases where it works (which is most of them in practice),
this permits us to skip the IsInitialized check.
Change-Id: I6b70953a333033a5e64fb7ca37a59786cb0f75a0
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/215878
Reviewed-by: Joe Tsai <joetsai@google.com>
Add functions to the proto package which plumb through the fast-path state.
As a sample use case: A followup CL adds an Initialized field to
protoiface.UnmarshalOutput, permitting the unmarshaller to report back
when it can confirm that a message is fully initialized. We want to
preserve that information when an unmarshal operation threads through
the proto package (such as when unmarshaling extensions).
To allow these functions to be added as methods of MarshalOptions and
UnmarshalOptions rather than top-level functions, separate the options
from the input structs.
Also update options passed to fast-path methods to set AllowPartial and
Merge to reflect the expected behavior of those methods. (Always allow
partial, never merge.)
Change-Id: I482477b0c9340793be533e75a86d0bb88708716a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/215877
Reviewed-by: Joe Tsai <joetsai@google.com>
The fast-path unmarshal funcs return the number of bytes consumed.
Change these functions to return an unmarshalOutput struct instead, to
make it easier to add to the results. This is groundwork for allowing
the fast-path unmarshaler to indicate when the unmarshaled message is
known to be initialized.
Change-Id: Ia8c44731a88f5be969a55cd98ea26282f412c7ae
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/215720
Reviewed-by: Joe Tsai <joetsai@google.com>
We may want to make changes to the inputs and outputs of the fast-path
functions in the future. For example, we likely want to add the ability
for the fast-path unmarshal to report back whether the unmarshaled
message is known to be initialized.
Change the signatures of these functions to take in and return struct
types which can be extended with whatever fields we want in the future.
Change-Id: Idead360785df730283a4630ea405265b72482e62
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/215719
Reviewed-by: Joe Tsai <joetsai@google.com>
Promote the fast-path magic ProtoMethods method to first-class citizen
of the protoreflect.Message interface.
To avoid polluting the protoreflect package with the various types
required by this method, make the necessary protoiface types unnamed and
duplicate them in protoreflect.
Updates golang/protobuf#1022.
Change-Id: I9595bae40b3bc7536d727fb6f99b3bce8f73da87
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/215718
Reviewed-by: Joe Tsai <joetsai@google.com>
Remove a trivial difference in the definition of the resolver
unmarshaler option to avoid a relatively expensive interface->interface
type conversion.
Change-Id: Iecf9a686af5d17fe3e2d9b80f886c644bf8a25df
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/215657
Reviewed-by: Joe Tsai <joetsai@google.com>
The fast-path unmarshal was getting confused when parsing a group
containing a field with a number the same as the group's own field
number. Separate the handling of EndGroup tags.
Change-Id: I637702b42c94a26102e693ee29a55e80b37d7f28
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/214737
Reviewed-by: Joe Tsai <joetsai@google.com>
Stash fast-path information for extensions on the ExtensionInfo. In
the usual case where an ExtensionType's underlying implementation is
an *ExtensionInfo, fetching the fast-path information becomes a type
assertion rather than a mutex-guarded map access.
Maintain a global sync.Map for the case where an ExtensionType isn't an
*ExtensionInfo.
Substantially improves performance for fast-path operations on
extensions:
Encode/MessageSet_type_id_before_message_content-12 267ns ± 1% 185ns ± 1% -30.44% (p=0.001 n=7+7)
Encode/basic_scalar_types_(*test.TestAllExtensions)-12 1.94µs ± 1% 0.40µs ± 1% -79.32% (p=0.000 n=8+7)
Change-Id: If048b521deb3665a090ea3d0a178c61691d4201e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/210540
Reviewed-by: Joe Tsai <joetsai@google.com>
The deprecated messageset format permits extension fields with numbers
greater than the usual maximum (1<<29-1). To support this, the
internal/encoding/wire package has disabled field number validation when
legacy support is enabled.
We shouldn't skip validating all field numbers for validity just because
we support larger ones in messagesets.
This change drops range validation from the wire package (other than
checking that numbers fit in an int32) and adds it to the wire
unmarshalers instead. This gives us validation where we care
about it (when unmarshaling a wire-format message) and allows for
best-effort handling of out-of-range numbers everywhere else.
Fixesgolang/protobuf#996
Change-Id: I4e11b8a8aa177dd60e89723570af074a317c2451
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/210290
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
In the v1 implementation, unknown MessageSet items are stored in a
message's unknown fields section in non-MessageSet format. For example,
consider a MessageSet containing an item with type_id T and value V.
If the type_id is not resolvable, the item will be placed in the unknown
fields as a bytes-valued field with number T and contents V. This
conversion is then reversed when marshaling a MessageSet containing
unknown fields.
Preserve this behavior in v2.
One consequence of this change is that actual unknown fields in a
MessageSet (any field other than 1) are now discarded. This matches
the previous behavior.
Change-Id: I3d913613f84e0ae82481078dbc91cb25628651cc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/205697
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
We change Unmarshal to reset a message by default.
* We add a Merge option to UnmarshalOptions for explicit merging.
* We speed up Reset by checking for the Reset method.
* Remove TODOs in prototext and protojson about reset behavior.
Fixesgolang/protobuf#890
Change-Id: Ibd8963c741053f564acf061fbdb846699942109c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/195457
Reviewed-by: Damien Neil <dneil@google.com>
Change the storage type of ExtensionField from interface{} to
protoreflect.Value.
Replace the codec functions operating on interface{}s with ones
operating on Values.
Values are potentially more efficient, since they can represent
non-pointer types without allocation. This also reduces the number of
types used to represent field values.
Additionally, this change lays groundwork for changing the
user-visible representation of repeated extension fields from
*[]T to []T. The storage type for extension fields must support mutation
(thus *[]T currently); changing the storage type to a Value permits this
without the need to introduce yet another view on field values.
Change-Id: Ida336be14112bb940f655236eb58df21bf312525
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/192218
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Remove the remaining uses of the prototype package.
The most significant change is to impl.MessageInfo, which now directly
implements the MessageType interface. This involves two notable changes
to exported fields of MessageInfo:
- PBType is now Desc.
- GoType is now GoReflectType. (Name changed to avoid a conflict with
the GoType method of the MessageType interface.)
Fixesgolang/protobuf#911
Change-Id: Ie2aa4766d6887ceaa9cf06b1f109aa6e6e2a208f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189340
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Change protoiface.ExtensionDescV1 to implement protoreflect.ExtensionType.
ExtensionDescV1's Name field conflicts with the Descriptor Name method,
so change the protoreflect.{Message,Enum,Extension}Type types to no
longer implement the corresponding Descriptor interface. This also leads
to a clearer distinction between the two types.
Introduce a protoreflect.ExtensionTypeDescriptor type which bridges
between ExtensionType and ExtensionDescriptor.
Add extension accessor functions to the proto package:
proto.{Has,Clear,Get,Set}Extension. These functions take a
protoreflect.ExtensionType parameter, which allows writing the
same function call using either the old or new API:
proto.GetExtension(message, somepb.E_ExtensionFoo)
Fixesgolang/protobuf#908
Change-Id: Ibc65d12a46666297849114fd3aefbc4a597d9f08
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189199
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
