The conditional was accidentally inverted.
This function provides dubious support for encoding/json.
Change-Id: Ib4131a229afa14d9aef1ad31fec51f4dac417a3b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/210638
Reviewed-by: Damien Neil <dneil@google.com>
Both the generator and the runtime need to agree upon the names of
specialized Go struct fields. Centralize that information in an
internal genname package.
In the mean time, also change the XXX_weak field name to match
the name used internally at Google.
Change-Id: I026bf354418c363482e5902f21aa5e0cacae24b0
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/207080
Reviewed-by: Damien Neil <dneil@google.com>
Change size, marshal, and isinit operations on oneofs to look up the
currently-set oneof type in a map rather than testing for each possible
oneof field in turn.
Significantly improves oneof encoding speed for oneofs with a
substantial number of fields:
go test ./proto -bench=./oneof.*string.*test.TestAll -benchmem -count=8 -cpu=1
name old time/op new time/op delta
Encode/oneof_(string)_(*test.TestAllTypes) 911ns ± 1% 397ns ± 3% -56.45% (p=0.000 n=8+7)
Decode/oneof_(string)_(*test.TestAllTypes) 899ns ± 1% 922ns ± 1% +2.49% (p=0.001 n=7+7)
Fixesgolang/protobuf#950
Change-Id: I9393a87975ce09011d885a8af4a63a639ea8452f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/210281
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Stash fast-path information for extensions on the ExtensionInfo. In
the usual case where an ExtensionType's underlying implementation is
an *ExtensionInfo, fetching the fast-path information becomes a type
assertion rather than a mutex-guarded map access.
Maintain a global sync.Map for the case where an ExtensionType isn't an
*ExtensionInfo.
Substantially improves performance for fast-path operations on
extensions:
Encode/MessageSet_type_id_before_message_content-12 267ns ± 1% 185ns ± 1% -30.44% (p=0.001 n=7+7)
Encode/basic_scalar_types_(*test.TestAllExtensions)-12 1.94µs ± 1% 0.40µs ± 1% -79.32% (p=0.000 n=8+7)
Change-Id: If048b521deb3665a090ea3d0a178c61691d4201e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/210540
Reviewed-by: Joe Tsai <joetsai@google.com>
The deprecated messageset format permits extension fields with numbers
greater than the usual maximum (1<<29-1). To support this, the
internal/encoding/wire package has disabled field number validation when
legacy support is enabled.
We shouldn't skip validating all field numbers for validity just because
we support larger ones in messagesets.
This change drops range validation from the wire package (other than
checking that numbers fit in an int32) and adds it to the wire
unmarshalers instead. This gives us validation where we care
about it (when unmarshaling a wire-format message) and allows for
best-effort handling of out-of-range numbers everywhere else.
Fixesgolang/protobuf#996
Change-Id: I4e11b8a8aa177dd60e89723570af074a317c2451
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/210290
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The fast-path decoder decodes zero-length repeated bytes values as
non-nil, zero-length []bytes. Do the same in the reflection decoder.
This isn't really a correctness issue, since there's no ambiguity about what a
nil entry in a [][]byte means. Still a good idea for consistency, and
retains v1 behavior.
Change-Id: Icd2cb726d14ff1f2b9f142e65756777a359971f3
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/210257
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The protoreflect documentation states that Get on an empty repeated or
map field returns an invalid (empty, read-only) List or Map. We weren't
doing this; fix it.
The documentation also states that an invalid List or Map may not be
assigned via Set. We were permitting Set with an invalid map; fix this.
Add tests for this behavior in prototest.
Change-Id: I4678af532e192210af0bde7c96a1439a4cd26efa
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/209019
Reviewed-by: Joe Tsai <joetsai@google.com>
Various protoreflect methods can return an "empty, read-only" message,
list, or map value. Provide a method to test if a value is one of these.
Fixesgolang/protobuf#966
Change-Id: I793d8426d6e2201755983c06f024412a7e09bc4c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/209018
Reviewed-by: Joe Tsai <joetsai@google.com>
This CL adds the following helper options:
func IgnoreEnums(...protoreflect.Enum) cmp.Option
func IgnoreMessages(...proto.Message) cmp.Option
func IgnoreFields(proto.Message, ...protoreflect.Name) cmp.Option
func IgnoreOneofs(proto.Message, ...protoreflect.Name) cmp.Option
func IgnoreDescriptors(...protoreflect.Descriptor) cmp.Option
func IgnoreDefaultScalars() cmp.Option
func IgnoreEmptyMessages() cmp.Option
func IgnoreUnknown() cmp.Option
It also augments transformMessage to unmarshal and expand Any messages
with the value of the underlying message. At this moment in time
we do not provide an API to provide a custom type resolver.
Change-Id: I51e1d9ff0d56d71161e510f366a7dcc32236d760
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/204577
Reviewed-by: Damien Neil <dneil@google.com>
Accessing Number as a method, rather than a field paves the
way to have Enum potentially implement protoreflect.Enum
in the future.
Change-Id: Iebe9c0ec12e067decf2121d12fe2fb1549477b32
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/207077
Reviewed-by: Damien Neil <dneil@google.com>
Simplify the implementation of makeStructInfo by checking the
names of the internal fields in a switch statement.
Also, add "weakFields" as the unexported name for weak fields
in preparation for actually unexporting it.
Change-Id: Ide970a39e9caa5a24bc288ba3e3a0d223a6bfcb6
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/207057
Reviewed-by: Damien Neil <dneil@google.com>
The v1 wire marshaler sorts fields as follows:
- All extensions, sorted by field number.
- All non-oneof fields, sorted by field number.
- All oneof fields, in indeterminate order.
We already make some steps toward supporting this ordering: The
fast path encoder places extensions in sorted order at the start
of the message.
This commit moves oneof fields to the end of the message, makes the
reflection-based encoder use this ordering when deterministic marshaling
is enabled, and adds a test to catch unintentional changes to the
ordering.
Users SHOULD NOT depend on stability of the marshal output. It is
subject to change over time. Without deterministic marshaling enabled,
it is subject to change over calls to Marshal.
Change-Id: I6cfd89090d790a3bb50785f32b94d2781d7d08db
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/206800
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
When creating a MessageDescriptor for a legacy message with a
Descriptor method, we call that method on the type's zero value to
get the message's DescriptorProto. Some existing dynamic message
types have a Descriptor method which panics in this case.
Catch the panic and continue as if the Descriptor method wasn't present.
Change-Id: I98d4625d6917cc1ec25737e5670a443f5d02a404
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/206637
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
In the v1 implementation, unknown MessageSet items are stored in a
message's unknown fields section in non-MessageSet format. For example,
consider a MessageSet containing an item with type_id T and value V.
If the type_id is not resolvable, the item will be placed in the unknown
fields as a bytes-valued field with number T and contents V. This
conversion is then reversed when marshaling a MessageSet containing
unknown fields.
Preserve this behavior in v2.
One consequence of this change is that actual unknown fields in a
MessageSet (any field other than 1) are now discarded. This matches
the previous behavior.
Change-Id: I3d913613f84e0ae82481078dbc91cb25628651cc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/205697
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Remove previously deprecated types, functions, and methods.
Update github.com/golang/protobuf module version to one which does not
depend on any deprecated APIs.
Fixexs golang/protobuf#963
Change-Id: Ida451ef5ef3f34830808f737cc0d1c98f32ce76a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/206017
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Avoid a deadlock when registering a legacy ExtensionType, caused by
initialization of the "internal/impl".ExtensionInfo calling IsMessageSet
on the MessageDescriptor of the type being extended.
We can avoid this deadlock either by initializing the ExtensionType
outside of the GlobalTypes mutex, or by moving IsMessageSet to L1
initialization of the MessageDescriptor so that it doesn't trigger lazy
init.
CL 204804 takes the former approach; this CL takes the latter.
Change-Id: Idfc1ed36a23a139839290ea32492142a17f68cf5
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/205957
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The v1 implementation calls Marshal methods when deterministic
serialization is requested, even though it has no way to verify that the
method supports determinism. Preserve this behavior.
Change-Id: I383f2ec4bd4d5b996d96d604e92dfa43cb6f1bdc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/205719
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The resolverOnly type embeds a *protoregistry.Files and overrides one of
of its methods. When that method was renamed (Register -> RegisterFile),
the resolverOnly type continued to compile but no longer overrode the
desired method.
Update the method name, and change this type to explicitly forward
methods to the underlying *protoregistry.Files to catch errors of this
nature in the future.
Change-Id: I09a529034b6e2f310977bc635288fa15ce14cc7e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/205242
Reviewed-by: Joe Tsai <joetsai@google.com>
Remove repeated extension fields from the set of nullable fields,
so that Has reports false and Range does not visit a a zero-length
repeated extension field.
This corrects a fuzzer-detected case where unmarshaling and remarshaling
a wire-format message could result in a semantic change. For a repeated
extension field in non-packed encoding, unmarshaling a packed
representation of the field would result in a message which Has the
extension. Remarshaling it would discard the the field.
Fixesgolang.org/protobuf#975
Change-Id: Ie836559c93d218db5b5201742a3b8ebbaacf54ed
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/204897
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Reviewed-by: Joe Tsai <joetsai@google.com>
Add type-safe methods to register message, enum, and extension types.
Deprecate the NewTypes function and the (*Types).Register method.
Add (*File).RegisterFile and deprecate the NewFiles function and
the (*File).Register method.
Updates golang/protobuf#963
Change-Id: Ie89e77526e0874539e9bd929ca0ba8d758e65a6e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/199898
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Remove compiler flags that originally existed to assist the internal
divergences of protoc-gen-go. However, these ended up not being used
since the internal "patch" ended up being an entirely different fork.
Change-Id: I936d1db96bc7d737a2f2e5b90d2bd09fcc8c7b88
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/201738
Reviewed-by: Herbie Ong <herbie@google.com>
High-level API:
func Transform() cmp.Option
type Enum struct{ ... }
type Message map[string]interface{}
The Transform function transform messages into a Message type that
cmp.Equal and cmp.Diff then knows how to traverse and compare.
Change-Id: I445f3b5c69f054b6984f28c205cda69e44af3b89
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/164680
Reviewed-by: Damien Neil <dneil@google.com>
Test the fuzzer with a minimal seed corpus. (Currently one file
containing a valid TestAllTypes messge with most fields set.)
Change-Id: I8dcec4e26f1e8374993cc3a4bef0496f36cccd41
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/201639
Reviewed-by: Herbie Ong <herbie@google.com>
The MessageInfo cache, once set, must not be cleared, otherwise
there exists a *messageState value where the MessageInfo value is nil.
Fix the generation of the Reset method to avoid clearing this value.
Change-Id: Ic84ca8b2640a43e967c36993da1ccd3f2b7096c4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/201478
Reviewed-by: Damien Neil <dneil@google.com>
The Message.String method is intended for debugging,
so outputting <nil> for typed nil messages actually has use.
This matches the current behavior of Message.String
for messages generated by the v1 protoc-gen-go.
Change-Id: I6e31183961c83d7bce6338eb99aa7758cfda1ff4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/199397
Reviewed-by: Damien Neil <dneil@google.com>
Change tests which use private types registries to use the global one.
Except in cases where we want to explicitly test that the private
registry is used, it's simpler to use the global registry.
Change-Id: I998fb463b6beef91c7f5ce2ca2083251ae24d1db
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/199897
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Reviewed-by: Herbie Ong <herbie@google.com>
These methods are difficult or impossible to use correctly; types
created by the dynamicpb package, for example, all have the same GoType.
Fixesgolang/protobuf#938
Change-Id: I33d4ef381579ff18569b11df501d0ba7f38a6b5c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/199060
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The global registry is initialized via generated code.
The Go language guarantees that these are serialized (non concurrently).
The main concern is when a concurrent read operation occurs while
registration is still ongoing. In such a case, we do need a lock to
serialize the read with regard to the writes (i.e. registrations).
Change-Id: Ied35d6f8d2620f448cb281c3ec46d8de893b5671
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/199217
Reviewed-by: Damien Neil <dneil@google.com>
The protolegacy package is a minimal version of the v1 proto package.
This allows us to use this stub version as the dependency for
internal/testprotos/legacy packages and avoid a dependency
on the real v1 proto package.
The implementation of most v1 functionality will panic if called.
This way, we know if we the v2 code depends on one of those
unimplemented functions.
Updates golang/protobuf#962
Change-Id: I20b4091706fd456e4b01ae0931cce30a872639b0
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/199297
Reviewed-by: Damien Neil <dneil@google.com>
Fix a reversed error check in impl.Export{}.WeakNil.
Check to see if we have a type for the weak field on marshal/size.
Treat a typed nil valued in XXX_Weak as not indicating presence for
the field.
Change-Id: Id667ac7eb4f53236be9e181017082bd8cd21d115
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/198717
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Support, to some limited degree, types which implement protoV1.Message
but which are not struct pointers. Our ability to work with these types
is largely limited to calling Marshal or Unmarshal methods, when
present.
Change-Id: Ie1b851d9e753e2b2cb189b17ffeefebe2d8b3a8f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/198237
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
When allocating more space for the destination message in MarshalAppend,
use the same slice growth algorithm as the Go runtime's append rather
than allocating precisely the desired space.
Change-Id: If6033f6f7abdca473bc5188c4d3938ce57d3bdd2
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/197758
Reviewed-by: Joe Tsai <joetsai@google.com>
Call the Marshal or Unmarshal method on legacy messages implementing
protoV1.Marshaler or protoV2.Unmarshaler.
We do this in the impl package by creating an appropriate function in
the protoiface.Methods struct for legacy messages.
In proto.MarshalAppend, return the bytes provided by the fast-path
marshal function even when the returned error is non-nil.
Fixesgolang/protobuf#955
Change-Id: I36924af9ff959a946c43f2295ef3202216e81b32
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/197357
Reviewed-by: Joe Tsai <joetsai@google.com>
Modify Equal to treat nil messages as equal iff both are nil.
Of special note, a typed nil pointer to T is equal to a new(T)
since they are indistinguishable from a protobuf reflection.
Change-Id: Ibf90b43a982e7376e07b4159be198f06230ec194
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/196618
Reviewed-by: Damien Neil <dneil@google.com>
While odd, it is possible to merge a message into itself.
In such a situation, the material impact is that repeated
and unknown fields are duplicated. The previous logic would
inifinite loop since the list iteration logic uses the current
length, but since the current length is ever growing, this loop
will never terminate. Instead, record the list length once
and iterate exactly that many times.
Change-Id: Ief98afa1b20bd950a9c2422d4462b170dbe6fa11
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/196857
Reviewed-by: Damien Neil <dneil@google.com>
Some existing targets (whether correctly or not) rely on it Merge
being safe to call concurrently so long as the set of fields being
merged are disjoint.
Change-Id: I4db9e64efccc7a2d44a5f9b52261b611cce461b0
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/196737
Reviewed-by: Damien Neil <dneil@google.com>
Add support for an ExtensionDesc with only Field populated as returned by
protoV1.ExtensionDescs. Rather than panicking when TypeDescriptor is called,
return a placeholder that preserves the name and/or number.
Change-Id: I60352a7aec8ccd8a0c1fb08db5891043a441695f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/193725
Reviewed-by: Damien Neil <dneil@google.com>
Suppose a oneof has N fields, the previous marshaling logic
would traverse every field checking for presence. This is O(N).
Using the protoreflect.Message.WhichOneof method, we can reduce
this to O(1). This optimization is exceptionally useful for oneofs
with a large number of fields.
Change-Id: I5f4aa8b1a899930f5c95e9cf1d68bac4b0b7884d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/196121
Reviewed-by: Herbie Ong <herbie@google.com>
This change performs two optimizations:
* It uses a pre-constructed rangeInfos slice to iterate over
all the fields. This is more performant since iterating over a slice
is faster than iterating over a map. Furthermore, this slice
does not contain fields that are part of a oneof. If a oneof has
N fields, the time to check presence on the oneof is now O(1)
instead of O(N).
* It uses a dense field info slice that is optmized for the common
case where the field number is relatively low and close in value
to the index itself.
We also fix a minor bug in the construction of oneofInfo where
it wasn't treating a typed nil pointer to a wrapper struct as if
it were unset. This ensures WhichOneof and Has always agree.
name old time/op new time/op delta
Reflect/Has-4 7.81µs ± 3% 6.74µs ± 3% -13.61% (p=0.000 n=9+9)
Reflect/Get-4 12.7µs ± 1% 11.3µs ± 4% -10.85% (p=0.000 n=8+10)
Reflect/Set-4 19.5µs ± 5% 17.8µs ± 2% -8.99% (p=0.000 n=10+10)
Reflect/Clear-4 12.0µs ± 4% 10.2µs ± 3% -14.86% (p=0.000 n=9+10)
Reflect/Range-4 6.58µs ± 1% 4.17µs ± 2% -36.65% (p=0.000 n=8+9)
Change-Id: I2c48b4d3fb6103ab238924950529ded0d37f8c8a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/196358
Reviewed-by: Damien Neil <dneil@google.com>
Ironically, the "real" protobuf name of a group is not the name
of the field descriptor, but the message descriptor.
Change-Id: I26ab546a94e934766fa6af6252cacd294442a221
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/195780
Reviewed-by: Herbie Ong <herbie@google.com>
We change Unmarshal to reset a message by default.
* We add a Merge option to UnmarshalOptions for explicit merging.
* We speed up Reset by checking for the Reset method.
* Remove TODOs in prototext and protojson about reset behavior.
Fixesgolang/protobuf#890
Change-Id: Ibd8963c741053f564acf061fbdb846699942109c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/195457
Reviewed-by: Damien Neil <dneil@google.com>
If an enum dependency is unresolved, there will be no enum values.
In such a case, just fall back to using number 0.
No tests because this situation is hard to replicate.
Change-Id: I41f5e1a5c6c8afd1ab6173d2dcfa33e611fa9560
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/195982
Reviewed-by: Damien Neil <dneil@google.com>
This CL adds support for discarding unknown fields from the input.
We add support for parsing and resolving field numbers, so that
the DiscardUnknown option can ignore all unresolvable fields.
We continue to reject known fields identified by field number
since there are a number of edge cases that a difficult to resolve.
Change-Id: I5c88b7bae8656ce20e85e4b5c92d8564a5ff8bb6
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/195779
Reviewed-by: Herbie Ong <herbie@google.com>
