I've come to agree with Joe about the proper location for this.
Change-Id: Ia5adbd1cd18f8cf40f7c3cc6bf8c7833dac37f20
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/214041
Reviewed-by: Joe Tsai <joetsai@google.com>
If a map value is a message with required fields, the validator should
note that it is uninitialized if a map item contains no value. In this
case, the value is an empty message which obviously does not have the
required field set.
Change-Id: I7698e60765e3c95478f293e121bba3ad7fc88e27
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/213900
Reviewed-by: Joe Tsai <joetsai@google.com>
For large positive integers, add check for number of decimal digits
before converting number to plain integer w/o exponent.
If exponent value is large, previous implementation may end up
constructing a large string with lots of zeroes that is not useful as it
will fail later on when called with strconv.Parse{Uint,Int} anyways.
Fixesgolang/protobuf#1002.
Change-Id: I65bfad304401e076743853d7501786b7231b083b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/213717
Reviewed-by: Damien Neil <dneil@google.com>
This adds a experimental function to the internal/impl package which
validates a wire-format message against a message type. The validator
reports whether the message can be successfully unmarshaled, and whether
the result is initialized (all required fields are set). In some cases,
the validator returns ambiguous results when full validation would be
expensive.
The validator is unused outside of tests. In the future, it may be used
to permit lazy unmarshaling of some data. It is being added now for
testing; in particular, the wire fuzzer now checks the validator output
for consistency with the unmarshaler.
The validator adds a small amount of unused per-MessageType state. If
this becomes a concern, we could conditionalize it with a build tag.
Change-Id: I4216ef81d6a9ed975302eed189b02d08608858b4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/212302
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Implementations of the legacy Marshaler type have no way to efficiently
compute the size of the message. Rather than generating an inefficient
fast-path Size method which marshals the message and examines the
length of the result, don't generate a fast-path at all.
Drop the requirement that a fast-path MarshalAppend requires a
corresponding Size.
Avoids O(N^2) behavior when marshaling a legacy Marshaler that
recursively calls proto.Marshal.
Change-Id: I4793cf32275d08f29c8e1a1a44a193d9a5724058
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/213443
Reviewed-by: Joe Tsai <joetsai@google.com>
Ensure that EnumOf, EnumDescriptorOf, EnumTypeOf, ProtoMessageV1Of,
ProtoMessageV2Of, MessageOf, MessageDescriptorOf, and MessageTypeOf
all return nil if passed a nil interface.
This parallels the behavior of reflect.TypeOf or reflect.ValueOf,
which return nil or an invalid value rather than panicking.
Change-Id: I461f15542f16cb0922d627bca6fcad5fc27d87e2
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/213239
Reviewed-by: Damien Neil <dneil@google.com>
The JSONName constructor returns a struct value which shallow copies
a sync.Once within it; this is a dubious pattern.
Instead, add a jsonName.Init method to initialize the value.
Change-Id: I190a7239b1b62a8041ee7e4e09c0fe37b64ff623
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/213237
Reviewed-by: Damien Neil <dneil@google.com>
The size calculation for packed repeated extension fields was
considering a zero-length list as encoding to a zero-length
wire.BytesType field, rather than being omitted entirely.
Change-Id: I7d4424a21ca8afd4fa81391caede49cadb4e2505
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/212297
Reviewed-by: Joe Tsai <joetsai@google.com>
All the fuzzers have the same test, which runs the fuzzer against every
entry in the corpus. Move the test logic into a separate package.
Change-Id: I3a7e2ca75d20a5ff6d51ed9e6151629e6667684b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/212258
Reviewed-by: Joe Tsai <joetsai@google.com>
Add a new Fuzz message containing all the message types we want to make
available to fuzzers. Previously, testing (for example) required fields
would require modifying the fuzzer; now, it's just a matter of adding a
message with required fields as a field of the top-level Fuzz message.
Add internal/cmd/generate-corpus to codify where the fuzz seed corpus
comes from. This will simplify adding text and json fuzzers.
Rename internal/fuzz/wire to internal/fuzz/wirefuzz to minimize package
name ambiguity. Also, the addition of the Fuzz container message
invalidates the existing corpus, so using a new name seems like a good
idea.
Change-Id: I94f8f64ba93596c8e8cecb4d42bcc5b98c17d838
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/212218
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Fix a bug in handling the case where the encoding for a map item places
the value field (2) before the key field (1).
Change-Id: I2e6ad9af729a199e960e566ed7ef96bba3726990
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/211804
Reviewed-by: Joe Tsai <joetsai@google.com>
In very old messages predating the existence of the size cache or the
proto3 unknown fields, it is possible that the generated struct lacks both
XXX_ fields and ones tagged with "protobuf". This can happen with a message
that only contains oneofs. As such, check for the "protobuf_oneof" tag as well.
Change-Id: I1981cd7dde68aece1a013356b6bc91cc5529f951
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/210747
Reviewed-by: Damien Neil <dneil@google.com>
These are not necessary now that weak fields are unexported.
Change-Id: Ida18b984abedfdf52fd3d5f3cb2f4ca580659a5c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/210745
Reviewed-by: Damien Neil <dneil@google.com>
When loading a *MessageInfo for a legacy message type, check to see if
the Go type contains at least one field which looks like a message
field. Specifically, look for at least one field with a `protobuf:` tag,
or an XXX_unrecognized field.
If a message has no recognizable fields, assume that it's something we
don't know how to interpret and treat it as an aberrant message.
Change-Id: If5c09087f1a0187271c98539d761395a2ee70a9e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/210617
Reviewed-by: Joe Tsai <joetsai@google.com>
The conditional was accidentally inverted.
This function provides dubious support for encoding/json.
Change-Id: Ib4131a229afa14d9aef1ad31fec51f4dac417a3b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/210638
Reviewed-by: Damien Neil <dneil@google.com>
Both the generator and the runtime need to agree upon the names of
specialized Go struct fields. Centralize that information in an
internal genname package.
In the mean time, also change the XXX_weak field name to match
the name used internally at Google.
Change-Id: I026bf354418c363482e5902f21aa5e0cacae24b0
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/207080
Reviewed-by: Damien Neil <dneil@google.com>
Change size, marshal, and isinit operations on oneofs to look up the
currently-set oneof type in a map rather than testing for each possible
oneof field in turn.
Significantly improves oneof encoding speed for oneofs with a
substantial number of fields:
go test ./proto -bench=./oneof.*string.*test.TestAll -benchmem -count=8 -cpu=1
name old time/op new time/op delta
Encode/oneof_(string)_(*test.TestAllTypes) 911ns ± 1% 397ns ± 3% -56.45% (p=0.000 n=8+7)
Decode/oneof_(string)_(*test.TestAllTypes) 899ns ± 1% 922ns ± 1% +2.49% (p=0.001 n=7+7)
Fixesgolang/protobuf#950
Change-Id: I9393a87975ce09011d885a8af4a63a639ea8452f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/210281
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Stash fast-path information for extensions on the ExtensionInfo. In
the usual case where an ExtensionType's underlying implementation is
an *ExtensionInfo, fetching the fast-path information becomes a type
assertion rather than a mutex-guarded map access.
Maintain a global sync.Map for the case where an ExtensionType isn't an
*ExtensionInfo.
Substantially improves performance for fast-path operations on
extensions:
Encode/MessageSet_type_id_before_message_content-12 267ns ± 1% 185ns ± 1% -30.44% (p=0.001 n=7+7)
Encode/basic_scalar_types_(*test.TestAllExtensions)-12 1.94µs ± 1% 0.40µs ± 1% -79.32% (p=0.000 n=8+7)
Change-Id: If048b521deb3665a090ea3d0a178c61691d4201e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/210540
Reviewed-by: Joe Tsai <joetsai@google.com>
The deprecated messageset format permits extension fields with numbers
greater than the usual maximum (1<<29-1). To support this, the
internal/encoding/wire package has disabled field number validation when
legacy support is enabled.
We shouldn't skip validating all field numbers for validity just because
we support larger ones in messagesets.
This change drops range validation from the wire package (other than
checking that numbers fit in an int32) and adds it to the wire
unmarshalers instead. This gives us validation where we care
about it (when unmarshaling a wire-format message) and allows for
best-effort handling of out-of-range numbers everywhere else.
Fixesgolang/protobuf#996
Change-Id: I4e11b8a8aa177dd60e89723570af074a317c2451
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/210290
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The fast-path decoder decodes zero-length repeated bytes values as
non-nil, zero-length []bytes. Do the same in the reflection decoder.
This isn't really a correctness issue, since there's no ambiguity about what a
nil entry in a [][]byte means. Still a good idea for consistency, and
retains v1 behavior.
Change-Id: Icd2cb726d14ff1f2b9f142e65756777a359971f3
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/210257
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The protoreflect documentation states that Get on an empty repeated or
map field returns an invalid (empty, read-only) List or Map. We weren't
doing this; fix it.
The documentation also states that an invalid List or Map may not be
assigned via Set. We were permitting Set with an invalid map; fix this.
Add tests for this behavior in prototest.
Change-Id: I4678af532e192210af0bde7c96a1439a4cd26efa
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/209019
Reviewed-by: Joe Tsai <joetsai@google.com>
Various protoreflect methods can return an "empty, read-only" message,
list, or map value. Provide a method to test if a value is one of these.
Fixesgolang/protobuf#966
Change-Id: I793d8426d6e2201755983c06f024412a7e09bc4c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/209018
Reviewed-by: Joe Tsai <joetsai@google.com>
Simplify the implementation of makeStructInfo by checking the
names of the internal fields in a switch statement.
Also, add "weakFields" as the unexported name for weak fields
in preparation for actually unexporting it.
Change-Id: Ide970a39e9caa5a24bc288ba3e3a0d223a6bfcb6
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/207057
Reviewed-by: Damien Neil <dneil@google.com>
The v1 wire marshaler sorts fields as follows:
- All extensions, sorted by field number.
- All non-oneof fields, sorted by field number.
- All oneof fields, in indeterminate order.
We already make some steps toward supporting this ordering: The
fast path encoder places extensions in sorted order at the start
of the message.
This commit moves oneof fields to the end of the message, makes the
reflection-based encoder use this ordering when deterministic marshaling
is enabled, and adds a test to catch unintentional changes to the
ordering.
Users SHOULD NOT depend on stability of the marshal output. It is
subject to change over time. Without deterministic marshaling enabled,
it is subject to change over calls to Marshal.
Change-Id: I6cfd89090d790a3bb50785f32b94d2781d7d08db
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/206800
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
When creating a MessageDescriptor for a legacy message with a
Descriptor method, we call that method on the type's zero value to
get the message's DescriptorProto. Some existing dynamic message
types have a Descriptor method which panics in this case.
Catch the panic and continue as if the Descriptor method wasn't present.
Change-Id: I98d4625d6917cc1ec25737e5670a443f5d02a404
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/206637
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
In the v1 implementation, unknown MessageSet items are stored in a
message's unknown fields section in non-MessageSet format. For example,
consider a MessageSet containing an item with type_id T and value V.
If the type_id is not resolvable, the item will be placed in the unknown
fields as a bytes-valued field with number T and contents V. This
conversion is then reversed when marshaling a MessageSet containing
unknown fields.
Preserve this behavior in v2.
One consequence of this change is that actual unknown fields in a
MessageSet (any field other than 1) are now discarded. This matches
the previous behavior.
Change-Id: I3d913613f84e0ae82481078dbc91cb25628651cc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/205697
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Avoid a deadlock when registering a legacy ExtensionType, caused by
initialization of the "internal/impl".ExtensionInfo calling IsMessageSet
on the MessageDescriptor of the type being extended.
We can avoid this deadlock either by initializing the ExtensionType
outside of the GlobalTypes mutex, or by moving IsMessageSet to L1
initialization of the MessageDescriptor so that it doesn't trigger lazy
init.
CL 204804 takes the former approach; this CL takes the latter.
Change-Id: Idfc1ed36a23a139839290ea32492142a17f68cf5
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/205957
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The v1 implementation calls Marshal methods when deterministic
serialization is requested, even though it has no way to verify that the
method supports determinism. Preserve this behavior.
Change-Id: I383f2ec4bd4d5b996d96d604e92dfa43cb6f1bdc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/205719
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The resolverOnly type embeds a *protoregistry.Files and overrides one of
of its methods. When that method was renamed (Register -> RegisterFile),
the resolverOnly type continued to compile but no longer overrode the
desired method.
Update the method name, and change this type to explicitly forward
methods to the underlying *protoregistry.Files to catch errors of this
nature in the future.
Change-Id: I09a529034b6e2f310977bc635288fa15ce14cc7e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/205242
Reviewed-by: Joe Tsai <joetsai@google.com>
Remove repeated extension fields from the set of nullable fields,
so that Has reports false and Range does not visit a a zero-length
repeated extension field.
This corrects a fuzzer-detected case where unmarshaling and remarshaling
a wire-format message could result in a semantic change. For a repeated
extension field in non-packed encoding, unmarshaling a packed
representation of the field would result in a message which Has the
extension. Remarshaling it would discard the the field.
Fixesgolang.org/protobuf#975
Change-Id: Ie836559c93d218db5b5201742a3b8ebbaacf54ed
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/204897
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Reviewed-by: Joe Tsai <joetsai@google.com>
Add type-safe methods to register message, enum, and extension types.
Deprecate the NewTypes function and the (*Types).Register method.
Add (*File).RegisterFile and deprecate the NewFiles function and
the (*File).Register method.
Updates golang/protobuf#963
Change-Id: Ie89e77526e0874539e9bd929ca0ba8d758e65a6e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/199898
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Test the fuzzer with a minimal seed corpus. (Currently one file
containing a valid TestAllTypes messge with most fields set.)
Change-Id: I8dcec4e26f1e8374993cc3a4bef0496f36cccd41
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/201639
Reviewed-by: Herbie Ong <herbie@google.com>
The MessageInfo cache, once set, must not be cleared, otherwise
there exists a *messageState value where the MessageInfo value is nil.
Fix the generation of the Reset method to avoid clearing this value.
Change-Id: Ic84ca8b2640a43e967c36993da1ccd3f2b7096c4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/201478
Reviewed-by: Damien Neil <dneil@google.com>
The Message.String method is intended for debugging,
so outputting <nil> for typed nil messages actually has use.
This matches the current behavior of Message.String
for messages generated by the v1 protoc-gen-go.
Change-Id: I6e31183961c83d7bce6338eb99aa7758cfda1ff4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/199397
Reviewed-by: Damien Neil <dneil@google.com>
These methods are difficult or impossible to use correctly; types
created by the dynamicpb package, for example, all have the same GoType.
Fixesgolang/protobuf#938
Change-Id: I33d4ef381579ff18569b11df501d0ba7f38a6b5c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/199060
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The protolegacy package is a minimal version of the v1 proto package.
This allows us to use this stub version as the dependency for
internal/testprotos/legacy packages and avoid a dependency
on the real v1 proto package.
The implementation of most v1 functionality will panic if called.
This way, we know if we the v2 code depends on one of those
unimplemented functions.
Updates golang/protobuf#962
Change-Id: I20b4091706fd456e4b01ae0931cce30a872639b0
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/199297
Reviewed-by: Damien Neil <dneil@google.com>
Fix a reversed error check in impl.Export{}.WeakNil.
Check to see if we have a type for the weak field on marshal/size.
Treat a typed nil valued in XXX_Weak as not indicating presence for
the field.
Change-Id: Id667ac7eb4f53236be9e181017082bd8cd21d115
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/198717
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Support, to some limited degree, types which implement protoV1.Message
but which are not struct pointers. Our ability to work with these types
is largely limited to calling Marshal or Unmarshal methods, when
present.
Change-Id: Ie1b851d9e753e2b2cb189b17ffeefebe2d8b3a8f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/198237
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Call the Marshal or Unmarshal method on legacy messages implementing
protoV1.Marshaler or protoV2.Unmarshaler.
We do this in the impl package by creating an appropriate function in
the protoiface.Methods struct for legacy messages.
In proto.MarshalAppend, return the bytes provided by the fast-path
marshal function even when the returned error is non-nil.
Fixesgolang/protobuf#955
Change-Id: I36924af9ff959a946c43f2295ef3202216e81b32
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/197357
Reviewed-by: Joe Tsai <joetsai@google.com>
Add support for an ExtensionDesc with only Field populated as returned by
protoV1.ExtensionDescs. Rather than panicking when TypeDescriptor is called,
return a placeholder that preserves the name and/or number.
Change-Id: I60352a7aec8ccd8a0c1fb08db5891043a441695f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/193725
Reviewed-by: Damien Neil <dneil@google.com>
