+ This change introduce a default and configurable depth limit for
proto.Unmarshal. If a message is nested deeper than the limit,
unmarshaling will fail. There are two ways to nest messages. Either by
having fields which are message types itself or by using groups.
+ The default limit is 10,000 for now. This might change in the future
to align it with other language implementation (C++ and Java use 100
as limit).
+ If pure groups (groups that don't contain message fields) are nested
deeper than the default limit the unmarshaling fails with:
proto: cannot parse invalid wire-format data
+ Note: the configured limit does not apply to pure groups.
+ This change is introduced to improve security and robustness. Because
unmarshaling is implemented using recursion it can lead to stack overflows
for certain inputs. The introduced limit protects against this.
+ A secondary motivation for this limit is the alignment with other
languages. Protocol buffers are a language interoperability mechanism
and thus either all implementations should accept the input or all
implementation should reject the input.
Change-Id: I14bdb44d06e4bd1aa90d6336c2cf6446003b2037
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/385854
Trust: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Damien Neil <dneil@google.com>
Trust: Damien Neil <dneil@google.com>
Reviewed-by: Nicolas Hillegeer <aktau@google.com>
Reviewed-by: Chressie Himpel <chressie@google.com>
Instead of lifting values to interface and then do equality test,
compare scalar values with defined types to reduce allocation
from interface.
Change-Id: Ieb777fbd1a48c83d896d0ebe6ad605433f44c16c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/253100
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Trust: Damien Neil <dneil@google.com>
Trust: Joe Tsai <thebrokentoaster@gmail.com>
The MessageName returns the full name of a message.
It is a shorthand for:
var name protoreflect.FullName
if m != nil {
name = m.ProtoReflect().Descriptor().FullName()
}
Generally, we avoid the addition of helper functions unless
their use is justified. Arguments for this helper:
• The legacy proto.MessageName is widely used.
For example, inside Google, there are thousands of usages of it.
It is commonly used in logging and error construction to
report the name of a protobuf message.
• The fact that a nil-check may be neccessary means that users
either forget the nil-check and risk a nil-panic in production code,
or users have to write cumbersome logic to check for it.
For example, compare use with the helper:
return fmt.Errorf("invalid message type: %v", proto.MessageName(m))
to use without the helper:
var messageName protoreflect.FullName
if m != nil {
messageName = m.ProtoReflect().Descriptor().FullName()
}
return fmt.Errorf("invalid message type: %v", messageName)
A point of consideration is whether this should return a string
or a protoreflect.FullName. This CL returns the latter type:
• Most uses of it are for logging via log.Printf or error construction
via fmt.Errorf where it does not matter what the exact string type is
since its formatted the same way.
• Another use of it is to check for message type equality by doing
something similar to:
if proto.MessageName(got) != proto.MessageName(want) {
...
}
in which case it still does not matter what the exact string type is.
• The other major use of proto.MessageName is to call proto APIs
that actually do expect a protoreflect.FullName
(e.g., protoregistry.GlobalTypes.FindExtensionByNumber) where
the user currently type-casts the legacy proto.MessageName
to a protoreflect.FullName anyways.
As such, there does not seem to be much justified use for MessageName
as a string. The rare cases that need a string can trivially type cast
it to a string.
Change-Id: I840758df828eef72e7e63620569363a496366afa
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/242377
Reviewed-by: Damien Neil <dneil@google.com>
When proto.Unmarshal fails, it is almost always due to being passed
input which is not a wire-format message. (A text-format message, a
message with framing left intact, and so forth.) An error like
"variable length integer overflow" can be confusing in this case, since
it implies the problem is the varint rather than the input being
entirely wrong.
Replace all Unmarshal parse errors with "cannot parse invalid
wire-format data".
Change-Id: Id97253bd39ac604e569df71778194f37b3c86c28
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/244297
Reviewed-by: Joe Tsai <joetsai@google.com>
Fix the tests to work on GOARCH=386.
Also add another test suite that tests on GOARCH=386.
Change-Id: I41290998053dc059347adb8af70012dc5cc31811
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/241537
Reviewed-by: Herbie Ong <herbie@google.com>
The order package replaces the mapsort and fieldsort packages.
It presents a common API for ordered iteration over message fields
and map fields.
It has a number of pre-defined orderings.
Change-Id: Ie6cd423da30b4757864c352cb04454f21fe07ee2
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/239837
Reviewed-by: Herbie Ong <herbie@google.com>
The genid package unifies the genname, fieldnum, and detectknown
packages into a single package.
Whenever possible use the generated constants rather than
hard-coded literals. This makes it easier to search the entire
module for special logic that deal with well-known types.
Change-Id: I13beff1f4149444a0c0b9e607ebf759657f000f4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/235301
Reviewed-by: Herbie Ong <herbie@google.com>
Some companies (e.g., Google) run a profiling service where they may
choose to special-case certain symbols in a binary to classify
commonly used libraries like protobufs.
This CL funnels similar functionality through a single function
so that they can be more easily identified. This is by no means a
firm statement that these identifiers will never change names,
but at least the code documents warnings to avoid changing the
name of certain identifiers.
This CL provides the following semi-stable symbol names:
"google.golang.org/protobuf/proto".MarshalOptions.size
"google.golang.org/protobuf/proto".MarshalOptions.marshal
"google.golang.org/protobuf/proto".UnmarshalOptions.unmarshal
"google.golang.org/protobuf/encoding/prototext".MarshalOptions.marshal
"google.golang.org/protobuf/encoding/prototext".UnmarshalOptions.unmarshal
"google.golang.org/protobuf/encoding/protojson".MarshalOptions.marshal
"google.golang.org/protobuf/encoding/protojson".UnmarshalOptions.unmarshal
Merge and Clone are not part of the above set since there is a
possibility that MergeOptions will be added in the future.
We use an unexported method so that we have the freedom to change the
method however we want since profilers do not care about that.
Change-Id: Ia79af260d00125f48139420e1e18a86482bd1829
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/234079
Reviewed-by: Damien Neil <dneil@google.com>
In the upcoming 3.12.x release of protoc, the proto3 language will be
amended to support true presence for scalars. This CL adds support
to both the generator and runtime to support these semantics.
Newly added public API:
protogen.Plugin.SupportedFeatures
protoreflect.FieldDescriptor.HasPresence
protoreflect.FieldDescriptor.HasOptionalKeyword
protoreflect.OneofDescriptor.IsSynthetic
Change-Id: I7c86bf66d0ae56642109beb5f2132184593747ad
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/230698
Reviewed-by: Damien Neil <dneil@google.com>
Changes made:
* Ensure protoreflect.ExtensionType.IsValidInterface never panics,
especially if given a nil interface value.
* Have protoreflect.ExtensionType.IsValid{Interface,Value} only
perform type-checks. It does not do value checks (i.e., whether the
value itself is valid). Value validity is left to when an actual
protoreflect.Message.Set operation is performed.
* Add special-casing on proto.SetExtension to treat an invalid
message or list as functionally equivalent to Clear. This is to
be more consistent with the legacy SetExtension implementation
which never panicked when given such values.
* Add special-casing on proto.HasExtension to treat a mismatched
extension descriptor as simply not being present in the message.
This is also to be more consistent with the legacy HasExtension
implementation which did the same thing.
Change-Id: Idf0419abf27b9f85d9b92bd2ff8088e25b7990cc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/229558
Reviewed-by: Damien Neil <dneil@google.com>
Two changes:
* Add RangeExtensions as a more suitable replacement for legacy
proto.ClearExtensions and proto.ExtensionDescs functions.
* Make HasExtension and GetExtension treat nil message interface
as an empty message to more consistently match legacy behavior.
Change-Id: I8eb1887a33d0737f2f80a2b80358cc296087ba3b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/229157
Reviewed-by: Damien Neil <dneil@google.com>
It is a sufficiently common pattern to do something like the following:
m.Proto2BytesField, ... = proto.Marshal(m2)
where the user is relying on Marshal to never return a nil byte slice,
otherwise it subtly changes the semantics of how the generated API
handles whether a proto2 "optional bytes" fields is populated or not.
Change-Id: Ie7508dbcdcc5f3295885609a91907c6eb4f04c1e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/228838
Reviewed-by: Damien Neil <dneil@google.com>
To assist users in migrating from github.com/golang/protobuf
to google.golang.org/protobuf, make it such that functiionality like
proto.Marshal doesn't panic on nil interfaces.
Similar to how the new implementation treats a typed nil message
as an empty message, we treat a nil interface as being equivalent
to an "untyped" empty message.
Change-Id: Ic037f386f855b122f732b34d370e524b7c0d76f1
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/228837
Reviewed-by: Damien Neil <dneil@google.com>
When combining multiple message fields in a MessageSet item (a case
which should never happen in practice), unmarshal could modify the input
data. Fix it to not do so. Add a general check to ensure that unmarshal
operations don't modify the input.
Change-Id: Idde46e6132a1dc96c374f9146efff81783c3bef3
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/223818
Reviewed-by: Joe Tsai <joetsai@google.com>
The pseudo-internal MarshalState and UnmarshalState method should
not have a seperate Message argument since it is passed in through
the extensible MarshalInput and UnmarshalInput values.
Change-Id: I838aadaee30e91cdf888ab024e65348c73c1cd7e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/222678
Reviewed-by: Damien Neil <dneil@google.com>
For historical reasons, MessageSets items are allowed to have field
numbers outside the usual valid range. Detect the case where the field
number cannot fit in an int32 and report an error. Also check for
a field number of 0 (always invalid).
Handle the case where a MessageSet item includes an unknown field.
We have no place to put the contents of the field, so drop it. This is,
I believe, consistent with other implementations.
Change-Id: Ic403427e1c276cbfa232ca577e7a799cce706bc7
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/221939
Reviewed-by: Herbie Ong <herbie@google.com>
TestNil checks for panic behavior for all top-level functions that
accept messages to ensure that we can detect when behavior changes
whether accidentally or deliberately.
This test discovered that the pure protobuf reflect behavior
differs with the fast-path implementation for a few cases.
For the protobuf reflection implementation, we explicitly check
for invalid messages in Merge and Reset. Previously, these would
not panic if the message was already empty (i.e., had no need to
actually set/clear any fields in the receiver message).
Change-Id: I571c0a819366bae993ce7c99b05fb4707e55cd3e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/220958
Reviewed-by: Damien Neil <dneil@google.com>
The v1 behavior of Clone returned nil as is.
Replicate this behavior in v2 for easier migration.
Change-Id: I79d93f15dd9913604027e31e5d9ff309e0c2da61
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/220437
Reviewed-by: Damien Neil <dneil@google.com>
The size cache is an int32. Store a -1 in it if the message size
overflows, and fall back to recomputing the size if the value is
negative. This means lamentable O(N^2) costs in marshaling,
but that's better than silently producing invalid output.
Also considered: Return an error. Avoids O(N^2) behavior, but gives the
user no good choices if they don't care the output being slow. Encoding
costs of messages this large are likely to be dominated by copying the
bytes rather than the size operation anyway, so slow-but-correct seems
like the most generally useful option.
We could store valid values for the range (0x7fffffff,0xfffffffe)
reserving only 0xffffffff as the overflow sentinel, but optimizing this
case seems less important than the code being obviously correct.
Fixesgolang/protobuf#970.
Change-Id: I44f59ff81fdfbc8672dd5aec959d5153a081aab9
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/220593
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
An Is prefix implies it returns a boolean.
A Check prefix better suggests that it could return an error.
Change-Id: I6ffcb32099a944c656c07654c294a0980efb2d0e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/220338
Reviewed-by: Damien Neil <dneil@google.com>
Remove a stray bit of punctuation that crept into one of the license
headers and got copied around everywhere.
Change-Id: Iebe4e882650ab6dab28f132b5e324e2ab0b99a73
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/220339
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Move all fast-path inputs and outputs into the Input/Output structs.
Collapse all booleans into bitfields.
Change-Id: I79ebfbac9cd1d8ef5ec17c4f955311db007391ca
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219505
Reviewed-by: Joe Tsai <joetsai@google.com>
Use protobuild to generate test messages, both to simplify generating
proto2/proto3/extension variants of each test where appropriate and
to make it easier to test alternative message generators in the future.
Add various additional merge tests.
Change-Id: I4ba3ce232304e1d8325543680e2b6aae61de7364
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219146
Reviewed-by: Joe Tsai <joetsai@google.com>
We're still debating what the semantics of shallow merges should be, and
what the use case for them (if any) is. There's no need to commit
ourselves to this feature at this time; drop it for now.
Remove the exported MergeOptions type, since the only option is gone. We
can trivially add it back in the future if we need it.
Change-Id: I4c8697cae69cc66eed1ea65de84b982f20e1be44
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219145
Reviewed-by: Joe Tsai <joetsai@google.com>
The generated Go code does not perfectly represent the protobuf data model.
As such, it is possible to construct Go message values that could not
otherwise be constructed through Unmarshal or Merge.
This CL adds a test to exercise some of the boundary conditions.
The tests do not necessarily lock in the current behavior, but provides
a way for us to know when they have changed (either intentionally or not).
Change-Id: I389ef97c2fcf037595ad553baf5270d7bee9673b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/196738
Reviewed-by: Damien Neil <dneil@google.com>
TODOs that we do not intend to address have been deleted.
Those that are blocking v2 release are marked with "blocks".
Change-Id: I7efa9e546d0637b562101d0edc7009893d762722
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/218878
Reviewed-by: Damien Neil <dneil@google.com>
Add a sentinel proto.Error error which matches all errors returned by
packages in this module.
Document that protoregistry.NotFound is an exact sentinel value for
performance reasons.
Add a Wrap function to the internal/errors package and use it to wrap
errors from outside sources (resolvers). Wrapped errors match
proto.Error.
Fixesgolang/protobuf#1021.
Change-Id: I45567df3fd6c8dc9a5caafdb55654827f6fb1941
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/215338
Reviewed-by: Joe Tsai <joetsai@google.com>
Add a test to confirm that extensions are lazily decoded when we expect.
Drop the UnmarshalDefaultResolver flag. I added it thinking for some
reason that internal/impl couldn't depend on protoregistry; since it can
(and does), it's simpler to just test if the resolver is the expected
value.
Use a default set of options when lazily unmarshaling extensions.
Change-Id: Ied7666ffdc3bf90630260a80c9568d9a945048bc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/218038
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Return the size of the field read from the validator, permitting us to
avoid an extra parse when skipping over groups.
Return an UnmarshalOutput from the validator, since it already combines
two of the validator outputs: bytes read and initialization status.
Remove initialization status from the ValidationStatus enum, since it's
covered by the UnmarshalOutput.
Change-Id: I3e684c45d15aa1992d8dc3bde0f608880d34a94b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/217763
Reviewed-by: Joe Tsai <joetsai@google.com>
Tweak the "nested unknown extension" test case's resolver to not depend
on the exact message being tested. Useful for if/when we want to run
these tests on other message implementations.
Change-Id: Id1722afd8e094ddb59cb3e5440f7994c20cfa681
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/217760
Reviewed-by: Joe Tsai <joetsai@google.com>
