related to golang/protobuf#1613
Change-Id: Ie4255c24c1b79b13aab763a75125836191088d26
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/585096
Reviewed-by: Lasse Folger <lassefolger@google.com>
Auto-Submit: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
This used to not be necessary, but a subsequent change will change behavior
based on MarshalOptions.Deterministic, so it is important that we do not
accidentally drop MarshalOptions anywhere.
related to golang/protobuf#1609
Change-Id: I6b53352707d93939642a627eb41c930f8ac3157f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/579995
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
There currently is no risk of producing invalid wire format,
but that will change with subsequent changes regarding lazy decoding.
We have been running this change in production for about a month,
without ever triggering the check (until lazy decoding is involved).
related to golang/protobuf#1609
Change-Id: I3c5c956aee2fa81f99dea03ed2a977a1547081fc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/579595
Auto-Submit: Michael Stapelberg <stapelberg@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
When I implemented this initially, I thought the parent of an extension is the
extendee. This is incorrect. The parent is the scope in which the extension is
defined. This CL changes the code to use the correct parent. This also allows
us to reduce some complexity in the implementation because we don't need to
wait until the extendee is resolved before we can resolve the features.
Change-Id: I6d7012f7502ef95457ab96f3e8abc4ab763d5bcb
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/579275
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
Hopefully this gives users a better understanding of the MarshalAppend
entrypoint and what it can be used for, as well as the typical Size usage.
Change-Id: I26c9705c3d1dbfea5f30820d41ccabbb88fbb772
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/573361
Reviewed-by: Lasse Folger <lassefolger@google.com>
Auto-Submit: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Cassondra Foesch <cfoesch@gmail.com>
Reviewed-by: Damien Neil <dneil@google.com>
Extensions are unmarshaled lazily if protolegacy is true. The current
implementation of proto.HasExtension forces this unmarshaling to happen.
Change that.
Lazy message extensions are unmarshaled on first access, see
(*ExtensionField).Value. This leads to an (expensive) unmarshal
operation even if the user only wanted to know whether the extension is
present.
Granted, in most cases a HasExtension returning true will be followed by
a GetExtension. Due to memoization (see (*ExtensionField).lazyInit), the
cost will just shift from HasExtension to GetExtension. But, this CL
allows building cheaper functionality that only needs to know about
extension existence.
Why can this validation be removed?
- All tests pass.
- This check was added in CL 229558. The author (Joe Tsai) noted:
> Technically this shouldn't be needed, but I couldn't adequately reason
> whether a nil message value would ever be set through non-reflection
> means.
Like the author, I believe it's not needed:
- `proto.SetExtension` does not allow setting invalid messages (see
proto/extension.go).
- Likewise, (*extensionMap).Set panics when attempting to set an
invalid value.
- Unmarshaling does not produce submessages for which `IsValid` is
false.
The added test fails without the fix:
$ go test -tags=protolegacy -test.v -test.run=TestHasExtensionNoAlloc proto/extension_test.go
=== RUN TestHasExtensionNoAlloc
=== RUN TestHasExtensionNoAlloc/Nil
=== RUN TestHasExtensionNoAlloc/Eager
=== RUN TestHasExtensionNoAlloc/Lazy
extension_test.go:156: proto.HasExtension should not allocate, but allocated 3.00B per run
--- FAIL: TestHasExtensionNoAlloc (0.00s)
--- PASS: TestHasExtensionNoAlloc/Nil (0.00s)
--- PASS: TestHasExtensionNoAlloc/Eager (0.00s)
--- FAIL: TestHasExtensionNoAlloc/Lazy (0.00s)
FAIL
FAIL command-line-arguments 0.018s
The tests are disabled in race mode because the race instrumentation for
closures et al. always allocates. The protolegacy tests were previously
only run in race mode. I added a non-race variant in
integration_test.go.
Change-Id: Idbc67c1cf0aea8833a2735ca7bfc8d2466ceaf44
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/575035
Reviewed-by: Nicolas Hillegeer <aktau@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
Auto-Submit: Nicolas Hillegeer <aktau@google.com>
This example uses the same protobuf and wire format encoding
as the corresponding Marshal example added in commit
https://go.googlesource.com/protobuf/+/c69658e23457d4e09
Change-Id: Ifd64a93a14589595cbe9b218235b57fb15d423c2
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/574635
Reviewed-by: Lasse Folger <lassefolger@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
The example has been written so that it can be run, meaning it only uses
packages that are included in the protobuf module (durationpb specifically).
I included an Output: comment in the example so that pkgsite displays the
program output even without having to run it (but running it is of course
possible).
I added a brief tip to protoscope, which is often mentioned in the protobuf.dev
docs for illustrative purposes, and I think it really makes much clearer to the
reader how the protobuf wire format looks like and what information survives the
encoding process (field numbers and their values, but not field names like in
JSON).
The struct literal contains only one field so that we don’t need to marshal this
message deterministically for stable wire format, which is not the point of the
example and would be distracting.
The value was chosen such that the wire format hex representation contains at
least one byte that is clearly identifiable as hexadecimal, to avoid confusion.
Change-Id: I86103abfd7d5b3f654aca3bfbb452f8ef7e49828
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/574455
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
This change removes most usages of Syntax() from the repository and uses
edition features for instead. The appropriate edition feature defaults are
loaded for proto2/proto3 when the initialization of the descriptors
start.
All of these changes were tested on the Google corpus.
Change-Id: Ieca076a2b38ca8e50e084cd32e725b7b3dcb4171
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/572435
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
This makes it a little easier to track down test failures.
Also add a note to TestMarshalAppendAllocations to explain what it tests.
Change-Id: Ie35f3ddd7c7d5cb300294f0fe665c6711d45d186
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/569775
Auto-Submit: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
fuzztest seeds are executed as part of `go test` so there is no need for
an explicit test and having this as a seed in the fuzztest is sufficient
to get test coverage for this case.
Change-Id: Ia4a2d721e544e1a1ad0ad3ef9aa9d0af6bfe2db8
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/567755
Auto-Submit: Lasse Folger <lassefolger@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Before this change the implementation would ignore editions options set on
extension declarations.
This change also add much more coverage for valid messages.
Change-Id: I0e7029b1430a39859cdd4a93667ac20c7e0ff20e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/565396
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
This fixes more minor issue when constructing the descriptor in that
delimited encoded message fields are not considered properly as groups.
Change-Id: I714a227a0f8d256fa7430c526844e8e94acdda33
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/564816
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
There are several minor fixes:
* fix the unmarshaler for the field options (it used the wrong field numbers).
* fix the unmarshaler to always initialize the file options with the appropriate defaults.
* fix the coder selection to work properly for fields with implicit presence tracking (they used the coders for explicit tracking).
* fix the dynamic message created from a descriptor for an editions using proto to use HasPresence instead of a syntax check for proto3
Change-Id: Ic13bc22a71aa9f93b476e5edd650d4d2ab5dcb98
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/564455
Auto-Submit: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Modernize the documentation across the entire module
to make use of the newer ability to linkify declarations.
Change-Id: I440f9a3f025ec6fadfd9cba59b1dfb57028bf3f1
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/309430
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
The doc comment of proto.Equal now explicitly describes the behavior
in the case when one message is valid and other is invalid.
Change-Id: Ifeba40277c7da25193d46ad496e910f9b63eb795
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/464275
Reviewed-by: Joseph Tsai <joetsai@digital-static.net>
Reviewed-by: Chressie Himpel <chressie@google.com>
The Value.Equal method compares Values in a way that is
analogous to the reflect.DeepEqual function.
Most of the implementation is moved from the "proto" package,
with proto.Equal implemented in terms of protoreflect.Value.Equal.
Change-Id: Ie9d5f6c073dc49fa59f78385c441db8137de5ec5
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/450775
TryBot-Bypass: Heschi Kreinick <heschi@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Oleksii Prokopchuk <prokopchuk@google.com>
Reviewed-by: Christian Höppner <hoeppi@google.com>
Run-TryBot: Lasse Folger <lassefolger@google.com>
This reverts commit 4d6d2135eba33eaef1c020129188ce7e82aa4270.
Post-submit discussion on CL 448876 suggests that this API should
change; revert it pending a decision.
Change-Id: I2e81be847c62ea7de5dfef36bb10ab193132e612
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/450955
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Run-TryBot: Damien Neil <dneil@google.com>
Reviewed-by: Joseph Tsai <joetsai@digital-static.net>
Reviewed-by: Lasse Folger <lassefolger@google.com>
Run-TryBot: Lasse Folger <lassefolger@google.com>
TryBot-Bypass: Michael Stapelberg <stapelberg@google.com>
There is a demand for comparing protobuf.Value-s.
One use case is when one wants to compare just one field via reflection.
Another is when one wants to iterate over fields and check for diffs.
Existing diff packages are recommended only for tests as they panic on error.
Exposing equalField will prevent excessive branching of proto/equal.go code.
Change-Id: Iec8843dae96c9ae3c45858e8b97e7aa963473ce2
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/448876
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Run-TryBot: Michael Stapelberg <stapelberg@google.com>
TryBot-Bypass: Michael Stapelberg <stapelberg@google.com>
Apply go1.19 gofmt to non-generated files.
Generated .pb.go files are created by generate.bash using Go 1.18,
so leave them unchanged for now.
Change-Id: Ied36c83cf99704988d059bf0412e677f0fbc71b0
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/418676
Reviewed-by: Ian Lance Taylor <iant@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
reflect.Pointer introduced on go1.18. Support 1.17 and earlier.
Change-Id: I62dcdb580a7976068e86df432bc44ee21c2cda81
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/412354
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Joseph Tsai <joetsai@digital-static.net>
Reviewed-by: Lasse Folger <lassefolger@google.com>
+ This change introduce a default and configurable depth limit for
proto.Unmarshal. If a message is nested deeper than the limit,
unmarshaling will fail. There are two ways to nest messages. Either by
having fields which are message types itself or by using groups.
+ The default limit is 10,000 for now. This might change in the future
to align it with other language implementation (C++ and Java use 100
as limit).
+ If pure groups (groups that don't contain message fields) are nested
deeper than the default limit the unmarshaling fails with:
proto: cannot parse invalid wire-format data
+ Note: the configured limit does not apply to pure groups.
+ This change is introduced to improve security and robustness. Because
unmarshaling is implemented using recursion it can lead to stack overflows
for certain inputs. The introduced limit protects against this.
+ A secondary motivation for this limit is the alignment with other
languages. Protocol buffers are a language interoperability mechanism
and thus either all implementations should accept the input or all
implementation should reject the input.
Change-Id: I14bdb44d06e4bd1aa90d6336c2cf6446003b2037
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/385854
Trust: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Damien Neil <dneil@google.com>
Trust: Damien Neil <dneil@google.com>
Reviewed-by: Nicolas Hillegeer <aktau@google.com>
Reviewed-by: Chressie Himpel <chressie@google.com>
Instead of lifting values to interface and then do equality test,
compare scalar values with defined types to reduce allocation
from interface.
Change-Id: Ieb777fbd1a48c83d896d0ebe6ad605433f44c16c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/253100
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Trust: Damien Neil <dneil@google.com>
Trust: Joe Tsai <thebrokentoaster@gmail.com>
The MessageName returns the full name of a message.
It is a shorthand for:
var name protoreflect.FullName
if m != nil {
name = m.ProtoReflect().Descriptor().FullName()
}
Generally, we avoid the addition of helper functions unless
their use is justified. Arguments for this helper:
• The legacy proto.MessageName is widely used.
For example, inside Google, there are thousands of usages of it.
It is commonly used in logging and error construction to
report the name of a protobuf message.
• The fact that a nil-check may be neccessary means that users
either forget the nil-check and risk a nil-panic in production code,
or users have to write cumbersome logic to check for it.
For example, compare use with the helper:
return fmt.Errorf("invalid message type: %v", proto.MessageName(m))
to use without the helper:
var messageName protoreflect.FullName
if m != nil {
messageName = m.ProtoReflect().Descriptor().FullName()
}
return fmt.Errorf("invalid message type: %v", messageName)
A point of consideration is whether this should return a string
or a protoreflect.FullName. This CL returns the latter type:
• Most uses of it are for logging via log.Printf or error construction
via fmt.Errorf where it does not matter what the exact string type is
since its formatted the same way.
• Another use of it is to check for message type equality by doing
something similar to:
if proto.MessageName(got) != proto.MessageName(want) {
...
}
in which case it still does not matter what the exact string type is.
• The other major use of proto.MessageName is to call proto APIs
that actually do expect a protoreflect.FullName
(e.g., protoregistry.GlobalTypes.FindExtensionByNumber) where
the user currently type-casts the legacy proto.MessageName
to a protoreflect.FullName anyways.
As such, there does not seem to be much justified use for MessageName
as a string. The rare cases that need a string can trivially type cast
it to a string.
Change-Id: I840758df828eef72e7e63620569363a496366afa
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/242377
Reviewed-by: Damien Neil <dneil@google.com>
When proto.Unmarshal fails, it is almost always due to being passed
input which is not a wire-format message. (A text-format message, a
message with framing left intact, and so forth.) An error like
"variable length integer overflow" can be confusing in this case, since
it implies the problem is the varint rather than the input being
entirely wrong.
Replace all Unmarshal parse errors with "cannot parse invalid
wire-format data".
Change-Id: Id97253bd39ac604e569df71778194f37b3c86c28
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/244297
Reviewed-by: Joe Tsai <joetsai@google.com>
Fix the tests to work on GOARCH=386.
Also add another test suite that tests on GOARCH=386.
Change-Id: I41290998053dc059347adb8af70012dc5cc31811
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/241537
Reviewed-by: Herbie Ong <herbie@google.com>
The order package replaces the mapsort and fieldsort packages.
It presents a common API for ordered iteration over message fields
and map fields.
It has a number of pre-defined orderings.
Change-Id: Ie6cd423da30b4757864c352cb04454f21fe07ee2
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/239837
Reviewed-by: Herbie Ong <herbie@google.com>
The genid package unifies the genname, fieldnum, and detectknown
packages into a single package.
Whenever possible use the generated constants rather than
hard-coded literals. This makes it easier to search the entire
module for special logic that deal with well-known types.
Change-Id: I13beff1f4149444a0c0b9e607ebf759657f000f4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/235301
Reviewed-by: Herbie Ong <herbie@google.com>
Some companies (e.g., Google) run a profiling service where they may
choose to special-case certain symbols in a binary to classify
commonly used libraries like protobufs.
This CL funnels similar functionality through a single function
so that they can be more easily identified. This is by no means a
firm statement that these identifiers will never change names,
but at least the code documents warnings to avoid changing the
name of certain identifiers.
This CL provides the following semi-stable symbol names:
"google.golang.org/protobuf/proto".MarshalOptions.size
"google.golang.org/protobuf/proto".MarshalOptions.marshal
"google.golang.org/protobuf/proto".UnmarshalOptions.unmarshal
"google.golang.org/protobuf/encoding/prototext".MarshalOptions.marshal
"google.golang.org/protobuf/encoding/prototext".UnmarshalOptions.unmarshal
"google.golang.org/protobuf/encoding/protojson".MarshalOptions.marshal
"google.golang.org/protobuf/encoding/protojson".UnmarshalOptions.unmarshal
Merge and Clone are not part of the above set since there is a
possibility that MergeOptions will be added in the future.
We use an unexported method so that we have the freedom to change the
method however we want since profilers do not care about that.
Change-Id: Ia79af260d00125f48139420e1e18a86482bd1829
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/234079
Reviewed-by: Damien Neil <dneil@google.com>
In the upcoming 3.12.x release of protoc, the proto3 language will be
amended to support true presence for scalars. This CL adds support
to both the generator and runtime to support these semantics.
Newly added public API:
protogen.Plugin.SupportedFeatures
protoreflect.FieldDescriptor.HasPresence
protoreflect.FieldDescriptor.HasOptionalKeyword
protoreflect.OneofDescriptor.IsSynthetic
Change-Id: I7c86bf66d0ae56642109beb5f2132184593747ad
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/230698
Reviewed-by: Damien Neil <dneil@google.com>
