Weak fields were the predecessor to extensions (many many years ago) and were
entirely removed from Google’s production usage by now. (The corresponding field
in descriptor.proto was always documented as “// For Google-internal migration
only. Do not use.”)
Before this change, Go Protobuf still contained support for weak fields behind
the `protolegacy` build tag.
The `protolegacy` build tag was always documented as not being part of the
compatibility agreement:
// WARNING: The compatibility agreement covers nothing provided by this flag.
// As such, functionality may suddenly be removed or changed at our discretion.
const ProtoLegacy = protoLegacy
Fixesgolang/protobuf#1666
Change-Id: Ie1675424bc80d9f44345ccb96a858ef847ee1018
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/641655
Reviewed-by: Chressie Himpel <chressie@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
This is a no-op change in preparation of removing support for weak fields.
It allows users to keep enabling ProtoLegacy in general,
but also disable weak field support independently.
For golang/protobuf#1666
Change-Id: Ic3cb90d937e21a817ddbbb36029274be8b5f2513
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/641635
Reviewed-by: Chressie Himpel <chressie@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
This parameter became unused with golang/protobuf#1640.
Change-Id: Iab50ffb757f5b6f017bc32c37b25294894c1c6d9
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/641019
Reviewed-by: Chressie Himpel <chressie@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
This is the equivalent of CL 638495, but for the Opaque API.
While the behavior is the same for non-synthetic oneofs between
the Open Struct API and Opaque API, the behavior for synthetic oneofs
is not the same: Because the Opaque API uses a bitfield to store
presence, we need to use the Opaque presence check in WhichOneof().
This CL extends the testproto generator to use the protoc
command-line flag for the test3 testproto (which specifies
syntax = "proto3";), as the in-file api_level is only available
in .proto files using editions (but editions does not use synthetic
oneofs for optional fields, only proto3 does).
For golang/protobuf#1659
Change-Id: I0a1fd6e5fc6f96eeab043f966728ce2a14dbd446
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/638515
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Chressie Himpel <chressie@google.com>
This reverts change https://go.dev/cl/632735, in which
I misunderstood what the Protobuf documentation wanted to convey:
The quoted docs in CL 632735 refer to reflection for proto3 optional
fields, not to reflection for proto3 synthetic oneofs.
Synthetic oneofs should remain visible through reflection.
For the Open API, this change restores the old behavior.
For the Opaque API, another fix is needed and will be sent
in a separate, follow-up CL (follow golang/protobuf#1659).
For golang/protobuf#1659
Thanks to Joshua Humphries for the report and reproducer!
Change-Id: I3a924eed6d2425581adc65651395a68fc1576f4d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/638495
Reviewed-by: Chressie Himpel <chressie@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
For golang/protobuf#1657
Change-Id: I7b2b0c30506706015ce278e6054439c9ad9ef727
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/634815
TryBot-Bypass: Michael Stapelberg <stapelberg@google.com>
Reviewed-by: Joseph Tsai <joetsai@digital-static.net>
Reviewed-by: Damien Neil <dneil@google.com>
Calling WhichOneof should not be possible for synthetic oneofs;
on the reflection level, these fields should work as if they
were regular fields, not as if they were oneofs:
> Reflection for proto3 optional fields should work properly. For example, a
> method like Reflection::HasField() should know to look for the hasbit for a
> proto3 optional field. It should not be fooled by the synthetic oneof into
> thinking that there is a case member for the oneof.
From the Protobuf docs at:
https://github.com/protocolbuffers/protobuf/blob/main/docs/implementing_proto3_presence.md#updating-reflection
This change was tested Google-internally as CL 701866153.
Change-Id: Id9500d4aa492acf4ebc6d2d84be07ed81201d2aa
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/632735
Reviewed-by: Chressie Himpel <chressie@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
This test was accidentally migrated from proto.GetExtension to
proto.HasExtension in a semi-automated change and nobody noticed.
proto.HasExtension does not actually trigger lazy extension decoding.
I verified this using the dlv debugger:
% go test -tags protolegacy -c -gcflags "all=-N -l"
% dlv exec ./impl.test
Type 'help' for list of commands.
(dlv) r -test.run=TestMarshalMessageSetLazyRace -test.v
Process restarted with PID 3586699
(dlv) b TestMarshalMessageSetLazyRace
Breakpoint 1 set at 0xac9bf6 for google.golang.org/protobuf/internal/impl_test.TestMarshalMessageSetLazyRace() ./lazy_test.go:500
(dlv) c
=== RUN TestMarshalMessageSetLazyRace
> [Breakpoint 1] google.golang.org/protobuf/internal/impl_test.TestMarshalMessageSetLazyRace() ./lazy_test.go:500 (hits goroutine(32):1 total:1) (PC: 0xac9bf6)
=> 500: func TestMarshalMessageSetLazyRace(t *testing.T) {
(dlv) b ExtensionField.lazyInit
Breakpoint 2 set at 0x8da076 for google.golang.org/protobuf/internal/impl.(*ExtensionField).lazyInit() ./codec_extension.go:123
(dlv) c
--- PASS: TestMarshalMessageSetLazyRace (4.07s)
PASS
Process 3586699 has exited with status 0
(dlv) exit
Note how breakpoint 2 is unexpectedly not hit!
Here is how the output changes with my fix:
% go test -tags protolegacy -c -gcflags "all=-N -l"
% dlv --check-go-version=false exec ./impl.test
Type 'help' for list of commands.
(dlv) r -test.run=TestMarshalMessageSetLazyRace -test.v
Process restarted with PID 3587344
(dlv) b TestMarshalMessageSetLazyRace
Breakpoint 1 set at 0xac9bf6 for google.golang.org/protobuf/internal/impl_test.TestMarshalMessageSetLazyRace() ./lazy_test.go:500
(dlv) c
=== RUN TestMarshalMessageSetLazyRace
> [Breakpoint 1] google.golang.org/protobuf/internal/impl_test.TestMarshalMessageSetLazyRace() ./lazy_test.go:500 (hits goroutine(28):1 total:1) (PC: 0xac9bf6)
=> 500: func TestMarshalMessageSetLazyRace(t *testing.T) {
(dlv) b ExtensionField.lazyInit
Breakpoint 2 set at 0x8da076 for google.golang.org/protobuf/internal/impl.(*ExtensionField).lazyInit() ./codec_extension.go:123
(dlv) c
> [Breakpoint 2] google.golang.org/protobuf/internal/impl.(*ExtensionField).lazyInit() ./codec_extension.go:123 (hits goroutine(68):1 total:4) (PC: 0x8da076)
> [Breakpoint 2] google.golang.org/protobuf/internal/impl.(*ExtensionField).lazyInit() ./codec_extension.go:123 (hits goroutine(70):1 total:4) (PC: 0x8da076)
> [Breakpoint 2] google.golang.org/protobuf/internal/impl.(*ExtensionField).lazyInit() ./codec_extension.go:123 (hits goroutine(58):1 total:4) (PC: 0x8da076)
> [Breakpoint 2] google.golang.org/protobuf/internal/impl.(*ExtensionField).lazyInit() ./codec_extension.go:123 (hits goroutine(54):1 total:4) (PC: 0x8da076)
=> 123: func (f *ExtensionField) lazyInit() {
(dlv) bt
0 0x00000000008da076 in google.golang.org/protobuf/internal/impl.(*ExtensionField).lazyInit
at ./codec_extension.go:123
1 0x00000000008dac3e in google.golang.org/protobuf/internal/impl.(*ExtensionField).Value
at ./codec_extension.go:180
2 0x000000000094e5d9 in google.golang.org/protobuf/internal/impl.(*extensionMap).Get
at ./message_reflect.go:276
3 0x000000000095a2ee in google.golang.org/protobuf/internal/impl.(*messageState).Get
at ./message_reflect_gen.go:90
4 0x0000000000890099 in google.golang.org/protobuf/proto.GetExtension
at /usr/local/google/home/stapelberg/protobuf/proto/extension.go:90
5 0x0000000000aca7d4 in google.golang.org/protobuf/internal/impl_test.TestMarshalMessageSetLazyRace.func2.1
at ./lazy_test.go:545
6 0x0000000000aca625 in google.golang.org/protobuf/internal/impl_test.TestMarshalMessageSetLazyRace.func2
at ./lazy_test.go:550
7 0x00000000006001c1 in runtime.goexit
at /usr/lib/google-golang/src/runtime/asm_amd64.s:1700
Change-Id: Ie7a8621064af412a1db7efb3ad6b8473f9db58e8
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/624416
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Chressie Himpel <chressie@google.com>
Also adds better benchmark cases for large message where some fields are
actually populated.
This change was previously done in Google internal cl/660848520.
Change-Id: I682aae0c9c2850bfe7638de29ab743ad7d7b119a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/609035
Reviewed-by: Christian Höppner <hoeppi@google.com>
Reviewed-by: Cassondra Foesch <cfoesch@gmail.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Most of our imports already comply with the style, but this change makes it
consistent across the entire repository.
I suspect that these unnecessary imports are left-overs from differences between
Google-internal (where many imports are renamed) and Open Source.
Change-Id: Id5f6eabd22bcc72dd4e36dd018b6cc5138df7696
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/609875
Reviewed-by: Christian Höppner <hoeppi@google.com>
Auto-Submit: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
This change was originally done by Patrik Nyblom in Google-internal cl/308876443
(The corresponding test is TestDecodeFastCheckInitialized in methods_test.go.)
Change-Id: I312dff53db4d6f87decf152eb20ec4dd280f03c3
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/608256
Reviewed-by: Christian Höppner <hoeppi@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
This cleanup was done in June 2020 by Dan Scales,
but never upstreamed to the open source Go Protobuf repository.
Change-Id: I175d9588b5bb536fa276133f8fe6073278e6d3b0
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/605895
Reviewed-by: Christian Höppner <hoeppi@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
The IsLazy() method is intentionally not added to the
protoreflect.FieldDescriptor interface because users should not need it.
The Go Protobuf runtime will use type assertion to dynamically call the method.
There is no change in behavior, this is preparation only.
Change-Id: I4500c17f5edd1bdc40447eb89bff115b4e8eab06
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/596539
Reviewed-by: Chressie Himpel <chressie@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
When a message (within an extension) is lazily decoded, its size cache is
initialized to 0 (the zero value for an int32). This doesn’t mean the size cache
reads 0, but rather that it was not initialized.
This fixes TestExtensionGetRace being flaky since CL 580015.
related to golang/protobuf#1609
Change-Id: Ia305badadd300679975f230005c3e33c94050e4a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/586396
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
Extensions will be kept in wire format over proto.Size and proto.Marshal.
This change is a significant performance optimization for jobs that read and
write Protobuf messages of the same type, but do not need to process extensions.
This change is based on work by Patrik Nyblom.
Note that the proto.Size semantics for lazy messages might be surprising;
see https://protobuf.dev/reference/go/size/ for details.
We have been running this change for about two weeks in Google,
all known breakages have already been addressed with CL 579995.
related to golang/protobuf#1609
Change-Id: I16be78d15304d775bb30e76356a1a61d61300b43
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/580015
Reviewed-by: Lasse Folger <lassefolger@google.com>
Auto-Submit: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Change-Id: I2db557669ada6e031140a09b3a92bd901220f8f3
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/580975
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Auto-Submit: Michael Stapelberg <stapelberg@google.com>
This used to not be necessary, but a subsequent change will change behavior
based on MarshalOptions.Deterministic, so it is important that we do not
accidentally drop MarshalOptions anywhere.
related to golang/protobuf#1609
Change-Id: I6b53352707d93939642a627eb41c930f8ac3157f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/579995
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
There currently is no risk of producing invalid wire format,
but that will change with subsequent changes regarding lazy decoding.
We have been running this change in production for about a month,
without ever triggering the check (until lazy decoding is involved).
related to golang/protobuf#1609
Change-Id: I3c5c956aee2fa81f99dea03ed2a977a1547081fc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/579595
Auto-Submit: Michael Stapelberg <stapelberg@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
messageInfo() looks like this:
func (ms *messageState) messageInfo() *MessageInfo {
mi := ms.LoadMessageInfo()
if mi == nil {
panic("invalid nil message info; this suggests memory corruption due to a race or shallow copy on the message struct")
}
return mi
}
func (ms *messageState) LoadMessageInfo() *MessageInfo {
return (*MessageInfo)(atomic.LoadPointer((*unsafe.Pointer)(unsafe.Pointer(&ms.atomicMessageInfo))))
}
Which is an atomic load and a predictable branch. On x86, this 64-bit
load is just a MOV. On other platforms, like ARM64, there's actual
atomics involved (LDAR).
Meaning, it's cheap, but not free. Eliminate redundant copies of this
(Common Subexpression Elimination).
The newly added benchmarks improve by (geomean) 2.5%:
$ benchstat pre post | head -10
goarch: amd64
cpu: AMD Ryzen Threadripper PRO 3995WX 64-Cores
│ pre │ post │
│ sec/op │ sec/op vs base │
Extension/Has/None-12 106.4n ± 2% 104.0n ± 2% -2.21% (p=0.020 n=10)
Extension/Has/Set-12 116.4n ± 1% 114.4n ± 2% -1.76% (p=0.017 n=10)
Extension/Get/None-12 184.2n ± 1% 181.0n ± 1% -1.68% (p=0.003 n=10)
Extension/Get/Set-12 144.5n ± 3% 140.7n ± 2% -2.63% (p=0.041 n=10)
Extension/Set-12 227.2n ± 2% 218.6n ± 2% -3.81% (p=0.000 n=10)
geomean 149.6n 145.9n -2.42%
I didn't test on ARM64, but the difference should be larger due to the
reduced atomics.
Change-Id: I8eebeb6f753425b743368a7f5c7be4d48537e5c3
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/575036
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
Commit-Queue: Nicolas Hillegeer <aktau@google.com>
Auto-Submit: Nicolas Hillegeer <aktau@google.com>
Extensions are unmarshaled lazily if protolegacy is true. The current
implementation of proto.HasExtension forces this unmarshaling to happen.
Change that.
Lazy message extensions are unmarshaled on first access, see
(*ExtensionField).Value. This leads to an (expensive) unmarshal
operation even if the user only wanted to know whether the extension is
present.
Granted, in most cases a HasExtension returning true will be followed by
a GetExtension. Due to memoization (see (*ExtensionField).lazyInit), the
cost will just shift from HasExtension to GetExtension. But, this CL
allows building cheaper functionality that only needs to know about
extension existence.
Why can this validation be removed?
- All tests pass.
- This check was added in CL 229558. The author (Joe Tsai) noted:
> Technically this shouldn't be needed, but I couldn't adequately reason
> whether a nil message value would ever be set through non-reflection
> means.
Like the author, I believe it's not needed:
- `proto.SetExtension` does not allow setting invalid messages (see
proto/extension.go).
- Likewise, (*extensionMap).Set panics when attempting to set an
invalid value.
- Unmarshaling does not produce submessages for which `IsValid` is
false.
The added test fails without the fix:
$ go test -tags=protolegacy -test.v -test.run=TestHasExtensionNoAlloc proto/extension_test.go
=== RUN TestHasExtensionNoAlloc
=== RUN TestHasExtensionNoAlloc/Nil
=== RUN TestHasExtensionNoAlloc/Eager
=== RUN TestHasExtensionNoAlloc/Lazy
extension_test.go:156: proto.HasExtension should not allocate, but allocated 3.00B per run
--- FAIL: TestHasExtensionNoAlloc (0.00s)
--- PASS: TestHasExtensionNoAlloc/Nil (0.00s)
--- PASS: TestHasExtensionNoAlloc/Eager (0.00s)
--- FAIL: TestHasExtensionNoAlloc/Lazy (0.00s)
FAIL
FAIL command-line-arguments 0.018s
The tests are disabled in race mode because the race instrumentation for
closures et al. always allocates. The protolegacy tests were previously
only run in race mode. I added a non-race variant in
integration_test.go.
Change-Id: Idbc67c1cf0aea8833a2735ca7bfc8d2466ceaf44
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/575035
Reviewed-by: Nicolas Hillegeer <aktau@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
Auto-Submit: Nicolas Hillegeer <aktau@google.com>
This change removes most usages of Syntax() from the repository and uses
edition features for instead. The appropriate edition feature defaults are
loaded for proto2/proto3 when the initialization of the descriptors
start.
All of these changes were tested on the Google corpus.
Change-Id: Ieca076a2b38ca8e50e084cd32e725b7b3dcb4171
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/572435
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
There are several minor fixes:
* fix the unmarshaler for the field options (it used the wrong field numbers).
* fix the unmarshaler to always initialize the file options with the appropriate defaults.
* fix the coder selection to work properly for fields with implicit presence tracking (they used the coders for explicit tracking).
* fix the dynamic message created from a descriptor for an editions using proto to use HasPresence instead of a syntax check for proto3
Change-Id: Ic13bc22a71aa9f93b476e5edd650d4d2ab5dcb98
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/564455
Auto-Submit: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Resolvesgolang/protobuf#1521
In the internal/impl package, a helper computes metadata used for
serialization and deserialization of extension values. It features a
package var of type sync.Map that is used as cache. This ostensibly
was for performance, however it has never worked, because the code
that updates the cache inserts entries into the wrong map. (These
erroneous entries do not cause any issues because they are keys that
never conflict with those used in valid queries.)
Instead of a one-line fix to have this code update the correct cache,
this change removes the cache altogether. The existence of the cache
means that once a protoreflect.ExtensionType is used to serialize or
deserialize a message, it can *never* be garbage collected. Workloads
that are long-lived servers using dynamic messages and extensions
based on user requests will exhibit unbounded growth in memory usage
as the cache only gets larger and larger.
Since the cache has never worked, any advantages it ostensibly
conferred have not been missed. So this fixes the unbounded memory
growth instead.
Change-Id: I15957fd8521852f9f7f9f89db7ebfd7170d85202
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/560095
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Damien Neil <dneil@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
The composites x/tools/analysis pass errors on this use of
reflect.ValueError with:
```
internal/impl/message_reflect_field.go:541:10: reflect.ValueError struct literal uses unkeyed fields (composites)
```
This patch adds the missing field names.
Change-Id: I1a175aad6e9724c27f2469adc1f3a9b1e8c61e85
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/553915
Reviewed-by: Christian Höppner <hoeppi@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Auto-Submit: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Calling the ProtoReflect method of the newly-constructed
message avoids an allocation in MessageInfo.MessageOf in
the common case of a generated message with an optimized
ProtoReflect method.
Benchmark for creating an empty message, darwin/arm64 M1 laptop:
name old time/op new time/op delta
EmptyMessage/New-10 32.1ns ± 2% 23.7ns ± 2% -26.06% (p=0.000 n=10+9)
name old alloc/op new alloc/op delta
EmptyMessage/New-10 64.0B ± 0% 48.0B ± 0% -25.00% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
EmptyMessage/New-10 2.00 ± 0% 1.00 ± 0% -50.00% (p=0.000 n=10+10)
Change-Id: Ifa3c3ffa8edc76f78399306d0f4964eae4aacd28
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/418677
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
Apply go1.19 gofmt to non-generated files.
Generated .pb.go files are created by generate.bash using Go 1.18,
so leave them unchanged for now.
Change-Id: Ied36c83cf99704988d059bf0412e677f0fbc71b0
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/418676
Reviewed-by: Ian Lance Taylor <iant@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
+ This change introduce a default and configurable depth limit for
proto.Unmarshal. If a message is nested deeper than the limit,
unmarshaling will fail. There are two ways to nest messages. Either by
having fields which are message types itself or by using groups.
+ The default limit is 10,000 for now. This might change in the future
to align it with other language implementation (C++ and Java use 100
as limit).
+ If pure groups (groups that don't contain message fields) are nested
deeper than the default limit the unmarshaling fails with:
proto: cannot parse invalid wire-format data
+ Note: the configured limit does not apply to pure groups.
+ This change is introduced to improve security and robustness. Because
unmarshaling is implemented using recursion it can lead to stack overflows
for certain inputs. The introduced limit protects against this.
+ A secondary motivation for this limit is the alignment with other
languages. Protocol buffers are a language interoperability mechanism
and thus either all implementations should accept the input or all
implementation should reject the input.
Change-Id: I14bdb44d06e4bd1aa90d6336c2cf6446003b2037
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/385854
Trust: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Damien Neil <dneil@google.com>
Trust: Damien Neil <dneil@google.com>
Reviewed-by: Nicolas Hillegeer <aktau@google.com>
Reviewed-by: Chressie Himpel <chressie@google.com>
When merging aberrant messages with legacy Marshal and Unmarshal
methods, check for a typed nil source before calling Marshal.
Add an aberrant message with Marshal/Unmarshal methods to
internal/testprotos/nullable and use it to test the internal/impl
support for these methods.
Fixesgolang/protobuf#1324
Change-Id: Ib6ce85b30b46e3392a226ca6abe411932a371f02
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/321529
Trust: Damien Neil <dneil@google.com>
Run-TryBot: Damien Neil <dneil@google.com>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Implement support in the protobuf runtime to better understand
message types that are not generated by the official generator.
In particular:
* Add a best-effort implementation of protobuf reflection for
"non-nullable" fields which are supposed to be represented by *T,
but are instead represented by a T. "Non-nullable" message fields
report presence based on whether the message is the zero Go value.
* We do NOT implement support for "non-nullable" fields in the
table-driven implementation since we assume that the aberrant messages
that we care about have a Marshal and Unmarshal method.
* We better handle custom messages that implement Marshal and Unmarshal,
but do NOT implement Merge. In that case, we implement merge in terms of
a back-to-back marshal and unmarshal.
* We better tolerate the situations where a protobuf message field
cannot be mapped to a Go struct field since the latter is missing.
In such cases, reflection treats the field as if it were unpopulated.
Setting such fields will panic.
This change allows the runtime to handle all message types declared
in the "go.etcd.io/etcd" and "k8s.io" modules where protobuf reflection,
Marshal, Unmarshal, Reset, Merge, and Equal all work.
The only types that still do not fully work are:
* "k8s.io/api/authentication/v1".ExtraValue
* "k8s.io/api/authentication/v1beta1".ExtraValue
* "k8s.io/api/authorization/v1".ExtraValue
* "k8s.io/api/authorization/v1beta1".ExtraValue
* "k8s.io/api/certificates/v1".ExtraValue
* "k8s.io/api/certificates/v1beta1".ExtraValue
* "k8s.io/apimachinery/pkg/apis/meta/v1".MicroTime
* "k8s.io/apimachinery/pkg/apis/meta/v1".Time
* "k8s.io/apimachinery/pkg/apis/meta/v1".Verbs
While Marshal, Unmarshal, Reset, and Merge continue to work,
protobuf reflection and any functionality that depends on it
(e.g., prototext, protojson, Equal, etc.) will not work.
Change-Id: I67a9d2f1bec35248045ad0c16220d02fc2e0e172
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/300869
Trust: Joe Tsai <joetsai@digital-static.net>
Trust: Joe Tsai <thebrokentoaster@gmail.com>
Reviewed-by: Damien Neil <dneil@google.com>
The protowire.{ConsumeBytes,ConsumeString} funcs are identical except
that the latter allocates a string by implicitly converting the []byte.
Avoid using ConsumeString since we can do the conversion ourselves
at a latter point and sometimes avoid the allocation.
Change-Id: Idf31edc013b72ee5ee8461a68d10e57ad461d95c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/263628
Trust: Joe Tsai <joetsai@digital-static.net>
Reviewed-by: Damien Neil <dneil@google.com>
This trivially adds a name for marshalOptions and unmarshalOptions
instead of always having it be underscore.
This assists the use of patches that rely on this variable.
Change-Id: I4c23901c93f0362aff412339912522f92749d659
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/263627
Trust: Joe Tsai <joetsai@digital-static.net>
Reviewed-by: Damien Neil <dneil@google.com>
Move the declaration of the slice variable
as close to its actual use as possible.
Change-Id: Ibe26460b5ca81edbed1073b32faf1c5f5d2c922a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/263719
Trust: Joe Tsai <joetsai@digital-static.net>
Trust: Joe Tsai <joetsai@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
This CL adds runtime support for unknown fields to be represented
as *[]byte in addition to the current representation as []byte.
This CL does not change generated code to use *[]byte.
Comparison between using *[]byte and []byte:
• Every message supports unknown fields, so use of []byte
expands a message size by 24B (for 64-bit systems).
In contrast, *[]byte only expands a message by 8B.
This has significant memory implications for small messages.
• If unknown fields are encountered, *[]byte has extra overhead
allocating the 24B slice header. However, it is assumed
that messages rarely see any unknown fields at runtime,
or generally do so for a temporary period of time.
Change-Id: I81935e4ea7394166e61ff4579f76f59fa792dfc9
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/244937
Reviewed-by: Damien Neil <dneil@google.com>
This variable is only used by the aberrantMessage type,
rename it to properly reflect what it is used for.
Change-Id: I081e8954eb1b282f01f6d2d764bffaf31c6677e9
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/245077
Reviewed-by: Damien Neil <dneil@google.com>
The MessageFieldTypes interface (if implemented by a MessageType)
provides Go type information about the fields if they are
an enum or message type.
Change-Id: I68b20f5726377f6b0f2c20a8b6e45f9802b43f67
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/236777
Reviewed-by: Damien Neil <dneil@google.com>
When proto.Unmarshal fails, it is almost always due to being passed
input which is not a wire-format message. (A text-format message, a
message with framing left intact, and so forth.) An error like
"variable length integer overflow" can be confusing in this case, since
it implies the problem is the varint rather than the input being
entirely wrong.
Replace all Unmarshal parse errors with "cannot parse invalid
wire-format data".
Change-Id: Id97253bd39ac604e569df71778194f37b3c86c28
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/244297
Reviewed-by: Joe Tsai <joetsai@google.com>
