Extensions are unmarshaled lazily if protolegacy is true. The current
implementation of proto.HasExtension forces this unmarshaling to happen.
Change that.
Lazy message extensions are unmarshaled on first access, see
(*ExtensionField).Value. This leads to an (expensive) unmarshal
operation even if the user only wanted to know whether the extension is
present.
Granted, in most cases a HasExtension returning true will be followed by
a GetExtension. Due to memoization (see (*ExtensionField).lazyInit), the
cost will just shift from HasExtension to GetExtension. But, this CL
allows building cheaper functionality that only needs to know about
extension existence.
Why can this validation be removed?
- All tests pass.
- This check was added in CL 229558. The author (Joe Tsai) noted:
> Technically this shouldn't be needed, but I couldn't adequately reason
> whether a nil message value would ever be set through non-reflection
> means.
Like the author, I believe it's not needed:
- `proto.SetExtension` does not allow setting invalid messages (see
proto/extension.go).
- Likewise, (*extensionMap).Set panics when attempting to set an
invalid value.
- Unmarshaling does not produce submessages for which `IsValid` is
false.
The added test fails without the fix:
$ go test -tags=protolegacy -test.v -test.run=TestHasExtensionNoAlloc proto/extension_test.go
=== RUN TestHasExtensionNoAlloc
=== RUN TestHasExtensionNoAlloc/Nil
=== RUN TestHasExtensionNoAlloc/Eager
=== RUN TestHasExtensionNoAlloc/Lazy
extension_test.go:156: proto.HasExtension should not allocate, but allocated 3.00B per run
--- FAIL: TestHasExtensionNoAlloc (0.00s)
--- PASS: TestHasExtensionNoAlloc/Nil (0.00s)
--- PASS: TestHasExtensionNoAlloc/Eager (0.00s)
--- FAIL: TestHasExtensionNoAlloc/Lazy (0.00s)
FAIL
FAIL command-line-arguments 0.018s
The tests are disabled in race mode because the race instrumentation for
closures et al. always allocates. The protolegacy tests were previously
only run in race mode. I added a non-race variant in
integration_test.go.
Change-Id: Idbc67c1cf0aea8833a2735ca7bfc8d2466ceaf44
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/575035
Reviewed-by: Nicolas Hillegeer <aktau@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
Auto-Submit: Nicolas Hillegeer <aktau@google.com>
I have choosen to implement this via interace assertion so that
other implementers of protoreflect.FileDescriptor can implement
this as well.
Change-Id: Ib907895044e89bdba1009cc09129d3dd1224561f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/573055
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
This change removes the remaining usages of Syntax() from Go Protobuf
and uses edition features instead.
It also adds a new function to the EnumDescriptor interface checking if
the enum is using a Closed semantics.
All of these changes were tested on the Google corpus.
Change-Id: I7a8110f6f3b6ed24bf7ece500b4942371302c56c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/572695
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
This change removes most usages of Syntax() from the repository and uses
edition features for instead. The appropriate edition feature defaults are
loaded for proto2/proto3 when the initialization of the descriptors
start.
All of these changes were tested on the Google corpus.
Change-Id: Ieca076a2b38ca8e50e084cd32e725b7b3dcb4171
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/572435
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
In internal/encoding/json, report an error when encountering a }
when we are expecting an object field value. For example, the input
`{"":}` now correctly results in an error at the closing } token.
In encoding/protojson, check for an unexpected EOF token in
skipJSONValue. This is redundant with the check in internal/encoding/json,
but adds a bit more defense against any other similar bugs that
might exist.
Fixes CVE-2024-24786
Change-Id: I03d52512acb5091c8549e31ca74541d57e56c99d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/569356
TryBot-Bypass: Damien Neil <dneil@google.com>
Reviewed-by: Roland Shoemaker <roland@golang.org>
Commit-Queue: Damien Neil <dneil@google.com>
Before this change there was a data race if you initialized the
descriptor of a message in parallel to the descriptor of any extension of
that message because the extendees feature set is read during the
initialization of the extension.
Change-Id: Id896b9fbf848209fce7dae8a7f40e2d61a3b2825
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/568695
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
In scope of this change, I had to fix the `IsPacked()` implementation
for field descriptors because it was not returning the correct values
for edition protos.
Change-Id: Ic1ba9d0b3552ddf16360a80336c14632f2ce6f16
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/566039
Auto-Submit: Lasse Folger <lassefolger@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Before this change the implementation would ignore editions options set on
extension declarations.
This change also add much more coverage for valid messages.
Change-Id: I0e7029b1430a39859cdd4a93667ac20c7e0ff20e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/565396
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
This fixes more minor issue when constructing the descriptor in that
delimited encoded message fields are not considered properly as groups.
Change-Id: I714a227a0f8d256fa7430c526844e8e94acdda33
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/564816
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
There are several minor fixes:
* fix the unmarshaler for the field options (it used the wrong field numbers).
* fix the unmarshaler to always initialize the file options with the appropriate defaults.
* fix the coder selection to work properly for fields with implicit presence tracking (they used the coders for explicit tracking).
* fix the dynamic message created from a descriptor for an editions using proto to use HasPresence instead of a syntax check for proto3
Change-Id: Ic13bc22a71aa9f93b476e5edd650d4d2ab5dcb98
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/564455
Auto-Submit: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
This change add parsing editions specific descriptor parts to
internal/filedesc which makes it possible to load file descriptors of
protos using editions at runtime.
Change-Id: I7a0c23303acddd2cff115859d4cf82bf0102b14c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/563615
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
I'll add the runtime implementations and tests in a followup change to
make it easier to review.
Change-Id: I2917e0d40a99e81799d89a66584a680ed7e8dbf7
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/561939
Auto-Submit: Lasse Folger <lassefolger@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
This change adds the first Go specific editions feature and the
associated proto. The feature is used to control if the legacy
UnmarshalJSON method should be generated for enums. This change only
affects protos using editions.
More tests will be added in a followup change.
Change-Id: Ifb62454d7568bd6d90d0b93f8953adcfe46c45fd
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/561938
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
Resolvesgolang/protobuf#1521
In the internal/impl package, a helper computes metadata used for
serialization and deserialization of extension values. It features a
package var of type sync.Map that is used as cache. This ostensibly
was for performance, however it has never worked, because the code
that updates the cache inserts entries into the wrong map. (These
erroneous entries do not cause any issues because they are keys that
never conflict with those used in valid queries.)
Instead of a one-line fix to have this code update the correct cache,
this change removes the cache altogether. The existence of the cache
means that once a protoreflect.ExtensionType is used to serialize or
deserialize a message, it can *never* be garbage collected. Workloads
that are long-lived servers using dynamic messages and extensions
based on user requests will exhibit unbounded growth in memory usage
as the cache only gets larger and larger.
Since the cache has never worked, any advantages it ostensibly
conferred have not been missed. So this fixes the unbounded memory
growth instead.
Change-Id: I15957fd8521852f9f7f9f89db7ebfd7170d85202
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/560095
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Damien Neil <dneil@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
+ The previous commit forgot to resolve the LegacyRequired feature
Change-Id: I05a2847be3d52277c65df004363ea3c87944364e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/555976
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Before this CL we only used features where they were applicable, e.g.
file features were only set on the file descriptor and field features
only on the fields descriptor. After this CL all descriptors (file,
message and field) contain always all features and when initializing
either of them we merge the features specified by options with the
features of the parent descriptor and use the result as the effective
feature. This way we initialize all the descriptors the same way and the
proto spec can change on which level features can be specified without
us having to change the code (at the moment most features can only be
specified on file or field level, in the future message might also be
possible).
Change-Id: I822202bf6a2bba5167bf9622c91a65c4523e78f9
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/555975
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
The composites x/tools/analysis pass errors on this use of
reflect.ValueError with:
```
internal/impl/message_reflect_field.go:541:10: reflect.ValueError struct literal uses unkeyed fields (composites)
```
This patch adds the missing field names.
Change-Id: I1a175aad6e9724c27f2469adc1f3a9b1e8c61e85
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/553915
Reviewed-by: Christian Höppner <hoeppi@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Auto-Submit: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
This is the first of multiple changes to add protobuf editions support
to the Go implementation.
This change includes the feature resolutions for protobuf editions.
The plugin does not yet report editions support and protoc would fail
if it invoked this plugin with a proto that uses editions.
Change-Id: I7d31909366c3433b21abab74ec92263e08145434
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/547315
Auto-Submit: Lasse Folger <lassefolger@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Commit-Queue: Lasse Folger <lassefolger@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
Prior to this change, one could either use go generate or
./test.bash -regenerate to re-generate the files — but only
the latter way would work.
With this change, we make ./test.bash -regenerate call go generate
so that the two ways cannot diverge again.
Change-Id: I940df66ebc6a4200847cf751929b3611309b67bc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/547256
Reviewed-by: Lasse Folger <lassefolger@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
This is slightly more efficient and removes explicit dependence
on string layout and alignment. Build tags control use.
Because of what is arguably a bug in go 1.20's treatment of build tags,
is build-tagged to go1.21, not go1.20. (1.20 sees that it matches the
build tag and tries to compile the file, but then demands that it only use
language features from the version mentioned in go.mod. Go 1.21 and 1.22
behave more sensibly.)
Change-Id: I0cfc3b2e5651595edd3a82d9071d918700f961cd
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/545195
Reviewed-by: Cassondra Foesch <cfoesch@gmail.com>
Run-TryBot: David Chase <drchase@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
This updates all generated code to match the contents of the latest
v25.0-rc2 release candidate of Protobuf. This provides access to the
various new fields and types defined in google/protobuf/descriptor.proto.
Change-Id: I209f8509f2d3e9a0ac7d9e134caef38c48a0d16b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/538075
Reviewed-by: Lasse Folger <lassefolger@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
This updates all generated code to match the contents of the latest
v24.0-rc2 release candidate of Protobuf. This provides access to the
various new fields and types defined in google/protobuf/descriptor.proto.
Change-Id: I8ec56b308dd7eed52fb9b17b258a45d0669c47ff
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/512795
Reviewed-by: Lasse Folger <lassefolger@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Before this change, synthetic oneofs were ignored in the inOneof() helper
function, but not in the remainder of the LegacyFieldOrder function.
Change-Id: Ia60e5244ae1000e98bbba9dd26f1d1583337fab4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/497935
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
Reviewed-by: Cassondra Foesch <cfoesch@gmail.com>
Adds MarshalAppend methods to allow for byte slices to be reused.
Copies signature from the binary encoding.
Small changes to internal json and text libraries to use strconv
AppendInt and AppendUint for number encoding.
Change-Id: Ife7c8979c1c153a0a0bf9b70b296b8158d38dffc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/489615
Reviewed-by: Edward McFarlane <emcfarlane000@gmail.com>
Reviewed-by: Joseph Tsai <joetsai@digital-static.net>
Reviewed-by: Lasse Folger <lassefolger@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
Update to a protobuf version that contains a fix for
https://github.com/protocolbuffers/protobuf/issues/12173
(macOS build error).
Set "CC=clang" when building the protobuf repo on macOS,
which fixes build errors on my machine (macOS 13.3.1, M1).
Change-Id: Id8abb054479454b7ccbded5a6fa4d74f3d3c236d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/489315
Run-TryBot: Damien Neil <dneil@google.com>
Reviewed-by: Joseph Tsai <joetsai@digital-static.net>
Reviewed-by: Lasse Folger <lassefolger@google.com>
