Change-Id: I2db557669ada6e031140a09b3a92bd901220f8f3
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/580975
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Auto-Submit: Michael Stapelberg <stapelberg@google.com>
messageInfo() looks like this:
func (ms *messageState) messageInfo() *MessageInfo {
mi := ms.LoadMessageInfo()
if mi == nil {
panic("invalid nil message info; this suggests memory corruption due to a race or shallow copy on the message struct")
}
return mi
}
func (ms *messageState) LoadMessageInfo() *MessageInfo {
return (*MessageInfo)(atomic.LoadPointer((*unsafe.Pointer)(unsafe.Pointer(&ms.atomicMessageInfo))))
}
Which is an atomic load and a predictable branch. On x86, this 64-bit
load is just a MOV. On other platforms, like ARM64, there's actual
atomics involved (LDAR).
Meaning, it's cheap, but not free. Eliminate redundant copies of this
(Common Subexpression Elimination).
The newly added benchmarks improve by (geomean) 2.5%:
$ benchstat pre post | head -10
goarch: amd64
cpu: AMD Ryzen Threadripper PRO 3995WX 64-Cores
│ pre │ post │
│ sec/op │ sec/op vs base │
Extension/Has/None-12 106.4n ± 2% 104.0n ± 2% -2.21% (p=0.020 n=10)
Extension/Has/Set-12 116.4n ± 1% 114.4n ± 2% -1.76% (p=0.017 n=10)
Extension/Get/None-12 184.2n ± 1% 181.0n ± 1% -1.68% (p=0.003 n=10)
Extension/Get/Set-12 144.5n ± 3% 140.7n ± 2% -2.63% (p=0.041 n=10)
Extension/Set-12 227.2n ± 2% 218.6n ± 2% -3.81% (p=0.000 n=10)
geomean 149.6n 145.9n -2.42%
I didn't test on ARM64, but the difference should be larger due to the
reduced atomics.
Change-Id: I8eebeb6f753425b743368a7f5c7be4d48537e5c3
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/575036
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
Commit-Queue: Nicolas Hillegeer <aktau@google.com>
Auto-Submit: Nicolas Hillegeer <aktau@google.com>
This change removes most usages of Syntax() from the repository and uses
edition features for instead. The appropriate edition feature defaults are
loaded for proto2/proto3 when the initialization of the descriptors
start.
All of these changes were tested on the Google corpus.
Change-Id: Ieca076a2b38ca8e50e084cd32e725b7b3dcb4171
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/572435
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
In scope of this change, I had to fix the `IsPacked()` implementation
for field descriptors because it was not returning the correct values
for edition protos.
Change-Id: Ic1ba9d0b3552ddf16360a80336c14632f2ce6f16
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/566039
Auto-Submit: Lasse Folger <lassefolger@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
This fixes more minor issue when constructing the descriptor in that
delimited encoded message fields are not considered properly as groups.
Change-Id: I714a227a0f8d256fa7430c526844e8e94acdda33
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/564816
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
This change add parsing editions specific descriptor parts to
internal/filedesc which makes it possible to load file descriptors of
protos using editions at runtime.
Change-Id: I7a0c23303acddd2cff115859d4cf82bf0102b14c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/563615
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
This change adds the first Go specific editions feature and the
associated proto. The feature is used to control if the legacy
UnmarshalJSON method should be generated for enums. This change only
affects protos using editions.
More tests will be added in a followup change.
Change-Id: Ifb62454d7568bd6d90d0b93f8953adcfe46c45fd
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/561938
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
This is the first of multiple changes to add protobuf editions support
to the Go implementation.
This change includes the feature resolutions for protobuf editions.
The plugin does not yet report editions support and protoc would fail
if it invoked this plugin with a proto that uses editions.
Change-Id: I7d31909366c3433b21abab74ec92263e08145434
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/547315
Auto-Submit: Lasse Folger <lassefolger@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Commit-Queue: Lasse Folger <lassefolger@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
Prior to this change, one could either use go generate or
./test.bash -regenerate to re-generate the files — but only
the latter way would work.
With this change, we make ./test.bash -regenerate call go generate
so that the two ways cannot diverge again.
Change-Id: I940df66ebc6a4200847cf751929b3611309b67bc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/547256
Reviewed-by: Lasse Folger <lassefolger@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
This updates all generated code to match the contents of the latest
v22.0 release of Protobuf.
This involved a couple of changes to the script that does the sync'ing:
1. The new Protobuf version no longer includes autoconf configuration
and instead requires using bazel to build things.
2. The new Protobuf release does not have an artifact named
"protobuf-all-${VERSION}.tar.gz", but the one named
"protobuf-${VERSION}.tar.gz" has all of the sources and was
sufficient for the regenerate.bash script to complete.
This change does NOT regenerate the protos related to benchmarks.
The Protobuf repo no longer includes benchmarks. The CL removing them
says they are superceded by google/fleetbench:
83c499de86
But that project's proto benchmark files are very different:
https://github.com/google/fleetbench/tree/main/fleetbench/proto
So I commented out those steps in the generation code since the benchmarks
will need some work to reconcile with fleetbench.
This code adds known failing tests for conformance. New test cases were
added in https://github.com/protocolbuffers/protobuf/pull/9534, but the
Go protojson package does not behave according to the new tests. There
is an existing issue in GitHub about this:
https://github.com/golang/protobuf/issues/1208
Change-Id: Iad796ec7889bc2a74b58db5224facf850cd1a1cd
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/469255
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
Fix typos in comments and unexported function name.
Change-Id: I239fff85bc68d7f0ef6f8853acdc9bbe19d86304
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/466535
Reviewed-by: Lasse Folger <lassefolger@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Reviewed-by: Joseph Tsai <joetsai@digital-static.net>
The protobuf wire format is insufficiently self-decribing such that
it is impossible to know whether for sure whether an unknown bytes value
is a sub-message or not. However, protopack is primarily used for debugging
where a best-effort guess is still very useful.
The Message.UnmarshalAbductive unmarshals an unknown bytes value as a message
if it is syntactically well-formed. Otherwise, it is left as is.
Change-Id: I5e2b4b995e2b5eb60942a242558bf4cea1da9891
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/309669
Trust: Joe Tsai <joetsai@digital-static.net>
Reviewed-by: Damien Neil <dneil@google.com>
This updates the dependency on the protobuf toolchain to an unreleased
version that has the go_package option for relevant .proto files in
that repository to reference this module.
Change-Id: Ie1ac8f81a323285efbd14a17d02c7105e810af2d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/235283
Trust: Joe Tsai <joetsai@digital-static.net>
Reviewed-by: Damien Neil <dneil@google.com>
The protowire.{ConsumeBytes,ConsumeString} funcs are identical except
that the latter allocates a string by implicitly converting the []byte.
Avoid using ConsumeString since we can do the conversion ourselves
at a latter point and sometimes avoid the allocation.
Change-Id: Idf31edc013b72ee5ee8461a68d10e57ad461d95c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/263628
Trust: Joe Tsai <joetsai@digital-static.net>
Reviewed-by: Damien Neil <dneil@google.com>
This trivially adds a name for marshalOptions and unmarshalOptions
instead of always having it be underscore.
This assists the use of patches that rely on this variable.
Change-Id: I4c23901c93f0362aff412339912522f92749d659
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/263627
Trust: Joe Tsai <joetsai@digital-static.net>
Reviewed-by: Damien Neil <dneil@google.com>
Move the declaration of the slice variable
as close to its actual use as possible.
Change-Id: Ibe26460b5ca81edbed1073b32faf1c5f5d2c922a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/263719
Trust: Joe Tsai <joetsai@digital-static.net>
Trust: Joe Tsai <joetsai@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
Add an outfile flag that specifies that the contents of
the provided file should generated to stdout.
When using this mode, it only generates one file at a time
and does not modify any files on disk.
Change-Id: I80ba1764a7f0ceb180b8b54f4197dd57fec86434
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/263625
Trust: Joe Tsai <joetsai@digital-static.net>
Reviewed-by: Patrik Nyblom <pnyb@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
Added API:
SourceLocations.ByPath
SourceLocations.ByDescriptor
SourceLocation.Next
SourcePath.String
SourcePath.Equal
We modify compiler/protogen to use SourceLocations.ByDescriptor.
In retrospect, if this had existed during the development of protogen,
we would not have exposed protogen.Location and related fields.
Change-Id: I58f17e59f90b9ba16f0982c4b71c2542e4ff6e75
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/238000
Reviewed-by: Damien Neil <dneil@google.com>
When proto.Unmarshal fails, it is almost always due to being passed
input which is not a wire-format message. (A text-format message, a
message with framing left intact, and so forth.) An error like
"variable length integer overflow" can be confusing in this case, since
it implies the problem is the varint rather than the input being
entirely wrong.
Replace all Unmarshal parse errors with "cannot parse invalid
wire-format data".
Change-Id: Id97253bd39ac604e569df71778194f37b3c86c28
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/244297
Reviewed-by: Joe Tsai <joetsai@google.com>
Add a new TextName accessor that returns the field name that should
be used for the text format. It is usually just the field name, except:
1) it uses the inlined message name for groups,
2) uses the full name surrounded by brackets for extensions, and
3) strips the "message_set_extension" for well-formed extensions
to the proto1 MessageSet.
We make similar adjustments to the JSONName accessor so that it applies
similar semantics for extensions.
The two changes simplifies all logic that wants the humanly readable
name for a field.
Change-Id: I524b6e017fb955146db81819270fe197f8f97980
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/239838
Reviewed-by: Herbie Ong <herbie@google.com>
The genid package unifies the genname, fieldnum, and detectknown
packages into a single package.
Whenever possible use the generated constants rather than
hard-coded literals. This makes it easier to search the entire
module for special logic that deal with well-known types.
Change-Id: I13beff1f4149444a0c0b9e607ebf759657f000f4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/235301
Reviewed-by: Herbie Ong <herbie@google.com>
This is step 2 of 6 in a multi-stage migration
to move the well-known types
from the google.golang.org/genproto module
to the google.golang.org/protobuf module.
The generated Go packages for field_mask.proto, api.proto,
type.proto, and source_context.proto are being moved over
to this module alongside all the other well-known types.
In order to move these types between two modules,
there needs to be a sequence of changes submitted in
decently rapid succession. It is impossible to atomically
make these changes, so a brief breakage is inevitable.
The steps are as follows:
Step 1: Submit a change to cloud.google.com/go/internal/gapicgen
to avoid generating the well-known types. Otherwise, the tool
will undo the changes made in step 3.
See https://code-review.googlesource.com/c/gocloud/+/56810
Step 2: Submit a change to google.golang.org/protobuf that
adds the generated well-known types being migrated to that module.
In order to prevent the situation where a user links in
too old a version of the genproto module such that
duplicate registration occurs for the well-known types,
the registry is specially modified to provide an error
message that instructs users to upgrade the genproto module.
See https://golang.org/cl/234937
Step 3: Submit a change to google.golang.org/genproto that
switches all generated well-known types to be aliases to the
ones declared in google.golang.org/protobuf from the previous step.
This will cause the genproto module to incur an dependency
on an unreleased version of the protobuf module.
See https://github.com/googleapis/go-genproto/pull/372
Step 4: Submit a change to google.golang.org/protobuf that
adds a weak module depdency on the genproto module at the
revision from the previous step.
Step 5: Release google.golang.org/protobuf@v1.24.0.
Step 6: Submit a change to google.golang.org/genproto that
updates the protobuf module dependency to v1.24.0.
Change-Id: I36a19049d2240b67a37dfad20e154505aee7c784
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/234937
Reviewed-by: Damien Neil <dneil@google.com>
Some companies (e.g., Google) run a profiling service where they may
choose to special-case certain symbols in a binary to classify
commonly used libraries like protobufs.
This CL funnels similar functionality through a single function
so that they can be more easily identified. This is by no means a
firm statement that these identifiers will never change names,
but at least the code documents warnings to avoid changing the
name of certain identifiers.
This CL provides the following semi-stable symbol names:
"google.golang.org/protobuf/proto".MarshalOptions.size
"google.golang.org/protobuf/proto".MarshalOptions.marshal
"google.golang.org/protobuf/proto".UnmarshalOptions.unmarshal
"google.golang.org/protobuf/encoding/prototext".MarshalOptions.marshal
"google.golang.org/protobuf/encoding/prototext".UnmarshalOptions.unmarshal
"google.golang.org/protobuf/encoding/protojson".MarshalOptions.marshal
"google.golang.org/protobuf/encoding/protojson".UnmarshalOptions.unmarshal
Merge and Clone are not part of the above set since there is a
possibility that MergeOptions will be added in the future.
We use an unexported method so that we have the freedom to change the
method however we want since profilers do not care about that.
Change-Id: Ia79af260d00125f48139420e1e18a86482bd1829
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/234079
Reviewed-by: Damien Neil <dneil@google.com>
In the upcoming 3.12.x release of protoc, the proto3 language will be
amended to support true presence for scalars. This CL adds support
to both the generator and runtime to support these semantics.
Newly added public API:
protogen.Plugin.SupportedFeatures
protoreflect.FieldDescriptor.HasPresence
protoreflect.FieldDescriptor.HasOptionalKeyword
protoreflect.OneofDescriptor.IsSynthetic
Change-Id: I7c86bf66d0ae56642109beb5f2132184593747ad
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/230698
Reviewed-by: Damien Neil <dneil@google.com>
Now we'll get a generator error if a file is missing a go_package
option.
Change-Id: I89eec716f86956e6c164a61f5531c140b74bc099
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/222378
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
This fixes the conformance test failures which occur because the
conformance protos have not been regenerated.
The generator script has been modified with a sanity check that
files do not exist outside the expected sub-tree.
Change-Id: I473efec4a016f6bc96ddf7e20d54bcf5ff9b55fe
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/223538
Reviewed-by: Damien Neil <dneil@google.com>
Under some rare circumstances registration can deadlock
when lazy descriptor initialization consults the registry.
Move the call triggering the lazy init out of the critical section.
Fixesgolang/protobuf#1052.
Change-Id: Ic266e06b0db99fea65e797b879ce53e5342fff95
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/204804
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Throughout the module options usually expressed as a struct where the
acting function is a method hanging off the options type.
Use this pattern for protogen as well.
Change-Id: I533a61387cb74971e4efc9313d400b66b8aac451
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/221424
Reviewed-by: Damien Neil <dneil@google.com>
Printing the filenames of regenerated files only when the contents
change makes it easier to tell when a change has produced the expected
result in the generated file output.
Change-Id: I74d0020be1de3d287bb8400ba290131008859bf1
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/220356
Reviewed-by: Joe Tsai <joetsai@google.com>
Remove the generated proto packages that already exist in
google.golang.org/genproto. We want to eventually move these
packages here, but it doesn't need to happen yet.
Add a local copy of fieldmaskpb for use in tests.
Refactor proto generation to override import paths using the
M<source>=<import_path> compiler option instead of by patching the
source files.
Change-Id: I8d31f67e931d70140182f19f3e0106111f71c4b4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219598
Reviewed-by: Joe Tsai <joetsai@google.com>
Refactor the fast-path size, marshal, unmarshal, and isinit functions to
take the *coderFieldInfo for the field as input.
This replaces a number of closures capturing field-specific information
with functions taking that information as an explicit parameter.
Change-Id: I8cb39701265edb7b673f6f04a0152d5f4dbb4d5d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/218937
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Change the protoc flags such that when one of our test .proto files
imports another, the filename is consistently specified relative to the
module root.
Change-Id: I690282795cef23347c8794c1c6357e4fe9560d8a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/217762
Reviewed-by: Joe Tsai <joetsai@google.com>
The fast-path unmarshal funcs return the number of bytes consumed.
Change these functions to return an unmarshalOutput struct instead, to
make it easier to add to the results. This is groundwork for allowing
the fast-path unmarshaler to indicate when the unmarshaled message is
known to be initialized.
Change-Id: Ia8c44731a88f5be969a55cd98ea26282f412c7ae
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/215720
Reviewed-by: Joe Tsai <joetsai@google.com>
The encoding/testprotos and reflect/protoregistry/testprotos are
accessible by other modules. Move them under internal/testprotos
to dissuade programmers who are too lazy to use their own test protos
when they need one.
Change-Id: I3dbfbce74e68ef033ec252bed076861cb47dd21e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/214341
Reviewed-by: Damien Neil <dneil@google.com>
Avoid dots and dashes in the directory to avoid issues on
build systems that cannot support them well.
Change-Id: I7ea5e6ce0b16c7158c7e53bcf5c3c1a334fe4718
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/214342
Reviewed-by: Damien Neil <dneil@google.com>
