This updates all generated code to match the contents of the latest
v22.0 release of Protobuf.
This involved a couple of changes to the script that does the sync'ing:
1. The new Protobuf version no longer includes autoconf configuration
and instead requires using bazel to build things.
2. The new Protobuf release does not have an artifact named
"protobuf-all-${VERSION}.tar.gz", but the one named
"protobuf-${VERSION}.tar.gz" has all of the sources and was
sufficient for the regenerate.bash script to complete.
This change does NOT regenerate the protos related to benchmarks.
The Protobuf repo no longer includes benchmarks. The CL removing them
says they are superceded by google/fleetbench:
83c499de86
But that project's proto benchmark files are very different:
https://github.com/google/fleetbench/tree/main/fleetbench/proto
So I commented out those steps in the generation code since the benchmarks
will need some work to reconcile with fleetbench.
This code adds known failing tests for conformance. New test cases were
added in https://github.com/protocolbuffers/protobuf/pull/9534, but the
Go protojson package does not behave according to the new tests. There
is an existing issue in GitHub about this:
https://github.com/golang/protobuf/issues/1208
Change-Id: Iad796ec7889bc2a74b58db5224facf850cd1a1cd
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/469255
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
Fix typos in comments and unexported function name.
Change-Id: I239fff85bc68d7f0ef6f8853acdc9bbe19d86304
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/466535
Reviewed-by: Lasse Folger <lassefolger@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Reviewed-by: Joseph Tsai <joetsai@digital-static.net>
The protobuf wire format is insufficiently self-decribing such that
it is impossible to know whether for sure whether an unknown bytes value
is a sub-message or not. However, protopack is primarily used for debugging
where a best-effort guess is still very useful.
The Message.UnmarshalAbductive unmarshals an unknown bytes value as a message
if it is syntactically well-formed. Otherwise, it is left as is.
Change-Id: I5e2b4b995e2b5eb60942a242558bf4cea1da9891
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/309669
Trust: Joe Tsai <joetsai@digital-static.net>
Reviewed-by: Damien Neil <dneil@google.com>
This updates the dependency on the protobuf toolchain to an unreleased
version that has the go_package option for relevant .proto files in
that repository to reference this module.
Change-Id: Ie1ac8f81a323285efbd14a17d02c7105e810af2d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/235283
Trust: Joe Tsai <joetsai@digital-static.net>
Reviewed-by: Damien Neil <dneil@google.com>
The protowire.{ConsumeBytes,ConsumeString} funcs are identical except
that the latter allocates a string by implicitly converting the []byte.
Avoid using ConsumeString since we can do the conversion ourselves
at a latter point and sometimes avoid the allocation.
Change-Id: Idf31edc013b72ee5ee8461a68d10e57ad461d95c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/263628
Trust: Joe Tsai <joetsai@digital-static.net>
Reviewed-by: Damien Neil <dneil@google.com>
This trivially adds a name for marshalOptions and unmarshalOptions
instead of always having it be underscore.
This assists the use of patches that rely on this variable.
Change-Id: I4c23901c93f0362aff412339912522f92749d659
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/263627
Trust: Joe Tsai <joetsai@digital-static.net>
Reviewed-by: Damien Neil <dneil@google.com>
Move the declaration of the slice variable
as close to its actual use as possible.
Change-Id: Ibe26460b5ca81edbed1073b32faf1c5f5d2c922a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/263719
Trust: Joe Tsai <joetsai@digital-static.net>
Trust: Joe Tsai <joetsai@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
Add an outfile flag that specifies that the contents of
the provided file should generated to stdout.
When using this mode, it only generates one file at a time
and does not modify any files on disk.
Change-Id: I80ba1764a7f0ceb180b8b54f4197dd57fec86434
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/263625
Trust: Joe Tsai <joetsai@digital-static.net>
Reviewed-by: Patrik Nyblom <pnyb@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
Added API:
SourceLocations.ByPath
SourceLocations.ByDescriptor
SourceLocation.Next
SourcePath.String
SourcePath.Equal
We modify compiler/protogen to use SourceLocations.ByDescriptor.
In retrospect, if this had existed during the development of protogen,
we would not have exposed protogen.Location and related fields.
Change-Id: I58f17e59f90b9ba16f0982c4b71c2542e4ff6e75
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/238000
Reviewed-by: Damien Neil <dneil@google.com>
When proto.Unmarshal fails, it is almost always due to being passed
input which is not a wire-format message. (A text-format message, a
message with framing left intact, and so forth.) An error like
"variable length integer overflow" can be confusing in this case, since
it implies the problem is the varint rather than the input being
entirely wrong.
Replace all Unmarshal parse errors with "cannot parse invalid
wire-format data".
Change-Id: Id97253bd39ac604e569df71778194f37b3c86c28
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/244297
Reviewed-by: Joe Tsai <joetsai@google.com>
Add a new TextName accessor that returns the field name that should
be used for the text format. It is usually just the field name, except:
1) it uses the inlined message name for groups,
2) uses the full name surrounded by brackets for extensions, and
3) strips the "message_set_extension" for well-formed extensions
to the proto1 MessageSet.
We make similar adjustments to the JSONName accessor so that it applies
similar semantics for extensions.
The two changes simplifies all logic that wants the humanly readable
name for a field.
Change-Id: I524b6e017fb955146db81819270fe197f8f97980
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/239838
Reviewed-by: Herbie Ong <herbie@google.com>
The genid package unifies the genname, fieldnum, and detectknown
packages into a single package.
Whenever possible use the generated constants rather than
hard-coded literals. This makes it easier to search the entire
module for special logic that deal with well-known types.
Change-Id: I13beff1f4149444a0c0b9e607ebf759657f000f4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/235301
Reviewed-by: Herbie Ong <herbie@google.com>
This is step 2 of 6 in a multi-stage migration
to move the well-known types
from the google.golang.org/genproto module
to the google.golang.org/protobuf module.
The generated Go packages for field_mask.proto, api.proto,
type.proto, and source_context.proto are being moved over
to this module alongside all the other well-known types.
In order to move these types between two modules,
there needs to be a sequence of changes submitted in
decently rapid succession. It is impossible to atomically
make these changes, so a brief breakage is inevitable.
The steps are as follows:
Step 1: Submit a change to cloud.google.com/go/internal/gapicgen
to avoid generating the well-known types. Otherwise, the tool
will undo the changes made in step 3.
See https://code-review.googlesource.com/c/gocloud/+/56810
Step 2: Submit a change to google.golang.org/protobuf that
adds the generated well-known types being migrated to that module.
In order to prevent the situation where a user links in
too old a version of the genproto module such that
duplicate registration occurs for the well-known types,
the registry is specially modified to provide an error
message that instructs users to upgrade the genproto module.
See https://golang.org/cl/234937
Step 3: Submit a change to google.golang.org/genproto that
switches all generated well-known types to be aliases to the
ones declared in google.golang.org/protobuf from the previous step.
This will cause the genproto module to incur an dependency
on an unreleased version of the protobuf module.
See https://github.com/googleapis/go-genproto/pull/372
Step 4: Submit a change to google.golang.org/protobuf that
adds a weak module depdency on the genproto module at the
revision from the previous step.
Step 5: Release google.golang.org/protobuf@v1.24.0.
Step 6: Submit a change to google.golang.org/genproto that
updates the protobuf module dependency to v1.24.0.
Change-Id: I36a19049d2240b67a37dfad20e154505aee7c784
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/234937
Reviewed-by: Damien Neil <dneil@google.com>
Some companies (e.g., Google) run a profiling service where they may
choose to special-case certain symbols in a binary to classify
commonly used libraries like protobufs.
This CL funnels similar functionality through a single function
so that they can be more easily identified. This is by no means a
firm statement that these identifiers will never change names,
but at least the code documents warnings to avoid changing the
name of certain identifiers.
This CL provides the following semi-stable symbol names:
"google.golang.org/protobuf/proto".MarshalOptions.size
"google.golang.org/protobuf/proto".MarshalOptions.marshal
"google.golang.org/protobuf/proto".UnmarshalOptions.unmarshal
"google.golang.org/protobuf/encoding/prototext".MarshalOptions.marshal
"google.golang.org/protobuf/encoding/prototext".UnmarshalOptions.unmarshal
"google.golang.org/protobuf/encoding/protojson".MarshalOptions.marshal
"google.golang.org/protobuf/encoding/protojson".UnmarshalOptions.unmarshal
Merge and Clone are not part of the above set since there is a
possibility that MergeOptions will be added in the future.
We use an unexported method so that we have the freedom to change the
method however we want since profilers do not care about that.
Change-Id: Ia79af260d00125f48139420e1e18a86482bd1829
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/234079
Reviewed-by: Damien Neil <dneil@google.com>
In the upcoming 3.12.x release of protoc, the proto3 language will be
amended to support true presence for scalars. This CL adds support
to both the generator and runtime to support these semantics.
Newly added public API:
protogen.Plugin.SupportedFeatures
protoreflect.FieldDescriptor.HasPresence
protoreflect.FieldDescriptor.HasOptionalKeyword
protoreflect.OneofDescriptor.IsSynthetic
Change-Id: I7c86bf66d0ae56642109beb5f2132184593747ad
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/230698
Reviewed-by: Damien Neil <dneil@google.com>
Now we'll get a generator error if a file is missing a go_package
option.
Change-Id: I89eec716f86956e6c164a61f5531c140b74bc099
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/222378
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
This fixes the conformance test failures which occur because the
conformance protos have not been regenerated.
The generator script has been modified with a sanity check that
files do not exist outside the expected sub-tree.
Change-Id: I473efec4a016f6bc96ddf7e20d54bcf5ff9b55fe
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/223538
Reviewed-by: Damien Neil <dneil@google.com>
Under some rare circumstances registration can deadlock
when lazy descriptor initialization consults the registry.
Move the call triggering the lazy init out of the critical section.
Fixesgolang/protobuf#1052.
Change-Id: Ic266e06b0db99fea65e797b879ce53e5342fff95
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/204804
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Throughout the module options usually expressed as a struct where the
acting function is a method hanging off the options type.
Use this pattern for protogen as well.
Change-Id: I533a61387cb74971e4efc9313d400b66b8aac451
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/221424
Reviewed-by: Damien Neil <dneil@google.com>
Printing the filenames of regenerated files only when the contents
change makes it easier to tell when a change has produced the expected
result in the generated file output.
Change-Id: I74d0020be1de3d287bb8400ba290131008859bf1
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/220356
Reviewed-by: Joe Tsai <joetsai@google.com>
Remove the generated proto packages that already exist in
google.golang.org/genproto. We want to eventually move these
packages here, but it doesn't need to happen yet.
Add a local copy of fieldmaskpb for use in tests.
Refactor proto generation to override import paths using the
M<source>=<import_path> compiler option instead of by patching the
source files.
Change-Id: I8d31f67e931d70140182f19f3e0106111f71c4b4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219598
Reviewed-by: Joe Tsai <joetsai@google.com>
Refactor the fast-path size, marshal, unmarshal, and isinit functions to
take the *coderFieldInfo for the field as input.
This replaces a number of closures capturing field-specific information
with functions taking that information as an explicit parameter.
Change-Id: I8cb39701265edb7b673f6f04a0152d5f4dbb4d5d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/218937
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Change the protoc flags such that when one of our test .proto files
imports another, the filename is consistently specified relative to the
module root.
Change-Id: I690282795cef23347c8794c1c6357e4fe9560d8a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/217762
Reviewed-by: Joe Tsai <joetsai@google.com>
The fast-path unmarshal funcs return the number of bytes consumed.
Change these functions to return an unmarshalOutput struct instead, to
make it easier to add to the results. This is groundwork for allowing
the fast-path unmarshaler to indicate when the unmarshaled message is
known to be initialized.
Change-Id: Ia8c44731a88f5be969a55cd98ea26282f412c7ae
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/215720
Reviewed-by: Joe Tsai <joetsai@google.com>
The encoding/testprotos and reflect/protoregistry/testprotos are
accessible by other modules. Move them under internal/testprotos
to dissuade programmers who are too lazy to use their own test protos
when they need one.
Change-Id: I3dbfbce74e68ef033ec252bed076861cb47dd21e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/214341
Reviewed-by: Damien Neil <dneil@google.com>
Avoid dots and dashes in the directory to avoid issues on
build systems that cannot support them well.
Change-Id: I7ea5e6ce0b16c7158c7e53bcf5c3c1a334fe4718
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/214342
Reviewed-by: Damien Neil <dneil@google.com>
The size calculation for packed repeated extension fields was
considering a zero-length list as encoding to a zero-length
wire.BytesType field, rather than being omitted entirely.
Change-Id: I7d4424a21ca8afd4fa81391caede49cadb4e2505
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/212297
Reviewed-by: Joe Tsai <joetsai@google.com>
Add a new Fuzz message containing all the message types we want to make
available to fuzzers. Previously, testing (for example) required fields
would require modifying the fuzzer; now, it's just a matter of adding a
message with required fields as a field of the top-level Fuzz message.
Add internal/cmd/generate-corpus to codify where the fuzz seed corpus
comes from. This will simplify adding text and json fuzzers.
Rename internal/fuzz/wire to internal/fuzz/wirefuzz to minimize package
name ambiguity. Also, the addition of the Fuzz container message
invalidates the existing corpus, so using a new name seems like a good
idea.
Change-Id: I94f8f64ba93596c8e8cecb4d42bcc5b98c17d838
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/212218
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The fast-path decoder decodes zero-length repeated bytes values as
non-nil, zero-length []bytes. Do the same in the reflection decoder.
This isn't really a correctness issue, since there's no ambiguity about what a
nil entry in a [][]byte means. Still a good idea for consistency, and
retains v1 behavior.
Change-Id: Icd2cb726d14ff1f2b9f142e65756777a359971f3
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/210257
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Various protoreflect methods can return an "empty, read-only" message,
list, or map value. Provide a method to test if a value is one of these.
Fixesgolang/protobuf#966
Change-Id: I793d8426d6e2201755983c06f024412a7e09bc4c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/209018
Reviewed-by: Joe Tsai <joetsai@google.com>
This change performs two optimizations:
* It uses a pre-constructed rangeInfos slice to iterate over
all the fields. This is more performant since iterating over a slice
is faster than iterating over a map. Furthermore, this slice
does not contain fields that are part of a oneof. If a oneof has
N fields, the time to check presence on the oneof is now O(1)
instead of O(N).
* It uses a dense field info slice that is optmized for the common
case where the field number is relatively low and close in value
to the index itself.
We also fix a minor bug in the construction of oneofInfo where
it wasn't treating a typed nil pointer to a wrapper struct as if
it were unset. This ensures WhichOneof and Has always agree.
name old time/op new time/op delta
Reflect/Has-4 7.81µs ± 3% 6.74µs ± 3% -13.61% (p=0.000 n=9+9)
Reflect/Get-4 12.7µs ± 1% 11.3µs ± 4% -10.85% (p=0.000 n=8+10)
Reflect/Set-4 19.5µs ± 5% 17.8µs ± 2% -8.99% (p=0.000 n=10+10)
Reflect/Clear-4 12.0µs ± 4% 10.2µs ± 3% -14.86% (p=0.000 n=9+10)
Reflect/Range-4 6.58µs ± 1% 4.17µs ± 2% -36.65% (p=0.000 n=8+9)
Change-Id: I2c48b4d3fb6103ab238924950529ded0d37f8c8a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/196358
Reviewed-by: Damien Neil <dneil@google.com>
Use internal/detrand in the construction of our error messages.
This alters whether there is one or two spaces following the "proto:" prefix.
While it is easy for users to still work around this mutation,
sit at least forces them to write test infrastructure to more fuzzily
match on error strings.
Change-Id: I4ddca717526ee3fc4dbb1e0b36cfca8c6e0df36d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/194038
Reviewed-by: Herbie Ong <herbie@google.com>
Remove the Go type from consideration when creating Value codecs, as it
is unnecessary. Value codecs convert between wire form and Values,
while Converters convert between Values and the Go type.
Change-Id: Iaa4bc7db81ad0a29dabd42c2229e6f33a0c91c67
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/193457
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
This interface is only ever used within the impl package.
Unexport it to avoid potential abuse by users.
Change-Id: I58d0cc2e8cb01c8c9d36cdec469c8d2196c4f836
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/193197
Reviewed-by: Damien Neil <dneil@google.com>
Change the storage type of ExtensionField from interface{} to
protoreflect.Value.
Replace the codec functions operating on interface{}s with ones
operating on Values.
Values are potentially more efficient, since they can represent
non-pointer types without allocation. This also reduces the number of
types used to represent field values.
Additionally, this change lays groundwork for changing the
user-visible representation of repeated extension fields from
*[]T to []T. The storage type for extension fields must support mutation
(thus *[]T currently); changing the storage type to a Value permits this
without the need to introduce yet another view on field values.
Change-Id: Ida336be14112bb940f655236eb58df21bf312525
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/192218
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Passing a non-pointer type to protoreflect.NewValue causes an
unnecessary allocation in order to store the value in an interface{}.
While this allocation could be avoided by a smarter compiler, no such
compiler exists today.
Add functions for creating new values of a specific type, avoiding the
allocation. (And also adding a small amount of type safety, although
this is unlikely to be important.)
Update the proto and internal/impl packages to use these functions.
Change-Id: Ic733de22ddf19c530189166c853348e1b54b7391
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/191457
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
