Before this change the implementation would ignore editions options set on
extension declarations.
This change also add much more coverage for valid messages.
Change-Id: I0e7029b1430a39859cdd4a93667ac20c7e0ff20e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/565396
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
This fixes more minor issue when constructing the descriptor in that
delimited encoded message fields are not considered properly as groups.
Change-Id: I714a227a0f8d256fa7430c526844e8e94acdda33
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/564816
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
There are several minor fixes:
* fix the unmarshaler for the field options (it used the wrong field numbers).
* fix the unmarshaler to always initialize the file options with the appropriate defaults.
* fix the coder selection to work properly for fields with implicit presence tracking (they used the coders for explicit tracking).
* fix the dynamic message created from a descriptor for an editions using proto to use HasPresence instead of a syntax check for proto3
Change-Id: Ic13bc22a71aa9f93b476e5edd650d4d2ab5dcb98
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/564455
Auto-Submit: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
This change add parsing editions specific descriptor parts to
internal/filedesc which makes it possible to load file descriptors of
protos using editions at runtime.
Change-Id: I7a0c23303acddd2cff115859d4cf82bf0102b14c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/563615
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
This change adds the first Go specific editions feature and the
associated proto. The feature is used to control if the legacy
UnmarshalJSON method should be generated for enums. This change only
affects protos using editions.
More tests will be added in a followup change.
Change-Id: Ifb62454d7568bd6d90d0b93f8953adcfe46c45fd
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/561938
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
+ The previous commit forgot to resolve the LegacyRequired feature
Change-Id: I05a2847be3d52277c65df004363ea3c87944364e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/555976
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Before this CL we only used features where they were applicable, e.g.
file features were only set on the file descriptor and field features
only on the fields descriptor. After this CL all descriptors (file,
message and field) contain always all features and when initializing
either of them we merge the features specified by options with the
features of the parent descriptor and use the result as the effective
feature. This way we initialize all the descriptors the same way and the
proto spec can change on which level features can be specified without
us having to change the code (at the moment most features can only be
specified on file or field level, in the future message might also be
possible).
Change-Id: I822202bf6a2bba5167bf9622c91a65c4523e78f9
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/555975
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
This is the first of multiple changes to add protobuf editions support
to the Go implementation.
This change includes the feature resolutions for protobuf editions.
The plugin does not yet report editions support and protoc would fail
if it invoked this plugin with a proto that uses editions.
Change-Id: I7d31909366c3433b21abab74ec92263e08145434
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/547315
Auto-Submit: Lasse Folger <lassefolger@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Commit-Queue: Lasse Folger <lassefolger@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
This updates all generated code to match the contents of the latest
v22.0 release of Protobuf.
This involved a couple of changes to the script that does the sync'ing:
1. The new Protobuf version no longer includes autoconf configuration
and instead requires using bazel to build things.
2. The new Protobuf release does not have an artifact named
"protobuf-all-${VERSION}.tar.gz", but the one named
"protobuf-${VERSION}.tar.gz" has all of the sources and was
sufficient for the regenerate.bash script to complete.
This change does NOT regenerate the protos related to benchmarks.
The Protobuf repo no longer includes benchmarks. The CL removing them
says they are superceded by google/fleetbench:
83c499de86
But that project's proto benchmark files are very different:
https://github.com/google/fleetbench/tree/main/fleetbench/proto
So I commented out those steps in the generation code since the benchmarks
will need some work to reconcile with fleetbench.
This code adds known failing tests for conformance. New test cases were
added in https://github.com/protocolbuffers/protobuf/pull/9534, but the
Go protojson package does not behave according to the new tests. There
is an existing issue in GitHub about this:
https://github.com/golang/protobuf/issues/1208
Change-Id: Iad796ec7889bc2a74b58db5224facf850cd1a1cd
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/469255
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
This method existed in the past to avoid a cyclic dependency
on descriptorpb from the legacy proto package.
This restriction no longer applies.
Change-Id: I5c61fa7de370063a126d8ea15c376b0882417b43
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/259900
Reviewed-by: Damien Neil <dneil@google.com>
Trust: Joe Tsai <thebrokentoaster@gmail.com>
FormatList previously formatted output inconsistently when called on
EnumDescriptors, ServiceDescriptors and ExtensionDescriptors: other
types included the type name before {{; these types did not.
Add a default case that includes the type in these (and any other)
cases also.
Expanded tests to demonstrate and verify the new behaviour.
Change-Id: I9f53cb6548c4448a12c17d519639357ebdfa809b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/286132
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Trust: Joe Tsai <joetsai@google.com>
Trust: Matthew Dempsky <mdempsky@google.com>
Added API:
SourceLocations.ByPath
SourceLocations.ByDescriptor
SourceLocation.Next
SourcePath.String
SourcePath.Equal
We modify compiler/protogen to use SourceLocations.ByDescriptor.
In retrospect, if this had existed during the development of protogen,
we would not have exposed protogen.Location and related fields.
Change-Id: I58f17e59f90b9ba16f0982c4b71c2542e4ff6e75
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/238000
Reviewed-by: Damien Neil <dneil@google.com>
Add a new TextName accessor that returns the field name that should
be used for the text format. It is usually just the field name, except:
1) it uses the inlined message name for groups,
2) uses the full name surrounded by brackets for extensions, and
3) strips the "message_set_extension" for well-formed extensions
to the proto1 MessageSet.
We make similar adjustments to the JSONName accessor so that it applies
similar semantics for extensions.
The two changes simplifies all logic that wants the humanly readable
name for a field.
Change-Id: I524b6e017fb955146db81819270fe197f8f97980
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/239838
Reviewed-by: Herbie Ong <herbie@google.com>
This adds the use of the staticcheck tool to check our code.
The use of staticcheck would have caught a bug in CL/211737.
Also, we adjust one minor code patterns to avoid triggering
one of the staticcheck checks.
This CL also modifies the integration script to check SHA256
checksums for archives downloaded from a third-party source.
Change-Id: Ieb69ae79464de62970448195c77cdd88e21fc48f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/212220
Reviewed-by: Damien Neil <dneil@google.com>
The genid package unifies the genname, fieldnum, and detectknown
packages into a single package.
Whenever possible use the generated constants rather than
hard-coded literals. This makes it easier to search the entire
module for special logic that deal with well-known types.
Change-Id: I13beff1f4149444a0c0b9e607ebf759657f000f4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/235301
Reviewed-by: Herbie Ong <herbie@google.com>
In the upcoming 3.12.x release of protoc, the proto3 language will be
amended to support true presence for scalars. This CL adds support
to both the generator and runtime to support these semantics.
Newly added public API:
protogen.Plugin.SupportedFeatures
protoreflect.FieldDescriptor.HasPresence
protoreflect.FieldDescriptor.HasOptionalKeyword
protoreflect.OneofDescriptor.IsSynthetic
Change-Id: I7c86bf66d0ae56642109beb5f2132184593747ad
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/230698
Reviewed-by: Damien Neil <dneil@google.com>
Remove a stray bit of punctuation that crept into one of the license
headers and got copied around everywhere.
Change-Id: Iebe4e882650ab6dab28f132b5e324e2ab0b99a73
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/220339
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Look up a ProtoFile via a message's ParentFile method. Makes it easier
to run the test in environments (blaze, bazel) where the file path may
have changed.
Change-Id: I824f8412ef7db8299961e8df6edac14a13f3e263
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/217761
Reviewed-by: Joe Tsai <joetsai@google.com>
Avoid dots and dashes in the directory to avoid issues on
build systems that cannot support them well.
Change-Id: I7ea5e6ce0b16c7158c7e53bcf5c3c1a334fe4718
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/214342
Reviewed-by: Damien Neil <dneil@google.com>
The JSONName constructor returns a struct value which shallow copies
a sync.Once within it; this is a dubious pattern.
Instead, add a jsonName.Init method to initialize the value.
Change-Id: I190a7239b1b62a8041ee7e4e09c0fe37b64ff623
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/213237
Reviewed-by: Damien Neil <dneil@google.com>
Avoid a deadlock when registering a legacy ExtensionType, caused by
initialization of the "internal/impl".ExtensionInfo calling IsMessageSet
on the MessageDescriptor of the type being extended.
We can avoid this deadlock either by initializing the ExtensionType
outside of the GlobalTypes mutex, or by moving IsMessageSet to L1
initialization of the MessageDescriptor so that it doesn't trigger lazy
init.
CL 204804 takes the former approach; this CL takes the latter.
Change-Id: Idfc1ed36a23a139839290ea32492142a17f68cf5
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/205957
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Add type-safe methods to register message, enum, and extension types.
Deprecate the NewTypes function and the (*Types).Register method.
Add (*File).RegisterFile and deprecate the NewFiles function and
the (*File).Register method.
Updates golang/protobuf#963
Change-Id: Ie89e77526e0874539e9bd929ca0ba8d758e65a6e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/199898
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
If an enum dependency is unresolved, there will be no enum values.
In such a case, just fall back to using number 0.
No tests because this situation is hard to replicate.
Change-Id: I41f5e1a5c6c8afd1ab6173d2dcfa33e611fa9560
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/195982
Reviewed-by: Damien Neil <dneil@google.com>
This CL:
* Make the meaning of impl/ExtensionInfo.goType consistent. Before,
it was sometimes a T and other times a []T depending on the current
state of initialization. Change it so that it is the constructor's
responsibility to pass in a []T if it is repeated.
* Make internal/filetype responsible for constructing a []T for
repeated extension fields.
* Makes filedesc/Extension.Cardinality one of the eagerly initialized
pieces of information since it is useful to internal/filetype.
* Unify ExtensionInfo.desc and ExtensionInfo.tdesc.ExtensionField,
which held the same information.
* Remove the internal implementation for impl.X.ExtensionDescFromType
since we are dropping support for this from v1.
Change-Id: Ie95c4de66cd674c1d886da4f63b133b7d763c7ef
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/195777
Reviewed-by: Damien Neil <dneil@google.com>
Suppose you have a generated message with an enum field that
has a default value, where the enum is declared in another proto file
that does not register its descriptors.
In such a situation, the enum dependency cannot be resolved and will
be left as a placeholder.
When unmarshalDefault is called, it will attempt to parse the enum
name stored as the default value, but panic since the placeholder enum
has no enum values known to it.
Instead of panicking, use a placeholder enum value that preserves the
name of default enum value and just use the enum number of 0.
This is not ideal, but is a better alternative than panicking.
Change-Id: I5ff6c351adbdd6fe7b41f2d4f215be4aa09e751f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/192500
Reviewed-by: Damien Neil <dneil@google.com>
Returning an anonymous struct is tickling a gccgo bug:
https://github.com/golang/go/issues/33866
Change to a named type. We could make the type unexpoerted, but this is
an internal package anyway so leave it exported for general style
cleanliness.
Change-Id: Idf33f1e354c0e07066ab68640308af1498a01447
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/191960
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Reduce the stutter in the name since the type of Builder
is obvious from the package it is from.
Change-Id: I0046a5122717536cc6bb5ebdb32b67a1560cfc23
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189020
Reviewed-by: Damien Neil <dneil@google.com>
Suppose some init logic performs protobuf reflection.
If so, it will cause the lazy descriptor init logic to be executed.
This is problematic for weak fields since we can not be certain that
all weak references have been registered at this point in time.
This is a fundamental issue with with weak dependencies,
since it means that we cannot enforce any ordering constraints
on the weak dependency unless we directly import the weakly referenced package
(which would defeat the point of weak imports).
Alleviate the problem by pushing evaluation of weak reference to
actual usage. This does not completely fix the problem,
but signifcantly reduces the probability of it being problematic.
In general, people should avoid interacting with weak fields at init time.
Change-Id: Ie5957ddedd61333e72ee9a1bba0c53dace65547c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/188982
Reviewed-by: Damien Neil <dneil@google.com>
It is possible for filedesc to construct a lazy options decoder before
the descriptor package has been imported. For example, top-level enum
values are eagerly decoded, so a generated proto package can construct a
lazy options decoder for an enum value at init time.
Don't close over the variables in descopts. Instead, close over a pointer
to the variable.
Panic with an informative message in the case where options are decoded
before the descriptor package has been initialized.
Change-Id: I277a57602b083cb7bbf92c8114c50b467e59521f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/188820
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The ignoreConflict function provides the ability to ignore certain conflicts.
By default, all conflicts are ignored with a log message produced instead.
Change-Id: I67fe56eef492e12421e5c8cb8d618dc2a46c82ed
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186658
Reviewed-by: Damien Neil <dneil@google.com>
Rather than panicking at init time due to registration failures,
print a warning to stderr. Historically, the Go protobuf implementation
has not been strict about registration conflicts, which has led users
to unknowningly tolerating conflicts that may or may not expose
themselvs as a bug.
Registration conlicts now produce a log message:
<<<
2019/07/17 17:36:42 WARNING: proto: file "path/to/example.proto" is already registered
previously from: "example.com/company/example_proto"
currently from: "example.com/user/example_proto"
A future release will panic on registration conflicts.
>>>
Change-Id: I2d583f04977c8bc8cb6bbd33d239277690bbec54
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186181
Reviewed-by: Damien Neil <dneil@google.com>
The v2 implementation strictly enforces that there are no conflicts at
all in the protobuf namespace unlike the prior v1 implementation.
This change is almost certainly going to cause loud failures for users
that were unknowingly tolerating registration conflicts.
We modify internal/filedesc to be able to record the Go package path
that the file descriptor is declared within. This information is used
by reflect/protoregistry to print both the previous Go package that
registered some declaration, and current Go package that is attempting
to register some declaration.
Change-Id: Ib5eb21c1c98495afc51aa08bd4404bd9d64b5b57
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186177
Reviewed-by: Damien Neil <dneil@google.com>
In 2014, when proto3 was being developed, there were a number of early
adopters of the new syntax. Before the finalization of proto3 when
it was released in open-source in July 2016, a decision was made to
strictly validate strings in proto3. However, some of the early adopters
were already using invalid UTF-8 with string fields.
The google.protobuf.FieldOptions.enforce_utf8 option only exists to support
those grandfathered users where they can opt-out of the validation logic.
Practical use of that option in open source is impossible even if a user
specifies the proto1_legacy build tag since it requires a hacked
variant of descriptor.proto that is not externally available.
This CL supports enforce_utf8 by modifiyng internal/filedesc to
expose the flag if it detects it in the raw descriptor.
We add an strs.EnforceUTF8 function as a centralized place to determine
whether to perform validation. Validation opt-out is supported
only in builds with legacy support.
We implement support for validating UTF-8 in all proto3 string fields,
even if they are backed by a Go []byte.
Change-Id: I9c0628b84909bc7181125f09db730c80d490e485
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186002
Reviewed-by: Damien Neil <dneil@google.com>
This implements generation of and reflection support for weak fields.
Weak fields are a proto1 feature where the "weak" option can be specified
on a singular message field. A weak reference results in generated code
that does not directly link in the dependency containing the weak message.
Weak field support is not added to any of the serialization logic.
Change-Id: I08ccfa72bc80b2ffb6af527a1677a0a81dcf33fb
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185399
Reviewed-by: Damien Neil <dneil@google.com>
This adds minimal support for preserving the source context information.
Change-Id: I4b3cac9690b7469ecb4e5434251a809be4d7894c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/183157
Reviewed-by: Damien Neil <dneil@google.com>
Usage of these is pervasive in code which works with proto2, and proto2
will be with us for a long, long time to come. Move them to the proto
package.
Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Create a new internal/strs package that unifies common functionality:
* Since protobuf itself pseudo-specifies at least 4 different camel-case
and snake-case conversion functions, we define all variants in one place.
* We move the internal/filedesc.nameBuilder function to this package.
We simplify its implementation to not depend on a strings.Builder fork
under the hood since the semantics we desire is simpler than what
strings.Builder provides.
* We use strs.Builder in reflect/protodesc in its construction of all
the full names. This is perfect use case of strs.Builder since all
full names within a file descriptor share the same lifetime.
* Add an UnsafeString and UnsafeBytes cast function that will be useful
in the near future for optimizing encoding/prototext and encoding/protojson.
Change-Id: I2cf07cbaf6f72e5f9fd6ae3d37b0d46f6af2ad59
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185198
Reviewed-by: Damien Neil <dneil@google.com>
The protoreflect.Descriptor.Options method is currently documented as
returning a reference to the options, where the user must not mutate
the returned message. This changes internal/filedesc to avoid returning
a copy of the options by caching the first unmarshal.
See golang/protobuf#877
Change-Id: I15701d33fbda7535b21b2add72628b02992c373f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185197
Reviewed-by: Benny Siegert <bsiegert@gmail.com>
Hyrum's Law dictates that if we do not prevent naughty behavior,
people will rely on it. If we do not validate that the provided
file descriptor is correct today, it will be near impossible
to add proper validation checks later on.
The logic added validates that the provided file descriptor is
correct according to the same semantics as protoc,
which was reversed engineered to derive the set of rules implemented here.
The rules are unfortunately complicated because protobuf is a language
full of many non-orthogonal features. While our logic is complicated,
it is still 1/7th the size of the equivalent C++ code!
Change-Id: I6acc5dc3bd2e4c6bea6cd9e81214f8104402602a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/184837
Reviewed-by: Damien Neil <dneil@google.com>
