248 Commits

Author SHA1 Message Date
Lasse Folger
f2cb7f136e encoding/protojson: add protojson editions tests including fuzztests
Change-Id: I478bf6a945cb2c86c71fd20b64dedb4b2e585f1d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/566035
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
2024-02-23 08:35:31 +00:00
Lasse Folger
f9eb6c71e7 go_features.proto: move .proto file next to generated .pb.go file
Change-Id: Ia24ca1ad2186bcc95681081d2719ee834b385728
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/565955
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
2024-02-22 08:56:15 +00:00
Lasse Folger
997075a7f5 reflect/protodesc: add editions support
Change-Id: Ie7fee936bdb480802cbaca34cf63c0dad34590b8
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/564815
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
2024-02-19 07:03:53 +00:00
Lasse Folger
e39100579e reflect/protodesc: propagate field_presence legacy_required properly
Change-Id: I3f067eb564349bc1ff867a03d9f1aa94b3428236
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/563735
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
2024-02-13 15:44:19 +00:00
Lasse Folger
020ec10f7b internal/filedesc: implement runtime editions support
This change add parsing editions specific descriptor parts to
internal/filedesc which makes it possible to load file descriptors of
protos using editions at runtime.

Change-Id: I7a0c23303acddd2cff115859d4cf82bf0102b14c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/563615
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-02-13 12:20:49 +00:00
Lasse Folger
f387b036d8 editions: add some test protos for editions
I'll add the runtime implementations and tests in a followup change to
make it easier to review.

Change-Id: I2917e0d40a99e81799d89a66584a680ed7e8dbf7
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/561939
Auto-Submit: Lasse Folger <lassefolger@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-02-12 14:14:45 +00:00
Lasse Folger
2040e8671b all: implement Go-specific GenerateLegacyUnmarshalJSON
This change adds the first Go specific editions feature and the
associated proto. The feature is used to control if the legacy
UnmarshalJSON method should be generated for enums. This change only
affects protos using editions.

More tests will be added in a followup change.

Change-Id: Ifb62454d7568bd6d90d0b93f8953adcfe46c45fd
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/561938
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
2024-02-12 13:54:51 +00:00
Lasse Folger
99e193e5dd all: update to protobuf-26.0-rc2 to implement go edition features
Change-Id: I190b1550e3e2e630392fed155716af9d651c74bc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/562615
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
2024-02-08 09:23:48 +00:00
Lasse Folger
b70f02be53 editions: minor fix to resolve editions features correctly
+ The previous commit forgot to resolve the LegacyRequired feature

Change-Id: I05a2847be3d52277c65df004363ea3c87944364e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/555976
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-01-16 15:43:50 +00:00
Lasse Folger
9e454d6514 internal/filedesc: refactor editions feature resolution
Before this CL we only used features where they were applicable, e.g.
file features were only set on the file descriptor and field features
only on the fields descriptor. After this CL all descriptors (file,
message and field) contain always all features and when initializing
either of them we merge the features specified by options with the
features of the parent descriptor and use the result as the effective
feature. This way we initialize all the descriptors the same way and the
proto spec can change on which level features can be specified without
us having to change the code (at the moment most features can only be
specified on file or field level, in the future message might also be
possible).

Change-Id: I822202bf6a2bba5167bf9622c91a65c4523e78f9
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/555975
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
2024-01-16 12:12:51 +00:00
Michael Stapelberg
26a52f39da reflect/protodesc: fix packed field conditional
Before this change, calling dynamicpb.NewMessage() on a
protoreflect.MessageDescriptor that uses Editions would fail with:

message field "[…].relatedqueries" is not packable

(While Go Protobuf does not yet support Editions, C++ programs
can save descriptors to a file that is later loaded by a Go program.)

This was an oversight in commit e8baad6b6c9e2bb1c48e4963866248d4f35d4fd7.

Change-Id: I45d2b07a85ee40cb7c028098b81c4e76bf2ff555
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/554896
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Cassondra Foesch <cfoesch@gmail.com>
Reviewed-by: Christian Höppner <hoeppi@google.com>
Auto-Submit: Michael Stapelberg <stapelberg@google.com>
2024-01-09 15:05:22 +00:00
Michael Stapelberg
7c85df2652 reflect/protodesc: fix group validation: not supported in proto3
Before this change, calling dynamicpb.NewMessage() on a
protoreflect.MessageDescriptor that uses Editions would fail with:

message field "[…].relatedqueries" is an invalid group: proto: invalid under
proto2 semantics

(While Go Protobuf does not yet support Editions, C++ programs
can save descriptors to a file that is later loaded by a Go program.)

Given that groups are only supported in proto2, but not in proto3
(see https://protobuf.dev/editions/features/#message_encoding),
I think the conditional meant to say "proto3 semantics".

Change-Id: I00368e4433ca60bb883b726e34ee6b8862cfd1e7
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/554955
Reviewed-by: Cassondra Foesch <cfoesch@gmail.com>
Reviewed-by: Christian Höppner <hoeppi@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-01-09 14:54:38 +00:00
Lasse Folger
e8baad6b6c reflect/protodesc: support editions feature resolution
This is the first of multiple changes to add protobuf editions support
to the Go implementation.
This change includes the feature resolutions for protobuf editions.
The plugin does not yet report editions support and protoc would fail
if it invoked this plugin with a proto that uses editions.

Change-Id: I7d31909366c3433b21abab74ec92263e08145434
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/547315
Auto-Submit: Lasse Folger <lassefolger@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Commit-Queue: Lasse Folger <lassefolger@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
2023-12-08 13:18:14 +00:00
David Chase
c9d255971a reflect/protoreflect: use go1.20 unsafe interface for strings and slices
This should be slightly more efficient and insulate this code from
changes to the internal representation/alignment of slices and strings.

The interface header lacks an unsafe constructor but is also tweaked
to follow any future alignment changes, if they should occur.

Change-Id: I54a3a93f248d3eef883d5ebf9ed75cc1f5c20977
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/547336
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Run-TryBot: David Chase <drchase@google.com>
Auto-Submit: Michael Stapelberg <stapelberg@google.com>
2023-12-06 13:41:21 +00:00
Josh Humphries
a8317fb7c5 reflect/protodesc: don't generate invalid syntax values
Fixes golang/protobuf#1575

When converting a protoreflect.FileDescriptor
to a FileDescriptorProto, don't emit an invalid syntax.

This can happen for placeholder files, which represent
missing dependencies of a file.

Change-Id: I1d2ae1bd9f89c8a92e751871ce5c02727505b5e2
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/540455
Reviewed-by: Lasse Folger <lassefolger@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
2023-11-08 07:09:48 +00:00
Josh Humphries
f4a6c1f6e5 types/descriptorpb: regenerate using latest protobuf v25.0-rc2 release candidate
This updates all generated code to match the contents of the latest
v25.0-rc2 release candidate of Protobuf. This provides access to the
various new fields and types defined in google/protobuf/descriptor.proto.

Change-Id: I209f8509f2d3e9a0ac7d9e134caef38c48a0d16b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/538075
Reviewed-by: Lasse Folger <lassefolger@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
2023-10-27 08:25:48 +00:00
Joe Tsai
f9212a8dfa all: modernize documentation
Modernize the documentation across the entire module
to make use of the newer ability to linkify declarations.

Change-Id: I440f9a3f025ec6fadfd9cba59b1dfb57028bf3f1
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/309430
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
2023-09-05 14:55:28 +00:00
Josh Humphries
6d0a5dbd95 types/descriptorpb: regenerate using latest protobuf v24.0-rc2 release candidate
This updates all generated code to match the contents of the latest
v24.0-rc2 release candidate of Protobuf. This provides access to the
various new fields and types defined in google/protobuf/descriptor.proto.

Change-Id: I8ec56b308dd7eed52fb9b17b258a45d0669c47ff
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/512795
Reviewed-by: Lasse Folger <lassefolger@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
2023-07-27 12:38:59 +00:00
Damien Neil
736947c4de all: update protobuf release version to fix macOS tests
Update to a protobuf version that contains a fix for
https://github.com/protocolbuffers/protobuf/issues/12173
(macOS build error).

Set "CC=clang" when building the protobuf repo on macOS,
which fixes build errors on my machine (macOS 13.3.1, M1).

Change-Id: Id8abb054479454b7ccbded5a6fa4d74f3d3c236d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/489315
Run-TryBot: Damien Neil <dneil@google.com>
Reviewed-by: Joseph Tsai <joetsai@digital-static.net>
Reviewed-by: Lasse Folger <lassefolger@google.com>
2023-04-26 20:34:47 +00:00
Josh Humphries
bc1253ad37 types/descriptorpb: regenerate using latest protobuf v22.0 release
This updates all generated code to match the contents of the latest
v22.0 release of Protobuf.

This involved a couple of changes to the script that does the sync'ing:
  1. The new Protobuf version no longer includes autoconf configuration
     and instead requires using bazel to build things.
  2. The new Protobuf release does not have an artifact named
     "protobuf-all-${VERSION}.tar.gz", but the one named
     "protobuf-${VERSION}.tar.gz" has all of the sources and was
     sufficient for the regenerate.bash script to complete.

This change does NOT regenerate the protos related to benchmarks.
The Protobuf repo no longer includes benchmarks. The CL removing them
says they are superceded by google/fleetbench:
    83c499de86
But that project's proto benchmark files are very different:
    https://github.com/google/fleetbench/tree/main/fleetbench/proto
So I commented out those steps in the generation code since the benchmarks
will need some work to reconcile with fleetbench.

This code adds known failing tests for conformance. New test cases were
added in https://github.com/protocolbuffers/protobuf/pull/9534, but the
Go protojson package does not behave according to the new tests. There
is an existing issue in GitHub about this:
    https://github.com/golang/protobuf/issues/1208

Change-Id: Iad796ec7889bc2a74b58db5224facf850cd1a1cd
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/469255
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
2023-02-22 09:33:03 +00:00
Oleksandr Redko
246af0d153 internal,reflect: correct typos
Fix typos in comments and unexported function name.

Change-Id: I239fff85bc68d7f0ef6f8853acdc9bbe19d86304
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/466535
Reviewed-by: Lasse Folger <lassefolger@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Reviewed-by: Joseph Tsai <joetsai@digital-static.net>
2023-02-09 09:27:51 +00:00
Oleksandr Redko
49eaa78c6c all: update links to Protocol Buffer documentation
https://developers.google.com/protocol-buffers/ states that
this site will be deprecated January 2023 and provides
https://protobuf.dev/ as a link to the new location.

Generated .pb.go files are created by generate.bash using Go 1.18,
so leave them unchanged for now.

Fixes golang/protobuf#1520.

Change-Id: I1ddf6f329c96ba05d7c7d8562ad2945f479ee09b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/466375
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
2023-02-08 13:52:20 +00:00
Joe Tsai
b2a7dfe48b reflect/protoreflect: add Value.Equal method
The Value.Equal method compares Values in a way that is
analogous to the reflect.DeepEqual function.

Most of the implementation is moved from the "proto" package,
with proto.Equal implemented in terms of protoreflect.Value.Equal.

Change-Id: Ie9d5f6c073dc49fa59f78385c441db8137de5ec5
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/450775
TryBot-Bypass: Heschi Kreinick <heschi@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Oleksii Prokopchuk <prokopchuk@google.com>
Reviewed-by: Christian Höppner <hoeppi@google.com>
Run-TryBot: Lasse Folger <lassefolger@google.com>
2022-12-01 15:36:18 +00:00
Michael Stapelberg
7a48e2b662 protoreflect: fix list example code
Change-Id: I2c8afba104ae97be5e575226a47f6a7f20198abe
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/431798
Reviewed-by: Nicolas Hillegeer <aktau@google.com>
2022-09-20 08:06:00 +00:00
Josh Humphries
692f4a24f8 types/descriptorpb: update *.pb.go to use latest protoc release, 21.5
I updated the protoc version to 21.5 and ran regenerate.bash.

This updates the various *.pb.go files to use the sources from
the latest protobuf release. In particular, this updates the
descriptorpb, so it includes some changes to descriptor.proto
in the past year and a half since this was last re-generated.
The change also pulled over several other generated changes.

Change-Id: Icd9c4e5d47ba85e1f4bae1d0a025f8dec15a3014
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/426054
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
2022-08-27 06:10:12 +00:00
Damien Neil
b0a944684d all: reformat with go1.19 gofmt
Apply go1.19 gofmt to non-generated files.

Generated .pb.go files are created by generate.bash using Go 1.18,
so leave them unchanged for now.

Change-Id: Ied36c83cf99704988d059bf0412e677f0fbc71b0
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/418676
Reviewed-by: Ian Lance Taylor <iant@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
2022-07-21 16:01:13 +00:00
Koichi Shiraishi
784c482554 all: remove shorthand import aliases
Change-Id: I62ec79419354a2c65ac69e4bcc1cc864a6a40a1a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/407934
Reviewed-by: Lasse Folger <lassefolger@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
2022-05-24 20:05:50 +00:00
Joe Tsai
8a7ba0762c reflect/protoreflect: improve documentation for message types
There are a number of message types, which can be confusing.
Expand the documentation. In particular, use Go's reflection model
as an analogy, which helps users if they are at least familiar
with how Go reflection works. After all, protobuf reflection
was modeled after Go reflection to a large degree.

Change-Id: I388ae41c792905145ccc86dc938b901a2b79f408
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/399654
Reviewed-by: Damien Neil <dneil@google.com>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Auto-Submit: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Joseph Tsai <joetsai@digital-static.net>
Auto-Submit: Joseph Tsai <joetsai@digital-static.net>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
2022-04-12 06:05:53 +00:00
Lasse Folger
3992ea83a2 all: implement depth limit for unmarshaling
+ This change introduce a default and configurable depth limit for
  proto.Unmarshal. If a message is nested deeper than the limit,
  unmarshaling will fail. There are two ways to nest messages. Either by
  having fields which are message types itself or by using groups.
+ The default limit is 10,000 for now. This might change in the future
  to align it with other language implementation (C++ and Java use 100
  as limit).
+ If pure groups (groups that don't contain message fields) are nested
  deeper than the default limit the unmarshaling fails with:
  proto: cannot parse invalid wire-format data
+ Note: the configured limit does not apply to pure groups.
+ This change is introduced to improve security and robustness. Because
  unmarshaling is implemented using recursion it can lead to stack overflows
  for certain inputs. The introduced limit protects against this.
+ A secondary motivation for this limit is the alignment with other
  languages. Protocol buffers are a language interoperability mechanism
  and thus either all implementations should accept the input or all
  implementation should reject the input.

Change-Id: I14bdb44d06e4bd1aa90d6336c2cf6446003b2037
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/385854
Trust: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Damien Neil <dneil@google.com>
Trust: Damien Neil <dneil@google.com>
Reviewed-by: Nicolas Hillegeer <aktau@google.com>
Reviewed-by: Chressie Himpel <chressie@google.com>
2022-02-17 17:07:31 +00:00
Damien Neil
3a9e1dc314 all: gofmt all
Change-Id: I1d42c8d784440a0b7e40b4f0b8da54be0fb338a6
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/370054
Trust: Damien Neil <dneil@google.com>
Run-TryBot: Damien Neil <dneil@google.com>
Reviewed-by: Joseph Tsai <joetsai@digital-static.net>
Reviewed-by: Florian Zenker <floriank@google.com>
2022-01-04 22:57:54 +00:00
Damien Neil
05be61fde3 reflect/protoreflect: add more docs on Value aliasing
Fixes golang/protobuf#1346

Change-Id: I9deb36e1c5e4f28c1c7e99ca64260c7a86af4ea1
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/339569
Trust: Damien Neil <dneil@google.com>
Trust: Joe Tsai <joetsai@digital-static.net>
Run-TryBot: Damien Neil <dneil@google.com>
Reviewed-by: Joe Tsai <joetsai@digital-static.net>
2021-08-05 18:00:43 +00:00
Damien Neil
21e33cc910 reflect/protoregistry: restore conflicting file names check
There are inconsistencies between implementations on what happens when
a single program contains generated code for multiple files with the
same source path. At least one canonical implementation (C++) will fail
at link time. Others print warnings. Some silently resolve the registry
conflict in favor of one file or the other. The protobuf maintainers
agree, however, that the desired behavior is for this condition to be an
error.

Updates golang/protobuf#1122

Change-Id: I716708f16ef90210bdfceb0888691e47783df172
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/322729
Trust: Damien Neil <dneil@google.com>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2021-06-02 01:04:00 +00:00
Damien Neil
febffdd88e reflect/protoregistry: permit conflicting file names
Remove the requirement that all registered files have unique names.

Files.FindFileByPath now returns an error if multiple files match
the path. Files.RangeFiles iterates over all files, including ones with
the same name.

Unlike descriptor names, file names are not part of any of the wire
encodings. Conflicts in file names only affect file lookups by path,
which is an uncommon operation.

The Files.RegisterFile documentation already states (inaccurately),
"It is permitted for multiple files to have the same file path."
This change brings the behavior of RegisterFile in line with the
documentation.

Fixes golang/protobuf#1122

Change-Id: I2750e35736fd34def23fa6ab6d6a829095a65a26
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/321531
Trust: Damien Neil <dneil@google.com>
Run-TryBot: Damien Neil <dneil@google.com>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2021-05-25 00:53:49 +00:00
sryoya
0e358a402f reflect/protorange: fix typo of func name in comment
Change-Id: I80ffc34e29757e0cbdccdd530404fa215cfaf75f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/317989
Reviewed-by: Alberto Donizetti <alb.donizetti@gmail.com>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Trust: Alberto Donizetti <alb.donizetti@gmail.com>
2021-05-07 17:05:00 +00:00
Joe Tsai
e4fcb9f7e7 reflect: add protopath and protorange packages
The protopath package provides a means to programmatically represent
a sequence of protobuf reflection operations.
The protorange package traverses through a message and
calls a user-provided function as it iterates.

This feature sets the groundwork for the often requested feature
of being able to exclude certain fields when merging or serializing.

package protopath
    type Path []Step
    type Step struct{ ... }
        func Root(protoreflect.MessageDescriptor) Step
        func FieldAccess(protoreflect.FieldDescriptor) Step
        func UnknownAccess() Step
        func ListIndex(int) Step
        func MapIndex(protoreflect.MapKey) Step
        func AnyExpand(protoreflect.MessageDescriptor) Step
        func (Step) Kind() StepKind
        func (Step) FieldDescriptor() protoreflect.FieldDescriptor
        func (Step) MessageDescriptor() protoreflect.MessageDescriptor
        func (Step) ListIndex() int
        func (Step) MapIndex() protoreflect.MapKey
        func (Step) String() string
    type StepKind int
        const RootStep StepKind
        const FieldAccessStep StepKind
        const UnknownAccessStep StepKind
        const ListIndexStep StepKind
        const MapIndexStep StepKind
        const AnyExpandStep StepKind
    type Values struct {
        Path   Path
        Values []protoreflect.Value
    }
    func (Values) Index(int) (out struct{ ... })
    func (Values) Len() int
    func (Values) String() string

package protorange
    var Break error
    var Terminate error
    func Range(protoreflect.Message, func(protopath.Values) error) error
    type Options struct {
        Stable bool
        Resolver interface { ... }
    }
    func (Options) Range(m protoreflect.Message, push, pop func(protopath.Values) error) error

Change-Id: I29cbd5142fe169d78367d54a95d37801888b64f4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/236540
Trust: Joe Tsai <joetsai@digital-static.net>
Reviewed-by: Damien Neil <dneil@google.com>
2021-04-01 23:09:19 +00:00
Joe Tsai
77dfd04893 reflect/protoregistry: add compile-time opt-out for registration conflicts
The next release of this module will panic on registration conflicts.
While we have been warning that this will become a fatal error for about
a year now, it's almost certainly the case that some users still have
ambiguous registrations.

In addition to the GOLANG_PROTOBUF_REGISTRATION_CONFLICT environment variable,
also define a linker-initialized variable that users can set.

Change-Id: I3325d841b9b6c692e2862100d81aebdcfda069fc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/295349
Trust: Joe Tsai <joetsai@digital-static.net>
Trust: Joe Tsai <thebrokentoaster@gmail.com>
Reviewed-by: Damien Neil <dneil@google.com>
2021-03-02 23:29:07 +00:00
Joe Tsai
aa45c46752 reflect/protoregistry: panic upon registration conflicts
In this CL, we switch the behavior of registration conflicts
from simply logging a warning to panicking.
This is a change that we have warned about ever since
the release of v1.20.0 in 2020-03-02.

For users that have not fixed their dependencies or are unable to,
they may set the GOLANG_PROTOBUF_REGISTRATION_CONFLICT environment
to "warn" in order to obtain the prior behavior of simply logging
and ignoring subsequent conflicts (first wins).

Change-Id: Ied7937f0fd5cb5d875858411714ebeb5ce145574
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/235298
Trust: Joe Tsai <joetsai@digital-static.net>
Reviewed-by: Damien Neil <dneil@google.com>
2020-10-16 22:00:47 +00:00
Joe Tsai
9712361eda all: rely on dynamic dependency check for genproto
The genproto module incurs an unfortunate amount of dependency bloat
since it has a relatively large set of transitive dependencies.
Even though these are brought in as a weak dependency
(i.e., there is no hard code dependency on genproto),
it complicates dependency inspection since users need to vet
modules that they may not actually be using.

Avoid this weak dependency and instead rely on a dynamic check
to ensure that a sufficiently new version of genproto is used.
While it can't statically prevent old versions of genproto from being
linked into a binary, it guarantees that it will panic at init
with a helpful message on what's wrong.

Change-Id: I2b5a9759a5cd4c4c57aced54278502b1b6352ba7
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/262679
Trust: Joe Tsai <thebrokentoaster@gmail.com>
Reviewed-by: Damien Neil <dneil@google.com>
2020-10-15 21:46:17 +00:00
Joe Tsai
687df2333b reflect/protodesc: fix round-tripping for package field
If a .proto file does not have a package statement,
then protoc will not populate the package field.
Emulate the same behavior for ToFileDescriptorProto.

Fixes golang/protobuf#1195

Change-Id: I617102e50c96f0fa1bac6a6bc4de1cc485750bb9
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/259899
Reviewed-by: Damien Neil <dneil@google.com>
Trust: Joe Tsai <thebrokentoaster@gmail.com>
2020-10-07 22:44:53 +00:00
Joe Tsai
42cc4c592f reflect/protoreflect: improve source information usability
Added API:
  SourceLocations.ByPath
  SourceLocations.ByDescriptor
  SourceLocation.Next
  SourcePath.String
  SourcePath.Equal

We modify compiler/protogen to use SourceLocations.ByDescriptor.
In retrospect, if this had existed during the development of protogen,
we would not have exposed protogen.Location and related fields.

Change-Id: I58f17e59f90b9ba16f0982c4b71c2542e4ff6e75
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/238000
Reviewed-by: Damien Neil <dneil@google.com>
2020-07-23 21:50:46 +00:00
Joe Tsai
1a290e9a0e reflect/protoreflect: add MessageFieldTypes
The MessageFieldTypes interface (if implemented by a MessageType)
provides Go type information about the fields if they are
an enum or message type.

Change-Id: I68b20f5726377f6b0f2c20a8b6e45f9802b43f67
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/236777
Reviewed-by: Damien Neil <dneil@google.com>
2020-07-23 21:24:58 +00:00
Joe Tsai
e14d6b3cdc reflect/protoreflect: add FieldDescriptor.TextName
Add a new TextName accessor that returns the field name that should
be used for the text format. It is usually just the field name, except:
1) it uses the inlined message name for groups,
2) uses the full name surrounded by brackets for extensions, and
3) strips the "message_set_extension" for well-formed extensions
to the proto1 MessageSet.

We make similar adjustments to the JSONName accessor so that it applies
similar semantics for extensions.

The two changes simplifies all logic that wants the humanly readable
name for a field.

Change-Id: I524b6e017fb955146db81819270fe197f8f97980
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/239838
Reviewed-by: Herbie Ong <herbie@google.com>
2020-07-08 23:23:57 +00:00
Martin Möhrmann
91eb7d3a63 protoregistry: avoid checking for '/' in FindMessageByName
Profiling shows FindMessageByName spending considerable time
in strings.LastIndexByte in FindMessageByURL.

This overhead can be avoided by giving FindMessageByName
its own implementation not pruning the name string up
to the last '/' which can not appear in a protobuf name.

Change-Id: Ifec07689f72790ead8b7552cd04dd99fa67b61e4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/240378
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2020-07-01 19:55:25 +00:00
Joe Tsai
b78321453d reflect/protoregistry: centralize MessageSet extension resolution logic
Centralize the MessageSet extension resolution logic in the registry.
This avoids needless replication of this exact logic in multiple places
(for JSON and text) and elsewhere.

Change-Id: I70bfea899e295e8c589f418965bf0dd099f93628
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/240077
Reviewed-by: Herbie Ong <herbie@google.com>
2020-07-01 17:56:02 +00:00
Joe Tsai
44e4150b30 reflect/protoreflect: optimize Name.IsValid and FullName.IsValid
For simplicity, IsValid was implemented in terms of regular expressions,
which are useful for verifying a string according to a strict grammar,
but performs poorly. Implement the check in terms of hand-written code,
which provides a 20x improvement to validate the name "google.protobuf.Any".

name               old time/op  new time/op  delta
FullNameIsValid-8   683ns ± 2%    35ns ± 1%  -94.86%  (p=0.000 n=10+10)

Change-Id: I980403befca0b72cea22acd274064a46cb02644b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/238002
Reviewed-by: Damien Neil <dneil@google.com>
2020-06-16 19:13:20 +00:00
Joe Tsai
e0b77db13b internal/genid: add new package for generated identifiers
The genid package unifies the genname, fieldnum, and detectknown
packages into a single package.

Whenever possible use the generated constants rather than
hard-coded literals. This makes it easier to search the entire
module for special logic that deal with well-known types.

Change-Id: I13beff1f4149444a0c0b9e607ebf759657f000f4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/235301
Reviewed-by: Herbie Ong <herbie@google.com>
2020-05-29 07:08:23 +00:00
Joe Tsai
27ea64ff2c reflect/protoreflect: adjust ValueOf panic message
A common mistake is to pass proto.Message to protoreflect.ValueOf
instead of a protoreflect.Message. Specialize for this case
and provide a panic message that is more helpful.

Change-Id: I5def43341aa9607182edde309dea2823e61c1fdb
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/234117
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-26 22:22:26 +00:00
Joe Tsai
0567a55b47 all: add weak dependency on google.golang.org/genproto module
This is step 4 of 6 in a multi-stage migration
to move the well-known types
from the google.golang.org/genproto module
to the google.golang.org/protobuf module.

The generated Go packages for field_mask.proto, api.proto,
type.proto, and source_context.proto are being moved over
to this module alongside all the other well-known types.

In order to move these types between two modules,
there needs to be a sequence of changes submitted in
decently rapid succession. It is impossible to atomically
make these changes, so a brief breakage is inevitable.
The steps are as follows:

Step 1: Submit a change to cloud.google.com/go/internal/gapicgen
to avoid generating the well-known types. Otherwise, the tool
will undo the changes made in step 3.
See https://code-review.googlesource.com/c/gocloud/+/56810

Step 2: Submit a change to google.golang.org/protobuf that
adds the generated well-known types being migrated to that module.
In order to prevent the situation where a user links in
too old a version of the genproto module such that
duplicate registration occurs for the well-known types,
the registry is specially modified to provide an error
message that instructs users to upgrade the genproto module.
See https://golang.org/cl/234937

Step 3: Submit a change to google.golang.org/genproto that
switches all generated well-known types to be aliases to the
ones declared in google.golang.org/protobuf from the previous step.
This will cause the genproto module to incur an dependency
on an unreleased version of the protobuf module.
See https://github.com/googleapis/go-genproto/pull/372

Step 4: Submit a change to google.golang.org/protobuf that
adds a weak module depdency on the genproto module at the
revision from the previous step.
See https://golang.org/cl/235297

Step 5: Release google.golang.org/protobuf@v1.24.0.
See https://golang.org/cl/235299

Step 6: Submit a change to google.golang.org/genproto that
updates the protobuf module dependency to v1.24.0.

Change-Id: I22d496e7496446becad742eab05b169abb735ae2
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/235297
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-26 22:06:12 +00:00
Joe Tsai
81db48ad09 all: move well-known types
This is step 2 of 6 in a multi-stage migration
to move the well-known types
from the google.golang.org/genproto module
to the google.golang.org/protobuf module.

The generated Go packages for field_mask.proto, api.proto,
type.proto, and source_context.proto are being moved over
to this module alongside all the other well-known types.

In order to move these types between two modules,
there needs to be a sequence of changes submitted in
decently rapid succession. It is impossible to atomically
make these changes, so a brief breakage is inevitable.
The steps are as follows:

Step 1: Submit a change to cloud.google.com/go/internal/gapicgen
to avoid generating the well-known types. Otherwise, the tool
will undo the changes made in step 3.
See https://code-review.googlesource.com/c/gocloud/+/56810

Step 2: Submit a change to google.golang.org/protobuf that
adds the generated well-known types being migrated to that module.
In order to prevent the situation where a user links in
too old a version of the genproto module such that
duplicate registration occurs for the well-known types,
the registry is specially modified to provide an error
message that instructs users to upgrade the genproto module.
See https://golang.org/cl/234937

Step 3: Submit a change to google.golang.org/genproto that
switches all generated well-known types to be aliases to the
ones declared in google.golang.org/protobuf from the previous step.
This will cause the genproto module to incur an dependency
on an unreleased version of the protobuf module.
See https://github.com/googleapis/go-genproto/pull/372

Step 4: Submit a change to google.golang.org/protobuf that
adds a weak module depdency on the genproto module at the
revision from the previous step.

Step 5: Release google.golang.org/protobuf@v1.24.0.

Step 6: Submit a change to google.golang.org/genproto that
updates the protobuf module dependency to v1.24.0.

Change-Id: I36a19049d2240b67a37dfad20e154505aee7c784
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/234937
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-26 19:51:55 +00:00
Joe Tsai
1f5b6fe645 all: improve panic messages for better debugability
Change-Id: If3e505e715d5ce2c9a81249c868d26226a25b724
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/232339
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-05 21:10:54 +00:00