161 Commits

Author SHA1 Message Date
Michael Stapelberg
cbc3dd69c1 all: replace interface{} by any now that we are on Go 1.21
I generated this change using:

  % sed -i 's,interface{},any,g' **/*.go
  % git checkout -- **/*.pb.go
  % $EDITOR cmd/protoc-gen-go/internal_gengo/well_known_types.go
  % ./regenerate.bash

Change-Id: I728f4b69c87ffc8f3b19bf807bf9bf1479bdbab4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/585735
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
2024-05-15 12:42:15 +00:00
Michael Stapelberg
0e932930c8 internal/impl: enable fully lazy extensions (over Size and Marshal)
Extensions will be kept in wire format over proto.Size and proto.Marshal.

This change is a significant performance optimization for jobs that read and
write Protobuf messages of the same type, but do not need to process extensions.

This change is based on work by Patrik Nyblom.

Note that the proto.Size semantics for lazy messages might be surprising;
see https://protobuf.dev/reference/go/size/ for details.

We have been running this change for about two weeks in Google,
all known breakages have already been addressed with CL 579995.

related to golang/protobuf#1609

Change-Id: I16be78d15304d775bb30e76356a1a61d61300b43
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/580015
Reviewed-by: Lasse Folger <lassefolger@google.com>
Auto-Submit: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-05-15 08:56:13 +00:00
Lasse Folger
6c3ebca21f all: update to protobuf 27.0-rc1 and regenerate protos
This change required some changes to the editions default handling code
because the descriptor.proto changed upstream [2]. The defaults are no
longer one feature set but are split into overridable and
not-overridable features which have to be merged.

I had to do bootstraping in 4 phases but the results should be correct:

1. generate everything depending on descriptor.proto
2. generate new defaults binary proto
3. adjust all code that works with defaults (*/edition.go files)
4. generate everything else

The was required because 1. is a prerequisite for 3. while 2. and 3. are
a prerequisite for 4. (2. and 3. can probably be done in parallel).

The new release also introduced new conformance tests. The go
implementation is not yet conformant and the tests will be fixed in a
follow up change because they require changes to the protojson and
protoext encoders.

[1] e5502c746e

Change-Id: Iddf248f6582a0402ab31256f6e64755d870ed82c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/582635
Auto-Submit: Lasse Folger <lassefolger@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Nicolas Hillegeer <aktau@google.com>
2024-05-03 08:08:45 +00:00
Lasse Folger
e4ad8f9dfc types/gofeaturespb: move go_feature.proto to be consistent with out languages
https://github.com/golang/protobuf/issues/1608

Change-Id: Ie4e60d8c6876e1d308a6119d3ff918bfeadd733e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/579975
Auto-Submit: Lasse Folger <lassefolger@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-04-22 16:37:39 +00:00
Lasse Folger
671c2db939 [proto] use the correct parent when resolving features for extensions
When I implemented this initially, I thought the parent of an extension is the
extendee. This is incorrect. The parent is the scope in which the extension is
defined. This CL changes the code to use the correct parent. This also allows
us to reduce some complexity in the implementation because we don't need to
wait until the extendee is resolved before we can resolve the features.

Change-Id: I6d7012f7502ef95457ab96f3e8abc4ab763d5bcb
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/579275
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
2024-04-16 13:33:29 +00:00
Mike Kruskal
a18684df74 protobuf: fix delimited fields under editions in go
This brings go into conformance with other implementations.  Group-like message fields with delimited encoding will continue to use the type name for text-format, but everything else will use the field name.

Change-Id: Ib6d07f19ccfa853ce0370392c89fd24fb7148793
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/575896
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Commit-Queue: Michael Stapelberg <stapelberg@google.com>
2024-04-04 07:31:06 +00:00
Lasse Folger
3ebf7dd8a5 proto: fix HasPresence for extensions and missing plugin response
Change-Id: I9ee81f7a88cf91f3041e89936cb03511bd224603
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/574375
Auto-Submit: Lasse Folger <lassefolger@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
2024-03-26 09:28:59 +00:00
Lasse Folger
7259b46773 all: implement proto2/proto3 as editions [1/2]
This change removes most usages of Syntax() from the repository and uses
edition features for instead. The appropriate edition feature defaults are
loaded for proto2/proto3 when the initialization of the descriptors
start.

All of these changes were tested on the Google corpus.

Change-Id: Ieca076a2b38ca8e50e084cd32e725b7b3dcb4171
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/572435
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
2024-03-19 09:40:08 +00:00
Lasse Folger
fe89159266 internal/filedesc: make descriptor initialization goroutine-safe
Before this change there was a data race if you initialized the
descriptor of a message in parallel to the descriptor of any extension of
that message because the extendees feature set is read during the
initialization of the extension.

Change-Id: Id896b9fbf848209fce7dae8a7f40e2d61a3b2825
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/568695
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
2024-03-04 08:49:27 +00:00
Lasse Folger
416d517afa internal/conformance: make conformance tests work with editions
In scope of this change, I had to fix the `IsPacked()` implementation
for field descriptors because it was not returning the correct values
for edition protos.

Change-Id: Ic1ba9d0b3552ddf16360a80336c14632f2ce6f16
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/566039
Auto-Submit: Lasse Folger <lassefolger@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-02-26 08:14:29 +00:00
Lasse Folger
055c812a4f encoding/prototext: add proto editions and fuzz tests
Change-Id: I2afc5ae83bf68600def3568e1d3ad51ef00e7671
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/566395
Auto-Submit: Lasse Folger <lassefolger@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-02-23 12:14:08 +00:00
Lasse Folger
2caa6b02a2 all: format all .proto files
Change-Id: Ied684945de38ab1895c3ce8afaa1d84cda1e24f9
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/566037
Auto-Submit: Lasse Folger <lassefolger@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-02-23 09:30:28 +00:00
Lasse Folger
f2cb7f136e encoding/protojson: add protojson editions tests including fuzztests
Change-Id: I478bf6a945cb2c86c71fd20b64dedb4b2e585f1d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/566035
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
2024-02-23 08:35:31 +00:00
Lasse Folger
da0e6634fb proto: add editions tests for invalid messages
Change-Id: I97dff474afa7465ffbf1ac554cad8127bd4717bf
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/565975
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-02-22 08:59:49 +00:00
Lasse Folger
f9eb6c71e7 go_features.proto: move .proto file next to generated .pb.go file
Change-Id: Ia24ca1ad2186bcc95681081d2719ee834b385728
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/565955
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
2024-02-22 08:56:15 +00:00
Lasse Folger
11de0a6409 all: add editions support for extensions
Before this change the implementation would ignore editions options set on
extension declarations.
This change also add much more coverage for valid messages.

Change-Id: I0e7029b1430a39859cdd4a93667ac20c7e0ff20e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/565396
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
2024-02-22 08:53:07 +00:00
Lasse Folger
7d98b0e8ab all: implement features.field_presence = LEGACY_REQUIRED support
Change-Id: Iaba3f4b81ac1b73f7542397a479642d081152c39
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/565656
Auto-Submit: Lasse Folger <lassefolger@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-02-21 14:39:44 +00:00
Lasse Folger
6e0d73fd79 proto: add extension tests for editions protos
Change-Id: Ib3a00d8bf6fbf4841959ad9099fdc7b0d59ee0d3
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/565655
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
2024-02-21 09:19:43 +00:00
Lasse Folger
194457763e proto: add more editions tests for proto.Size and proto.Equal
Change-Id: I8c7065d413b3a8d90ef25a2f5b9b9868c18906ec
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/565395
Auto-Submit: Lasse Folger <lassefolger@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
2024-02-21 08:36:29 +00:00
Lasse Folger
5e8da94149 proto: add decoding fuzz test for proto3 to editions conversion
Change-Id: Ic2536373eeb5c814059ddf5c7c0d675d0e06153c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/564817
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
2024-02-20 10:20:29 +00:00
Lasse Folger
8f6a6615d6 proto: add fuzztest for proto2 and equivalent editions proto
This fixes more minor issue when constructing the descriptor in that
delimited encoded message fields are not considered properly as groups.

Change-Id: I714a227a0f8d256fa7430c526844e8e94acdda33
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/564816
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
2024-02-20 09:18:14 +00:00
Lasse Folger
f102ec6532 all: add more test for editions and fix some bugs in the implementation
There are several minor fixes:
* fix the unmarshaler for the field options (it used the wrong field numbers).
* fix the unmarshaler to always initialize the file options with the appropriate defaults.
* fix the coder selection to work properly for fields with implicit presence tracking (they used the coders for explicit tracking).
* fix the dynamic message created from a descriptor for an editions using proto to use HasPresence instead of a syntax check for proto3

Change-Id: Ic13bc22a71aa9f93b476e5edd650d4d2ab5dcb98
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/564455
Auto-Submit: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
2024-02-16 14:47:55 +00:00
Lasse Folger
f387b036d8 editions: add some test protos for editions
I'll add the runtime implementations and tests in a followup change to
make it easier to review.

Change-Id: I2917e0d40a99e81799d89a66584a680ed7e8dbf7
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/561939
Auto-Submit: Lasse Folger <lassefolger@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-02-12 14:14:45 +00:00
Josh Humphries
f4a6c1f6e5 types/descriptorpb: regenerate using latest protobuf v25.0-rc2 release candidate
This updates all generated code to match the contents of the latest
v25.0-rc2 release candidate of Protobuf. This provides access to the
various new fields and types defined in google/protobuf/descriptor.proto.

Change-Id: I209f8509f2d3e9a0ac7d9e134caef38c48a0d16b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/538075
Reviewed-by: Lasse Folger <lassefolger@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
2023-10-27 08:25:48 +00:00
Damien Neil
736947c4de all: update protobuf release version to fix macOS tests
Update to a protobuf version that contains a fix for
https://github.com/protocolbuffers/protobuf/issues/12173
(macOS build error).

Set "CC=clang" when building the protobuf repo on macOS,
which fixes build errors on my machine (macOS 13.3.1, M1).

Change-Id: Id8abb054479454b7ccbded5a6fa4d74f3d3c236d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/489315
Run-TryBot: Damien Neil <dneil@google.com>
Reviewed-by: Joseph Tsai <joetsai@digital-static.net>
Reviewed-by: Lasse Folger <lassefolger@google.com>
2023-04-26 20:34:47 +00:00
Josh Humphries
bc1253ad37 types/descriptorpb: regenerate using latest protobuf v22.0 release
This updates all generated code to match the contents of the latest
v22.0 release of Protobuf.

This involved a couple of changes to the script that does the sync'ing:
  1. The new Protobuf version no longer includes autoconf configuration
     and instead requires using bazel to build things.
  2. The new Protobuf release does not have an artifact named
     "protobuf-all-${VERSION}.tar.gz", but the one named
     "protobuf-${VERSION}.tar.gz" has all of the sources and was
     sufficient for the regenerate.bash script to complete.

This change does NOT regenerate the protos related to benchmarks.
The Protobuf repo no longer includes benchmarks. The CL removing them
says they are superceded by google/fleetbench:
    83c499de86
But that project's proto benchmark files are very different:
    https://github.com/google/fleetbench/tree/main/fleetbench/proto
So I commented out those steps in the generation code since the benchmarks
will need some work to reconcile with fleetbench.

This code adds known failing tests for conformance. New test cases were
added in https://github.com/protocolbuffers/protobuf/pull/9534, but the
Go protojson package does not behave according to the new tests. There
is an existing issue in GitHub about this:
    https://github.com/golang/protobuf/issues/1208

Change-Id: Iad796ec7889bc2a74b58db5224facf850cd1a1cd
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/469255
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
2023-02-22 09:33:03 +00:00
Michael Stapelberg
a9481185b3 cmd/protoc-gen-go: make deprecated messages more descriptive
Also respect the file-wide deprecation option by treating every
identifier within the file as deprecated.

Change-Id: Ic3a5e84c2d90bbcbefc0a1b3272cc50554b94218
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/462315
Reviewed-by: Lasse Folger <lassefolger@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
2023-01-18 09:34:59 +00:00
Josh Humphries
692f4a24f8 types/descriptorpb: update *.pb.go to use latest protoc release, 21.5
I updated the protoc version to 21.5 and ran regenerate.bash.

This updates the various *.pb.go files to use the sources from
the latest protobuf release. In particular, this updates the
descriptorpb, so it includes some changes to descriptor.proto
in the past year and a half since this was last re-generated.
The change also pulled over several other generated changes.

Change-Id: Icd9c4e5d47ba85e1f4bae1d0a025f8dec15a3014
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/426054
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
2022-08-27 06:10:12 +00:00
Koichi Shiraishi
784c482554 all: remove shorthand import aliases
Change-Id: I62ec79419354a2c65ac69e4bcc1cc864a6a40a1a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/407934
Reviewed-by: Lasse Folger <lassefolger@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
2022-05-24 20:05:50 +00:00
Damien Neil
3a9e1dc314 all: gofmt all
Change-Id: I1d42c8d784440a0b7e40b4f0b8da54be0fb338a6
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/370054
Trust: Damien Neil <dneil@google.com>
Run-TryBot: Damien Neil <dneil@google.com>
Reviewed-by: Joseph Tsai <joetsai@digital-static.net>
Reviewed-by: Florian Zenker <floriank@google.com>
2022-01-04 22:57:54 +00:00
Joe Tsai
aa432c0868 cmd/protoc-gen-go: fix generation of enum defaults
Previously, this call strconv.FormatInt with base 32,
when the intention is to call it with base 10.

Change-Id: I31cdd2415b3a80936cdcdeb5612a486204404ecb
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/331149
Trust: Joe Tsai <joetsai@digital-static.net>
Reviewed-by: Damien Neil <dneil@google.com>
2021-06-27 06:07:42 +00:00
Joe Tsai
acaef6adb7 cmd/protoc-gen-go: avoid referencing remote enum values by name
The Go generator has historically always prefixed an enum value
with the name of the enum type, when it was unnecessary to do so.

For example:
	enum Status {
		STATUS_FAILED = 0;
		STATUS_PASSED = 1;
	}
would be generated as:
	type Status int32
	const (
		Status_STATUS_FAILED Status = 0
		Status_STATUS_PASSED Status = 1
	)

It is common for the enum values to be manually prefixed by the
enum type since protobuf enums use C++ namespace rules where
enum types and enum values are in the same namespace scope.
Thus, having the Go generator add a prefix is redundant.
See https://github.com/golang/protobuf/issues/513.

Some custom Go generators like protoc-gen-gogo allow removing
the prefix with the gogoproto.goproto_enum_prefix feature.
However, this leads to interoperability issues between
protoc-gen-go and protoc-gen-gogo, where the enum value names
cannot be accurately inferred.

Avoid this problem by just hard-coding the enum value number
for values declared in other packages. This provides benefits
in interoperability at the small cost of enum values possibly
being stale if their value were ever changed in a remote package.
However, this would only occur with use of proto2 enums and
default values, which seems to be an exceptionally rare situation.

Before:
	Default_MyMessage_MyField = remotepb.FooEnum_FOO_ENUM
After:
	Default_MyMessage_MyField = remotepb.FooEnum(4) // remotepb.FooEnum_FOO_ENUM

Before:
	func (x *MyMessage) GetField() remotepb.FooEnum {
		...
		return remotepb.FooEnum_FOO_ZERO
	}
After:
	func (x *MyMessage) GetField() remotepb.FooEnum {
		...
		return remotepb.FooEnum(0) // always 0 for proto3 and often 0 for proto2
	}

Change-Id: I3a06cd553f2eaf6124666f6c36c196d500d35718
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/319649
Trust: Joe Tsai <joetsai@digital-static.net>
Reviewed-by: Damien Neil <dneil@google.com>
2021-05-24 22:21:56 +00:00
Damien Neil
24d799b3c1 internal/impl: support typed nil source for Merge of aberrant messages
When merging aberrant messages with legacy Marshal and Unmarshal
methods, check for a typed nil source before calling Marshal.

Add an aberrant message with Marshal/Unmarshal methods to
internal/testprotos/nullable and use it to test the internal/impl
support for these methods.

Fixes golang/protobuf#1324

Change-Id: Ib6ce85b30b46e3392a226ca6abe411932a371f02
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/321529
Trust: Damien Neil <dneil@google.com>
Run-TryBot: Damien Neil <dneil@google.com>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2021-05-20 18:23:20 +00:00
Joe Tsai
e4fcb9f7e7 reflect: add protopath and protorange packages
The protopath package provides a means to programmatically represent
a sequence of protobuf reflection operations.
The protorange package traverses through a message and
calls a user-provided function as it iterates.

This feature sets the groundwork for the often requested feature
of being able to exclude certain fields when merging or serializing.

package protopath
    type Path []Step
    type Step struct{ ... }
        func Root(protoreflect.MessageDescriptor) Step
        func FieldAccess(protoreflect.FieldDescriptor) Step
        func UnknownAccess() Step
        func ListIndex(int) Step
        func MapIndex(protoreflect.MapKey) Step
        func AnyExpand(protoreflect.MessageDescriptor) Step
        func (Step) Kind() StepKind
        func (Step) FieldDescriptor() protoreflect.FieldDescriptor
        func (Step) MessageDescriptor() protoreflect.MessageDescriptor
        func (Step) ListIndex() int
        func (Step) MapIndex() protoreflect.MapKey
        func (Step) String() string
    type StepKind int
        const RootStep StepKind
        const FieldAccessStep StepKind
        const UnknownAccessStep StepKind
        const ListIndexStep StepKind
        const MapIndexStep StepKind
        const AnyExpandStep StepKind
    type Values struct {
        Path   Path
        Values []protoreflect.Value
    }
    func (Values) Index(int) (out struct{ ... })
    func (Values) Len() int
    func (Values) String() string

package protorange
    var Break error
    var Terminate error
    func Range(protoreflect.Message, func(protopath.Values) error) error
    type Options struct {
        Stable bool
        Resolver interface { ... }
    }
    func (Options) Range(m protoreflect.Message, push, pop func(protopath.Values) error) error

Change-Id: I29cbd5142fe169d78367d54a95d37801888b64f4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/236540
Trust: Joe Tsai <joetsai@digital-static.net>
Reviewed-by: Damien Neil <dneil@google.com>
2021-04-01 23:09:19 +00:00
Joe Tsai
31610fd913 cmd/protoc-gen-go: remove generation of the ExtensionRangeArray method
The ExtensionRangeArray was a pseudo-internal API that was used by
the protobuf runtime implementation to know what the extension ranges are.
For some time now, the runtime implementation does not make use of this.

According to all the latest modules known by the module proxy,
no code depends directly on this method being present.

In the extremely rare case where some user code is depending on this method,
the user can place a .go file next to the generated .pb.go file that
injects this method.

Change-Id: Iae40a9a33b8c9b5a243d48db14f25b05ca24e3dc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/305574
Trust: Joe Tsai <joetsai@digital-static.net>
Reviewed-by: Damien Neil <dneil@google.com>
2021-03-29 20:25:28 +00:00
Joe Tsai
23ccb359e1 internal/impl: add runtime support for aberrant messages
Implement support in the protobuf runtime to better understand
message types that are not generated by the official generator.

In particular:
* Add a best-effort implementation of protobuf reflection for
"non-nullable" fields which are supposed to be represented by *T,
but are instead represented by a T. "Non-nullable" message fields
report presence based on whether the message is the zero Go value.
* We do NOT implement support for "non-nullable" fields in the
table-driven implementation since we assume that the aberrant messages
that we care about have a Marshal and Unmarshal method.
* We better handle custom messages that implement Marshal and Unmarshal,
but do NOT implement Merge. In that case, we implement merge in terms of
a back-to-back marshal and unmarshal.
* We better tolerate the situations where a protobuf message field
cannot be mapped to a Go struct field since the latter is missing.
In such cases, reflection treats the field as if it were unpopulated.
Setting such fields will panic.

This change allows the runtime to handle all message types declared
in the "go.etcd.io/etcd" and "k8s.io" modules where protobuf reflection,
Marshal, Unmarshal, Reset, Merge, and Equal all work.

The only types that still do not fully work are:
	* "k8s.io/api/authentication/v1".ExtraValue
	* "k8s.io/api/authentication/v1beta1".ExtraValue
	* "k8s.io/api/authorization/v1".ExtraValue
	* "k8s.io/api/authorization/v1beta1".ExtraValue
	* "k8s.io/api/certificates/v1".ExtraValue
	* "k8s.io/api/certificates/v1beta1".ExtraValue
	* "k8s.io/apimachinery/pkg/apis/meta/v1".MicroTime
	* "k8s.io/apimachinery/pkg/apis/meta/v1".Time
	* "k8s.io/apimachinery/pkg/apis/meta/v1".Verbs
While Marshal, Unmarshal, Reset, and Merge continue to work,
protobuf reflection and any functionality that depends on it
(e.g., prototext, protojson, Equal, etc.) will not work.

Change-Id: I67a9d2f1bec35248045ad0c16220d02fc2e0e172
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/300869
Trust: Joe Tsai <joetsai@digital-static.net>
Trust: Joe Tsai <thebrokentoaster@gmail.com>
Reviewed-by: Damien Neil <dneil@google.com>
2021-03-16 02:41:12 +00:00
Joe Tsai
85f47621e0 all: update protobuf toolchain dependency
This updates the dependency on the protobuf toolchain to an unreleased
version that has the go_package option for relevant .proto files in
that repository to reference this module.

Change-Id: Ie1ac8f81a323285efbd14a17d02c7105e810af2d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/235283
Trust: Joe Tsai <joetsai@digital-static.net>
Reviewed-by: Damien Neil <dneil@google.com>
2021-03-03 01:59:11 +00:00
Joe Tsai
1a290e9a0e reflect/protoreflect: add MessageFieldTypes
The MessageFieldTypes interface (if implemented by a MessageType)
provides Go type information about the fields if they are
an enum or message type.

Change-Id: I68b20f5726377f6b0f2c20a8b6e45f9802b43f67
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/236777
Reviewed-by: Damien Neil <dneil@google.com>
2020-07-23 21:24:58 +00:00
Joe Tsai
e7c3f89377 cmd/protoc-gen-go: remove name mangling for MessageSet extensions
MessageSet are a proto1 feature only used by Google.
We no longer rely on the name mangling logic performed here,
so remove it.

Change-Id: I5d66ebd86875894632f0d0c1e9816ae47ee0d5f4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/242657
Reviewed-by: Herbie Ong <herbie@google.com>
2020-07-14 22:22:58 +00:00
Joe Tsai
81db48ad09 all: move well-known types
This is step 2 of 6 in a multi-stage migration
to move the well-known types
from the google.golang.org/genproto module
to the google.golang.org/protobuf module.

The generated Go packages for field_mask.proto, api.proto,
type.proto, and source_context.proto are being moved over
to this module alongside all the other well-known types.

In order to move these types between two modules,
there needs to be a sequence of changes submitted in
decently rapid succession. It is impossible to atomically
make these changes, so a brief breakage is inevitable.
The steps are as follows:

Step 1: Submit a change to cloud.google.com/go/internal/gapicgen
to avoid generating the well-known types. Otherwise, the tool
will undo the changes made in step 3.
See https://code-review.googlesource.com/c/gocloud/+/56810

Step 2: Submit a change to google.golang.org/protobuf that
adds the generated well-known types being migrated to that module.
In order to prevent the situation where a user links in
too old a version of the genproto module such that
duplicate registration occurs for the well-known types,
the registry is specially modified to provide an error
message that instructs users to upgrade the genproto module.
See https://golang.org/cl/234937

Step 3: Submit a change to google.golang.org/genproto that
switches all generated well-known types to be aliases to the
ones declared in google.golang.org/protobuf from the previous step.
This will cause the genproto module to incur an dependency
on an unreleased version of the protobuf module.
See https://github.com/googleapis/go-genproto/pull/372

Step 4: Submit a change to google.golang.org/protobuf that
adds a weak module depdency on the genproto module at the
revision from the previous step.

Step 5: Release google.golang.org/protobuf@v1.24.0.

Step 6: Submit a change to google.golang.org/genproto that
updates the protobuf module dependency to v1.24.0.

Change-Id: I36a19049d2240b67a37dfad20e154505aee7c784
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/234937
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-26 19:51:55 +00:00
Joe Tsai
2d80e9b3ab encoding/prototext: adjust handling of invalid UTF-8
The following changes are made:
* Permit invalid UTF-8 in proto2. This goes against specified behavior,
but matches functional behavior in wire marshaling (not just for Go,
but also in the other major language implementations as well).
* The Format function is specified as ignoring errors since its intended
purpose is to surface information to the human user even if it's not
exactly parsible back into a message. As such, add an unexported
allowInvalidUTF8 option that is specially used by Format.
* Add an EmitASCII option that forces the formatting of
strings and bytes to always be encoded as ASCII.
This ensures that the entire output is always ASCII as well.

Note that we do not replicate this behavior for protojson since:
* The JSON format fundamentally has a stricter and well-specified
grammar for exactly what is valid/invalid, while the text format
has not had a well-specified grammar for the longest time,
leading to all sorts of weird usages due to Hyrum's law.
* This is to ease migration from the legacy implementation,
which did permit invalid UTF-8 in proto2.
* The EmitASCII option relies on the ability to always escape
Unicode characters using ASCII escape sequences, but this is not
possible in JSON since the grammar only has an escape sequence defined
for Unicode characters \u0000 to \uffff, inclusive.
However, Unicode v12.0.0 defines characters up to \U0010FFFF,
which is beyond what the JSON grammar provides escape sequences for.

Change-Id: I2b524a904e9ec59f9ed5500e299613bc27c31a14
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/233077
Reviewed-by: Herbie Ong <herbie@google.com>
2020-05-13 05:25:02 +00:00
Joe Tsai
3ebaa92e92 all: use v2 Message interface for weak fields
Cleanup the generated logic by having the implementation be backed
by protoimpl rather that directly generated.

Weak fields are a deprecated feature of protobufs and
have entirely be superceded by extensions.
Unfortunately, there are still some usages of it.

Change-Id: Ie1a4b7da253e2ccf5e56627775d9b2fb4090d59a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/229717
Reviewed-by: Damien Neil <dneil@google.com>
2020-04-29 22:34:29 +00:00
Joe Tsai
387873dd53 all: implement support for proto3 optional semantics
In the upcoming 3.12.x release of protoc, the proto3 language will be
amended to support true presence for scalars. This CL adds support
to both the generator and runtime to support these semantics.

Newly added public API:
	protogen.Plugin.SupportedFeatures
	protoreflect.FieldDescriptor.HasPresence
	protoreflect.FieldDescriptor.HasOptionalKeyword
	protoreflect.OneofDescriptor.IsSynthetic

Change-Id: I7c86bf66d0ae56642109beb5f2132184593747ad
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/230698
Reviewed-by: Damien Neil <dneil@google.com>
2020-04-29 20:02:24 +00:00
Joe Tsai
b738ac9285 all: regenerate remote .proto files
This fixes the conformance test failures which occur because the
conformance protos have not been regenerated.
The generator script has been modified with a sanity check that
files do not exist outside the expected sub-tree.

Change-Id: I473efec4a016f6bc96ddf7e20d54bcf5ff9b55fe
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/223538
Reviewed-by: Damien Neil <dneil@google.com>
2020-03-18 20:17:11 +00:00
Joe Tsai
f92988f900 all: upgrade protobuf to v3.11.4
This introduces some conformance test failures,
which occur not because our implementation changed behavior,
but because new cases were added.

Future work will be to investigate these failuress.

Change-Id: Ifb17465883c417acd46865744572f8cd0c858383
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/222857
Reviewed-by: Damien Neil <dneil@google.com>
2020-03-13 19:49:20 +00:00
Joe Tsai
ea6e3da5ef internal/testprotos/legacy: avoid direct dependency on github.com/golang/protobuf
Change-Id: I42b17ae672a76d91ecbeeb2c12949002d7720524
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/222677
Reviewed-by: Damien Neil <dneil@google.com>
2020-03-09 19:39:04 +00:00
Damien Neil
13e7f145cc reflect/protoregistry: avoid deadlock in registration
Under some rare circumstances registration can deadlock
when lazy descriptor initialization consults the registry.

Move the call triggering the lazy init out of the critical section.

Fixes golang/protobuf#1052.

Change-Id: Ic266e06b0db99fea65e797b879ce53e5342fff95
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/204804
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2020-03-05 17:35:16 +00:00
Damien Neil
3a18560b88 all: add a test for copyright headers
Fix a few files with no or unusual copyright headers.

Manually add copyright headers to the legacy testprotos, which will
(probably) never be regenerated.

Change-Id: Ifb52fa0eb59bf9d2de40b8df7e581a3c9731c883
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/220498
Reviewed-by: Joe Tsai <joetsai@google.com>
2020-02-21 18:41:46 +00:00
Damien Neil
929c287d2b internal/testprotos: add a missing Copyright header line
Change-Id: Idb92e34af7c5342e4c4478707eb99fcb824d7606
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/220350
Reviewed-by: Joe Tsai <joetsai@google.com>
2020-02-20 22:40:12 +00:00
Damien Neil
c4051cd4ec types/known: remove packages present in genproto
Remove the generated proto packages that already exist in
google.golang.org/genproto. We want to eventually move these
packages here, but it doesn't need to happen yet.

Add a local copy of fieldmaskpb for use in tests.

Refactor proto generation to override import paths using the
M<source>=<import_path> compiler option instead of by patching the
source files.

Change-Id: I8d31f67e931d70140182f19f3e0106111f71c4b4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219598
Reviewed-by: Joe Tsai <joetsai@google.com>
2020-02-18 22:04:05 +00:00