132 Commits

Author SHA1 Message Date
Damien Neil
3a9e1dc314 all: gofmt all
Change-Id: I1d42c8d784440a0b7e40b4f0b8da54be0fb338a6
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/370054
Trust: Damien Neil <dneil@google.com>
Run-TryBot: Damien Neil <dneil@google.com>
Reviewed-by: Joseph Tsai <joetsai@digital-static.net>
Reviewed-by: Florian Zenker <floriank@google.com>
2022-01-04 22:57:54 +00:00
Joe Tsai
aa432c0868 cmd/protoc-gen-go: fix generation of enum defaults
Previously, this call strconv.FormatInt with base 32,
when the intention is to call it with base 10.

Change-Id: I31cdd2415b3a80936cdcdeb5612a486204404ecb
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/331149
Trust: Joe Tsai <joetsai@digital-static.net>
Reviewed-by: Damien Neil <dneil@google.com>
2021-06-27 06:07:42 +00:00
Joe Tsai
acaef6adb7 cmd/protoc-gen-go: avoid referencing remote enum values by name
The Go generator has historically always prefixed an enum value
with the name of the enum type, when it was unnecessary to do so.

For example:
	enum Status {
		STATUS_FAILED = 0;
		STATUS_PASSED = 1;
	}
would be generated as:
	type Status int32
	const (
		Status_STATUS_FAILED Status = 0
		Status_STATUS_PASSED Status = 1
	)

It is common for the enum values to be manually prefixed by the
enum type since protobuf enums use C++ namespace rules where
enum types and enum values are in the same namespace scope.
Thus, having the Go generator add a prefix is redundant.
See https://github.com/golang/protobuf/issues/513.

Some custom Go generators like protoc-gen-gogo allow removing
the prefix with the gogoproto.goproto_enum_prefix feature.
However, this leads to interoperability issues between
protoc-gen-go and protoc-gen-gogo, where the enum value names
cannot be accurately inferred.

Avoid this problem by just hard-coding the enum value number
for values declared in other packages. This provides benefits
in interoperability at the small cost of enum values possibly
being stale if their value were ever changed in a remote package.
However, this would only occur with use of proto2 enums and
default values, which seems to be an exceptionally rare situation.

Before:
	Default_MyMessage_MyField = remotepb.FooEnum_FOO_ENUM
After:
	Default_MyMessage_MyField = remotepb.FooEnum(4) // remotepb.FooEnum_FOO_ENUM

Before:
	func (x *MyMessage) GetField() remotepb.FooEnum {
		...
		return remotepb.FooEnum_FOO_ZERO
	}
After:
	func (x *MyMessage) GetField() remotepb.FooEnum {
		...
		return remotepb.FooEnum(0) // always 0 for proto3 and often 0 for proto2
	}

Change-Id: I3a06cd553f2eaf6124666f6c36c196d500d35718
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/319649
Trust: Joe Tsai <joetsai@digital-static.net>
Reviewed-by: Damien Neil <dneil@google.com>
2021-05-24 22:21:56 +00:00
Damien Neil
24d799b3c1 internal/impl: support typed nil source for Merge of aberrant messages
When merging aberrant messages with legacy Marshal and Unmarshal
methods, check for a typed nil source before calling Marshal.

Add an aberrant message with Marshal/Unmarshal methods to
internal/testprotos/nullable and use it to test the internal/impl
support for these methods.

Fixes golang/protobuf#1324

Change-Id: Ib6ce85b30b46e3392a226ca6abe411932a371f02
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/321529
Trust: Damien Neil <dneil@google.com>
Run-TryBot: Damien Neil <dneil@google.com>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2021-05-20 18:23:20 +00:00
Joe Tsai
e4fcb9f7e7 reflect: add protopath and protorange packages
The protopath package provides a means to programmatically represent
a sequence of protobuf reflection operations.
The protorange package traverses through a message and
calls a user-provided function as it iterates.

This feature sets the groundwork for the often requested feature
of being able to exclude certain fields when merging or serializing.

package protopath
    type Path []Step
    type Step struct{ ... }
        func Root(protoreflect.MessageDescriptor) Step
        func FieldAccess(protoreflect.FieldDescriptor) Step
        func UnknownAccess() Step
        func ListIndex(int) Step
        func MapIndex(protoreflect.MapKey) Step
        func AnyExpand(protoreflect.MessageDescriptor) Step
        func (Step) Kind() StepKind
        func (Step) FieldDescriptor() protoreflect.FieldDescriptor
        func (Step) MessageDescriptor() protoreflect.MessageDescriptor
        func (Step) ListIndex() int
        func (Step) MapIndex() protoreflect.MapKey
        func (Step) String() string
    type StepKind int
        const RootStep StepKind
        const FieldAccessStep StepKind
        const UnknownAccessStep StepKind
        const ListIndexStep StepKind
        const MapIndexStep StepKind
        const AnyExpandStep StepKind
    type Values struct {
        Path   Path
        Values []protoreflect.Value
    }
    func (Values) Index(int) (out struct{ ... })
    func (Values) Len() int
    func (Values) String() string

package protorange
    var Break error
    var Terminate error
    func Range(protoreflect.Message, func(protopath.Values) error) error
    type Options struct {
        Stable bool
        Resolver interface { ... }
    }
    func (Options) Range(m protoreflect.Message, push, pop func(protopath.Values) error) error

Change-Id: I29cbd5142fe169d78367d54a95d37801888b64f4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/236540
Trust: Joe Tsai <joetsai@digital-static.net>
Reviewed-by: Damien Neil <dneil@google.com>
2021-04-01 23:09:19 +00:00
Joe Tsai
31610fd913 cmd/protoc-gen-go: remove generation of the ExtensionRangeArray method
The ExtensionRangeArray was a pseudo-internal API that was used by
the protobuf runtime implementation to know what the extension ranges are.
For some time now, the runtime implementation does not make use of this.

According to all the latest modules known by the module proxy,
no code depends directly on this method being present.

In the extremely rare case where some user code is depending on this method,
the user can place a .go file next to the generated .pb.go file that
injects this method.

Change-Id: Iae40a9a33b8c9b5a243d48db14f25b05ca24e3dc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/305574
Trust: Joe Tsai <joetsai@digital-static.net>
Reviewed-by: Damien Neil <dneil@google.com>
2021-03-29 20:25:28 +00:00
Joe Tsai
23ccb359e1 internal/impl: add runtime support for aberrant messages
Implement support in the protobuf runtime to better understand
message types that are not generated by the official generator.

In particular:
* Add a best-effort implementation of protobuf reflection for
"non-nullable" fields which are supposed to be represented by *T,
but are instead represented by a T. "Non-nullable" message fields
report presence based on whether the message is the zero Go value.
* We do NOT implement support for "non-nullable" fields in the
table-driven implementation since we assume that the aberrant messages
that we care about have a Marshal and Unmarshal method.
* We better handle custom messages that implement Marshal and Unmarshal,
but do NOT implement Merge. In that case, we implement merge in terms of
a back-to-back marshal and unmarshal.
* We better tolerate the situations where a protobuf message field
cannot be mapped to a Go struct field since the latter is missing.
In such cases, reflection treats the field as if it were unpopulated.
Setting such fields will panic.

This change allows the runtime to handle all message types declared
in the "go.etcd.io/etcd" and "k8s.io" modules where protobuf reflection,
Marshal, Unmarshal, Reset, Merge, and Equal all work.

The only types that still do not fully work are:
	* "k8s.io/api/authentication/v1".ExtraValue
	* "k8s.io/api/authentication/v1beta1".ExtraValue
	* "k8s.io/api/authorization/v1".ExtraValue
	* "k8s.io/api/authorization/v1beta1".ExtraValue
	* "k8s.io/api/certificates/v1".ExtraValue
	* "k8s.io/api/certificates/v1beta1".ExtraValue
	* "k8s.io/apimachinery/pkg/apis/meta/v1".MicroTime
	* "k8s.io/apimachinery/pkg/apis/meta/v1".Time
	* "k8s.io/apimachinery/pkg/apis/meta/v1".Verbs
While Marshal, Unmarshal, Reset, and Merge continue to work,
protobuf reflection and any functionality that depends on it
(e.g., prototext, protojson, Equal, etc.) will not work.

Change-Id: I67a9d2f1bec35248045ad0c16220d02fc2e0e172
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/300869
Trust: Joe Tsai <joetsai@digital-static.net>
Trust: Joe Tsai <thebrokentoaster@gmail.com>
Reviewed-by: Damien Neil <dneil@google.com>
2021-03-16 02:41:12 +00:00
Joe Tsai
85f47621e0 all: update protobuf toolchain dependency
This updates the dependency on the protobuf toolchain to an unreleased
version that has the go_package option for relevant .proto files in
that repository to reference this module.

Change-Id: Ie1ac8f81a323285efbd14a17d02c7105e810af2d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/235283
Trust: Joe Tsai <joetsai@digital-static.net>
Reviewed-by: Damien Neil <dneil@google.com>
2021-03-03 01:59:11 +00:00
Joe Tsai
1a290e9a0e reflect/protoreflect: add MessageFieldTypes
The MessageFieldTypes interface (if implemented by a MessageType)
provides Go type information about the fields if they are
an enum or message type.

Change-Id: I68b20f5726377f6b0f2c20a8b6e45f9802b43f67
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/236777
Reviewed-by: Damien Neil <dneil@google.com>
2020-07-23 21:24:58 +00:00
Joe Tsai
e7c3f89377 cmd/protoc-gen-go: remove name mangling for MessageSet extensions
MessageSet are a proto1 feature only used by Google.
We no longer rely on the name mangling logic performed here,
so remove it.

Change-Id: I5d66ebd86875894632f0d0c1e9816ae47ee0d5f4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/242657
Reviewed-by: Herbie Ong <herbie@google.com>
2020-07-14 22:22:58 +00:00
Joe Tsai
81db48ad09 all: move well-known types
This is step 2 of 6 in a multi-stage migration
to move the well-known types
from the google.golang.org/genproto module
to the google.golang.org/protobuf module.

The generated Go packages for field_mask.proto, api.proto,
type.proto, and source_context.proto are being moved over
to this module alongside all the other well-known types.

In order to move these types between two modules,
there needs to be a sequence of changes submitted in
decently rapid succession. It is impossible to atomically
make these changes, so a brief breakage is inevitable.
The steps are as follows:

Step 1: Submit a change to cloud.google.com/go/internal/gapicgen
to avoid generating the well-known types. Otherwise, the tool
will undo the changes made in step 3.
See https://code-review.googlesource.com/c/gocloud/+/56810

Step 2: Submit a change to google.golang.org/protobuf that
adds the generated well-known types being migrated to that module.
In order to prevent the situation where a user links in
too old a version of the genproto module such that
duplicate registration occurs for the well-known types,
the registry is specially modified to provide an error
message that instructs users to upgrade the genproto module.
See https://golang.org/cl/234937

Step 3: Submit a change to google.golang.org/genproto that
switches all generated well-known types to be aliases to the
ones declared in google.golang.org/protobuf from the previous step.
This will cause the genproto module to incur an dependency
on an unreleased version of the protobuf module.
See https://github.com/googleapis/go-genproto/pull/372

Step 4: Submit a change to google.golang.org/protobuf that
adds a weak module depdency on the genproto module at the
revision from the previous step.

Step 5: Release google.golang.org/protobuf@v1.24.0.

Step 6: Submit a change to google.golang.org/genproto that
updates the protobuf module dependency to v1.24.0.

Change-Id: I36a19049d2240b67a37dfad20e154505aee7c784
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/234937
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-26 19:51:55 +00:00
Joe Tsai
2d80e9b3ab encoding/prototext: adjust handling of invalid UTF-8
The following changes are made:
* Permit invalid UTF-8 in proto2. This goes against specified behavior,
but matches functional behavior in wire marshaling (not just for Go,
but also in the other major language implementations as well).
* The Format function is specified as ignoring errors since its intended
purpose is to surface information to the human user even if it's not
exactly parsible back into a message. As such, add an unexported
allowInvalidUTF8 option that is specially used by Format.
* Add an EmitASCII option that forces the formatting of
strings and bytes to always be encoded as ASCII.
This ensures that the entire output is always ASCII as well.

Note that we do not replicate this behavior for protojson since:
* The JSON format fundamentally has a stricter and well-specified
grammar for exactly what is valid/invalid, while the text format
has not had a well-specified grammar for the longest time,
leading to all sorts of weird usages due to Hyrum's law.
* This is to ease migration from the legacy implementation,
which did permit invalid UTF-8 in proto2.
* The EmitASCII option relies on the ability to always escape
Unicode characters using ASCII escape sequences, but this is not
possible in JSON since the grammar only has an escape sequence defined
for Unicode characters \u0000 to \uffff, inclusive.
However, Unicode v12.0.0 defines characters up to \U0010FFFF,
which is beyond what the JSON grammar provides escape sequences for.

Change-Id: I2b524a904e9ec59f9ed5500e299613bc27c31a14
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/233077
Reviewed-by: Herbie Ong <herbie@google.com>
2020-05-13 05:25:02 +00:00
Joe Tsai
3ebaa92e92 all: use v2 Message interface for weak fields
Cleanup the generated logic by having the implementation be backed
by protoimpl rather that directly generated.

Weak fields are a deprecated feature of protobufs and
have entirely be superceded by extensions.
Unfortunately, there are still some usages of it.

Change-Id: Ie1a4b7da253e2ccf5e56627775d9b2fb4090d59a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/229717
Reviewed-by: Damien Neil <dneil@google.com>
2020-04-29 22:34:29 +00:00
Joe Tsai
387873dd53 all: implement support for proto3 optional semantics
In the upcoming 3.12.x release of protoc, the proto3 language will be
amended to support true presence for scalars. This CL adds support
to both the generator and runtime to support these semantics.

Newly added public API:
	protogen.Plugin.SupportedFeatures
	protoreflect.FieldDescriptor.HasPresence
	protoreflect.FieldDescriptor.HasOptionalKeyword
	protoreflect.OneofDescriptor.IsSynthetic

Change-Id: I7c86bf66d0ae56642109beb5f2132184593747ad
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/230698
Reviewed-by: Damien Neil <dneil@google.com>
2020-04-29 20:02:24 +00:00
Joe Tsai
b738ac9285 all: regenerate remote .proto files
This fixes the conformance test failures which occur because the
conformance protos have not been regenerated.
The generator script has been modified with a sanity check that
files do not exist outside the expected sub-tree.

Change-Id: I473efec4a016f6bc96ddf7e20d54bcf5ff9b55fe
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/223538
Reviewed-by: Damien Neil <dneil@google.com>
2020-03-18 20:17:11 +00:00
Joe Tsai
f92988f900 all: upgrade protobuf to v3.11.4
This introduces some conformance test failures,
which occur not because our implementation changed behavior,
but because new cases were added.

Future work will be to investigate these failuress.

Change-Id: Ifb17465883c417acd46865744572f8cd0c858383
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/222857
Reviewed-by: Damien Neil <dneil@google.com>
2020-03-13 19:49:20 +00:00
Joe Tsai
ea6e3da5ef internal/testprotos/legacy: avoid direct dependency on github.com/golang/protobuf
Change-Id: I42b17ae672a76d91ecbeeb2c12949002d7720524
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/222677
Reviewed-by: Damien Neil <dneil@google.com>
2020-03-09 19:39:04 +00:00
Damien Neil
13e7f145cc reflect/protoregistry: avoid deadlock in registration
Under some rare circumstances registration can deadlock
when lazy descriptor initialization consults the registry.

Move the call triggering the lazy init out of the critical section.

Fixes golang/protobuf#1052.

Change-Id: Ic266e06b0db99fea65e797b879ce53e5342fff95
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/204804
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2020-03-05 17:35:16 +00:00
Damien Neil
3a18560b88 all: add a test for copyright headers
Fix a few files with no or unusual copyright headers.

Manually add copyright headers to the legacy testprotos, which will
(probably) never be regenerated.

Change-Id: Ifb52fa0eb59bf9d2de40b8df7e581a3c9731c883
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/220498
Reviewed-by: Joe Tsai <joetsai@google.com>
2020-02-21 18:41:46 +00:00
Damien Neil
929c287d2b internal/testprotos: add a missing Copyright header line
Change-Id: Idb92e34af7c5342e4c4478707eb99fcb824d7606
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/220350
Reviewed-by: Joe Tsai <joetsai@google.com>
2020-02-20 22:40:12 +00:00
Damien Neil
c4051cd4ec types/known: remove packages present in genproto
Remove the generated proto packages that already exist in
google.golang.org/genproto. We want to eventually move these
packages here, but it doesn't need to happen yet.

Add a local copy of fieldmaskpb for use in tests.

Refactor proto generation to override import paths using the
M<source>=<import_path> compiler option instead of by patching the
source files.

Change-Id: I8d31f67e931d70140182f19f3e0106111f71c4b4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219598
Reviewed-by: Joe Tsai <joetsai@google.com>
2020-02-18 22:04:05 +00:00
Damien Neil
aa735f3cca internal/testprotos: add missing go_package option
Change-Id: If28f15266349b2c05c209202c38bc98dc43f7dbd
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219297
Reviewed-by: Joe Tsai <joetsai@google.com>
2020-02-14 00:45:09 +00:00
Damien Neil
ee206b9211 proto: add tests for groups in oneofs
Fixes golang/protobuf#1000.

Change-Id: I9904f6496240c544b0190a8a1bc3e6d067f98f82
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/218581
Reviewed-by: Joe Tsai <joetsai@google.com>
2020-02-07 23:35:43 +00:00
Damien Neil
3c5fb5f879 all: make .proto file names relative to module root
Change the protoc flags such that when one of our test .proto files
imports another, the filename is consistently specified relative to the
module root.

Change-Id: I690282795cef23347c8794c1c6357e4fe9560d8a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/217762
Reviewed-by: Joe Tsai <joetsai@google.com>
2020-02-04 23:20:20 +00:00
Damien Neil
f68f17085a internal/testprotos: minor .proto file fixes
Add a missing syntax and package statement.

Change-Id: I3c95aa055ad84fa0bbb2bbdd341b51df9359f499
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/217759
Reviewed-by: Joe Tsai <joetsai@google.com>
2020-02-04 19:28:20 +00:00
Damien Neil
d025c95110 proto, internal/protobuild: add test proto template builder
The proto package tests often test several variations of messages with a
similar shape. For example, most tests are performed with a proto2
message with a regular field, a proto2 message with an extension field,
and a proto3 message.

Add a protobuild package which can initialize all these variations from
a single template. For example, these three messages:

	&testpb.TestAllTypes{OptionalInt32: proto.Int32(1)}

	&test3pb.TestAllTypes{OptionalInt32: 1}

	m := &testpb.TestAllExtensions{}
	proto.SetExtension(m, &testpb.E_OptionalInt32, 1)

can all be constructed from the template:

	protobuild.Message{"optional_int32": 1}

This reduces redundancy in tests and will make it more practical to
test alternative code generators.

Change-Id: I3245a4bf74ee1bce957bc772fed513d427720677
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/217457
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2020-02-03 19:14:55 +00:00
Damien Neil
212b05b808 internal/testprotos: make TestAllExtensions recursive
Tweak the test message to allow creating messages with extensions that
contain extensions that contain extensions, etc.

Change-Id: I41844ae699c88ab96bf0d30db3a3fbaf09616161
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/216761
Reviewed-by: Joe Tsai <joetsai@google.com>
2020-01-28 23:28:36 +00:00
Damien Neil
8729675da7 internal/benchmarks/micro: add a place for microbenchmarks
Add a place to put microbenchmarks used to justify performance-related changes.

Change-Id: I6e90a3500594b3f6297cee0b8e321a50d0a556ca
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/216480
Reviewed-by: Joe Tsai <joetsai@google.com>
2020-01-26 22:22:44 +00:00
Damien Neil
0bf97b7e36 internal/impl: messageset validation and isinit fixes
Recognize messagesets in the validator. Currently, this just gives
up and reports an unknown validity rather than trying to descend
into the messageset.

Plumb fast-path initialization checks through messageset decoding.

Change-Id: Ice55f28e8555764e4ce2720251830e8cf475c133
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/216245
Reviewed-by: Joe Tsai <joetsai@google.com>
2020-01-24 20:27:57 +00:00
Damien Neil
f12fb45fd6 all: add ProtoMethods method to protoreflect.Message
Promote the fast-path magic ProtoMethods method to first-class citizen
of the protoreflect.Message interface.

To avoid polluting the protoreflect package with the various types
required by this method, make the necessary protoiface types unnamed and
duplicate them in protoreflect.

Updates golang/protobuf#1022.

Change-Id: I9595bae40b3bc7536d727fb6f99b3bce8f73da87
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/215718
Reviewed-by: Joe Tsai <joetsai@google.com>
2020-01-21 21:05:54 +00:00
Damien Neil
6635e7d00a internal/impl: recognized required bytes fields in validation
Add a missed case in validation so we correctly validate bytes fields.
Fixes a case where we would report required bytes fields as potentially
missing.

Change-Id: I3dc4196d6995942d32a795a64214b3679d60ab6c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/215000
Reviewed-by: Joe Tsai <joetsai@google.com>
2020-01-15 23:51:51 +00:00
Damien Neil
2ae60936c2 internal/impl: fix unmarshal of group containing their own field number
The fast-path unmarshal was getting confused when parsing a group
containing a field with a number the same as the group's own field
number. Separate the handling of EndGroup tags.

Change-Id: I637702b42c94a26102e693ee29a55e80b37d7f28
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/214737
Reviewed-by: Joe Tsai <joetsai@google.com>
2020-01-14 19:43:24 +00:00
Joe Tsai
1c8015fff5 all: minor tweaks
Fix a typo in legacy_enum.go.
Rename package in ancient legacy proto so that it doesn't confuse
tooling that assume that the package and directory names match.

Change-Id: I0b896045e74b0a7f998d3e5693b853eb3aa3839c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/214182
Reviewed-by: Joe Tsai <joetsai@google.com>
2020-01-12 09:18:34 +00:00
Joe Tsai
94e730bc38 internal/testprotos: hide all public testprotos
The encoding/testprotos and reflect/protoregistry/testprotos are
accessible by other modules. Move them under internal/testprotos
to dissuade programmers who are too lazy to use their own test protos
when they need one.

Change-Id: I3dbfbce74e68ef033ec252bed076861cb47dd21e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/214341
Reviewed-by: Damien Neil <dneil@google.com>
2020-01-12 08:32:55 +00:00
Joe Tsai
55f18259ef internal/testprotos/legacy: rename and regenerate
Avoid dots and dashes in the directory to avoid issues on
build systems that cannot support them well.

Change-Id: I7ea5e6ce0b16c7158c7e53bcf5c3c1a334fe4718
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/214342
Reviewed-by: Damien Neil <dneil@google.com>
2020-01-12 08:13:18 +00:00
Joe Tsai
4a7fc8200d cmd/protoc-gen-go: refactor package
Refactor the internal logic of protoc-gen-go to better plumb local
settings and parameters down the call tree.

Change-Id: I09fec188d7359f2b66be584aa8f10e682a7b6796
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/214357
Reviewed-by: Patrik Nyblom <pnyb@google.com>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2020-01-11 03:09:40 +00:00
Damien Neil
b0c26f1868 internal/impl: add message validator
This adds a experimental function to the internal/impl package which
validates a wire-format message against a message type. The validator
reports whether the message can be successfully unmarshaled, and whether
the result is initialized (all required fields are set). In some cases,
the validator returns ambiguous results when full validation would be
expensive.

The validator is unused outside of tests. In the future, it may be used
to permit lazy unmarshaling of some data. It is being added now for
testing; in particular, the wire fuzzer now checks the validator output
for consistency with the unmarshaler.

The validator adds a small amount of unused per-MessageType state. If
this becomes a concern, we could conditionalize it with a build tag.

Change-Id: I4216ef81d6a9ed975302eed189b02d08608858b4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/212302
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2020-01-07 21:36:47 +00:00
Damien Neil
2244bc5833 internal/testprotos/fuzz: add go_package option
Change-Id: Ic9ad6a5a96a3178cc6074e7e26b5ac864552d05f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/212863
Reviewed-by: Joe Tsai <joetsai@google.com>
2020-01-06 22:47:51 +00:00
Damien Neil
7f9c7d9fe4 internal/fuzz: refactor fuzzer
Add a new Fuzz message containing all the message types we want to make
available to fuzzers. Previously, testing (for example) required fields
would require modifying the fuzzer; now, it's just a matter of adding a
message with required fields as a field of the top-level Fuzz message.

Add internal/cmd/generate-corpus to codify where the fuzz seed corpus
comes from. This will simplify adding text and json fuzzers.

Rename internal/fuzz/wire to internal/fuzz/wirefuzz to minimize package
name ambiguity. Also, the addition of the Fuzz container message
invalidates the existing corpus, so using a new name seems like a good
idea.

Change-Id: I94f8f64ba93596c8e8cecb4d42bcc5b98c17d838
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/212218
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-12-20 09:01:58 +00:00
Joe Tsai
8517608739 cmd/protoc-gen-go: remove json ignore tags
These are not necessary now that weak fields are unexported.

Change-Id: Ida18b984abedfdf52fd3d5f3cb2f4ca580659a5c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/210745
Reviewed-by: Damien Neil <dneil@google.com>
2019-12-10 23:22:16 +00:00
Joe Tsai
4663ebc852 internal/genname: centralize the definitions for generated names
Both the generator and the runtime need to agree upon the names of
specialized Go struct fields. Centralize that information in an
internal genname package.

In the mean time, also change the XXX_weak field name to match
the name used internally at Google.

Change-Id: I026bf354418c363482e5902f21aa5e0cacae24b0
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/207080
Reviewed-by: Damien Neil <dneil@google.com>
2019-12-09 22:57:38 +00:00
Damien Neil
fe15dd4cdd all: don't allow invalid field numbers when legacy support is on
The deprecated messageset format permits extension fields with numbers
greater than the usual maximum (1<<29-1). To support this, the
internal/encoding/wire package has disabled field number validation when
legacy support is enabled.

We shouldn't skip validating all field numbers for validity just because
we support larger ones in messagesets.

This change drops range validation from the wire package (other than
checking that numbers fit in an int32) and adds it to the wire
unmarshalers instead. This gives us validation where we care
about it (when unmarshaling a wire-format message) and allows for
best-effort handling of out-of-range numbers everywhere else.

Fixes golang/protobuf#996

Change-Id: I4e11b8a8aa177dd60e89723570af074a317c2451
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/210290
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-12-09 18:35:13 +00:00
Damien Neil
82886da2b9 reflect/protoreflect: add {Message,List,Map}.IsValid
Various protoreflect methods can return an "empty, read-only" message,
list, or map value. Provide a method to test if a value is one of these.

Fixes golang/protobuf#966

Change-Id: I793d8426d6e2201755983c06f024412a7e09bc4c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/209018
Reviewed-by: Joe Tsai <joetsai@google.com>
2019-11-26 22:56:18 +00:00
Damien Neil
01c0e8d680 proto, internal/impl: make wire output more consistent with v1
The v1 wire marshaler sorts fields as follows:
  - All extensions, sorted by field number.
  - All non-oneof fields, sorted by field number.
  - All oneof fields, in indeterminate order.

We already make some steps toward supporting this ordering: The
fast path encoder places extensions in sorted order at the start
of the message.

This commit moves oneof fields to the end of the message, makes the
reflection-based encoder use this ordering when deterministic marshaling
is enabled, and adds a test to catch unintentional changes to the
ordering.

Users SHOULD NOT depend on stability of the marshal output. It is
subject to change over time. Without deterministic marshaling enabled,
it is subject to change over calls to Marshal.

Change-Id: I6cfd89090d790a3bb50785f32b94d2781d7d08db
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/206800
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-11-12 20:59:03 +00:00
Damien Neil
ce3384cd34 proto, internal/impl: store unknown MessageSet items in non-mset format
In the v1 implementation, unknown MessageSet items are stored in a
message's unknown fields section in non-MessageSet format. For example,
consider a MessageSet containing an item with type_id T and value V.
If the type_id is not resolvable, the item will be placed in the unknown
fields as a bytes-valued field with number T and contents V. This
conversion is then reversed when marshaling a MessageSet containing
unknown fields.

Preserve this behavior in v2.

One consequence of this change is that actual unknown fields in a
MessageSet (any field other than 1) are now discarded. This matches
the previous behavior.

Change-Id: I3d913613f84e0ae82481078dbc91cb25628651cc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/205697
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-11-11 19:40:27 +00:00
Joe Tsai
ae313d4af3 cmd/protoc-gen-go: fix Reset method
The MessageInfo cache, once set, must not be cleared, otherwise
there exists a *messageState value where the MessageInfo value is nil.
Fix the generation of the Reset method to avoid clearing this value.

Change-Id: Ic84ca8b2640a43e967c36993da1ccd3f2b7096c4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/201478
Reviewed-by: Damien Neil <dneil@google.com>
2019-10-17 17:10:14 +00:00
Damien Neil
3b65992c65 all: remove {Enum,Message,Extension}Type.GoType methods
These methods are difficult or impossible to use correctly; types
created by the dynamicpb package, for example, all have the same GoType.

Fixes golang/protobuf#938

Change-Id: I33d4ef381579ff18569b11df501d0ba7f38a6b5c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/199060
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-10-07 21:08:26 +00:00
Joe Tsai
8e9d5f6e8a internal/protolegacy: add stub v1 proto package for testing purposes
The protolegacy package is a minimal version of the v1 proto package.
This allows us to use this stub version as the dependency for
internal/testprotos/legacy packages and avoid a dependency
on the real v1 proto package.

The implementation of most v1 functionality will panic if called.
This way, we know if we the v2 code depends on one of those
unimplemented functions.

Updates golang/protobuf#962

Change-Id: I20b4091706fd456e4b01ae0931cce30a872639b0
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/199297
Reviewed-by: Damien Neil <dneil@google.com>
2019-10-05 23:12:32 +00:00
Damien Neil
47d5893acf internal/impl: support non-struct-pointer legacy message types
Support, to some limited degree, types which implement protoV1.Message
but which are not struct pointers. Our ability to work with these types
is largely limited to calling Marshal or Unmarshal methods, when
present.

Change-Id: Ie1b851d9e753e2b2cb189b17ffeefebe2d8b3a8f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/198237
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-10-04 16:13:03 +00:00
Joe Tsai
6e095998ae proto, internal/impl: implement support for weak fields
Change-Id: I0a3ff79542a3316295fd6c58e1447e597be97ab9
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189923
Reviewed-by: Damien Neil <dneil@google.com>
2019-09-19 22:41:12 +00:00