699 Commits

Author SHA1 Message Date
Damien Neil
4b3a82f6b1 internal/impl: clean up Value codecs
Remove the Go type from consideration when creating Value codecs, as it
is unnecessary. Value codecs convert between wire form and Values,
while Converters convert between Values and the Go type.

Change-Id: Iaa4bc7db81ad0a29dabd42c2229e6f33a0c91c67
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/193457
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-09-05 16:31:05 +00:00
Joe Tsai
fd528ff53f internal/impl: unexport Unwrapper
This interface is only ever used within the impl package.
Unexport it to avoid potential abuse by users.

Change-Id: I58d0cc2e8cb01c8c9d36cdec469c8d2196c4f836
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/193197
Reviewed-by: Damien Neil <dneil@google.com>
2019-09-05 06:07:39 +00:00
Damien Neil
d91c422d95 all: remove use of deprecated NewMessage
Replace NewMessage calls with NewField, NewElement, or NewValue.

Change-Id: I6d2bb4f11f0eb2ba7a52308b1addb111137ad4b9
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/193266
Reviewed-by: Joe Tsai <joetsai@google.com>
2019-09-04 17:55:31 +00:00
Damien Neil
dfae831ab7 types/dynamicpb: New->NewMessage, add NewMessageType
Rename New to NewMessage, since messages are not the only type in this
package.

Add NewMessageType to allow creating a MessageType without first
creating a Message.

Fixes golang/protobuf#934

Change-Id: I9c59594a436dec75d3e26078b0055a4aa08e0169
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/193157
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-09-04 04:34:35 +00:00
Damien Neil
0e00e666b1 cmd/protoc-gen-go: remove go version from header
The Go toolchain version isn't particularly interesting; short of bugs
in the toolchain, the protoc-gen-go output should be identical for any
version of the compiler and stdlib. Including it introduces pointless
variance in generated output when switching between compiler versions.

Change-Id: I74a709cf227ecf68dd62579947e03c07349f04de
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/193122
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-09-04 04:34:11 +00:00
Joe Tsai
6663f3a630 internal/impl: add ProtoMessageV1Of and ProtoMessageV2Of
These functions are used by v1 code to switch between
protoV1.Message and protoV2.Message.
Wrappers over them may be exposed in a hypothetical protoadapt package.

Change-Id: Ib5265420e34e9888b540307962c70189625d663c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/193180
Reviewed-by: Damien Neil <dneil@google.com>
2019-09-03 23:27:40 +00:00
Damien Neil
290ceea663 types/dynamicpb: support dynamic extensions
Add a dynamicpb.NewExtensionType function to permit creating extension
types from descriptors.

Also fix a some bugs around extension field handling:
When creating a new value for an extension field, use the
ExtensionType's Zero or New method to create the value.

Ensure that prototest exercises true zero-values of fields. (i.e.,
getting a list, map, or message from an empty message rather than
creating a new empty one with NewField.)

Change-Id: Idb8e87cdc92692610e12a4b8a68c34b129fae617
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186180
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-09-03 21:40:45 +00:00
Damien Neil
293dc761cb internal/impl: change Go representation of extension lists to []T
Change-Id: Iebcefe0330c8f858c7735f9362abfd87043ee39d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/192458
Reviewed-by: Joe Tsai <joetsai@google.com>
2019-09-03 21:19:03 +00:00
Damien Neil
68b81c3117 internal/impl: store extension values as Values
Change the storage type of ExtensionField from interface{} to
protoreflect.Value.

Replace the codec functions operating on interface{}s with ones
operating on Values.

Values are potentially more efficient, since they can represent
non-pointer types without allocation. This also reduces the number of
types used to represent field values.

Additionally, this change lays groundwork for changing the
user-visible representation of repeated extension fields from
*[]T to []T. The storage type for extension fields must support mutation
(thus *[]T currently); changing the storage type to a Value permits this
without the need to introduce yet another view on field values.

Change-Id: Ida336be14112bb940f655236eb58df21bf312525
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/192218
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-09-03 20:58:28 +00:00
Damien Neil
835b271169 reflect/protoreflect: add ExtensionType IsValid{Interface,Value} methods
Add a way to typecheck a Value or interface{} without converting it to
the other form.  This permits implementations which store field values as
a Value (such as dynamicpb, or (soon) extensions in generated messages)
to validate inputs without an unnecessary conversion.

Fixes golang/protobuf#905

Change-Id: I1b78612b22ae832efbb55f81ae420871729e3a02
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/192457
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-09-03 20:58:16 +00:00
Joe Tsai
50f860a45a internal/filedesc: fix dependency on legacy generated enums
Suppose you have a generated message with an enum field that
has a default value, where the enum is declared in another proto file
that does not register its descriptors.
In such a situation, the enum dependency cannot be resolved and will
be left as a placeholder.

When unmarshalDefault is called, it will attempt to parse the enum
name stored as the default value, but panic since the placeholder enum
has no enum values known to it.

Instead of panicking, use a placeholder enum value that preserves the
name of default enum value and just use the enum number of 0.
This is not ideal, but is a better alternative than panicking.

Change-Id: I5ff6c351adbdd6fe7b41f2d4f215be4aa09e751f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/192500
Reviewed-by: Damien Neil <dneil@google.com>
2019-09-03 20:42:35 +00:00
Damien Neil
95539ea04c internal/filetype: fix EnumInfo.New
Previously panicked due to operating on the *T instead of the T.

Change-Id: I2aa23a63d1451d1fb29351abbe230d8d2518bd69
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/192579
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-30 17:34:31 +00:00
Damien Neil
3cda377ed2 all: rename ExtensionType Descriptor method to TypeDescriptor (2/2)
Remove the ExtensionType Descriptor method.

Change-Id: I89c985c45f2a5abc4e4e3770f9652bc2a444251e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/192141
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-28 19:46:47 +00:00
Damien Neil
79bfdbe45b all: rename ExtensionType Descriptor method to TypeDescriptor (1/2)
Descriptor methods generally return a Descriptor with no Go type
information. ExtensionType's Descriptor is an exception, returning an
ExtensionTypeDescriptor containing both the proto descriptor and a
reference back to the ExtensionType. The pure descriptor is accessed
by xt.Descriptor().Descriptor().

Rename ExtensionType's Descriptor method to TypeDescriptor to make it
clear that it behaves a bit differently.

Change 1/2: Add the TypeDescriptor method and deprecate Descriptor.

Change-Id: I1806095044d35a474d60f94d2a28bdf528f12238
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/192139
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-28 18:34:29 +00:00
Damien Neil
95758c0892 internal/filetype, internal/filedesc: avoid gccgo bug
Returning an anonymous struct is tickling a gccgo bug:
https://github.com/golang/go/issues/33866

Change to a named type. We could make the type unexpoerted, but this is
an internal package anyway so leave it exported for general style
cleanliness.

Change-Id: Idf33f1e354c0e07066ab68640308af1498a01447
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/191960
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-27 19:57:09 +00:00
Joe Tsai
26aef9d6d6 cmd/protoc-gen-go: add support for field-tracking
Field-tracking is an experimental feature in the Go linker
where it statically tracks whether a Go struct field is used.
This CL modifies the generator to emit special markers that
tells the linker to know which fields to track.

Change-Id: I235da3f3d6c0ef2021b5fe1c16106ebb8fd6f557
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189558
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-26 17:54:17 +00:00
Joe Tsai
2e7817f117 compiler/protogen, internal/strs, internal/impl: expose enum Go name derivation
In order to migrate v1 to wrap v2, we need a way to reproduce
the awful enum "names" that v1 used, which was the concatenation of
the proto package with the Go identifier used for the enum.

To support this:
* Move the camel case logic from compiler/protogen to internal/strs
* Add a small stub in internal/impl to expose this functionality

Change-Id: I8ff31daa9ae541e5788dc04d2e89eae1574877e4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/191637
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-26 17:49:17 +00:00
Damien Neil
c5060d2fe6 reflect/protoreflect: add non-allocating Value constructors
Passing a non-pointer type to protoreflect.NewValue causes an
unnecessary allocation in order to store the value in an interface{}.
While this allocation could be avoided by a smarter compiler, no such
compiler exists today.

Add functions for creating new values of a specific type, avoiding the
allocation. (And also adding a small amount of type safety, although
this is unlikely to be important.)

Update the proto and internal/impl packages to use these functions.

Change-Id: Ic733de22ddf19c530189166c853348e1b54b7391
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/191457
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-26 17:48:05 +00:00
Tuo Shan
6e25d8c6a6 proto: add tests for unmarshalling invalid field numbers
This change adds tests for unmarshalling fields with various invalid field
numbers. Our current behavior is that proto.Unmarshal will return an error when
it sees zero and larger than max field numbers and return nil for reserved
ones, which matches the C++ behavior. (Note: depending on which parser helper
in the C++ implementation, one may need to call additional method to check the
result, which we don't have in Go)

Change-Id: I8791fd077f25656107556f5606d55d05c1b4a120
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/191459
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-23 21:35:40 +00:00
Joe Tsai
08ff730048 all: add go1.13rc1 to list of supported Go versions
Change-Id: I2ba52055bcb0e434b66dbcd4763afd2fce5b4264
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/191179
Reviewed-by: Herbie Ong <herbie@google.com>
2019-08-21 21:14:39 +00:00
Joe Tsai
ef6e524dca compiler/protogen: move name mangling logic to protogen
The name mangling logic should be unified in a single place
rather than being split between compiler/protogen and cmd/protoc-gen-go.
Move it over compiler/protogen.

Change-Id: Iea65e0b3fba45e0c95c76e3fc1f061e0fa8f84d7
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/191117
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-21 20:49:58 +00:00
Joe Tsai
4df99fd526 compiler/protogen: add EnumValue.Parent
EnumValues, Fields, Oneofs, and Methods are all declarations that exist
within some parent declaration. The later three have a Parent field.
It seems only consistent that EnumValue have one as well.

Change-Id: I774576565046ede4a96f86ceaa446a39613a39f5
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/191097
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-21 20:40:24 +00:00
Joe Tsai
2cec484ed7 compiler/protogen: export Plugin.FilesByPath
The Plugin.FileByName method seems like an unnecessary indirection that
provides little benefit over just exposing the map.
If the intention to provide some form of immutability, there are already
many leakages in the abstraction given that most things returned by
protogen is by pointer.

Change-Id: I0c793a8d7dab7632b92968a74f8c1ba5accb242f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/191039
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-21 20:35:20 +00:00
Joe Tsai
7762ec2098 compiler/protogen: reorder code
This CL reorders code in the protogen package to consistently match
the typical ordering of enum, message, extension, service seen in
the rest of the module.

Change-Id: I42c3a2ab7ff01857ce9a8e0a71c757b7ea791521
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/191038
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-21 20:31:39 +00:00
Joe Tsai
945a1706d0 all: rely on message_set_extension name mangling in ExtensionDescV1
Too much code inside Google rely on ExtensionDescV1.Name being the
target message name rather than the real name of the extension.
Double down on this assumption consistently across our codebase.

Change-Id: If0b7cd16dd1f2f7acfb2e562dbf1a2b5487162b5
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186980
Reviewed-by: Herbie Ong <herbie@google.com>
2019-08-21 02:47:33 +00:00
Joe Tsai
4f3de44102 release.bash: add support for building release binaries
Updates golang/protobuf#738

Change-Id: I6dd85ff0129bfcfb67b12b06549c568eebfd68e3
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189342
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-21 01:15:26 +00:00
Joe Tsai
769cbd0041 release.bash: add script for making new releases
Change-Id: Ic5754837d1bd93c8b23fc489c6682d1b20dc1c26
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189922
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-21 01:00:05 +00:00
Damien Neil
1de09cbe21 internal/conformance: change test package name to conformance_test
Since there's no conformance package aside from the test, it's cleaner
to keep the test in the conformance_test package.

Change-Id: If9ccfdfc074fa4f390956dd17d066b8613911147
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/190981
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-20 22:30:23 +00:00
Damien Neil
f247a903b9 internal/impl: clean up obsolete ExtensionInfo fields/funcs
This is change 5/5 in a series of commits changing protoV1.ExtensionDesc
to directly implement protoreflect.ExtensionType.

1. [v2] Add protoimpl.ExtensionInfo as an alias for
   protoiface.ExtensionDescV1.

2. [v1] Update references to protoimpl.ExtensionInfo to use
   protoiface.ExtensionInfo.

3. [v2] Create protoimpl.ExtensionInfo (an alias to a new type in
   the impl package) and remove protoiface.ExtensionDescV1.

4. [v1] Remove unneeded explicit conversions between ExtensionDesc and
   ExtensionType (since the former now directly implements the latter).

5. [v2] Remove stub conversion functions.

Change-Id: I1fdfb18c481e72a362a6f9ee0e440f8f909790ca
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189564
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-20 22:03:04 +00:00
Damien Neil
f1e905b042 all: unify protoV1.ExtensionDesc and proto.ExtensionType
Change protoV1.ExtensionDesc to directly implement ExtensionType
rather than delegating to one.

Unify the previous types protoiface.ExtensionDescV1 and
filetype.Extension in impl.ExtensionInfo. The protoV1.ExtensionDesc
type becomes an alias to ExtensionInfo.

This gives us:

  - Just one implementation of ExtensionType.
  - Generated foopb.E_Ext vars are canonical ExtensionTypes.
  - Generated foopb.E_Ext vars are also v1.ExtensionDescs for backwards
    compatibility.
  - Conversion between legacy and modern representations happens
    transparently when lazily initializing an ExtensionInfo.

Overall, a simplification for users of generated code, since they can
mostly ignore the ExtensionDesc/ExtentionType distinction and use the
same value in either the old or new API.

This is change 3/5 in a series of commits changing protoV1.ExtensionDesc
to directly implement protoreflect.ExtensionType.

1. [v2] Add protoimpl.ExtensionInfo as an alias for
   protoiface.ExtensionDescV1.

2. [v1] Update references to protoimpl.ExtensionInfo to use
   protoiface.ExtensionInfo.

3. [v2] Create protoimpl.ExtensionInfo (an alias to a new type in
   the impl package) and remove protoiface.ExtensionDescV1.

4. [v1] Remove unneeded explicit conversions between ExtensionDesc and
   ExtensionType (since the former now directly implements the latter).

5. [v2] Remove stub conversion functions.

Change-Id: I96ee890541ec11b2412e1a72c9d7b96e4d7f66b4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189563
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-20 21:32:57 +00:00
Damien Neil
c0f8c0a24e runtime/protoimpl: add ExtensionInfo alias
This is change 1/5 in a series of commits changing protoV1.ExtensionDesc
to directly implement protoreflect.ExtensionType.

1. [v2] Add protoimpl.ExtensionInfo as an alias for
   protoiface.ExtensionDescV1.

2. [v1] Update references to protoimpl.ExtensionInfo to use
   protoiface.ExtensionInfo.

3. [v2] Create protoimpl.ExtensionInfo (an alias to a new type in
   the impl package) and remove protoiface.ExtensionDescV1.

4. [v1] Remove unneeded explicit conversions between ExtensionDesc and
   ExtensionType (since the former now directly implements the latter).

5. [v2] Remove stub conversion functions.

Change-Id: If6c7fd5f55364613387a05e6f8e9aa38cbfcc5b5
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189562
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-20 20:36:59 +00:00
Joe Tsai
0080e68288 reflect/protoreflect: improve package documentation
There are many types in this package, which can get very confusing how
they all relate to each other. Add some Unicode box art to show the
relationships between XType, XDescriptor, and X.

Change-Id: I935e645907270cdbb70c561e6cb394078366f861
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/190379
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-20 18:44:49 +00:00
Joe Tsai
dd271b6b63 internal/conformance: make conformance test use specific go version
The use of internal/cmd/conformance/conformance.sh means that the test
is not hermetic since the script uses the system version of go,
rather than the specific versions chosen by integration_test.go.

Change the way the conformance test is invoked by turning it
into a Go unit test that integration_test.go can directly call.

Change-Id: I37d23341e1eda984f23f78757a38e862e5fac3d4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/190518
Reviewed-by: Herbie Ong <herbie@google.com>
2019-08-16 22:09:14 +00:00
Joe Tsai
abd06a8b30 all: update to protobuf-v3.9.1 and go-v1.11.13 and go-v1.12.9
A conformance test has been added for the text format.
Update our conformance runner to handle that format and
update with the list of current failures.

Change-Id: I36a271fcfbe328c4ee8c870caff4661659ad27ef
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/190517
Reviewed-by: Herbie Ong <herbie@google.com>
2019-08-16 18:12:10 +00:00
Joe Tsai
9b22b9382e internal/impl: treat a nil oneof wrapper as if it were unset
The old implementation had the behavior where a nil wrapper value:
	m := new(foopb.Message)
	m.OneofField = (*foopb.Message_OneofUint32)(nil)
was functionally equivalent to it being directly set to nil:
	m := new(foopb.Message)
	m.OneofField = nil
preserve this semantic in both the table-drive implementation
and the reflection implementation.

Change-Id: Ie44d51e044d4822e61d0e646fbc44aa8d9b90c1f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189559
Reviewed-by: Herbie Ong <herbie@google.com>
2019-08-16 00:24:53 +00:00
Joe Tsai
7328839f81 cmd/protoc-gen-go: annotate depIdxs list with index comments
Generate the current index into depIdxs for easier human debugging.

Change-Id: Ida42aa95137b2044a4dc267c31cebec5023bdfb1
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/190278
Reviewed-by: Herbie Ong <herbie@google.com>
2019-08-15 22:47:44 +00:00
Joe Tsai
0484b1a125 internal/impl: refactor MessageInfo
Changes made:
* Move MessageInfo.{methods,extensionFieldInfosMu,extensionFieldInfos}
to the coderMessageInfo type since they are fields all related to the
fast-path implementation, which is what the coderMessageInfo is for.
* Rename message_field.go -> message_reflect_field.go to make it obvious
from the file name that this only deals with the reflection implementation.
* Rename message_test.go -> message_reflect_test.go.
* Move reflection-specific implementation functions from message.go
to message_reflect.go. The intention is to make it such that message.go
is the entry point to message implementations and is agnostic towards
whether we are implementing reflection or the table-driven fast path.

Overall, there is no semantic changes. Just code movement.

Change-Id: I7743c39ba84dc63cd2a02934c3319285e16d6b1c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/190100
Reviewed-by: Herbie Ong <herbie@google.com>
2019-08-15 22:26:40 +00:00
Joe Tsai
17581daabb cmd/protoc-gen-go: fix struct tag formatting
If a string default value contains a backtick, it breaks the
struct tag formatting logic which assumes such characters never exist.
Fix this by adding a structTags type, whose String method knows how
to properly escape the value.

Change-Id: I96fe5a6630387eac89eef429da5f917125570e4c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189557
Reviewed-by: Herbie Ong <herbie@google.com>
2019-08-15 22:18:29 +00:00
Damien Neil
16163b4f67 all: drop reflect/prototype package
Remove the remaining uses of the prototype package.

The most significant change is to impl.MessageInfo, which now directly
implements the MessageType interface. This involves two notable changes
to exported fields of MessageInfo:

  - PBType is now Desc.
  - GoType is now GoReflectType. (Name changed to avoid a conflict with
    the GoType method of the MessageType interface.)

Fixes golang/protobuf#911

Change-Id: Ie2aa4766d6887ceaa9cf06b1f109aa6e6e2a208f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189340
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-09 19:06:30 +00:00
Damien Neil
4401a0de4b cmd/protoc-gen-go, internal/filetype: clean up EnumType construction
Drop the dependency from generated files on prototype.Enum: Generated
code should only depend on runtime/proto{iface,impl}.

Drop the Enums, Messages, and Extensions returns form
filetype.Builder.Build. Of these, only Enums was used by generated code.
Change the generated init function to pass the builder a slice of values
to fill in (as is done for messages and extensions).

Remove the filetype dependency on prototype in preparation for
eventually dropping the prototype package entirely.

Change-Id: I28a3420f5dfcc13fed531a64ef07b9afddfd9d55
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189200
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-09 19:04:41 +00:00
Joe Tsai
1799d1111a all: rename tag and flags for legacy support
Rename build tag "proto1_legacy" -> "protolegacy"
to be consistent with the "protoreflect" tag.

Rename flag constant "Proto1Legacy" -> "ProtoLegacy" since
it covers more than simply proto1 legacy features.
For example, it covers alpha-features of proto3 that
were eventually removed from the final proto3 release.

Change-Id: I0f4fcbadd4b5a61c87645e2e5be11d187e59157c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189345
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-08 20:49:00 +00:00
Joe Tsai
7164af5a9d test.bash: refactor command execution logic
Refactor the command execution logic to be more configurable.

Change-Id: I6d21f6c3f38691a8ffc992a170ff221c8a06f990
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189341
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-08 20:23:56 +00:00
Damien Neil
92f76189a3 all: refactor extensions, add proto.GetExtension etc.
Change protoiface.ExtensionDescV1 to implement protoreflect.ExtensionType.

ExtensionDescV1's Name field conflicts with the Descriptor Name method,
so change the protoreflect.{Message,Enum,Extension}Type types to no
longer implement the corresponding Descriptor interface. This also leads
to a clearer distinction between the two types.

Introduce a protoreflect.ExtensionTypeDescriptor type which bridges
between ExtensionType and ExtensionDescriptor.

Add extension accessor functions to the proto package:
proto.{Has,Clear,Get,Set}Extension. These functions take a
protoreflect.ExtensionType parameter, which allows writing the
same function call using either the old or new API:

  proto.GetExtension(message, somepb.E_ExtensionFoo)

Fixes golang/protobuf#908

Change-Id: Ibc65d12a46666297849114fd3aefbc4a597d9f08
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189199
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-08 18:20:51 +00:00
Damien Neil
d4f0800c42 all: make handling of zero-value composites more consistent
We occasionally need to work with immutable, empty lists, maps, and
messages. Notably, Message.Get on an empty repeated field will return a
"frozen" empty value.

Move handling of these immutable, zero-length composites into Converter,
to unify the behavior of regular and extension fields.

Add a Zero method to Converter, MessageType, and ExtensionType, to
provide a consistent way to get an empty, frozen value of a composite
type. Adding this method to the public {Message,Extension}Type
interfaces does increase our API surface, but lets us (for example)
cleanly represent an empty map as a nil map rather than a non-nil
one wrapped in a frozenMap type.

Drop the frozen{List,Map,Message} types as no longer necessary.
(These types did have support for creating a read-only view of a
non-empty value, but we are not currently using that feature.)

Change-Id: Ia76f149d591da07b40ce75b7404a7ab8a60cb9d8
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189339
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-08 18:20:25 +00:00
Joe Tsai
bab3d4084e runtime/protoimpl, cmd/protoc-gen-go: support release versioning
In order for protoc-gen-go to output the current version,
it needs to know what version it is currently running as.
However, we cannot rely on the git tags since the tags are not
made until *after* the commit has been submitted.
Instead, we manually encode the version into the code and
make sure that git tags match up with the version in the code.

The version.go file in runtime/protoimpl contains instructions
for how to make a release. Essentially:
* Every non-release commit has a version string with "devel" in it.
* Every release commit must not have "devel" in it and must be unique.
* The "release process" involves submitting two CLs.
The first CL creates a version string without "devel",
which is the commit that a git tag will actually reference.
The second CL follows immediately and re-introduces "devel"
into the version string.

The following example shows a possible sequence of VersionStrings
for git commits in time-ascending order:
	v1.19.0-devel      (this CL)
	v1.19.0-devel
	v1.19.0-devel
	v1.19.0-devel
	v1.20.0-rc.1       <- tagged
	v1.20.0-rc.1.devel
	v1.20.0-rc.1.devel
	v1.20.0-rc.1.devel
	v1.20.0-rc.2       <- tagged
	v1.20.0-rc.2.devel
	v1.20.0            <- tagged (future public release)
	v1.20.0-devel
	v1.20.0-devel
	v1.20.0-devel
	v1.20.0-devel
	v1.20.1            <- tagged
	v1.20.1-devel
	v1.20.1-devel
	v1.21.0            <- tagged
	v1.21.0-devel

Note that we start today with v1.19.0-devel, which means that our initial
release will be v1.20.0. This number was intentionally chosen since
1) the number 20 has some correlation to the fact that we keep calling
the new implementation the "v2" implementation, and
2) the set of tagged versions for github.com/golang/protobuf
and google.golang.org/protobuf are unlikely to ever overlap.
This way, the version of protoc-gen-go is never ambiguous which module
it was built from.

Now that we have version information, we add support for generating .pb.go
files with the version information recorded. However, we do not emit
these for .pb.go files in our own repository since they are always guaranteed
to be at the right version (enforced by integration_test.go).

Updates golang/protobuf#524

Change-Id: I25495a45042c2aa39a39cb7e7738ae8e831a9d26
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186117
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-07 22:59:30 +00:00
Joe Tsai
8d5e6d6927 cmd/protoc-gen-go: improve generation of comments
The following improvements were made:
* All standalone comments above the "syntax" marker are preserved
similar to Java and some other generators.
* All standalone comments above the "package" marker are preserved
to be consistent with our former behavior.
* Leading comments are now generated for enums and extension fields.
* Single-line trailing comments are now generated for
enum values, message fields, and extension fields.
* The leading comments for each field that is part of a oneof are now
generated with the wrapper types rather than being shoved into the
comment for the oneof itself in an unreadable way.
* The deprecation marker is always generated as being above the declaration
rather than sometimes being an inlined comment.
* The deprecation marker is now properly generated for weak field setters.

Updates golang/protobuf#666

Change-Id: I7fd832dd4f86d15bfff70d7c22c6ba4934c05fcf
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189238
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-07 17:33:08 +00:00
Joe Tsai
70fdd5d140 compiler/protogen, cmd/protoc-gen-go: use alternative comments API
This is a breaking change. High-level protogen API changes:
* remove GeneratedFile.PrintLeadingComments method
* add {Message,Field,Oneof,Enum,EnumValue,Service,Method}.Comments field
* add CommentSet and Comments type

CL/183157 added protoreflect.SourceLocations and it was discovered
that there can actually be duplicate locations for certain paths.
For that reason, we decided not to expose any helper methods
for looking up locations by path since it is unclear which location
to return if multiple matches.

The protogen.GeneratedFile.PrintLeadingComments has a similar dilemma
where it also needs to figure out what to do when duplicates exist.
Previously, it just chooses the first one with comments,
which may not be the right choice in a given context.

Analysis of current PrintLeadingComments usage shows that it is only
ever used (except once) for descriptor declarations.
In the case of descriptor declarations, they are guaranteed by protoc
to have only location.

Thus, we avoid the duplicate location problem by:
* Providing a CommentSet for every descriptor. The CommentSet contains
a set of leading and trailing comments of the Comments type.
* The Comments.String method knows how to interpret the comments
as provided by protoc and format them as // prefixed line comments.
* Values of the Comments type can be passed to the P method.

We drop direct support printing leading comments for non-descriptor locations,
but the exposure of the Comments type makes it easy for users to manually
handle other types of comments themselves.

Change-Id: Id4851456dc4e64d76bd6a30e8ad6137408dfb27a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189198
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-07 06:00:36 +00:00
Joe Tsai
4a7d633604 cmd/protoc-gen-go: group extension variable declarations
For better readability in godoc, group extension fields by the
target message that they are extending.

Change-Id: Icc0a247b37639e3dbf7a107810923b8ca8294724
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189257
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-07 04:26:34 +00:00
Damien Neil
f5274511fe all: add NewField, NewElement, NewValue
Add methods to protoreflect.{Message,List,Map} to constrict values
assignable to a message field, list element, or map value. These
methods return the default value for scalar fields, the zero value for
scalar list elements and map values, and an empty, mutable value for
messages, lists, and maps.

Deprecate the NewMessage methods on these types, which are superseded.

Updates golang/protobuf#879

Change-Id: I0f064f60c89a239330ccea81523f559f14fd2c4f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/188997
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-06 21:54:30 +00:00
Joe Tsai
c36f3ae703 .gitignore: add policy for new entries
Change-Id: Ib3592d0ed265ca75703eb2dceb9f26a6dbecfcb7
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189197
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-06 21:42:16 +00:00