736 Commits

Author SHA1 Message Date
Damien Neil
79bfdbe45b all: rename ExtensionType Descriptor method to TypeDescriptor (1/2)
Descriptor methods generally return a Descriptor with no Go type
information. ExtensionType's Descriptor is an exception, returning an
ExtensionTypeDescriptor containing both the proto descriptor and a
reference back to the ExtensionType. The pure descriptor is accessed
by xt.Descriptor().Descriptor().

Rename ExtensionType's Descriptor method to TypeDescriptor to make it
clear that it behaves a bit differently.

Change 1/2: Add the TypeDescriptor method and deprecate Descriptor.

Change-Id: I1806095044d35a474d60f94d2a28bdf528f12238
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/192139
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-28 18:34:29 +00:00
Damien Neil
95758c0892 internal/filetype, internal/filedesc: avoid gccgo bug
Returning an anonymous struct is tickling a gccgo bug:
https://github.com/golang/go/issues/33866

Change to a named type. We could make the type unexpoerted, but this is
an internal package anyway so leave it exported for general style
cleanliness.

Change-Id: Idf33f1e354c0e07066ab68640308af1498a01447
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/191960
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-27 19:57:09 +00:00
Joe Tsai
26aef9d6d6 cmd/protoc-gen-go: add support for field-tracking
Field-tracking is an experimental feature in the Go linker
where it statically tracks whether a Go struct field is used.
This CL modifies the generator to emit special markers that
tells the linker to know which fields to track.

Change-Id: I235da3f3d6c0ef2021b5fe1c16106ebb8fd6f557
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189558
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-26 17:54:17 +00:00
Joe Tsai
2e7817f117 compiler/protogen, internal/strs, internal/impl: expose enum Go name derivation
In order to migrate v1 to wrap v2, we need a way to reproduce
the awful enum "names" that v1 used, which was the concatenation of
the proto package with the Go identifier used for the enum.

To support this:
* Move the camel case logic from compiler/protogen to internal/strs
* Add a small stub in internal/impl to expose this functionality

Change-Id: I8ff31daa9ae541e5788dc04d2e89eae1574877e4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/191637
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-26 17:49:17 +00:00
Damien Neil
c5060d2fe6 reflect/protoreflect: add non-allocating Value constructors
Passing a non-pointer type to protoreflect.NewValue causes an
unnecessary allocation in order to store the value in an interface{}.
While this allocation could be avoided by a smarter compiler, no such
compiler exists today.

Add functions for creating new values of a specific type, avoiding the
allocation. (And also adding a small amount of type safety, although
this is unlikely to be important.)

Update the proto and internal/impl packages to use these functions.

Change-Id: Ic733de22ddf19c530189166c853348e1b54b7391
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/191457
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-26 17:48:05 +00:00
Tuo Shan
6e25d8c6a6 proto: add tests for unmarshalling invalid field numbers
This change adds tests for unmarshalling fields with various invalid field
numbers. Our current behavior is that proto.Unmarshal will return an error when
it sees zero and larger than max field numbers and return nil for reserved
ones, which matches the C++ behavior. (Note: depending on which parser helper
in the C++ implementation, one may need to call additional method to check the
result, which we don't have in Go)

Change-Id: I8791fd077f25656107556f5606d55d05c1b4a120
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/191459
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-23 21:35:40 +00:00
Joe Tsai
08ff730048 all: add go1.13rc1 to list of supported Go versions
Change-Id: I2ba52055bcb0e434b66dbcd4763afd2fce5b4264
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/191179
Reviewed-by: Herbie Ong <herbie@google.com>
2019-08-21 21:14:39 +00:00
Joe Tsai
ef6e524dca compiler/protogen: move name mangling logic to protogen
The name mangling logic should be unified in a single place
rather than being split between compiler/protogen and cmd/protoc-gen-go.
Move it over compiler/protogen.

Change-Id: Iea65e0b3fba45e0c95c76e3fc1f061e0fa8f84d7
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/191117
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-21 20:49:58 +00:00
Joe Tsai
4df99fd526 compiler/protogen: add EnumValue.Parent
EnumValues, Fields, Oneofs, and Methods are all declarations that exist
within some parent declaration. The later three have a Parent field.
It seems only consistent that EnumValue have one as well.

Change-Id: I774576565046ede4a96f86ceaa446a39613a39f5
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/191097
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-21 20:40:24 +00:00
Joe Tsai
2cec484ed7 compiler/protogen: export Plugin.FilesByPath
The Plugin.FileByName method seems like an unnecessary indirection that
provides little benefit over just exposing the map.
If the intention to provide some form of immutability, there are already
many leakages in the abstraction given that most things returned by
protogen is by pointer.

Change-Id: I0c793a8d7dab7632b92968a74f8c1ba5accb242f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/191039
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-21 20:35:20 +00:00
Joe Tsai
7762ec2098 compiler/protogen: reorder code
This CL reorders code in the protogen package to consistently match
the typical ordering of enum, message, extension, service seen in
the rest of the module.

Change-Id: I42c3a2ab7ff01857ce9a8e0a71c757b7ea791521
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/191038
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-21 20:31:39 +00:00
Joe Tsai
945a1706d0 all: rely on message_set_extension name mangling in ExtensionDescV1
Too much code inside Google rely on ExtensionDescV1.Name being the
target message name rather than the real name of the extension.
Double down on this assumption consistently across our codebase.

Change-Id: If0b7cd16dd1f2f7acfb2e562dbf1a2b5487162b5
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186980
Reviewed-by: Herbie Ong <herbie@google.com>
2019-08-21 02:47:33 +00:00
Joe Tsai
4f3de44102 release.bash: add support for building release binaries
Updates golang/protobuf#738

Change-Id: I6dd85ff0129bfcfb67b12b06549c568eebfd68e3
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189342
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-21 01:15:26 +00:00
Joe Tsai
769cbd0041 release.bash: add script for making new releases
Change-Id: Ic5754837d1bd93c8b23fc489c6682d1b20dc1c26
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189922
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-21 01:00:05 +00:00
Damien Neil
1de09cbe21 internal/conformance: change test package name to conformance_test
Since there's no conformance package aside from the test, it's cleaner
to keep the test in the conformance_test package.

Change-Id: If9ccfdfc074fa4f390956dd17d066b8613911147
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/190981
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-20 22:30:23 +00:00
Damien Neil
f247a903b9 internal/impl: clean up obsolete ExtensionInfo fields/funcs
This is change 5/5 in a series of commits changing protoV1.ExtensionDesc
to directly implement protoreflect.ExtensionType.

1. [v2] Add protoimpl.ExtensionInfo as an alias for
   protoiface.ExtensionDescV1.

2. [v1] Update references to protoimpl.ExtensionInfo to use
   protoiface.ExtensionInfo.

3. [v2] Create protoimpl.ExtensionInfo (an alias to a new type in
   the impl package) and remove protoiface.ExtensionDescV1.

4. [v1] Remove unneeded explicit conversions between ExtensionDesc and
   ExtensionType (since the former now directly implements the latter).

5. [v2] Remove stub conversion functions.

Change-Id: I1fdfb18c481e72a362a6f9ee0e440f8f909790ca
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189564
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-20 22:03:04 +00:00
Damien Neil
f1e905b042 all: unify protoV1.ExtensionDesc and proto.ExtensionType
Change protoV1.ExtensionDesc to directly implement ExtensionType
rather than delegating to one.

Unify the previous types protoiface.ExtensionDescV1 and
filetype.Extension in impl.ExtensionInfo. The protoV1.ExtensionDesc
type becomes an alias to ExtensionInfo.

This gives us:

  - Just one implementation of ExtensionType.
  - Generated foopb.E_Ext vars are canonical ExtensionTypes.
  - Generated foopb.E_Ext vars are also v1.ExtensionDescs for backwards
    compatibility.
  - Conversion between legacy and modern representations happens
    transparently when lazily initializing an ExtensionInfo.

Overall, a simplification for users of generated code, since they can
mostly ignore the ExtensionDesc/ExtentionType distinction and use the
same value in either the old or new API.

This is change 3/5 in a series of commits changing protoV1.ExtensionDesc
to directly implement protoreflect.ExtensionType.

1. [v2] Add protoimpl.ExtensionInfo as an alias for
   protoiface.ExtensionDescV1.

2. [v1] Update references to protoimpl.ExtensionInfo to use
   protoiface.ExtensionInfo.

3. [v2] Create protoimpl.ExtensionInfo (an alias to a new type in
   the impl package) and remove protoiface.ExtensionDescV1.

4. [v1] Remove unneeded explicit conversions between ExtensionDesc and
   ExtensionType (since the former now directly implements the latter).

5. [v2] Remove stub conversion functions.

Change-Id: I96ee890541ec11b2412e1a72c9d7b96e4d7f66b4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189563
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-20 21:32:57 +00:00
Damien Neil
c0f8c0a24e runtime/protoimpl: add ExtensionInfo alias
This is change 1/5 in a series of commits changing protoV1.ExtensionDesc
to directly implement protoreflect.ExtensionType.

1. [v2] Add protoimpl.ExtensionInfo as an alias for
   protoiface.ExtensionDescV1.

2. [v1] Update references to protoimpl.ExtensionInfo to use
   protoiface.ExtensionInfo.

3. [v2] Create protoimpl.ExtensionInfo (an alias to a new type in
   the impl package) and remove protoiface.ExtensionDescV1.

4. [v1] Remove unneeded explicit conversions between ExtensionDesc and
   ExtensionType (since the former now directly implements the latter).

5. [v2] Remove stub conversion functions.

Change-Id: If6c7fd5f55364613387a05e6f8e9aa38cbfcc5b5
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189562
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-20 20:36:59 +00:00
Joe Tsai
0080e68288 reflect/protoreflect: improve package documentation
There are many types in this package, which can get very confusing how
they all relate to each other. Add some Unicode box art to show the
relationships between XType, XDescriptor, and X.

Change-Id: I935e645907270cdbb70c561e6cb394078366f861
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/190379
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-20 18:44:49 +00:00
Joe Tsai
dd271b6b63 internal/conformance: make conformance test use specific go version
The use of internal/cmd/conformance/conformance.sh means that the test
is not hermetic since the script uses the system version of go,
rather than the specific versions chosen by integration_test.go.

Change the way the conformance test is invoked by turning it
into a Go unit test that integration_test.go can directly call.

Change-Id: I37d23341e1eda984f23f78757a38e862e5fac3d4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/190518
Reviewed-by: Herbie Ong <herbie@google.com>
2019-08-16 22:09:14 +00:00
Joe Tsai
abd06a8b30 all: update to protobuf-v3.9.1 and go-v1.11.13 and go-v1.12.9
A conformance test has been added for the text format.
Update our conformance runner to handle that format and
update with the list of current failures.

Change-Id: I36a271fcfbe328c4ee8c870caff4661659ad27ef
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/190517
Reviewed-by: Herbie Ong <herbie@google.com>
2019-08-16 18:12:10 +00:00
Joe Tsai
9b22b9382e internal/impl: treat a nil oneof wrapper as if it were unset
The old implementation had the behavior where a nil wrapper value:
	m := new(foopb.Message)
	m.OneofField = (*foopb.Message_OneofUint32)(nil)
was functionally equivalent to it being directly set to nil:
	m := new(foopb.Message)
	m.OneofField = nil
preserve this semantic in both the table-drive implementation
and the reflection implementation.

Change-Id: Ie44d51e044d4822e61d0e646fbc44aa8d9b90c1f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189559
Reviewed-by: Herbie Ong <herbie@google.com>
2019-08-16 00:24:53 +00:00
Joe Tsai
7328839f81 cmd/protoc-gen-go: annotate depIdxs list with index comments
Generate the current index into depIdxs for easier human debugging.

Change-Id: Ida42aa95137b2044a4dc267c31cebec5023bdfb1
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/190278
Reviewed-by: Herbie Ong <herbie@google.com>
2019-08-15 22:47:44 +00:00
Joe Tsai
0484b1a125 internal/impl: refactor MessageInfo
Changes made:
* Move MessageInfo.{methods,extensionFieldInfosMu,extensionFieldInfos}
to the coderMessageInfo type since they are fields all related to the
fast-path implementation, which is what the coderMessageInfo is for.
* Rename message_field.go -> message_reflect_field.go to make it obvious
from the file name that this only deals with the reflection implementation.
* Rename message_test.go -> message_reflect_test.go.
* Move reflection-specific implementation functions from message.go
to message_reflect.go. The intention is to make it such that message.go
is the entry point to message implementations and is agnostic towards
whether we are implementing reflection or the table-driven fast path.

Overall, there is no semantic changes. Just code movement.

Change-Id: I7743c39ba84dc63cd2a02934c3319285e16d6b1c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/190100
Reviewed-by: Herbie Ong <herbie@google.com>
2019-08-15 22:26:40 +00:00
Joe Tsai
17581daabb cmd/protoc-gen-go: fix struct tag formatting
If a string default value contains a backtick, it breaks the
struct tag formatting logic which assumes such characters never exist.
Fix this by adding a structTags type, whose String method knows how
to properly escape the value.

Change-Id: I96fe5a6630387eac89eef429da5f917125570e4c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189557
Reviewed-by: Herbie Ong <herbie@google.com>
2019-08-15 22:18:29 +00:00
Damien Neil
16163b4f67 all: drop reflect/prototype package
Remove the remaining uses of the prototype package.

The most significant change is to impl.MessageInfo, which now directly
implements the MessageType interface. This involves two notable changes
to exported fields of MessageInfo:

  - PBType is now Desc.
  - GoType is now GoReflectType. (Name changed to avoid a conflict with
    the GoType method of the MessageType interface.)

Fixes golang/protobuf#911

Change-Id: Ie2aa4766d6887ceaa9cf06b1f109aa6e6e2a208f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189340
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-09 19:06:30 +00:00
Damien Neil
4401a0de4b cmd/protoc-gen-go, internal/filetype: clean up EnumType construction
Drop the dependency from generated files on prototype.Enum: Generated
code should only depend on runtime/proto{iface,impl}.

Drop the Enums, Messages, and Extensions returns form
filetype.Builder.Build. Of these, only Enums was used by generated code.
Change the generated init function to pass the builder a slice of values
to fill in (as is done for messages and extensions).

Remove the filetype dependency on prototype in preparation for
eventually dropping the prototype package entirely.

Change-Id: I28a3420f5dfcc13fed531a64ef07b9afddfd9d55
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189200
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-09 19:04:41 +00:00
Joe Tsai
1799d1111a all: rename tag and flags for legacy support
Rename build tag "proto1_legacy" -> "protolegacy"
to be consistent with the "protoreflect" tag.

Rename flag constant "Proto1Legacy" -> "ProtoLegacy" since
it covers more than simply proto1 legacy features.
For example, it covers alpha-features of proto3 that
were eventually removed from the final proto3 release.

Change-Id: I0f4fcbadd4b5a61c87645e2e5be11d187e59157c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189345
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-08 20:49:00 +00:00
Joe Tsai
7164af5a9d test.bash: refactor command execution logic
Refactor the command execution logic to be more configurable.

Change-Id: I6d21f6c3f38691a8ffc992a170ff221c8a06f990
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189341
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-08 20:23:56 +00:00
Damien Neil
92f76189a3 all: refactor extensions, add proto.GetExtension etc.
Change protoiface.ExtensionDescV1 to implement protoreflect.ExtensionType.

ExtensionDescV1's Name field conflicts with the Descriptor Name method,
so change the protoreflect.{Message,Enum,Extension}Type types to no
longer implement the corresponding Descriptor interface. This also leads
to a clearer distinction between the two types.

Introduce a protoreflect.ExtensionTypeDescriptor type which bridges
between ExtensionType and ExtensionDescriptor.

Add extension accessor functions to the proto package:
proto.{Has,Clear,Get,Set}Extension. These functions take a
protoreflect.ExtensionType parameter, which allows writing the
same function call using either the old or new API:

  proto.GetExtension(message, somepb.E_ExtensionFoo)

Fixes golang/protobuf#908

Change-Id: Ibc65d12a46666297849114fd3aefbc4a597d9f08
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189199
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-08 18:20:51 +00:00
Damien Neil
d4f0800c42 all: make handling of zero-value composites more consistent
We occasionally need to work with immutable, empty lists, maps, and
messages. Notably, Message.Get on an empty repeated field will return a
"frozen" empty value.

Move handling of these immutable, zero-length composites into Converter,
to unify the behavior of regular and extension fields.

Add a Zero method to Converter, MessageType, and ExtensionType, to
provide a consistent way to get an empty, frozen value of a composite
type. Adding this method to the public {Message,Extension}Type
interfaces does increase our API surface, but lets us (for example)
cleanly represent an empty map as a nil map rather than a non-nil
one wrapped in a frozenMap type.

Drop the frozen{List,Map,Message} types as no longer necessary.
(These types did have support for creating a read-only view of a
non-empty value, but we are not currently using that feature.)

Change-Id: Ia76f149d591da07b40ce75b7404a7ab8a60cb9d8
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189339
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-08 18:20:25 +00:00
Joe Tsai
bab3d4084e runtime/protoimpl, cmd/protoc-gen-go: support release versioning
In order for protoc-gen-go to output the current version,
it needs to know what version it is currently running as.
However, we cannot rely on the git tags since the tags are not
made until *after* the commit has been submitted.
Instead, we manually encode the version into the code and
make sure that git tags match up with the version in the code.

The version.go file in runtime/protoimpl contains instructions
for how to make a release. Essentially:
* Every non-release commit has a version string with "devel" in it.
* Every release commit must not have "devel" in it and must be unique.
* The "release process" involves submitting two CLs.
The first CL creates a version string without "devel",
which is the commit that a git tag will actually reference.
The second CL follows immediately and re-introduces "devel"
into the version string.

The following example shows a possible sequence of VersionStrings
for git commits in time-ascending order:
	v1.19.0-devel      (this CL)
	v1.19.0-devel
	v1.19.0-devel
	v1.19.0-devel
	v1.20.0-rc.1       <- tagged
	v1.20.0-rc.1.devel
	v1.20.0-rc.1.devel
	v1.20.0-rc.1.devel
	v1.20.0-rc.2       <- tagged
	v1.20.0-rc.2.devel
	v1.20.0            <- tagged (future public release)
	v1.20.0-devel
	v1.20.0-devel
	v1.20.0-devel
	v1.20.0-devel
	v1.20.1            <- tagged
	v1.20.1-devel
	v1.20.1-devel
	v1.21.0            <- tagged
	v1.21.0-devel

Note that we start today with v1.19.0-devel, which means that our initial
release will be v1.20.0. This number was intentionally chosen since
1) the number 20 has some correlation to the fact that we keep calling
the new implementation the "v2" implementation, and
2) the set of tagged versions for github.com/golang/protobuf
and google.golang.org/protobuf are unlikely to ever overlap.
This way, the version of protoc-gen-go is never ambiguous which module
it was built from.

Now that we have version information, we add support for generating .pb.go
files with the version information recorded. However, we do not emit
these for .pb.go files in our own repository since they are always guaranteed
to be at the right version (enforced by integration_test.go).

Updates golang/protobuf#524

Change-Id: I25495a45042c2aa39a39cb7e7738ae8e831a9d26
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186117
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-07 22:59:30 +00:00
Joe Tsai
8d5e6d6927 cmd/protoc-gen-go: improve generation of comments
The following improvements were made:
* All standalone comments above the "syntax" marker are preserved
similar to Java and some other generators.
* All standalone comments above the "package" marker are preserved
to be consistent with our former behavior.
* Leading comments are now generated for enums and extension fields.
* Single-line trailing comments are now generated for
enum values, message fields, and extension fields.
* The leading comments for each field that is part of a oneof are now
generated with the wrapper types rather than being shoved into the
comment for the oneof itself in an unreadable way.
* The deprecation marker is always generated as being above the declaration
rather than sometimes being an inlined comment.
* The deprecation marker is now properly generated for weak field setters.

Updates golang/protobuf#666

Change-Id: I7fd832dd4f86d15bfff70d7c22c6ba4934c05fcf
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189238
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-07 17:33:08 +00:00
Joe Tsai
70fdd5d140 compiler/protogen, cmd/protoc-gen-go: use alternative comments API
This is a breaking change. High-level protogen API changes:
* remove GeneratedFile.PrintLeadingComments method
* add {Message,Field,Oneof,Enum,EnumValue,Service,Method}.Comments field
* add CommentSet and Comments type

CL/183157 added protoreflect.SourceLocations and it was discovered
that there can actually be duplicate locations for certain paths.
For that reason, we decided not to expose any helper methods
for looking up locations by path since it is unclear which location
to return if multiple matches.

The protogen.GeneratedFile.PrintLeadingComments has a similar dilemma
where it also needs to figure out what to do when duplicates exist.
Previously, it just chooses the first one with comments,
which may not be the right choice in a given context.

Analysis of current PrintLeadingComments usage shows that it is only
ever used (except once) for descriptor declarations.
In the case of descriptor declarations, they are guaranteed by protoc
to have only location.

Thus, we avoid the duplicate location problem by:
* Providing a CommentSet for every descriptor. The CommentSet contains
a set of leading and trailing comments of the Comments type.
* The Comments.String method knows how to interpret the comments
as provided by protoc and format them as // prefixed line comments.
* Values of the Comments type can be passed to the P method.

We drop direct support printing leading comments for non-descriptor locations,
but the exposure of the Comments type makes it easy for users to manually
handle other types of comments themselves.

Change-Id: Id4851456dc4e64d76bd6a30e8ad6137408dfb27a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189198
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-07 06:00:36 +00:00
Joe Tsai
4a7d633604 cmd/protoc-gen-go: group extension variable declarations
For better readability in godoc, group extension fields by the
target message that they are extending.

Change-Id: Icc0a247b37639e3dbf7a107810923b8ca8294724
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189257
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-07 04:26:34 +00:00
Damien Neil
f5274511fe all: add NewField, NewElement, NewValue
Add methods to protoreflect.{Message,List,Map} to constrict values
assignable to a message field, list element, or map value. These
methods return the default value for scalar fields, the zero value for
scalar list elements and map values, and an empty, mutable value for
messages, lists, and maps.

Deprecate the NewMessage methods on these types, which are superseded.

Updates golang/protobuf#879

Change-Id: I0f064f60c89a239330ccea81523f559f14fd2c4f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/188997
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-06 21:54:30 +00:00
Joe Tsai
c36f3ae703 .gitignore: add policy for new entries
Change-Id: Ib3592d0ed265ca75703eb2dceb9f26a6dbecfcb7
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189197
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-06 21:42:16 +00:00
Joe Tsai
9b8a433283 cmd/protoc-gen-go: group enum map vars in blocks
Since the enum maps are here to stay, group the declarations together
in a var block for better readability in godoc.

Change-Id: I9a313266539e9a60781f98b80a5293379f82607b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189077
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-06 21:33:36 +00:00
Joe Tsai
d29a71bff0 cmd/protoc-gen-go: group default consts and vars in blocks
Group the default constant and variable declarations together in a block
for better readability in godoc.

Change-Id: I6b62f5374f0302d0f7cb224cbe34102359c8c51d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189057
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-06 21:23:53 +00:00
Joe Tsai
e815d6a43b all: remove dead code
Change-Id: I1344d6afca9d3348db849c2b5f387ac18b80d2ba
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189021
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-06 21:16:48 +00:00
Joe Tsai
52ec1759a4 internal/filedesc, internal/filetype: rename {Desc,Type}Builder as Builder
Reduce the stutter in the name since the type of Builder
is obvious from the package it is from.

Change-Id: I0046a5122717536cc6bb5ebdb32b67a1560cfc23
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189020
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-05 23:43:37 +00:00
Damien Neil
954bd92c19 all: refactor Converter
A Converter converts between reflect.Values and protoreflect.Values.
The existing usage of Converter is somewhat confusing: The
internal/value package creates Converters for scalar types only, the
internal/impl package creates Converters for legacy messages and enums,
and the reflect/prototype package creates Converters for repeated fields.

Change the Converter type to an interface. The constructor for
Converter takes a FieldDescriptor and reflect.Type, and directly
handles conversions for all field types: Scalars, lists, maps, and
legacy types.

Move Converter into the internal/impl package, since that package
contains the necessary support for dealing with legacy messages and
enums. Drop the internal/value package.

Replace two uses of prototype.Extension with more focused
implementations, since the implementation is trivial with the
refactored Converter. Drop prototype.Extension for the moment since
it is now unused.

Change-Id: If0c570fefac002cc5925b3d56281b6eb17e90d5f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/187857
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-05 22:33:14 +00:00
Joe Tsai
f542220747 test.bash: add -failfast flag
Rather than waiting for all tests for all Go versions to finish
only to find that there was some minor breakage,
fail fast so that we stop wasting time.

Change-Id: Ie255ceb5ac2cbdc598a074c6da281f5e49eb1326
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189019
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-05 21:48:09 +00:00
Damien Neil
8003f08e51 proto, internal/impl: zero-length proto2 bytes fields should be non-nil
Fix decoding of zero-length bytes fields to produce a non-nil []byte.

Change-Id: Ifb7791a47df81091700f7226523371d1386fb1ad
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/188765
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-05 21:44:30 +00:00
Joe Tsai
afd3633ce3 internal/impl: defer evaluation of weak reference until actual use
Suppose some init logic performs protobuf reflection.
If so, it will cause the table-driven logic for protobuf reflection
to be initialized. This is problematic for weak fields since we
can not be certain that all weak references have been registered
at this point in time. This is a fundamental issue with with weak
dependencies, since it means that we cannot enforce any ordering
constraints on the weak dependency unless we directly import the weakly
referenced package (which would defeat the point of weak imports).

Alleviate the problem by pushing evaluation of weak reference to
actual usage. This does not completely fix the problem,
but signifcantly reduces the probability of it being problematic.
In general, people should avoid interacting with weak fields at init time.

Change-Id: Iebaefddde8cf07b5cd7dee49b7015b05b5428618
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/188980
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-05 21:38:46 +00:00
Joe Tsai
f98dd98c15 internal/filedesc: defer evaluation of weak reference until actual use
Suppose some init logic performs protobuf reflection.
If so, it will cause the lazy descriptor init logic to be executed.
This is problematic for weak fields since we can not be certain that
all weak references have been registered at this point in time.
This is a fundamental issue with with weak dependencies,
since it means that we cannot enforce any ordering constraints
on the weak dependency unless we directly import the weakly referenced package
(which would defeat the point of weak imports).

Alleviate the problem by pushing evaluation of weak reference to
actual usage. This does not completely fix the problem,
but signifcantly reduces the probability of it being problematic.
In general, people should avoid interacting with weak fields at init time.

Change-Id: Ie5957ddedd61333e72ee9a1bba0c53dace65547c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/188982
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-05 21:31:06 +00:00
Joe Tsai
2aea614c5e internal/impl: fix race over messageState.mi
The messageState.mi field is atomically checked and set
in generated code to the *MessageInfo associated with that message.
However, the messageState type accesses the mi field without
any atomic loads, thus being a potential race.
We fix this by always calling a messageInfo method that performs
a atomic.LoadPointer on the *MessageInfo.

There is no performance effect from this change on x86 since
an atomic.LoadPointer is identical to a MOV instruction.
From an assembly perspective, there was no memory race previously.
However, the lack of an atomic.LoadPointer meant that the compiler
could in theory reorder the "normal" load to produce truly racy code.

Change-Id: I8afefaf35c1916872781abc0239cbb63d62edf16
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189017
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-05 21:22:56 +00:00
Joe Tsai
d57568e763 cmd/protoc-gen-go: remove XXX_OneofWrappers
Change-Id: I31a311c9ea24e959d5d641c66c4ee77f0c98a2ed
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186917
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-05 21:08:32 +00:00
Joe Tsai
3efb138050 cmd/protoc-gen-go: remove XXX_NoUnkeyedLiteral
Change-Id: Ie098db74bab580903d813de05e9d015ea4b5ac75
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186898
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-05 21:06:36 +00:00
Joe Tsai
38b6196f13 cmd/protoc-gen-go: group generation of internal fields together
These were originally kept separate to assist Google-internal patches,
but it turns out that Google-internal patches do not use the
genMessageInternalFields function.

Change-Id: Idfa962b943d3bede9982b5b0875ba90c86c6d181
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/188979
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-05 20:58:04 +00:00