Commit Graph

64 Commits

Author SHA1 Message Date
Damien Neil
16163b4f67 all: drop reflect/prototype package
Remove the remaining uses of the prototype package.

The most significant change is to impl.MessageInfo, which now directly
implements the MessageType interface. This involves two notable changes
to exported fields of MessageInfo:

  - PBType is now Desc.
  - GoType is now GoReflectType. (Name changed to avoid a conflict with
    the GoType method of the MessageType interface.)

Fixes golang/protobuf#911

Change-Id: Ie2aa4766d6887ceaa9cf06b1f109aa6e6e2a208f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189340
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-09 19:06:30 +00:00
Damien Neil
4401a0de4b cmd/protoc-gen-go, internal/filetype: clean up EnumType construction
Drop the dependency from generated files on prototype.Enum: Generated
code should only depend on runtime/proto{iface,impl}.

Drop the Enums, Messages, and Extensions returns form
filetype.Builder.Build. Of these, only Enums was used by generated code.
Change the generated init function to pass the builder a slice of values
to fill in (as is done for messages and extensions).

Remove the filetype dependency on prototype in preparation for
eventually dropping the prototype package entirely.

Change-Id: I28a3420f5dfcc13fed531a64ef07b9afddfd9d55
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189200
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-09 19:04:41 +00:00
Damien Neil
92f76189a3 all: refactor extensions, add proto.GetExtension etc.
Change protoiface.ExtensionDescV1 to implement protoreflect.ExtensionType.

ExtensionDescV1's Name field conflicts with the Descriptor Name method,
so change the protoreflect.{Message,Enum,Extension}Type types to no
longer implement the corresponding Descriptor interface. This also leads
to a clearer distinction between the two types.

Introduce a protoreflect.ExtensionTypeDescriptor type which bridges
between ExtensionType and ExtensionDescriptor.

Add extension accessor functions to the proto package:
proto.{Has,Clear,Get,Set}Extension. These functions take a
protoreflect.ExtensionType parameter, which allows writing the
same function call using either the old or new API:

  proto.GetExtension(message, somepb.E_ExtensionFoo)

Fixes golang/protobuf#908

Change-Id: Ibc65d12a46666297849114fd3aefbc4a597d9f08
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189199
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-08 18:20:51 +00:00
Joe Tsai
bab3d4084e runtime/protoimpl, cmd/protoc-gen-go: support release versioning
In order for protoc-gen-go to output the current version,
it needs to know what version it is currently running as.
However, we cannot rely on the git tags since the tags are not
made until *after* the commit has been submitted.
Instead, we manually encode the version into the code and
make sure that git tags match up with the version in the code.

The version.go file in runtime/protoimpl contains instructions
for how to make a release. Essentially:
* Every non-release commit has a version string with "devel" in it.
* Every release commit must not have "devel" in it and must be unique.
* The "release process" involves submitting two CLs.
The first CL creates a version string without "devel",
which is the commit that a git tag will actually reference.
The second CL follows immediately and re-introduces "devel"
into the version string.

The following example shows a possible sequence of VersionStrings
for git commits in time-ascending order:
	v1.19.0-devel      (this CL)
	v1.19.0-devel
	v1.19.0-devel
	v1.19.0-devel
	v1.20.0-rc.1       <- tagged
	v1.20.0-rc.1.devel
	v1.20.0-rc.1.devel
	v1.20.0-rc.1.devel
	v1.20.0-rc.2       <- tagged
	v1.20.0-rc.2.devel
	v1.20.0            <- tagged (future public release)
	v1.20.0-devel
	v1.20.0-devel
	v1.20.0-devel
	v1.20.0-devel
	v1.20.1            <- tagged
	v1.20.1-devel
	v1.20.1-devel
	v1.21.0            <- tagged
	v1.21.0-devel

Note that we start today with v1.19.0-devel, which means that our initial
release will be v1.20.0. This number was intentionally chosen since
1) the number 20 has some correlation to the fact that we keep calling
the new implementation the "v2" implementation, and
2) the set of tagged versions for github.com/golang/protobuf
and google.golang.org/protobuf are unlikely to ever overlap.
This way, the version of protoc-gen-go is never ambiguous which module
it was built from.

Now that we have version information, we add support for generating .pb.go
files with the version information recorded. However, we do not emit
these for .pb.go files in our own repository since they are always guaranteed
to be at the right version (enforced by integration_test.go).

Updates golang/protobuf#524

Change-Id: I25495a45042c2aa39a39cb7e7738ae8e831a9d26
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186117
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-07 22:59:30 +00:00
Joe Tsai
8d5e6d6927 cmd/protoc-gen-go: improve generation of comments
The following improvements were made:
* All standalone comments above the "syntax" marker are preserved
similar to Java and some other generators.
* All standalone comments above the "package" marker are preserved
to be consistent with our former behavior.
* Leading comments are now generated for enums and extension fields.
* Single-line trailing comments are now generated for
enum values, message fields, and extension fields.
* The leading comments for each field that is part of a oneof are now
generated with the wrapper types rather than being shoved into the
comment for the oneof itself in an unreadable way.
* The deprecation marker is always generated as being above the declaration
rather than sometimes being an inlined comment.
* The deprecation marker is now properly generated for weak field setters.

Updates golang/protobuf#666

Change-Id: I7fd832dd4f86d15bfff70d7c22c6ba4934c05fcf
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189238
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-07 17:33:08 +00:00
Damien Neil
f5274511fe all: add NewField, NewElement, NewValue
Add methods to protoreflect.{Message,List,Map} to constrict values
assignable to a message field, list element, or map value. These
methods return the default value for scalar fields, the zero value for
scalar list elements and map values, and an empty, mutable value for
messages, lists, and maps.

Deprecate the NewMessage methods on these types, which are superseded.

Updates golang/protobuf#879

Change-Id: I0f064f60c89a239330ccea81523f559f14fd2c4f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/188997
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-06 21:54:30 +00:00
Joe Tsai
9b8a433283 cmd/protoc-gen-go: group enum map vars in blocks
Since the enum maps are here to stay, group the declarations together
in a var block for better readability in godoc.

Change-Id: I9a313266539e9a60781f98b80a5293379f82607b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189077
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-06 21:33:36 +00:00
Joe Tsai
d29a71bff0 cmd/protoc-gen-go: group default consts and vars in blocks
Group the default constant and variable declarations together in a block
for better readability in godoc.

Change-Id: I6b62f5374f0302d0f7cb224cbe34102359c8c51d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189057
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-06 21:23:53 +00:00
Joe Tsai
38b6196f13 cmd/protoc-gen-go: group generation of internal fields together
These were originally kept separate to assist Google-internal patches,
but it turns out that Google-internal patches do not use the
genMessageInternalFields function.

Change-Id: Idfa962b943d3bede9982b5b0875ba90c86c6d181
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/188979
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-05 20:58:04 +00:00
Joe Tsai
fb3ff727f0 cmd/protoc-gen-go: refactor logic to be more compartmentalized
This CL makes no feature changes except to move code around.
The only change to the actual generated code is the placement of
the default constants and variables. They move because the new logic
generates all methods together, while previously the constants
were interspersed in-between.

Change-Id: I45932d5aeec5ba45180fb255ea17898beb6c3bd2
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186878
Reviewed-by: Herbie Ong <herbie@google.com>
2019-07-19 08:16:51 +00:00
Joe Tsai
43761bdfe7 cmd/protoc-gen-go: update deprecation warning
Change-Id: Ie1a854bf6f47d4ca9e941b6ccc64dc24ff32bd19
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186657
Reviewed-by: Damien Neil <dneil@google.com>
2019-07-18 02:26:18 +00:00
Joe Tsai
af57087245 reflect/protoregistry: provide more informative errors for conflicts
The v2 implementation strictly enforces that there are no conflicts at
all in the protobuf namespace unlike the prior v1 implementation.
This change is almost certainly going to cause loud failures for users
that were unknowingly tolerating registration conflicts.

We modify internal/filedesc to be able to record the Go package path
that the file descriptor is declared within. This information is used
by reflect/protoregistry to print both the previous Go package that
registered some declaration, and current Go package that is attempting
to register some declaration.

Change-Id: Ib5eb21c1c98495afc51aa08bd4404bd9d64b5b57
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186177
Reviewed-by: Damien Neil <dneil@google.com>
2019-07-15 20:39:24 +00:00
Joe Tsai
d421150c3b reflect/protoreflect: add Enum.Type and Message.Type
CL/174938 removed these methods in favor of a method that returned
only the descriptors. This CL adds back in the Type methods alongside
the Descriptor methods.

In a vast majority of protobuf usages, only the descriptor information
is needed. However, there is a small percentage that legitimately needs
the Go type information. We should provide both, but document that the
descriptor-only information is preferred.

Change-Id: Ia0a098997fb1bd009994940ae8ea5257ccd87cae
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/184578
Reviewed-by: Damien Neil <dneil@google.com>
2019-07-12 23:43:21 +00:00
Joe Tsai
0f81b38d61 runtime/protoiface: move and rename XXX_Methods
This CL moves and renames the protoreflect.ProtoMessage.XXX_Methods
to protoreflect.Message.ProtoMethods.

Since one needs to obtain a protoreflect.Message now to get at
the fast-path methods, we modify the method signatures to take
in a protoreflect.Message instead of protoreflect.ProtoMessage.
Doing so also avoids the wrapper hack that was formerly done on
impl.messageReflectWrapper.

After this change the new protoc-gen-go no longer generates
any XXX fields or methods. All internal fields and methods are truly
hidden from the end-user.

name                                     old time/op    new time/op    delta
Wire/Unmarshal/google_message1_proto2-4    1.50µs ±10%    1.50µs ± 2%    ~     (p=0.483 n=10+9)
Wire/Unmarshal/google_message1_proto3-4    1.06µs ± 6%    1.06µs ± 4%    ~     (p=0.814 n=9+9)
Wire/Unmarshal/google_message2-4            734µs ±22%     689µs ±13%    ~     (p=0.133 n=10+9)
Wire/Marshal/google_message1_proto2-4       790ns ±46%     652ns ± 8%    ~     (p=0.590 n=10+9)
Wire/Marshal/google_message1_proto3-4       872ns ± 4%     857ns ± 3%    ~     (p=0.168 n=9+9)
Wire/Marshal/google_message2-4              232µs ±16%     221µs ± 3%  -4.75%  (p=0.014 n=9+9)
Wire/Size/google_message1_proto2-4          164ns ± 2%     167ns ± 4%  +1.87%  (p=0.046 n=9+10)
Wire/Size/google_message1_proto3-4          240ns ± 9%     229ns ± 1%  -4.81%  (p=0.000 n=9+8)
Wire/Size/google_message2-4                58.9µs ± 9%    59.6µs ± 2%  +1.23%  (p=0.040 n=9+9)

name                                     old alloc/op   new alloc/op   delta
Wire/Unmarshal/google_message1_proto2-4      912B ± 0%      912B ± 0%    ~     (all equal)
Wire/Unmarshal/google_message1_proto3-4      688B ± 0%      688B ± 0%    ~     (all equal)
Wire/Unmarshal/google_message2-4            470kB ± 0%     470kB ± 0%    ~     (p=0.215 n=10+10)
Wire/Marshal/google_message1_proto2-4        240B ± 0%      240B ± 0%    ~     (all equal)
Wire/Marshal/google_message1_proto3-4        224B ± 0%      224B ± 0%    ~     (all equal)
Wire/Marshal/google_message2-4             90.1kB ± 0%    90.1kB ± 0%    ~     (all equal)
Wire/Size/google_message1_proto2-4          0.00B          0.00B         ~     (all equal)
Wire/Size/google_message1_proto3-4          0.00B          0.00B         ~     (all equal)
Wire/Size/google_message2-4                 0.00B          0.00B         ~     (all equal)

name                                     old allocs/op  new allocs/op  delta
Wire/Unmarshal/google_message1_proto2-4      24.0 ± 0%      24.0 ± 0%    ~     (all equal)
Wire/Unmarshal/google_message1_proto3-4      6.00 ± 0%      6.00 ± 0%    ~     (all equal)
Wire/Unmarshal/google_message2-4            8.49k ± 0%     8.49k ± 0%    ~     (all equal)
Wire/Marshal/google_message1_proto2-4        1.00 ± 0%      1.00 ± 0%    ~     (all equal)
Wire/Marshal/google_message1_proto3-4        1.00 ± 0%      1.00 ± 0%    ~     (all equal)
Wire/Marshal/google_message2-4               1.00 ± 0%      1.00 ± 0%    ~     (all equal)
Wire/Size/google_message1_proto2-4           0.00           0.00         ~     (all equal)
Wire/Size/google_message1_proto3-4           0.00           0.00         ~     (all equal)
Wire/Size/google_message2-4                  0.00           0.00         ~     (all equal)

Change-Id: Ibf3263ad0f293326695c22020a92a6b938ef4f65
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185697
Reviewed-by: Damien Neil <dneil@google.com>
2019-07-12 19:31:58 +00:00
Joe Tsai
82760ceffa internal/impl: add MessageState to every generated message
We define MessageState, which is essentially an atomically set *MessageInfo.
By nesting this as the first field in every generated message, we can
implement the reflective methods on a *MessageState when obtained by
unsafe casting a concrete message pointer as a *MessageState.
The MessageInfo held by MessageState provides additional Go type information
to interpret the memory that comes after the contents of the MessageState.

Since we are nesting a MessageState in every message,
the memory use of every message instance grows by 8B.

On average, the body of ProtoReflect grows from 133B to 202B (+50%).
However, this is offset by XXX_Methods, which is 108B and
will be removed in a future CL. Taking into account the eventual removal
of XXX_Methods, this is a net reduction of 25%.

name          old time/op    new time/op    delta
Name/Value-4    70.3ns ± 2%    17.5ns ± 6%   -75.08%  (p=0.000 n=10+10)
Name/Nil-4      70.6ns ± 3%    33.4ns ± 2%   -52.66%  (p=0.000 n=10+10)

name          old alloc/op   new alloc/op   delta
Name/Value-4     16.0B ± 0%      0.0B       -100.00%  (p=0.000 n=10+10)
Name/Nil-4       16.0B ± 0%      0.0B       -100.00%  (p=0.000 n=10+10)

name          old allocs/op  new allocs/op  delta
Name/Value-4      1.00 ± 0%      0.00       -100.00%  (p=0.000 n=10+10)
Name/Nil-4        1.00 ± 0%      0.00       -100.00%  (p=0.000 n=10+10)

Change-Id: I92bd58dc681c57c92612fd5ba7fc066aea34e95a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185460
Reviewed-by: Damien Neil <dneil@google.com>
2019-07-10 19:44:24 +00:00
Joe Tsai
5455ef5ca4 cmd/protoc-gen-go: disable generation of XXX_WellKnownType
The logic for this will be removed entirely in a future CL.

Change-Id: I3a20fb2a207fc5bedc550b3f6deeab7f63eaa2c4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185243
Reviewed-by: Herbie Ong <herbie@google.com>
2019-07-08 21:38:58 +00:00
Joe Tsai
6ceeaab1ba cmd/protoc-gen-go: remove MessageSet hackery
The encoding/prototext and encoding/protojson are implemented entirely
in terms of protobuf reflection, which side-steps this information.
Remove the hacks in the generator to special-case MessageSet.

Change-Id: I708c4636b77672545a103b7ab686f103b9dfc514
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185240
Reviewed-by: Herbie Ong <herbie@google.com>
2019-07-08 21:17:17 +00:00
Joe Tsai
c0e4bb2054 cmd/protoc-gen-go: unexport implementation-specific XXX fields
We modify protoc-gen-go to stop generating exported XXX fields.
The unsafe implementation is unaffected by this change since unsafe
can access fields regardless of visibility. However, for the purego
implementation, we need to respect Go visibility rules as enforced
by the reflect package.

We work around this by generating a exporter function that given
a reference to the message and the field to export, returns a reference
to the unexported field value. This exporter function is protected by
a constant such that it is not linked into the final binary in non-purego
build environment.

Updates golang/protobuf#276

Change-Id: Idf5c1f158973fa1c61187ff41440acb21c5dac94
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185141
Reviewed-by: Damien Neil <dneil@google.com>
2019-07-08 20:45:09 +00:00
Joe Tsai
09912274cc cmd/protoc-gen-go: remove generation of XXX_OneofWrappers
Associate the oneof wrapper types with a message by conveying that
information to the associated MessageInfo.

Change-Id: Iabfca593850e1d6a89498a37eacbf22dbb73bd20
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185239
Reviewed-by: Damien Neil <dneil@google.com>
2019-07-08 20:36:12 +00:00
Damien Neil
a9940822d4 all: remove protoreflect.Message.Len
Len looks like it should be O(1), but the need to check for
non-zero-length repeated fields makes it at minimum O(n) where n is
the number of repeated fields. In practice, it's O(n) where n is the
number of fields altogether.

The Len function is not especially useful, easily duplicated with Range
and a counter, and can be surprisingly inefficient. Drop it.

Change-Id: I24b27433217e131e842bd18dd58475bcdf62ef97
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/183678
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-06-25 21:59:46 +00:00
Damien Neil
a3456c946b types/dynamicpb: add dynamic message type
The dynamicpb package permits creating Message values from a
MessageDescriptor.

Change-Id: Ice429ae45a0835dffb5a7ec8c0bd2c1df7aac8a2
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/174960
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-06-24 19:50:03 +00:00
Joe Tsai
7ea76d2a4c cmd/protoc-gen-go: un-deprecate enum maps
If we were starting from scratch, we would not have added the enum maps.
However, they already exist and see a fair amount of usage.
The effort to remove them is not worth it. Thus, remove the deprecation
warning since they are here to stay.

Note that the generated code does not refer to the generated enum maps.
One day, the linker should be able to elide them if unused by the user.
However, https://golang.org/issue/2559 would need to be resolved first.

Change-Id: Ia8b9b1812b5d8462ca2fa1d543170e4a09ff9e4f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/183177
Reviewed-by: Damien Neil <dneil@google.com>
2019-06-20 19:36:37 +00:00
Joe Tsai
d888139e7b internal/filedesc, internal/filetype: initial commit
The internal/fileinit package is split apart into two packages:
* internal/filedesc constructs descriptors from the raw proto.
It is very similar to the previous internal/fileinit package.
* internal/filetype wraps descriptors with Go type information

Overview:
* The internal/fileinit package will be deleted in a future CL.
It is kept around since the v1 repo currently depends on it.
* The internal/prototype package is deleted. All former usages of it
are now using internal/filedesc instead. Most significantly,
the reflect/protodesc package was almost entirely re-written.
* The internal/impl package drops support for messages that do not
have a Descriptor method (pre-2016). This removes a significant amount
of technical debt.
filedesc.Builder to parse raw descriptors.
* The internal/encoding/defval package now handles enum values by name.

Change-Id: I3957bcc8588a70470fd6c7de1122216b80615ab7
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/182360
Reviewed-by: Damien Neil <dneil@google.com>
2019-06-20 02:06:11 +00:00
Joe Tsai
7a5e02648a reflect/protoreflect: remove {Enum,Message}.Type methods
This is a breaking change in light of new methods added in CL/174918.

This removes:
	Enum.Type: replacement is Descriptor
	Message.Type: replacement is Descriptor and New

Change-Id: Iaa5328795407c8401ef14ed038bd5ace19d8e03b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/174938
Reviewed-by: Damien Neil <dneil@google.com>
2019-06-17 19:43:39 +00:00
Joe Tsai
58b42d8892 cmd/protoc-gen-go, runtime/protoimpl: enforce minimum and maximum versions
Generate the needed infrastructure to ensure that we can statically
enforce minimum and maximum versions. This enables us to have a policy
when we release v2 where it fails to build for:
* new generated code with really old runtimes
* new runtimes with really old generated code

Change-Id: Ib699ad62c06dff8f9285806394a741c18db00288
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/178546
Reviewed-by: Damien Neil <dneil@google.com>
2019-05-23 17:53:52 +00:00
Joe Tsai
4fe9663f4c internal/impl: rename MessageType as MessageInfo
The name MessageType is easily confused with protoreflect.MessageType.
Rename it as MessageInfo, which follows the pattern set by v1,
where the equivalent data structure is called InternalMessageInfo.

Change-Id: I535956e1f7c6e9b07e9585e889d5e93388d0d2ce
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/178478
Reviewed-by: Damien Neil <dneil@google.com>
2019-05-22 14:54:35 +00:00
Joe Tsai
dbab6c06ee cmd/protoc-gen-go: generate Enum method even on proto3
A proto2 message can reference a proto3 enum. It is currently cumbersome
to set proto3 enums in proto2 messages. Add the Enum method in all situations
to support this situation. It also removes yet another point of divergence
between proto2 and proto3.

We could consider removing this method if Go ever gets generics,
but that hypothetical future is long ways away.

Change-Id: Ib83bc87e46b49f3271b90bacb2893bb8e278e4f2
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/177257
Reviewed-by: Damien Neil <dneil@google.com>
2019-05-17 06:27:54 +00:00
Joe Tsai
5c62f675c5 types: remove stub packages
CL/177623 updates v1 to not depend on the previous import paths.

Change-Id: I46a61b8f8fa136c9c1fe367ff311732ba8841279
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/177604
Reviewed-by: Damien Neil <dneil@google.com>
2019-05-16 22:06:16 +00:00
Joe Tsai
a95b29fbf6 types: consistently name generated protos
Rename each generated protobuf package such that the base of the
Go package path is always equal to the Go package name to follow
proper Go package naming conventions.

The Go package name is derived from the .proto source file name by
replacing ".proto" with "pb" and stripping all underscores.

Change-Id: Iea05d1b5d94b1b2821ae10276ab771bb2df93c0e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/177380
Reviewed-by: Damien Neil <dneil@google.com>
2019-05-16 21:55:40 +00:00
Damien Neil
e89e6244e0 all: change module to google.golang.org/protobuf
Temporarily remove go.mod, since we can't generate an accurate one until
the corresponding v1 change is submitted.

Change-Id: I1e1ad97f2b455e33f61ffaeb8676289795e47e72
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/177000
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-05-14 17:28:29 +00:00
Joe Tsai
0fc49f8225 reflect/protoreflect: add Descriptor specific methods
Added methods:
	Enum.Descriptor
	Message.Descriptor
	EnumType.Descriptor
	MessageType.Descriptor
	ExtensionType.Descriptor
	Message.New

All functionality is switched over to use those methods instead of
implicitly relying on the fact that {Enum,Message}Type implicitly
implement the associated descriptor interface.

This CL does not yet remove {Enum,Message}.Type or prevent
{Enum,Message,Extension}Type from implementating a descriptor.
That is a subsequent CL.

The Message.New method is also added to replace functionality
that will be lost when the Type methods are removed.

Change-Id: I7fefde1673bbd40bfdac489aca05cec9a6c98eb1
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/174918
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Herbie Ong <herbie@google.com>
2019-05-13 19:34:41 +00:00
Joe Tsai
5e71dc95f2 cmd/protoc-gen-go: used named types for special fields
Using a named fields gives us the flexibility to change the underlying
representation of special fields without needing to regenerate user code.

We add a named type for ExtensionFields, UnknownFields, and SizeCache.

Change-Id: I107cf82899850ea76665310ce79def60f0f7ab97
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/172402
Reviewed-by: Damien Neil <dneil@google.com>
2019-04-16 22:04:16 +00:00
Joe Tsai
7ca7098113 cmd/protoc-gen-go: drop xxx_ prefix for package-level variables
Lower-casing the FileDescriptor variable provides for shorter and
more readable global variable names. This has a very minor benefit
that the read-only data section of a binary is slightly smaller
since function names do end up in the final binary.

Change-Id: I077364905e5c9adea69873b3ea580fddc68b9eb8
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/172119
Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-16 16:31:54 +00:00
Joe Tsai
a2dd228004 internal/fileinit: make fileinit responsible for registering concrete options types
Changes:
* Remove protoreflect.OptionsMessage and use protoreflect.ProtoMessage
now that the generated options natively implement reflection.
* Make registration of concrete option types the responsibility of
internal/fileinit since the init logic of descriptor.pb.go uses that.
* Remove equivalent logic in protoc-gen-go to special-case descriptor.

Change-Id: Id814651fafa4d888ff4532d59c6a4f9b68145157
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171465
Reviewed-by: Damien Neil <dneil@google.com>
2019-04-15 20:37:10 +00:00
Joe Tsai
09b5b46567 cmd/protoc-gen-go: revert Enum simplication
Go1.12 on GoLLVM with LLVM 9.0.0 has a bug where the simpler but
equivalent expression results in memory corruption.
Revert to the more complex expression to avoid breaking GoLLVM and
it is not worth waiting for a GoLLVM fix.

Change-Id: I8b42965ca9bd333deb8693021e7b22f966e13521
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171463
Reviewed-by: Damien Neil <dneil@google.com>
2019-04-10 22:39:46 +00:00
Joe Tsai
872b50047a cmd/protoc-gen-go: generate oneof types seperately
Seperate out the generation of the oneof wrapper types from the
code block that is about getter methods.

Change-Id: Ief44ef953d0b5ad8c998a8542c830ca70468a3bf
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171029
Reviewed-by: Damien Neil <dneil@google.com>
2019-04-08 21:41:25 +00:00
Damien Neil
0d3e8cc096 proto, runtime/protoiface: add support for fast-path marshaling
Allow message implementations to provide optimized versions of standard
operations. Generated messages now include a ProtoReflectMethods method,
returning a protoiface.Methods struct containing pointers to assorted
optional functions.

The Methods struct also includes a Flags field indicating support for
optional features such as deterministic marshaling.

Implementation of the fast paths (and tests) will come in later CLs.

Change-Id: Idd1beed0ecf43ec5e5e7b8da2ee1e08d3ce32213
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170340
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-04-02 21:23:04 +00:00
Joe Tsai
61968ce130 cmd/protoc-gen-go: perform some code cleanup
Minor changes:
* Use x as the receiver since "e" and "m" are meaningless in the presence
of user-defined enum and message names.
* Consistently keep enum methods together, rather awkwardly split apart
by the value maps.

Change-Id: I68e5666efb56ac7a4d062fb223b9f826dc72aba9
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170357
Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-01 20:44:00 +00:00
Joe Tsai
5d72cc2d37 cmd/protoc-gen-go: lazily GZIP-encode the raw descriptor
This reduces the init-time cost slightly since the GZIP'd
raw descriptor is constructed lazily on demand.

Change-Id: I482c6a2201b8786e425d7dee5612fdfd60ab1500
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169917
Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-01 20:24:54 +00:00
Joe Tsai
35ec98fdcb cmd/protoc-gen-go: generate for v2-only dependencies
This removes yet another set of dependencies of v2 on v1.
The only remaining dependency are in the _test.go files,
primarily for proto.Equal.

Changes made:
* cmd/protoc-gen-go no longer generates any functionality that depends
on the v1 package, and instead only depends on v2.
* internal/fileinit.FileBuilder.MessageOutputTypes is switched from
protoreflect.MessageType to protoimpl.MessageType since the
implementation must be fully inialized before registration occurs.
* The test for internal/legacy/file_test.go is switched to a legacy_test
package to avoid a cyclic dependency.
This requires Load{Enum,Message,File}Desc to be exported.

Change-Id: I43e2fe64cff4eea204258ce11e791aca5eb6e569
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169298
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-26 17:03:31 +00:00
Joe Tsai
4fddebafc0 all: move v1 types over to the v2 repository
As a goal, v2 should not depend on v1. As another step towards that end,
we move all the types that used to be in the v1 protoapi package over to v2.

For now, we place MessageV1, ExtensionRangeV1, and ExtensionDescV1
in runtime/protoiface since these are types that generated messages will
probably have to reference forever. An alternative location could be
reflect/protoreflect, but it seems unfortunate to have to dirty the
namespace of that package with these types.

We move ExtensionFieldV1, ExtensionFieldsV1, and ExtensionFieldsOf
to internal/impl, since these are related to the implementation of a
generated message.

Since moving these types from v1 to v2 implies that the v1 protoapi
package is useless, we update all usages of v1 protoapi in the v2
repository to point to the relevant v2 type or functionality.

CL/168538 is the corresponding change to alter v1.
There will be a temporary build failure as it is not possible
to submit CL/168519 and CL/168538 atomically.

Change-Id: Ide4025c1b6af5b7f0696f4b65b988b4d10a50f0b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168519
Reviewed-by: Herbie Ong <herbie@google.com>
2019-03-22 20:01:07 +00:00
Joe Tsai
1321a0e05b cmd/protoc-gen-go: generate descriptor proto using v2 textpb
Changes:
* Modify protoc-gen-go to use a helper from internal/impl
to print the message as text.
* Add a helper function to internal/impl that calls v2 textpb.
* Modify encoding/textpb to avoid depending on descriptor proto,
which would cause an import cycle.
* Modify internal/fileinit to populate a pseudo-internal
method on MessageDescriptor to check whether the message should
use the message set wire format. We avoid adding this to the
main MessageDescriptor interface since message sets are a
deprecated proto1 feature that we should avoid promoting.

Change-Id: Ibaf79a563af695756f11ddc4db69b38e25a8f1a7
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168439
Reviewed-by: Herbie Ong <herbie@google.com>
2019-03-20 20:58:24 +00:00
Joe Tsai
559d47f1da cmd/protoc-gen-go: fix init order for v1 registration
The v1 registration leaks the message types out to the proto package.
When doing that, it must ensure that the reflection data structures
for those types are properly initialized first. We achieve that by
doing v1 registration at the end of the reflection init function.

Change-Id: If6df18df59d05bad50ff39c2eff6beb19e7466cc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168348
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-20 04:16:33 +00:00
Joe Tsai
08cd8842f1 cmd/protoc-gen-go: register with the v2 registry for descriptor
By passing the global registries to fileinit.Builder, the Build
method can register the newly constructed descriptors on our behalf.
As a result, we can stop registering with the v1 registries as well.

We only do this for descriptor proto to remove another dependency on v1.
We deliberately keep the v1 registration logic for now to make it easy
to patch away this new behavior.

Change-Id: Ic6aa8ffba3d2d0abe08a61fc5e1c9ca7668e0988
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168000
Reviewed-by: Herbie Ong <herbie@google.com>
2019-03-19 05:04:43 +00:00
Joe Tsai
8e506a8704 cmd/protoc-gen-go: rely on protoimpl for basic helpers
The EnumName, UnmarshalJSONEnum, and CompressGZIP helpers currently live
in v1 protoapi, which would cause all generated messages to depend on v1.
In an effort to break the dependency of v2 on v1, we move these helper
functions to v2 (and re-written to take advantage of protobuf reflection).

These helpers are unfortunate, but we cannot eliminate the functionality
that they implement since they are exposed in the publicly generated API.

Since EnumName does not rely on the enum maps, it removes another dependency
on those variables. Eventually, we can get to the point where these variables
(though declared) are not linked into the binary if the user does not use them.

Also, we rely on the v1 proto package for registration instead of v1 protoapi.
This may re-introduce a cyclic dependency on descriptor proto again in the
future, but the better approach is to just start registering with v2.

Change-Id: Id755585a7a1df14e4a6a2dfa650df221a3c153fb
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/167921
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-18 18:50:16 +00:00
Joe Tsai
990b9f5919 internal/prototype: move from reflect/prototype
The prototype package was initially used by generated reflection support,
but has now been replaced by internal/fileinit.
Eventually, this functionality should be deleted and re-written in terms
of other components in the repo.

Usages that prototype currently provides (but should be moved) are:
* Constructing standalone messages and enums, which is behavior we should
provide in reflect/protodesc. The google.protobuf.{Enum,Type} are well-known
proto messages designed for this purpose.
* Constructing placeholder files, enums, and messages.
* Consructing protoreflect.{Message,Enum,Extension}Types, which are protobuf
descriptors with associated Go type information.

Change-Id: Id7dbefff952682781b439aa555508c59b2629f9e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/167383
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-13 20:17:00 +00:00
Joe Tsai
bb786ea59e test.bash: update to protobuf 3.7.0
Change-Id: Id1237eaf92d6a07c57b279a59fb6bd794af0f231
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/167041
Reviewed-by: Herbie Ong <herbie@google.com>
2019-03-12 08:34:56 +00:00
Damien Neil
0fc224513b cmd/protoc-gen-go: enforce init order within packages
Ensure that the init funcs for files within a Go package run in the
dependency order of the source .proto files. That is, if a.proto and b.proto
are part of the same Go package, and a.proto imports b.proto, then b.pb.go's
init funcs must run before a.pb.go's.

Change-Id: I0e86ff22e5c4cab9df7a73fe4805390fadd34b0d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/166419
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Reviewed-by: Herbie Ong <herbie@google.com>
2019-03-10 22:19:14 +00:00
Damien Neil
6bb8dec7f6 cmd/protoc-gen-go: change some arrays to slices to save bytes
Using arrays in the generated reflection information adds unnecessary
eq and hash functions being added to the package. Change to slices
to reduce bloat.

Change-Id: I1a4f6d59021644d93dd6c24679b9233141e89a75
Reviewed-on: https://go-review.googlesource.com/c/164640
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-01 23:53:03 +00:00
Joe Tsai
19058431cd internal/cmd/generate-protos: initial commit
Create a single binary for handling generation of protos.
This replaces previous logic spread throughout the repo in:
* regenerate.bash
* cmd/protoc-gen-go/golden_test.go
* cmd/protoc-gen-go-grpc/golden_test.go
* (indirectly) internal/protogen/goldentest

One of the problems with the former approaches is that they relied on
a version of protoc that was specific to a developer's workstation.
This meant that the result of generation was not hermetic.
To address this, we rely on the hard-coded version of protobuf specified
in the test.bash script.

A summary of changes in this CL are:
* The internal_gengo.GenerateFile and internal_gengogrpc.GenerateFile
functions are unified to have consistent signatures. It seems that the
former accepted a *protogen.GeneratedFile to support v1 where gRPC code
was generated into the same file as the base .pb.go file. However, the
same functionality can be achieved by having the function return
the generated file object.
* The test.bash script patches the protobuf toolchain to have properly
specified go_package options in each proto source file.
* The test.bash script accepts a "-regenerate" argument.
* Add generation for the well-known types. Contrary to how these were
laid out in the v1 repo, all the well-known types are placed in the
same Go package.
* Add generation for the conformance proto.
* Remove regenerate.bash
* Remove internal/protogen
* Remove cmd/protoc-gen-go/golden_test.go
* Remove cmd/protoc-gen-go-grpc/golden_test.go
* Add cmd/protoc-gen-go/annotation_test.go

Change-Id: I4a1a97ae6f66e2fabcf4e4d292c95ab2a2db0248
Reviewed-on: https://go-review.googlesource.com/c/164477
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-01 20:47:52 +00:00