55 Commits

Author SHA1 Message Date
Damien Neil
f1e905b042 all: unify protoV1.ExtensionDesc and proto.ExtensionType
Change protoV1.ExtensionDesc to directly implement ExtensionType
rather than delegating to one.

Unify the previous types protoiface.ExtensionDescV1 and
filetype.Extension in impl.ExtensionInfo. The protoV1.ExtensionDesc
type becomes an alias to ExtensionInfo.

This gives us:

  - Just one implementation of ExtensionType.
  - Generated foopb.E_Ext vars are canonical ExtensionTypes.
  - Generated foopb.E_Ext vars are also v1.ExtensionDescs for backwards
    compatibility.
  - Conversion between legacy and modern representations happens
    transparently when lazily initializing an ExtensionInfo.

Overall, a simplification for users of generated code, since they can
mostly ignore the ExtensionDesc/ExtentionType distinction and use the
same value in either the old or new API.

This is change 3/5 in a series of commits changing protoV1.ExtensionDesc
to directly implement protoreflect.ExtensionType.

1. [v2] Add protoimpl.ExtensionInfo as an alias for
   protoiface.ExtensionDescV1.

2. [v1] Update references to protoimpl.ExtensionInfo to use
   protoiface.ExtensionInfo.

3. [v2] Create protoimpl.ExtensionInfo (an alias to a new type in
   the impl package) and remove protoiface.ExtensionDescV1.

4. [v1] Remove unneeded explicit conversions between ExtensionDesc and
   ExtensionType (since the former now directly implements the latter).

5. [v2] Remove stub conversion functions.

Change-Id: I96ee890541ec11b2412e1a72c9d7b96e4d7f66b4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189563
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-20 21:32:57 +00:00
Joe Tsai
7328839f81 cmd/protoc-gen-go: annotate depIdxs list with index comments
Generate the current index into depIdxs for easier human debugging.

Change-Id: Ida42aa95137b2044a4dc267c31cebec5023bdfb1
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/190278
Reviewed-by: Herbie Ong <herbie@google.com>
2019-08-15 22:47:44 +00:00
Damien Neil
92f76189a3 all: refactor extensions, add proto.GetExtension etc.
Change protoiface.ExtensionDescV1 to implement protoreflect.ExtensionType.

ExtensionDescV1's Name field conflicts with the Descriptor Name method,
so change the protoreflect.{Message,Enum,Extension}Type types to no
longer implement the corresponding Descriptor interface. This also leads
to a clearer distinction between the two types.

Introduce a protoreflect.ExtensionTypeDescriptor type which bridges
between ExtensionType and ExtensionDescriptor.

Add extension accessor functions to the proto package:
proto.{Has,Clear,Get,Set}Extension. These functions take a
protoreflect.ExtensionType parameter, which allows writing the
same function call using either the old or new API:

  proto.GetExtension(message, somepb.E_ExtensionFoo)

Fixes golang/protobuf#908

Change-Id: Ibc65d12a46666297849114fd3aefbc4a597d9f08
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189199
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-08 18:20:51 +00:00
Joe Tsai
bab3d4084e runtime/protoimpl, cmd/protoc-gen-go: support release versioning
In order for protoc-gen-go to output the current version,
it needs to know what version it is currently running as.
However, we cannot rely on the git tags since the tags are not
made until *after* the commit has been submitted.
Instead, we manually encode the version into the code and
make sure that git tags match up with the version in the code.

The version.go file in runtime/protoimpl contains instructions
for how to make a release. Essentially:
* Every non-release commit has a version string with "devel" in it.
* Every release commit must not have "devel" in it and must be unique.
* The "release process" involves submitting two CLs.
The first CL creates a version string without "devel",
which is the commit that a git tag will actually reference.
The second CL follows immediately and re-introduces "devel"
into the version string.

The following example shows a possible sequence of VersionStrings
for git commits in time-ascending order:
	v1.19.0-devel      (this CL)
	v1.19.0-devel
	v1.19.0-devel
	v1.19.0-devel
	v1.20.0-rc.1       <- tagged
	v1.20.0-rc.1.devel
	v1.20.0-rc.1.devel
	v1.20.0-rc.1.devel
	v1.20.0-rc.2       <- tagged
	v1.20.0-rc.2.devel
	v1.20.0            <- tagged (future public release)
	v1.20.0-devel
	v1.20.0-devel
	v1.20.0-devel
	v1.20.0-devel
	v1.20.1            <- tagged
	v1.20.1-devel
	v1.20.1-devel
	v1.21.0            <- tagged
	v1.21.0-devel

Note that we start today with v1.19.0-devel, which means that our initial
release will be v1.20.0. This number was intentionally chosen since
1) the number 20 has some correlation to the fact that we keep calling
the new implementation the "v2" implementation, and
2) the set of tagged versions for github.com/golang/protobuf
and google.golang.org/protobuf are unlikely to ever overlap.
This way, the version of protoc-gen-go is never ambiguous which module
it was built from.

Now that we have version information, we add support for generating .pb.go
files with the version information recorded. However, we do not emit
these for .pb.go files in our own repository since they are always guaranteed
to be at the right version (enforced by integration_test.go).

Updates golang/protobuf#524

Change-Id: I25495a45042c2aa39a39cb7e7738ae8e831a9d26
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186117
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-07 22:59:30 +00:00
Joe Tsai
8d5e6d6927 cmd/protoc-gen-go: improve generation of comments
The following improvements were made:
* All standalone comments above the "syntax" marker are preserved
similar to Java and some other generators.
* All standalone comments above the "package" marker are preserved
to be consistent with our former behavior.
* Leading comments are now generated for enums and extension fields.
* Single-line trailing comments are now generated for
enum values, message fields, and extension fields.
* The leading comments for each field that is part of a oneof are now
generated with the wrapper types rather than being shoved into the
comment for the oneof itself in an unreadable way.
* The deprecation marker is always generated as being above the declaration
rather than sometimes being an inlined comment.
* The deprecation marker is now properly generated for weak field setters.

Updates golang/protobuf#666

Change-Id: I7fd832dd4f86d15bfff70d7c22c6ba4934c05fcf
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189238
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-07 17:33:08 +00:00
Joe Tsai
43761bdfe7 cmd/protoc-gen-go: update deprecation warning
Change-Id: Ie1a854bf6f47d4ca9e941b6ccc64dc24ff32bd19
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186657
Reviewed-by: Damien Neil <dneil@google.com>
2019-07-18 02:26:18 +00:00
Joe Tsai
ad8dff3eec all: update to github.com/golang/protobuf@b9f5089f
Change-Id: I6178e0552630f2eb11f0ad95d77f23527708e5ec
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186638
Reviewed-by: Damien Neil <dneil@google.com>
2019-07-18 00:01:40 +00:00
Joe Tsai
af57087245 reflect/protoregistry: provide more informative errors for conflicts
The v2 implementation strictly enforces that there are no conflicts at
all in the protobuf namespace unlike the prior v1 implementation.
This change is almost certainly going to cause loud failures for users
that were unknowingly tolerating registration conflicts.

We modify internal/filedesc to be able to record the Go package path
that the file descriptor is declared within. This information is used
by reflect/protoregistry to print both the previous Go package that
registered some declaration, and current Go package that is attempting
to register some declaration.

Change-Id: Ib5eb21c1c98495afc51aa08bd4404bd9d64b5b57
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186177
Reviewed-by: Damien Neil <dneil@google.com>
2019-07-15 20:39:24 +00:00
Joe Tsai
0f81b38d61 runtime/protoiface: move and rename XXX_Methods
This CL moves and renames the protoreflect.ProtoMessage.XXX_Methods
to protoreflect.Message.ProtoMethods.

Since one needs to obtain a protoreflect.Message now to get at
the fast-path methods, we modify the method signatures to take
in a protoreflect.Message instead of protoreflect.ProtoMessage.
Doing so also avoids the wrapper hack that was formerly done on
impl.messageReflectWrapper.

After this change the new protoc-gen-go no longer generates
any XXX fields or methods. All internal fields and methods are truly
hidden from the end-user.

name                                     old time/op    new time/op    delta
Wire/Unmarshal/google_message1_proto2-4    1.50µs ±10%    1.50µs ± 2%    ~     (p=0.483 n=10+9)
Wire/Unmarshal/google_message1_proto3-4    1.06µs ± 6%    1.06µs ± 4%    ~     (p=0.814 n=9+9)
Wire/Unmarshal/google_message2-4            734µs ±22%     689µs ±13%    ~     (p=0.133 n=10+9)
Wire/Marshal/google_message1_proto2-4       790ns ±46%     652ns ± 8%    ~     (p=0.590 n=10+9)
Wire/Marshal/google_message1_proto3-4       872ns ± 4%     857ns ± 3%    ~     (p=0.168 n=9+9)
Wire/Marshal/google_message2-4              232µs ±16%     221µs ± 3%  -4.75%  (p=0.014 n=9+9)
Wire/Size/google_message1_proto2-4          164ns ± 2%     167ns ± 4%  +1.87%  (p=0.046 n=9+10)
Wire/Size/google_message1_proto3-4          240ns ± 9%     229ns ± 1%  -4.81%  (p=0.000 n=9+8)
Wire/Size/google_message2-4                58.9µs ± 9%    59.6µs ± 2%  +1.23%  (p=0.040 n=9+9)

name                                     old alloc/op   new alloc/op   delta
Wire/Unmarshal/google_message1_proto2-4      912B ± 0%      912B ± 0%    ~     (all equal)
Wire/Unmarshal/google_message1_proto3-4      688B ± 0%      688B ± 0%    ~     (all equal)
Wire/Unmarshal/google_message2-4            470kB ± 0%     470kB ± 0%    ~     (p=0.215 n=10+10)
Wire/Marshal/google_message1_proto2-4        240B ± 0%      240B ± 0%    ~     (all equal)
Wire/Marshal/google_message1_proto3-4        224B ± 0%      224B ± 0%    ~     (all equal)
Wire/Marshal/google_message2-4             90.1kB ± 0%    90.1kB ± 0%    ~     (all equal)
Wire/Size/google_message1_proto2-4          0.00B          0.00B         ~     (all equal)
Wire/Size/google_message1_proto3-4          0.00B          0.00B         ~     (all equal)
Wire/Size/google_message2-4                 0.00B          0.00B         ~     (all equal)

name                                     old allocs/op  new allocs/op  delta
Wire/Unmarshal/google_message1_proto2-4      24.0 ± 0%      24.0 ± 0%    ~     (all equal)
Wire/Unmarshal/google_message1_proto3-4      6.00 ± 0%      6.00 ± 0%    ~     (all equal)
Wire/Unmarshal/google_message2-4            8.49k ± 0%     8.49k ± 0%    ~     (all equal)
Wire/Marshal/google_message1_proto2-4        1.00 ± 0%      1.00 ± 0%    ~     (all equal)
Wire/Marshal/google_message1_proto3-4        1.00 ± 0%      1.00 ± 0%    ~     (all equal)
Wire/Marshal/google_message2-4               1.00 ± 0%      1.00 ± 0%    ~     (all equal)
Wire/Size/google_message1_proto2-4           0.00           0.00         ~     (all equal)
Wire/Size/google_message1_proto3-4           0.00           0.00         ~     (all equal)
Wire/Size/google_message2-4                  0.00           0.00         ~     (all equal)

Change-Id: Ibf3263ad0f293326695c22020a92a6b938ef4f65
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185697
Reviewed-by: Damien Neil <dneil@google.com>
2019-07-12 19:31:58 +00:00
Joe Tsai
82760ceffa internal/impl: add MessageState to every generated message
We define MessageState, which is essentially an atomically set *MessageInfo.
By nesting this as the first field in every generated message, we can
implement the reflective methods on a *MessageState when obtained by
unsafe casting a concrete message pointer as a *MessageState.
The MessageInfo held by MessageState provides additional Go type information
to interpret the memory that comes after the contents of the MessageState.

Since we are nesting a MessageState in every message,
the memory use of every message instance grows by 8B.

On average, the body of ProtoReflect grows from 133B to 202B (+50%).
However, this is offset by XXX_Methods, which is 108B and
will be removed in a future CL. Taking into account the eventual removal
of XXX_Methods, this is a net reduction of 25%.

name          old time/op    new time/op    delta
Name/Value-4    70.3ns ± 2%    17.5ns ± 6%   -75.08%  (p=0.000 n=10+10)
Name/Nil-4      70.6ns ± 3%    33.4ns ± 2%   -52.66%  (p=0.000 n=10+10)

name          old alloc/op   new alloc/op   delta
Name/Value-4     16.0B ± 0%      0.0B       -100.00%  (p=0.000 n=10+10)
Name/Nil-4       16.0B ± 0%      0.0B       -100.00%  (p=0.000 n=10+10)

name          old allocs/op  new allocs/op  delta
Name/Value-4      1.00 ± 0%      0.00       -100.00%  (p=0.000 n=10+10)
Name/Nil-4        1.00 ± 0%      0.00       -100.00%  (p=0.000 n=10+10)

Change-Id: I92bd58dc681c57c92612fd5ba7fc066aea34e95a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185460
Reviewed-by: Damien Neil <dneil@google.com>
2019-07-10 19:44:24 +00:00
Joe Tsai
c0e4bb2054 cmd/protoc-gen-go: unexport implementation-specific XXX fields
We modify protoc-gen-go to stop generating exported XXX fields.
The unsafe implementation is unaffected by this change since unsafe
can access fields regardless of visibility. However, for the purego
implementation, we need to respect Go visibility rules as enforced
by the reflect package.

We work around this by generating a exporter function that given
a reference to the message and the field to export, returns a reference
to the unexported field value. This exporter function is protected by
a constant such that it is not linked into the final binary in non-purego
build environment.

Updates golang/protobuf#276

Change-Id: Idf5c1f158973fa1c61187ff41440acb21c5dac94
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185141
Reviewed-by: Damien Neil <dneil@google.com>
2019-07-08 20:45:09 +00:00
Joe Tsai
dfe160316f internal/fileinit: remove package
This package is no longer used.

Change-Id: Ib6d50490b07d6c5005f9e7e6c4b9cdc6ce6722fb
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/183101
Reviewed-by: Damien Neil <dneil@google.com>
2019-06-20 19:35:23 +00:00
Joe Tsai
d888139e7b internal/filedesc, internal/filetype: initial commit
The internal/fileinit package is split apart into two packages:
* internal/filedesc constructs descriptors from the raw proto.
It is very similar to the previous internal/fileinit package.
* internal/filetype wraps descriptors with Go type information

Overview:
* The internal/fileinit package will be deleted in a future CL.
It is kept around since the v1 repo currently depends on it.
* The internal/prototype package is deleted. All former usages of it
are now using internal/filedesc instead. Most significantly,
the reflect/protodesc package was almost entirely re-written.
* The internal/impl package drops support for messages that do not
have a Descriptor method (pre-2016). This removes a significant amount
of technical debt.
filedesc.Builder to parse raw descriptors.
* The internal/encoding/defval package now handles enum values by name.

Change-Id: I3957bcc8588a70470fd6c7de1122216b80615ab7
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/182360
Reviewed-by: Damien Neil <dneil@google.com>
2019-06-20 02:06:11 +00:00
Joe Tsai
dddbe8b487 reflect/protoregistry: remove Files.RangeFilesByPath
This is a breaking change in light of API added in CL/182497.

This removes:
	Files.RangeFilesByPath: replacement is Files.FindFileByPath

Change-Id: I47bf59b37c355844984661056212953853a0db51
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/182537
Reviewed-by: Damien Neil <dneil@google.com>
2019-06-17 19:43:11 +00:00
Joe Tsai
05e11b806f internal/impl: unexport ExtensionField.Desc
CL/180577 removes direct access to the Desc field in v1.

Since the only way to access the type is through {Has,Get,Set}Type,
we can now unexport the field and change the underlying type.

Change-Id: I6cc7018d4326c17228ba579d6161c5fb3f6d4127
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/180578
Reviewed-by: Damien Neil <dneil@google.com>
2019-06-05 20:15:29 +00:00
Joe Tsai
44b9a3083f cmd: fix go.mod files
Change-Id: I13cbb878dad1024bc1b5e07e9c10f53cacce0349
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/178717
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-05-23 18:29:14 +00:00
Joe Tsai
58b42d8892 cmd/protoc-gen-go, runtime/protoimpl: enforce minimum and maximum versions
Generate the needed infrastructure to ensure that we can statically
enforce minimum and maximum versions. This enables us to have a policy
when we release v2 where it fails to build for:
* new generated code with really old runtimes
* new runtimes with really old generated code

Change-Id: Ib699ad62c06dff8f9285806394a741c18db00288
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/178546
Reviewed-by: Damien Neil <dneil@google.com>
2019-05-23 17:53:52 +00:00
Joe Tsai
4fe9663f4c internal/impl: rename MessageType as MessageInfo
The name MessageType is easily confused with protoreflect.MessageType.
Rename it as MessageInfo, which follows the pattern set by v1,
where the equivalent data structure is called InternalMessageInfo.

Change-Id: I535956e1f7c6e9b07e9585e889d5e93388d0d2ce
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/178478
Reviewed-by: Damien Neil <dneil@google.com>
2019-05-22 14:54:35 +00:00
Joe Tsai
5c62f675c5 types: remove stub packages
CL/177623 updates v1 to not depend on the previous import paths.

Change-Id: I46a61b8f8fa136c9c1fe367ff311732ba8841279
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/177604
Reviewed-by: Damien Neil <dneil@google.com>
2019-05-16 22:06:16 +00:00
Joe Tsai
11caeff126 internal/impl: unexport ExtensionFieldV1.Value
CL/177620 modifies v1 to stop touching the Value field directly,
such that it is now unexport the Value field.

Change-Id: I0a05bbe59146862fc77c261349d7d90d6fa312e0
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/177621
Reviewed-by: Damien Neil <dneil@google.com>
2019-05-16 20:29:06 +00:00
Joe Tsai
838f1f50f9 encoding: remove jsonpb and textpb
Change-Id: Id057d697efecedb00bc07705a47cddb2d69866e9
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/177603
Reviewed-by: Damien Neil <dneil@google.com>
2019-05-16 19:31:22 +00:00
Damien Neil
934520f5b2 go.mod, go.sum: add back in after v2 module renaming
Change-Id: Iba5d25391870b4a6014fd90fc79ab101c75ea3c7
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/177042
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-05-14 18:44:45 +00:00
Damien Neil
e89e6244e0 all: change module to google.golang.org/protobuf
Temporarily remove go.mod, since we can't generate an accurate one until
the corresponding v1 change is submitted.

Change-Id: I1e1ad97f2b455e33f61ffaeb8676289795e47e72
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/177000
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-05-14 17:28:29 +00:00
Joe Tsai
4a58eb3f7c internal/impl: remove ExtensionFieldV1.Raw
CL/175579 switches v2 to use XXX_unrecognized exclusively.
CL/175838 switches v1 to use XXX_unrecognized exclusively.

This means that it is now safe to delete the Raw field.

Change-Id: Ic60c58147c2df4078e72c34d1202d226a7ea4bed
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/175839
Reviewed-by: Damien Neil <dneil@google.com>
2019-05-09 21:52:05 +00:00
Joe Tsai
c90b6ada23 reflect/protoreflect: remove methods with Type suffix
CL/172238 added equivalent methods without the Type suffix,
while keeping the old methods.
CL/172582 updates the v1 codebase to use the new methods.
This CL removes the methods with the Type suffix.

Change-Id: Iaaaa4fff11cac1694735657db2e5fd7cadc90afe
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/173138
Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-20 08:14:13 +00:00
Joe Tsai
00e50dc9c1 runtime/protoimpl: remove ExtensionFieldsV1
This is no longer directly used by v1.

Change-Id: I14a283dd2d1572d50ebd82fdf55f7d86bfacc272
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/172438
Reviewed-by: Damien Neil <dneil@google.com>
2019-04-16 23:53:07 +00:00
Joe Tsai
5e71dc95f2 cmd/protoc-gen-go: used named types for special fields
Using a named fields gives us the flexibility to change the underlying
representation of special fields without needing to regenerate user code.

We add a named type for ExtensionFields, UnknownFields, and SizeCache.

Change-Id: I107cf82899850ea76665310ce79def60f0f7ab97
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/172402
Reviewed-by: Damien Neil <dneil@google.com>
2019-04-16 22:04:16 +00:00
Joe Tsai
7ca7098113 cmd/protoc-gen-go: drop xxx_ prefix for package-level variables
Lower-casing the FileDescriptor variable provides for shorter and
more readable global variable names. This has a very minor benefit
that the read-only data section of a binary is slightly smaller
since function names do end up in the final binary.

Change-Id: I077364905e5c9adea69873b3ea580fddc68b9eb8
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/172119
Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-16 16:31:54 +00:00
Damien Neil
0d3e8cc096 proto, runtime/protoiface: add support for fast-path marshaling
Allow message implementations to provide optimized versions of standard
operations. Generated messages now include a ProtoReflectMethods method,
returning a protoiface.Methods struct containing pointers to assorted
optional functions.

The Methods struct also includes a Flags field indicating support for
optional features such as deterministic marshaling.

Implementation of the fast paths (and tests) will come in later CLs.

Change-Id: Idd1beed0ecf43ec5e5e7b8da2ee1e08d3ce32213
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170340
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-04-02 21:23:04 +00:00
Joe Tsai
61968ce130 cmd/protoc-gen-go: perform some code cleanup
Minor changes:
* Use x as the receiver since "e" and "m" are meaningless in the presence
of user-defined enum and message names.
* Consistently keep enum methods together, rather awkwardly split apart
by the value maps.

Change-Id: I68e5666efb56ac7a4d062fb223b9f826dc72aba9
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170357
Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-01 20:44:00 +00:00
Joe Tsai
5d72cc2d37 cmd/protoc-gen-go: lazily GZIP-encode the raw descriptor
This reduces the init-time cost slightly since the GZIP'd
raw descriptor is constructed lazily on demand.

Change-Id: I482c6a2201b8786e425d7dee5612fdfd60ab1500
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169917
Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-01 20:24:54 +00:00
Joe Tsai
35ec98fdcb cmd/protoc-gen-go: generate for v2-only dependencies
This removes yet another set of dependencies of v2 on v1.
The only remaining dependency are in the _test.go files,
primarily for proto.Equal.

Changes made:
* cmd/protoc-gen-go no longer generates any functionality that depends
on the v1 package, and instead only depends on v2.
* internal/fileinit.FileBuilder.MessageOutputTypes is switched from
protoreflect.MessageType to protoimpl.MessageType since the
implementation must be fully inialized before registration occurs.
* The test for internal/legacy/file_test.go is switched to a legacy_test
package to avoid a cyclic dependency.
This requires Load{Enum,Message,File}Desc to be exported.

Change-Id: I43e2fe64cff4eea204258ce11e791aca5eb6e569
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169298
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-26 17:03:31 +00:00
Joe Tsai
f503c300f7 all: re-add go.mod and go.sum files
Change-Id: I0eed4846a597daf112016bce1dcc198aa7c3a5f1
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168899
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-22 20:14:22 +00:00
Joe Tsai
4fddebafc0 all: move v1 types over to the v2 repository
As a goal, v2 should not depend on v1. As another step towards that end,
we move all the types that used to be in the v1 protoapi package over to v2.

For now, we place MessageV1, ExtensionRangeV1, and ExtensionDescV1
in runtime/protoiface since these are types that generated messages will
probably have to reference forever. An alternative location could be
reflect/protoreflect, but it seems unfortunate to have to dirty the
namespace of that package with these types.

We move ExtensionFieldV1, ExtensionFieldsV1, and ExtensionFieldsOf
to internal/impl, since these are related to the implementation of a
generated message.

Since moving these types from v1 to v2 implies that the v1 protoapi
package is useless, we update all usages of v1 protoapi in the v2
repository to point to the relevant v2 type or functionality.

CL/168538 is the corresponding change to alter v1.
There will be a temporary build failure as it is not possible
to submit CL/168519 and CL/168538 atomically.

Change-Id: Ide4025c1b6af5b7f0696f4b65b988b4d10a50f0b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168519
Reviewed-by: Herbie Ong <herbie@google.com>
2019-03-22 20:01:07 +00:00
Joe Tsai
559d47f1da cmd/protoc-gen-go: fix init order for v1 registration
The v1 registration leaks the message types out to the proto package.
When doing that, it must ensure that the reflection data structures
for those types are properly initialized first. We achieve that by
doing v1 registration at the end of the reflection init function.

Change-Id: If6df18df59d05bad50ff39c2eff6beb19e7466cc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168348
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-20 04:16:33 +00:00
Joe Tsai
8e506a8704 cmd/protoc-gen-go: rely on protoimpl for basic helpers
The EnumName, UnmarshalJSONEnum, and CompressGZIP helpers currently live
in v1 protoapi, which would cause all generated messages to depend on v1.
In an effort to break the dependency of v2 on v1, we move these helper
functions to v2 (and re-written to take advantage of protobuf reflection).

These helpers are unfortunate, but we cannot eliminate the functionality
that they implement since they are exposed in the publicly generated API.

Since EnumName does not rely on the enum maps, it removes another dependency
on those variables. Eventually, we can get to the point where these variables
(though declared) are not linked into the binary if the user does not use them.

Also, we rely on the v1 proto package for registration instead of v1 protoapi.
This may re-introduce a cyclic dependency on descriptor proto again in the
future, but the better approach is to just start registering with v2.

Change-Id: Id755585a7a1df14e4a6a2dfa650df221a3c153fb
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/167921
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-18 18:50:16 +00:00
Joe Tsai
d9bfe8bc52 all: fix travis after v1 update
Change-Id: I3194f4d2e57903d7ae0d6c20bae551dc6ab7d8ce
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/166891
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-12 00:05:04 +00:00
Joe Tsai
c93f494f4a cmd/protoc-gen-go/testdata: ignore go.sum
The go.sum is not particularly important for the testdata directory,
so ignore it.

Change-Id: I069613deda2af4944d09aed34008fec201df6bb4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/166879
Reviewed-by: Herbie Ong <herbie@google.com>
2019-03-11 20:19:03 +00:00
Joe Tsai
d56458e71b internal/cmd/generate-protos: generate test for testdata
Running "go build ./..." does not descend into testdata directories.
However, the testdata in this repository is source code that is
intended to build properly. We could rename the directory, but that does
not test whether the generated packages can initialize properly.

Thus, we generate a trivial test that simply blank imports all packages.

Doing this reveals that some of the generated files have incorrect imports,
leading to registration conflicts.

To avoid introducing a dependency on gRPC from our go.mod file, we put
the testdata directories in their own module. Also, we avoid running
internal/testprotos through the grpc plugin because the servie and method
definitions in that directory are more for testing proto file initialization
rather than testing grpc generation.

Change-Id: Iaa6a06449787a085200e31bc7606e3ac904d3180
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/164917
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-11 19:40:53 +00:00
Damien Neil
0fc224513b cmd/protoc-gen-go: enforce init order within packages
Ensure that the init funcs for files within a Go package run in the
dependency order of the source .proto files. That is, if a.proto and b.proto
are part of the same Go package, and a.proto imports b.proto, then b.pb.go's
init funcs must run before a.pb.go's.

Change-Id: I0e86ff22e5c4cab9df7a73fe4805390fadd34b0d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/166419
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Reviewed-by: Herbie Ong <herbie@google.com>
2019-03-10 22:19:14 +00:00
Damien Neil
6bb8dec7f6 cmd/protoc-gen-go: change some arrays to slices to save bytes
Using arrays in the generated reflection information adds unnecessary
eq and hash functions being added to the package. Change to slices
to reduce bloat.

Change-Id: I1a4f6d59021644d93dd6c24679b9233141e89a75
Reviewed-on: https://go-review.googlesource.com/c/164640
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-01 23:53:03 +00:00
Joe Tsai
19058431cd internal/cmd/generate-protos: initial commit
Create a single binary for handling generation of protos.
This replaces previous logic spread throughout the repo in:
* regenerate.bash
* cmd/protoc-gen-go/golden_test.go
* cmd/protoc-gen-go-grpc/golden_test.go
* (indirectly) internal/protogen/goldentest

One of the problems with the former approaches is that they relied on
a version of protoc that was specific to a developer's workstation.
This meant that the result of generation was not hermetic.
To address this, we rely on the hard-coded version of protobuf specified
in the test.bash script.

A summary of changes in this CL are:
* The internal_gengo.GenerateFile and internal_gengogrpc.GenerateFile
functions are unified to have consistent signatures. It seems that the
former accepted a *protogen.GeneratedFile to support v1 where gRPC code
was generated into the same file as the base .pb.go file. However, the
same functionality can be achieved by having the function return
the generated file object.
* The test.bash script patches the protobuf toolchain to have properly
specified go_package options in each proto source file.
* The test.bash script accepts a "-regenerate" argument.
* Add generation for the well-known types. Contrary to how these were
laid out in the v1 repo, all the well-known types are placed in the
same Go package.
* Add generation for the conformance proto.
* Remove regenerate.bash
* Remove internal/protogen
* Remove cmd/protoc-gen-go/golden_test.go
* Remove cmd/protoc-gen-go-grpc/golden_test.go
* Add cmd/protoc-gen-go/annotation_test.go

Change-Id: I4a1a97ae6f66e2fabcf4e4d292c95ab2a2db0248
Reviewed-on: https://go-review.googlesource.com/c/164477
Reviewed-by: Damien Neil <dneil@google.com>
2019-03-01 20:47:52 +00:00
Damien Neil
8012b444ee internal/fileinit: generate reflect data structures from raw descriptors
This CL takes a significantly different approach to generating support
for protobuf reflection. The previous approach involved generating a
large number of Go literals to represent the reflection information.
While that approach was correct, it resulted in too much binary bloat.

The approach taken here initializes the reflection information from
the raw descriptor proto, which is a relatively dense representation
of the protobuf reflection information. In order to keep initialization
cost low, several measures were taken:
* At program init, the bare minimum is parsed in order to initialize
naming information for enums, messages, extensions, and services declared
in the file. This is done because those top-level declarations are often
relevant for registration.
* Only upon first are most of the other data structures for protobuf
reflection actually initialized.
* Instead of using proto.Unmarshal, a hand-written unmarshaler is used.
This allows us to avoid a dependendency on the descriptor proto and also
because the API for the descriptor proto is fundamentally non-performant
since it requires an allocation for every primitive field.

At a high-level, the new implementation lives in internal/fileinit.

Several changes were made to other parts of the repository:
* cmd/protoc-gen-go:
  * Stop compressing the raw descriptors. While compression does reduce
the size of the descriptors by approximately 2x, it is a pre-mature
optimization since the descriptors themselves are around 1% of the total
binary bloat that is due to generated protobufs.
  * Seeding protobuf reflection from the raw descriptor significantly
simplifies the generator implementation since it is no longer responsible
for constructing a tree of Go literals to represent the same information.
  * We remove the generation of the shadow types and instead call
protoimpl.MessageType.MessageOf. Unfortunately, this incurs an allocation
for every call to ProtoReflect since we need to allocate a tuple that wraps
a pointer to the message value, and a pointer to message type.
* internal/impl:
  * We add a MessageType.GoType field and make it required that it is
set prior to first use. This is done so that we can avoid calling
MessageType.init except for when it is actually needed. The allows code
to call (*FooMessage)(nil).ProtoReflect().Type() without fearing that the
init code will run, possibly triggering a recursive deadlock (where the
init code depends on getting the Type of some dependency which may be
declared within the same file).
* internal/cmd/generate-types:
  * The code to generate reflect/prototype/protofile_list_gen.go was copied
and altered to generated internal/fileinit.desc_list_gen.go.

At a high-level this CL adds significant technical complexity.
However, this is offset by several possible future changes:
* The prototype package can be drastically simplified. We can probably
reimplement internal/legacy to use internal/fileinit instead, allowing us
to drop another dependency on the prototype package. As a result, we can
probably delete most of the constructor types in that package.
* With the prototype package significantly pruned, and the fact that generated
code no longer depend on depends on that package, we can consider merging
what's left of prototype into protodesc.

Change-Id: I6090f023f2e1b6afaf62bd3ae883566242e30715
Reviewed-on: https://go-review.googlesource.com/c/158539
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-01-30 01:33:46 +00:00
Joe Tsai
5681bb2587 protogen: use _protoFile suffix for file descriptor variable
A "_ProtoFile" suffix can potentially conflict with a sub-message named
"ProtoFile" nested within a message that matches the camel-cased
form of the basename of the .proto source file.

Avoid unlikely conflicts and rename this to use a "_protoFile" suffix,
which can never conflict except with an enum value that is also named
"protoFile" (which is a violation of the style guide).

Change-Id: Ie9d22f9f741a63021b8f76906b20c6c2f599885b
Reviewed-on: https://go-review.googlesource.com/c/157218
Reviewed-by: Damien Neil <dneil@google.com>
2019-01-14 20:23:59 +00:00
Joe Tsai
3bc7d6f5cd reflect: switch MessageType.New to return Message
Most usages of New actually prefer to interact with the reflective view
rather than the native Go type. Thus, change New to return that instead.
This parallels reflect.New, which returns the reflective view
(i.e., reflect.Value) instead of native type (i.e., interface{}).
We make the equivalent change to KnownFields.NewMessage, List.NewMessage,
and Map.NewMessage for consistency.

Since this is a subtle change where the type system will not always
catch the changed type, this change was made by both changing the type
and renaming the function to NewXXX and manually looking at every usage
of the the function to ensure that the usage correctly operates
on either the native Go type or the reflective view of the type.
After the entire codebase was cleaned up, a rename was performed to convert
NewXXX back to New.

Change-Id: I153fef627b4bf0a427e4039ce0aaec52e20c7950
Reviewed-on: https://go-review.googlesource.com/c/157077
Reviewed-by: Damien Neil <dneil@google.com>
2019-01-09 20:29:29 +00:00
Damien Neil
f25e14ca27 cmd/protoc-gen-go-grpc: regenerate golden file
File is out of date, regenerate it. (This didn't show up as a test failure
because we don't test the non-grpc golden files under the grpc directory.)

Change-Id: Ied485d28184c45bbca5b52138eb9cf300d813a57
Reviewed-on: https://go-review.googlesource.com/c/156345
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-01-05 01:25:02 +00:00
Damien Neil
692690ff4e cmd/protoc-gen-go-grpc: fix camel-casing of names in generated code
Fix a few points of divergence between the v1 and v2 generators around
when to apply camel-casing to service and method names.

Change-Id: I862f89c0995c540e4862013316d7af772e1ab0d8
Reviewed-on: https://go-review.googlesource.com/c/153658
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-11 19:20:28 +00:00
Damien Neil
08c3cdf59f cmd/protoc-gen-go-grpc: change test files to exercise name rewriting
Change the service and method names in grpc.proto to not use camel-case,
to make it explicit where rewriting occurs in the generated code.

Change-Id: I9e4a851097b0ee14817a589f5f959adcc5a14fe3
Reviewed-on: https://go-review.googlesource.com/c/153657
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-11 19:11:24 +00:00
Damien Neil
97e7f57dbb reflect/protoreflect: replace Mutable with NewMessage
Remove the Mutable methods from KnownFields, List, and Map, replacing
them with methods which return a new, empty message value without adding
that value to the collection.

The new API is simpler, since it clearly applies only to message values,
and more orthogonal, since it provides a way to create a value without
mutating the collection. This latter point is particularly useful in
map deserialization, where the key may be unknown at the time the value
is deserialized.

Drop the Mutable interface, since it is no longer necessary.

Change-Id: Ic5f3d06a2aa331a5d5cd2b4e670a3dba4a74f77c
Reviewed-on: https://go-review.googlesource.com/c/153278
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-10 21:17:16 +00:00
Joe Tsai
9667c4816d cmd/protoc-gen-go: reduce technical debt
The following TODOs were addressed:
* Consistently collect all enums, messages, and extensions in a breadth-first order.
The practical affect of this is that the declaration order in a Go file may change.
This simplifies reflection generation, which relies on consistent ordering.
* Removal of placeholder declarations (e.g., "var _ = proto.Marshal") since
protogen is intelligent about including imports as necessary.
* Always generate a default variable or constant for explicit empty strings.
The practical effect of this is the addition of new declarations in some cases.
However, it simplifies our logic such that it matches the protobuf data model.
* Generate the registration statements in a consistent order.

Change-Id: I627bb72589432bb65d53b50965ea88e5f7983977
Reviewed-on: https://go-review.googlesource.com/c/152778
Reviewed-by: Damien Neil <dneil@google.com>
2018-12-07 03:13:48 +00:00