1166 Commits

Author SHA1 Message Date
Joe Tsai
a4cbffe4bc reflect/prototype: add registration hooks for options
Add pseudo-hidden functions to register the concrete Go type used for
the optional types. Also, augment protoc-gen-go to specially generate the
descriptor proto to register these types.

This change does not add validation logic yet to ensure that the correct option
types are passed to the API.

Change-Id: I5decc897e14b4bf570a61cf17b57a066a2a0f9d7
Reviewed-on: https://go-review.googlesource.com/c/153017
Reviewed-by: Damien Neil <dneil@google.com>
2018-12-07 19:38:57 +00:00
Joe Tsai
9667c4816d cmd/protoc-gen-go: reduce technical debt
The following TODOs were addressed:
* Consistently collect all enums, messages, and extensions in a breadth-first order.
The practical affect of this is that the declaration order in a Go file may change.
This simplifies reflection generation, which relies on consistent ordering.
* Removal of placeholder declarations (e.g., "var _ = proto.Marshal") since
protogen is intelligent about including imports as necessary.
* Always generate a default variable or constant for explicit empty strings.
The practical effect of this is the addition of new declarations in some cases.
However, it simplifies our logic such that it matches the protobuf data model.
* Generate the registration statements in a consistent order.

Change-Id: I627bb72589432bb65d53b50965ea88e5f7983977
Reviewed-on: https://go-review.googlesource.com/c/152778
Reviewed-by: Damien Neil <dneil@google.com>
2018-12-07 03:13:48 +00:00
Joe Tsai
05d60cda2c reflect/prototype: disable detrand for test
Change-Id: I62cadc7f02afb9a6ee84c5ae2d7857be4b405360
Reviewed-on: https://go-review.googlesource.com/c/153021
Reviewed-by: Herbie Ong <herbie@google.com>
2018-12-07 00:53:04 +00:00
Joe Tsai
24ceb2b095 cmd/protoc-gen-go: generate descriptor and plugin with reflection
In CL/152020, we checked in pre-generated versions of descriptor and plugin.
This CL makes it such that they are generated by protoc-gen-go.

We modify protoc-gen-go to avoid reflection support by default
since the binary size increase is still an issue to investigate.
Reflection support is temporarily enabled by setting a special
PROTOC_GEN_GO_ENABLE_REFLECT environment variable.

Reflection support is always enabled for descriptor and plugin.
Furthermore, we change descriptor to depend on the protoapi package
instead of the proto package. The reason we do not switch to protoapi
for all generated protos is because we still depend on v1 proto
for the table-driven InternalMessageInfo type. Dropping it from descriptor
is semantically correct, but does incur slight performance cost.
It does not seem appropriate to drop it for all generated messages.

We could move InternalMessageInfo to protoapi, but the logic behind that
is significant.

Change-Id: I5c3fff7f6eab1a5a2399049d42fa6bf42d4c93f9
Reviewed-on: https://go-review.googlesource.com/c/152547
Reviewed-by: Damien Neil <dneil@google.com>
2018-12-06 17:47:27 +00:00
Joe Tsai
b3960a7de0 internal/typefmt: move descriptor formatting logic to typefmt
Custom descriptor types would want to benefit from descriptor formatting.
As such, move the logic out from prototype into an internal package for
the benefit of usages outside the prototype package.

Change-Id: I4bb2144221e656aa36909d33a77189fe084f700b
Reviewed-on: https://go-review.googlesource.com/c/152777
Reviewed-by: Herbie Ong <herbie@google.com>
2018-12-05 22:14:52 +00:00
Joe Tsai
b6405bd782 cmd/protoc-gen-go: add support for protobuf reflection
Implement support in protoc-gen-go for generating messages and enums
that satisfy the v2 protobuf reflection interfaces. Specifically, the following
are added:
* top-level variable representing the file descriptor
* ProtoReflect method on enums (to implement protoV2.Enum)
* ProtoReflect method on messages (to implement protoV2.Message)

The following are not supported yet:
* resolving transitive dependencies for file imports
* Extension descriptors
* Service descriptors

The implementation approach creates a single array for all the message and enum
declarations and references sections of that array to complete dependencies.
Since protobuf declarations can form a graph (a message may depend on itself),
it is difficult to construct a graph as a single literal. One way is to use
placeholder descriptors, but that is not efficient as it requires encoding
the full name of each dependent enum and message and then later resolving it;
thus, both expanding the binary size and also increasing initialization cost.
Instead, we add a prototype.{Enum,Message}.Reference method to obtain a
descriptor reference for the purposes for satisfying dependencies.
As such, nested declarations and dependencies are populated in an init function.

Other changes to support the implementation:
* Added a protoimpl package to expose the MessageType type and also the
MessageTypeOf and EnumTypeOf helper functions.
* Added a protogen.File.GoIdent field to provide a suggested variable name
for the file descriptor.
* Added prototype.{Enum,Message}.Reference that provides a descriptor reference
for the purposes for satisfying cyclic dependencies.
* Added protoreflect.{Syntax,Cardinality,Kind}.GoString to obtain a Go source
identifier that represents the given constant.

Change-Id: I9455764882dee6ad10f251901e7d419091e8bf1d
Reviewed-on: https://go-review.googlesource.com/c/150074
Reviewed-by: Damien Neil <dneil@google.com>
2018-12-05 19:32:12 +00:00
Joe Tsai
eeca8bb946 internal/legacy: drop dependency on protoV1 via vendoring
In order to transition more of v1 proto package to use the v2 API,
we need the v2 API to stop depending on v1 proto. The legacy package currently
depends on v1 proto because it needs to unmarshal the descriptor protos.

Ideally, we would switch this to use the v2 implementation of wire unmarshaling.
However, that is not available yet. So, instead, we vendor a minified version
of the v1 proto package that only supports unmarshaling.

The only changes to the vendored v1 code are:
* Delete code not needed to implement proto.Unmarshal
* Drop support for message sets
* Drop support for reporting the full field name for required not set errors

The unused tool was used to delete unrelated code:
	https://github.com/dominikh/go-tools/tree/master/cmd/unused

To verify that the dependency was dropped:
$ cd internal/legacy
$ go list -f "{{join .Deps \"\n\"}}" | sort | uniq | grep protobuf
github.com/golang/protobuf/protoapi
github.com/golang/protobuf/v2/internal/detrand
github.com/golang/protobuf/v2/internal/encoding/tag
github.com/golang/protobuf/v2/internal/encoding/text
github.com/golang/protobuf/v2/internal/encoding/wire
github.com/golang/protobuf/v2/internal/errors
github.com/golang/protobuf/v2/internal/flags
github.com/golang/protobuf/v2/internal/impl
github.com/golang/protobuf/v2/internal/legacy/protoV1
github.com/golang/protobuf/v2/internal/pragma
github.com/golang/protobuf/v2/internal/scalar
github.com/golang/protobuf/v2/internal/set
github.com/golang/protobuf/v2/internal/value
github.com/golang/protobuf/v2/reflect/protoreflect
github.com/golang/protobuf/v2/reflect/prototype
github.com/golang/protobuf/v2/runtime/protoimpl
github.com/golang/protobuf/v2/types/descriptor

Change-Id: I470865f1a987203574339fefc7d83843a12af966
Reviewed-on: https://go-review.googlesource.com/c/152545
Reviewed-by: Damien Neil <dneil@google.com>
2018-12-05 19:14:18 +00:00
Joe Tsai
e1f8d50e17 reflect/protodesc: split descriptor related functionality from prototype
In order to generate descriptor.proto, the generated code would want to depend
on the prototype package to construct the reflection data structures.
However, this is a problem since descriptor itself is one of the dependencies
for prototype. To break this dependency, we do the following:
* Avoid using concrete *descriptorpb.XOptions messages in the public API, and
instead just use protoreflect.ProtoMessage. We do lose some type safety here
as a result.
* Use protobuf reflection to interpret the Options message.
* Split out NewFileFromDescriptorProto into a separate protodesc package since
constructing protobuf reflection from the descriptor proto obviously depends
on the descriptor protos themselves.

As part of this CL, we check in a pre-generated version of descriptor and plugin
that supports protobuf reflection natively and switchover all usages of those
protos to the new definitions. These files were generated by protoc-gen-go
from CL/150074, but hand-modified to remove dependencies on the v1 proto runtime.

Change-Id: I81e03c42eeab480b03764e2fcbe1aae0e058fc57
Reviewed-on: https://go-review.googlesource.com/c/152020
Reviewed-by: Damien Neil <dneil@google.com>
2018-12-05 00:38:30 +00:00
Joe Tsai
6dbffb765a internal/legacy: drop legacy prefix
The legacy prefix made sense when this functionality was part of impl.
Now that it is in its own package called legacy, the legacy prefix is silly.

Change-Id: I9e6ddb6185ce1f701e02768b505e6a05f3986f77
Reviewed-on: https://go-review.googlesource.com/c/152543
Reviewed-by: Herbie Ong <herbie@google.com>
2018-12-05 00:24:04 +00:00
Joe Tsai
62517cc551 internal/legacy: improve performance of extension descriptor conversions
Converting to/from v1/v2 extension descriptor types is a common operation
for v1 and v2 interoperability. Optimize these operations with a cache.

Change-Id: I5feca810f60376847c791654982acd3b6a37a5db
Reviewed-on: https://go-review.googlesource.com/c/152542
Reviewed-by: Herbie Ong <herbie@google.com>
2018-12-05 00:14:16 +00:00
Damien Neil
4c057caa84 cmd/protoc-gen-go-grpc: switch to the stdlib "context" package
Also replace the ident function with the handy GoImportPath.Ident method.

Change-Id: Ie4e820556fb83e659ab7e7af98f27fc24cdcd760
Reviewed-on: https://go-review.googlesource.com/c/152177
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-03 20:00:03 +00:00
Joe Tsai
6cf80c4f76 internal/impl: allow reflection on typed nil pointers
Similar to how generated messages allow you to call Get methods on a
nil pointer, we permit similar functionality when protobuf reflection
is used on a nil pointer.

Change-Id: Ie2f596d39105c191073b42d7d689525c3b715240
Reviewed-on: https://go-review.googlesource.com/c/152021
Reviewed-by: Damien Neil <dneil@google.com>
2018-12-03 19:36:32 +00:00
Joe Tsai
f9123a3cd0 runtime: expose protoimpl and protolegacy
The internal/impl and internal/legacy packages are only visible from packages
within the v2 repository. However, selected parts of these packages need to
be visible from the v1 repository or from generated code.

Change-Id: I1d87227dacc38f5b1ac464e9b9dd43bd34559cda
Reviewed-on: https://go-review.googlesource.com/c/151858
Reviewed-by: Herbie Ong <herbie@google.com>
2018-11-30 23:16:22 +00:00
Joe Tsai
08e0030032 internal/legacy: extract legacy support out from the impl package
The impl package currently supports wrapping legacy v1 enums and messages
so that they implement the v2 reflective APIs. This functionality is necessary
for v1 and v2 to interoperate. However, the existence of this functionality
presents several problems:
	* A significant portion of the complexity in impl is for legacy wrapping.
	* This complexity is linked into a Go binary even if all the other messages
	in the binary natively support v2 reflection.
	* It presents a cyclic dependency when trying to generate descriptor proto.

Suppose you are generating descriptor.proto. The generated code would want to
depend on the impl package because impl is the runtime implementation for
protobuf messages. However, impl currently depends depends on descriptor in
order to wrap legacy enum and messages since it needs the ability to dynamically
create new protobuf descriptor types. In the case of descriptor.proto, it would
presumably be generated with native reflection support, so the legacy wrapping
logic is unneccessary.

To break the dependency of impl on descriptor, we move the legacy support logic
to a different package and instead add hooks in impl so that legacy support could
be dynamically registered at runtime. This is dependency injection.

Change-Id: I01a582908ed5629993f6699e9bf2f4bee93857a4
Reviewed-on: https://go-review.googlesource.com/c/151877
Reviewed-by: Herbie Ong <herbie@google.com>
2018-11-30 23:16:16 +00:00
Joe Tsai
ba0ef9a571 internal/value: rename the Unwrap method as ProtoUnwrap
Add a Proto prefix before the Unwrap method to reduce the probability that
it would ever conflict with a method of the same name that a
custom implementation of Enum, Message, List, or Map may have.

Change-Id: I628bf8335583f2747ab4589f3e6ff82e4501ce98
Reviewed-on: https://go-review.googlesource.com/c/151817
Reviewed-by: Herbie Ong <herbie@google.com>
2018-11-30 01:23:40 +00:00
Damien Neil
7e5c64778d cmd/protoc-gen-go: fix forwarding decls for inf and NaN defaults
Floating-point defaults of -inf, +inf, and NaN must be generated as
vars rather than consts. Fix the forwarding declaration for public
imports of these vars. (Was "const", is now "var".)

Change-Id: Ic6dc90ab7f88378ba477bff39e047de5f5193c35
Reviewed-on: https://go-review.googlesource.com/c/151757
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-11-29 23:43:35 +00:00
Herbie Ong
cddf8195e1 encoding/textpb: initial implementation of textproto marshaling
This initial implementation covers marshaling Message without use
of extensions, Any expansion, weak yet.

Change-Id: Ic787939c1d2a4e70e40c3a1654c6e7073052b7d3
Reviewed-on: https://go-review.googlesource.com/c/151677
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-11-29 23:06:35 +00:00
Joe Tsai
f18ab539ab all: make use of the protoapi package in v1
The new v1 protoapi package enables:
* Referencing types in the protoapi package instead of protoV1, which further
reduces the number of situations where we need to depend on protoV1.
This is for the goal of eventually breaking all cases where the v2 implementation
relies on v1, so that in the near future, proto v1 can rely on proto v2 instead.
* Removes the need for legacy_extension_hack.go since that functionality has now
been exported into the protoapi package.

Change-Id: If71002d9ec711bfabfe494636829df9abf19e23e
Reviewed-on: https://go-review.googlesource.com/c/151403
Reviewed-by: Herbie Ong <herbie@google.com>
2018-11-29 22:46:29 +00:00
Joe Tsai
25cc69d405 internal/impl: fix legacy logic to know about the new XXX_OneofWrappers method
The XXX_OneofWrappers method is a simplified way to obtain the wrapper structs
compared the previous XXX_OneofFuncs method which returned far more information
that was strictly necessary.

Change-Id: I2670506a2a8f7e8e724846b8c4083e7995371007
Reviewed-on: https://go-review.googlesource.com/c/151679
Reviewed-by: Herbie Ong <herbie@google.com>
2018-11-29 08:04:36 +00:00
Joe Tsai
71acbc7b7d internal/detrand: support disabling detrand
Since detrand is an internal package, we can safely provide a function
that can be called to disable its functionality for testing purposes.

Change-Id: I26383e12a5832eb5af01952898a4c73f627d7aa5
Reviewed-on: https://go-review.googlesource.com/c/151678
Reviewed-by: Herbie Ong <herbie@google.com>
2018-11-29 07:49:45 +00:00
Joe Tsai
009e067ed8 internal/scalar: add scalar package for primitive wrappers
Add the scalar package to reduce dependencies on the v1 proto runtime package.
It may very well be the case that these functions should be exposed in the
public API of v2, but that is not a decision we need to make now.

Change-Id: Ifbc6d15311ba5837909ac72af47c630a80a142ef
Reviewed-on: https://go-review.googlesource.com/c/151402
Reviewed-by: Herbie Ong <herbie@google.com>
2018-11-28 07:06:11 +00:00
Joe Tsai
f757390b52 cmd/protoc-gen-go: remove generation of RegisterMessageSetType
PR#741 in the v1 repository deprecated this behavior.

Change-Id: Ife48f1d586f178d875b9b3002a88b3336a2cd3b4
Reviewed-on: https://go-review.googlesource.com/c/151401
Reviewed-by: Damien Neil <dneil@google.com>
2018-11-28 00:40:08 +00:00
Joe Tsai
b6dbf23033 cmd/protoc-gen-go: remove generation of {Unmarshal,Marshal}JSON for MessageSets
PR#741 in the v1 repository deprecated this behavior.

Change-Id: Idffa3884d43f9cc5528ce8d9f303676e0e501b67
Reviewed-on: https://go-review.googlesource.com/c/151400
Reviewed-by: Damien Neil <dneil@google.com>
2018-11-27 23:47:23 +00:00
Joe Tsai
411f33952f protogen: clarify that GoIdent.GoName is a single idenfitier
While we're at it, also add the license header for the file.

Change-Id: If33f1764111aff96efcdb5a4e88c2a70ae28c09c
Reviewed-on: https://go-review.googlesource.com/c/150060
Reviewed-by: Damien Neil <dneil@google.com>
2018-11-27 19:56:42 +00:00
Joe Tsai
d7e97bc71b cmd/protoc-gen-go: generate XXX_OneofWrappers instead of XXX_OneofFuncs
The marshaler, unmarshaler, and sizer functions are unused ever since
the underlying implementation was switched to be table-driven.
Change the function to only return the wrapper structs.

This change:
* enables generated protos to drop dependencies on certain proto types
* reduces the size of generated protos
* simplifies the implementation of oneofs in protoc-gen-go

Updates #708

Change-Id: I845c9009bc0236d1b51d34b014dc3e184303c0f2
Reviewed-on: https://go-review.googlesource.com/c/151357
Reviewed-by: Damien Neil <dneil@google.com>
2018-11-27 19:36:27 +00:00
Joe Tsai
f6d4a4215f reflect/protoreflect: clarify Get semantics on unpopulated fields
Clearly specify that Get on an unpopulated field:
* returns the default value for scalars
* returns a mutable (but empty) List for repeated fields
* returns a mutable (but empty) Map for map fields
* returns an invalid value for message fields

The difference in semantics between List+Maps and Messages is because
protobuf semantics provide no distinction between an unpopulated and empty list
or map. On the other hand, there is a semantic difference between an unpopulated
message and an empty message.

Default values for scalars is trivial to implement with FieldDescriptor.Default.

A mutable, but empty List and Map is easy to implement for known fields since
known fields are generated as a slice or map field in a struct.
Since struct fields are addressable, the implementation can just return a
reference to the slice or map.

Repeated, extension fields are a little more tricky since extension fields
are implemented under the hood as a map[FieldNumber]Extension.
Rather than allocating an empty list in KnownFields.Get upon first retrieval
(which presents a race), delegate the work to ExtensionFieldTypes.Register,
which must occur before any Get operation. Register is not a concurrent-safe
operation, so that is an excellent time to initilize empty lists.
The implementation of extensions will need to be careful that Clear on a repeated
field simply truncates it zero instead of deleting the object.

For unpopulated messages, we return an invalid value, instead of the prior
behavior of returning a typed nil-pointer to the Go type for the message.
The approach is problematic because it assumes that
1) all messages are always implemented on a pointer reciever
2) a typed nil-pointer is an appropriate "read-only, but empty" message
These assumptions are not true of all message types (e.g., dynamic messages).

Change-Id: Ie96e6744c890308d9de738b6cf01d3b19e7e7c6a
Reviewed-on: https://go-review.googlesource.com/c/150319
Reviewed-by: Damien Neil <dneil@google.com>
2018-11-27 19:13:59 +00:00
Joe Tsai
492a476312 internal/detrand: new package for deterministically random functionality
The use of math/rand in serialization is to provide some form of instability
to the output to provide a clear signal to the user that the should not
depend on the the property of stability. However, it is reasonable that users
expect the output for these to be deterministic.

As such, add a detrand package that provides deterministic, yet unstable
randomization functionality.

Since this package hashes the binary, it does impose a small initialization cost:
	Benchmark    100000    20712 ns/op    480 B/op    6 allocs/op

Change-Id: I232d0fea1789a4278079837a67ee2f63474a4364
Reviewed-on: https://go-review.googlesource.com/c/151340
Reviewed-by: Herbie Ong <herbie@google.com>
2018-11-27 02:14:04 +00:00
Herbie Ong
c3f4d48629 internal/encoding/text: add extra random space to make output unstable.
Make output deliberately unstable so users don't rely on exactness.

For multi-line output, add another extra random space after <key>: for
at most one field per message.

-- example --
key1: field1
key2:  {
    foo:  bar
}

For single-line output, add another extra random space after a field per
message.

-- example --
key1:field1  key2:{foo:bar}

Change-Id: I3ab25d4d970fdebb88bbd9dd8fa6d73af84338ea
Reviewed-on: https://go-review.googlesource.com/c/150977
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-11-27 00:46:16 +00:00
Joe Tsai
ccf645912d reflect/prototype: print {Enum,Message,Extension}Type.GoType
Add GoType to the list of accessors that can be pretty-printed.
Useful for debugging.

Change-Id: Ib2a240249ca5bf0176cb77fa5edc639d44ae0561
Reviewed-on: https://go-review.googlesource.com/c/150537
Reviewed-by: Herbie Ong <herbie@google.com>
2018-11-20 20:02:59 +00:00
Joe Tsai
44e389ca6c internal/impl: fix Has for proto3 scalars
The definition of Has for proto3 scalars is whether the value is non-zero.

Change-Id: I6aee92dd518d63a66515ad35da84b2be7aa22527
Reviewed-on: https://go-review.googlesource.com/c/150320
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Joe Tsai <joetsai@google.com>
2018-11-19 23:34:00 +00:00
Joe Tsai
c6320b9f23 test.bash: update to go1.10.5 and go1.11.2
Change-Id: Ie62569d8544e262689dced53e9701859871962d9
Reviewed-on: https://go-review.googlesource.com/c/150076
Reviewed-by: Andrew Bonventre <andybons@golang.org>
Reviewed-by: Herbie Ong <herbie@google.com>
2018-11-19 18:05:39 +00:00
Joe Tsai
c1c17aa013 protogen: add GoImportPath.Ident helper
The GoImportPath.Ident helper creates a GoIdent using the receiver
as the GoImportPath in the GoIdent. This helper helps with the construction
of qualified identifiers.

Example usage:
	const protoPackage = protogen.GoImportPath("github.com/golang/protobuf/proto")
	protoPackage.Ident("ExtensionRange") // produces "proto.ExtensionRange"

The advantage of this helper is that usage of it looks similar to how
the identifier will eventually be rendered.

This is significantly more readable than the current approach:
	protogen.GoIdent{
		GoImportPath: protoPackage,
		GoName: "ExtensionRange",
	}

Change-Id: If7ecd7e60fad12bc491eee0dcb05f8fdebc9c94e
Reviewed-on: https://go-review.googlesource.com/c/150058
Reviewed-by: Damien Neil <dneil@google.com>
2018-11-16 19:34:14 +00:00
Joe Tsai
87b955ba7f internal/impl: add extensive tests for enum and messages
Add more extensive tests to ensure that the reflective API works for both
enums and messages. We tests the situation where a v2 message has dependencies
on v1 messages and vice versa.

Change-Id: Ib85d465711728ae13743bea700b678d9dda5e85c
Reviewed-on: https://go-review.googlesource.com/c/149758
Reviewed-by: Herbie Ong <herbie@google.com>
2018-11-15 23:10:55 +00:00
Joe Tsai
a31649d331 internal/impl: remove hack in legacy_extension_hack.go
This hack has now been upstreamed in the v1 codebase;
there is no longer any need to maintain the code in v2.
See https://github.com/golang/protobuf/pull/746

Change-Id: I2c59d11303f5465e65190d893c422c088c647691
Reviewed-on: https://go-review.googlesource.com/c/149659
Reviewed-by: Herbie Ong <herbie@google.com>
2018-11-15 02:11:35 +00:00
Joe Tsai
4b7aff630a all: rename Vector as List
The terminology Vector does not occur in protobuf documentation at all,
so we should rename the Go use of the term to something more recognizable.
As such, all instances that match the regexp "[Vv]ect(or)?" were replaced.

The C++ documentation uses the term "Repeated", which is a reasonable name.
However, the term became overloaded in 2014, when maps were added as a feature
and implementated under the hood as repeated fields. This is confusing as it
means "repeated" could either refer to repeated fields proper (i.e., explicitly
marked with the "repeated" label in the proto file) or map fields. In the case
of the C++ reflective API, this is not a problem since repeated fields proper
and map fields are interacted with through the same RepeatedField type.

In Go, we do not use a single type to handle both types of repeated fields:
1) We are coming up with the Go protobuf reflection API for the first time
and so do not need to piggy-back on the repeated fields API to remain backwards
compatible since no former usages of Go protobuf reflection exists.
2) Map fields are commonly represented in Go as the Go map type, which do not
preserve ordering information. As such it is fundamentally impossible to present
an unordered map as a consistently ordered list. Thus, Go needs two different
interfaces for lists and maps.

Given the above situation, "Repeated" is not a great term to use since it
refers to two different things (when we only want one of the meanings).
To distinguish between the two, we'll use the terms "List" and "Map" instead.
There is some precedence for the term "List" in the protobuf codebase
(e.g., "getRepeatedInt32List").

Change-Id: Iddcdb6b78e1e60c14fa4ca213c15f45e214b967b
Reviewed-on: https://go-review.googlesource.com/c/149657
Reviewed-by: Damien Neil <dneil@google.com>
2018-11-14 23:03:53 +00:00
Joe Tsai
f0c01e459b internal/impl: support legacy extension fields
Implement support for extension fields for messages that use the v1
data structures for extensions. The legacyExtensionFields type wraps a
v1 map to implement the v2 protoreflect.KnownFields interface.

Working on this change revealed a bug in the dynamic construction of
message types for protobuf messages that had cyclic dependencies (e.g.,
message Foo has a sub-field of message Bar, and Bar has a sub-field of Foo).
In such a situation, a deadlock occurs because initialization code depends on
the very initialization code that is currently running. To break these cycles,
we make some systematic changes listed in the following paragraphs.
Generally speaking, we separate the logic for construction and wrapping,
where constuction does not recursively rely on dependencies,
while wrapping may recursively inspect dependencies.

Promote the MessageType.MessageOf method as a standalone MessageOf function
that dynamically finds the proper *MessageType to use. We make it such that
MessageType only supports two forms of messages types:
* Those that fully implement the v2 API.
* Those that do not implement the v2 API at all.
This removes support for the hybrid form that was exploited by message_test.go

In impl/message_test.go, switch each message to look more like how future
generated messages will look like. This is done in reaction to the fact that
MessageType.MessageOf no longer exists.

In value/{map,vector}.go, fix Unwrap to return a pointer since the underlying
reflect.Value is addressable reference value, not a pointer value.

In value/convert.go, split the logic apart so that obtaining a v2 type and
wrapping a type as v2 are distinct operations. Wrapping requires further
initialization than simply creating the initial message type, and calling it
during initial construction would lead to a deadlock.

In protoreflect/go_type.go, we switch back to a lazy initialization of GoType
to avoid a deadlock since the user-provided fn may rely on the fact that
prototype.GoMessage returned.

Change-Id: I5dea00e36fe1a9899bd2ac0aed2c8e51d5d87420
Reviewed-on: https://go-review.googlesource.com/c/148826
Reviewed-by: Herbie Ong <herbie@google.com>
2018-11-14 18:37:45 +00:00
Joe Tsai
ea11813c05 internal/testprotos/legacy: initial commit
Add a corpus of generated protobuf messages generated at specific versions
of protoc-gen-go to ensure that we continue to support for generated messages
that have may never be updated.

Change-Id: I04a1b74306f471d7c99f5daf52399a5bd9adcbbc
Reviewed-on: https://go-review.googlesource.com/c/148831
Reviewed-by: Herbie Ong <herbie@google.com>
2018-11-13 22:37:52 +00:00
Joe Tsai
6f9095c675 internal/value: expose Converter.{MessageType,EnumType}
Rather than having the Converter carry a NewMessage method, have the struct
simply expose the MessageType or EnumType since they carry more information
and are retrieved anyways as part of the functionality of NewConverter.
While changing Converter, export the fields and remove all the methods.
Also, add an IsLegacy boolean, which is useful for the later implementation
of the extension fields.

Add a wrapLegacyEnum function which is used to wrap v1 enums as v2 enums.
We use this functionality in NewLegacyConverter to detrive the EnumType.
Additionally, modify wrapLegacyMessage to return a protoreflect.ProtoMessage
to be consistent with wrapLegacyEnum which must return a protoreflect.ProtoEnum.

Change-Id: Idc8989d07e4895d30de4ebc22c9ffa7357815cad
Reviewed-on: https://go-review.googlesource.com/c/148827
Reviewed-by: Herbie Ong <herbie@google.com>
2018-11-12 22:59:43 +00:00
Joe Tsai
1c40f4957d reflect/prototype: simplify Go type descriptor constructors
The Go type descriptors protoreflect.{Enum,Message,Extension}Type are simple
wrappers over protoreflect.{Enum,Message,Extension}Descriptor with a small
number of additional methods. It is very unlikely that more will be added in
the near future.

For this reason, construct the types directly using arguments to the constructor
function, as opposed to taking in another struct (which was originally done
to provide flexibility in-case we needed more fields).

Furthmore, rename GoNew and New.

Change-Id: Ic7fb5bc250cdb2761ae03b388b5147ff50f37d15
Reviewed-on: https://go-review.googlesource.com/c/148822
Reviewed-by: Herbie Ong <herbie@google.com>
2018-11-10 21:41:49 +00:00
Joe Tsai
2cdb23da01 reflect/prototype: remove TODO about custom extensions constructors
Most usages extensions are only for fields that have a singular message
as the type. In such a sitation, the Go type and constructors are determined
by the provided protoreflect.MessageType and is certainly what the user wanted.

Custom extensions constructors are mainly useful if the user wants to use their
own type to represent repeated fields. For example, we use *[]T to represent
repeated fields, while a user may want to use their own opaque type.

However, this is such a niche use-case that it is not worth supporting given
that there is a workaround:

	type MyExtensionType struct {
		protoreflect.ExtensionType
	}
	func (x MyExtensionType) GoNew() interface{} {
		... // return my custom value
	}
	func (x MyExtensionType) GoType() reflect.Type {
		... // return my custom type
	}
	func (x MyExtensionType) ValueOf(v interface{}) protoreflect.Value {
		... // convert my custom type to a protoreflect.Value
	}
	func (x MyExtensionType) InterfaceOf(v protoreflect.Value) interface{} {
		... // convert a protoreflect.Value to my custom type
	}

Change-Id: Iafc838b1c95b0ffc8583461ee9eb327bd3ce9c3c
Reviewed-on: https://go-review.googlesource.com/c/148820
Reviewed-by: Herbie Ong <herbie@google.com>
2018-11-10 00:59:49 +00:00
Joe Tsai
34eb7ef6d5 reflect/protoreflect: clarify Clear and Remove operation on missing entries
Follow the precedence of Go maps where deletion on a key without an entry in
the map is a noop. Similarly, document that the following methods are safe
to call with entries that do not exist:
* Map.Clear
* KnownFields.Clear
* ExtensionFieldTypes.Remove

Change the implementation for each of these to match the documented behavior.

Change-Id: Ifccff9b7b03baaeffdc366d05f6286ba60e14934
Reviewed-on: https://go-review.googlesource.com/c/148317
Reviewed-by: Herbie Ong <herbie@google.com>
2018-11-08 02:34:04 +00:00
Joe Tsai
1795eb16a4 internal/impl: rename legacy_extension.go as legacy_extension_hack.go
The current logic in this file is temporary and will be removed.
Rename the file as legacy_extension_hack.go so that the future
legacy_extension.go file can contain the actual legacy code that cannot be
removed anytime in the near future.

Change-Id: I7edd9d682836379c4c97d18ea2c2a2e3b15db5a2
Reviewed-on: https://go-review.googlesource.com/c/147918
Reviewed-by: Herbie Ong <herbie@google.com>
2018-11-06 21:02:36 +00:00
Joe Tsai
f0ad886d73 reflect/prototype: implement NewGoExtension
Implement the constructor for protoreflect.ExtensionType.
The constructor is more complicated than NewGoEnum and NewGoMessage because
it is responsible for providing the wrappers to present *[]T as a
protoreflect.Vector.

There are no tests since we need the follow-up logic in internal/impl to actually
make use of extensions. A subsequent CL will add that logic and comprehensively
test extensions.

Change-Id: I2d7893de299fe40be2ccedd8f39a92c40c41e59a
Reviewed-on: https://go-review.googlesource.com/c/147578
Reviewed-by: Herbie Ong <herbie@google.com>
2018-11-06 20:55:45 +00:00
Joe Tsai
88bc5a7d99 internal/value: extract Vector and Map logic as separate package
The implementation of reflect/protoreflect.NewGoExtension needs to be able to
provide a constructor for wrapping *[]T as a protoreflect.Vector.
However, it cannot depend on internal/impl since impl also depends on prototype.
Extract the common logic of Vector creation into a separate package that
has no dependencies on either impl or prototype.

Change-Id: I9295fde9b8861de11af085c91d9dfa56047d1b1e
Reviewed-on: https://go-review.googlesource.com/c/147446
Reviewed-by: Herbie Ong <herbie@google.com>
2018-11-06 01:20:32 +00:00
Joe Tsai
ce6edd3c71 internal/impl: support message and enum fields
Dynamically generate functions for handling message and enum fields,
regardless of whether they are of the v1 or v2 forms.

If a v1 message is encountered, it is automatically wrapped such that it
implements the v2 interface.

Change-Id: I457bc5286892e8fc00a61da7062dd33058daafd5
Reviewed-on: https://go-review.googlesource.com/c/143837
Reviewed-by: Damien Neil <dneil@google.com>
2018-11-05 22:25:52 +00:00
Joe Tsai
05828dba44 internal/encoding/tag: centralize logic for protobuf struct tag serialization
The bespoke text-serialization of field descriptors in protoc-gen-go is also
used in the legacy implementation of protobuf reflection to derive a
protoreflect.FieldDescriptor from legacy messages and also to convert to/from
protoreflect.ExtensionDescriptor and protoV1.ExtensionDesc.

Centralize this logic in a single place:
* to avoid reimplementing the same logic in internal/impl
* to keep the marshal and unmarshal logic co-located

Change-Id: I634c5afbb9dc6eda91d6cb6b0e68dbd724cb1ccb
Reviewed-on: https://go-review.googlesource.com/c/146758
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
2018-11-05 20:12:56 +00:00
Joe Tsai
95b0290ea8 internal/impl: support legacy unknown extension fields
The unknown fields in legacy messages is split across the XXX_unrecognized
field and also the XXX_InternalExtensions field. Implement support for
wrapping both fields and presenting it as if it were a unified set of
unknown fields.

Change-Id: If274fae2b48962520edd8a640080b6eced747684
Reviewed-on: https://go-review.googlesource.com/c/146517
Reviewed-by: Damien Neil <dneil@google.com>
2018-11-01 18:28:27 +00:00
Damien Neil
bb1d557b23 cmd/protoc-gen-go: factor out oneof field naming
A oneof is represented by a single struct field of interface type.
Pull out the decision of what to name that field into a separate
function.

The function is trivial (return oneof.GoName), but factoring out the
field name like this makes it a bit easier to experiment with changes to
the oneof implementation in the future.

Change-Id: I1114b68c85cb6608852fa1c6bf4103ff58fd5de6
Reviewed-on: https://go-review.googlesource.com/c/146397
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-10-31 20:40:31 +00:00
Joe Tsai
2d5a169f75 internal/impl: use linked-list for unknown field ordering
Unknown fields follow a policy where the latest field takes precedence when
it comes to the ordering. However, the current implementation is incorrect
as it uses a slice and simply swaps the current entry with the last entry.
While this ensures that the latest field seen remains last, it does not ensure
that the swapped out entry is second-to-last.

To provide the desired behavior, a linked-list is used.
For simplicity, we use the list package in the standard library even if it
is neither the most performant nor type safe.

Change-Id: I675145c61f6b5b624ed9e94bbe2251b5a71e2c48
Reviewed-on: https://go-review.googlesource.com/c/145241
Reviewed-by: Damien Neil <dneil@google.com>
2018-10-31 18:39:54 +00:00
Damien Neil
6238696abc cmd/protoc-gen-go: replicate v1 generator behavior for MessageSet extensions
Given:

  package foo
  extend proto2.bridge.MessageSet {
    optional Message message_set_extension = 100;
  }

Register the extension as a message set extension and give it the name
"foo.".

We really shouldn't do this in this case; the special-case treatment of
extensions to MessageSet is only for extensions nested in a parent
message. However, this is consistent with the behavior of the v1 generator.
Match that for now.

Change-Id: I919c409605a197904fd3227efc920192d484f431
Reviewed-on: https://go-review.googlesource.com/c/145957
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-10-30 18:46:20 +00:00