130 Commits

Author SHA1 Message Date
Joe Tsai
ea11813c05 internal/testprotos/legacy: initial commit
Add a corpus of generated protobuf messages generated at specific versions
of protoc-gen-go to ensure that we continue to support for generated messages
that have may never be updated.

Change-Id: I04a1b74306f471d7c99f5daf52399a5bd9adcbbc
Reviewed-on: https://go-review.googlesource.com/c/148831
Reviewed-by: Herbie Ong <herbie@google.com>
2018-11-13 22:37:52 +00:00
Joe Tsai
6f9095c675 internal/value: expose Converter.{MessageType,EnumType}
Rather than having the Converter carry a NewMessage method, have the struct
simply expose the MessageType or EnumType since they carry more information
and are retrieved anyways as part of the functionality of NewConverter.
While changing Converter, export the fields and remove all the methods.
Also, add an IsLegacy boolean, which is useful for the later implementation
of the extension fields.

Add a wrapLegacyEnum function which is used to wrap v1 enums as v2 enums.
We use this functionality in NewLegacyConverter to detrive the EnumType.
Additionally, modify wrapLegacyMessage to return a protoreflect.ProtoMessage
to be consistent with wrapLegacyEnum which must return a protoreflect.ProtoEnum.

Change-Id: Idc8989d07e4895d30de4ebc22c9ffa7357815cad
Reviewed-on: https://go-review.googlesource.com/c/148827
Reviewed-by: Herbie Ong <herbie@google.com>
2018-11-12 22:59:43 +00:00
Joe Tsai
1c40f4957d reflect/prototype: simplify Go type descriptor constructors
The Go type descriptors protoreflect.{Enum,Message,Extension}Type are simple
wrappers over protoreflect.{Enum,Message,Extension}Descriptor with a small
number of additional methods. It is very unlikely that more will be added in
the near future.

For this reason, construct the types directly using arguments to the constructor
function, as opposed to taking in another struct (which was originally done
to provide flexibility in-case we needed more fields).

Furthmore, rename GoNew and New.

Change-Id: Ic7fb5bc250cdb2761ae03b388b5147ff50f37d15
Reviewed-on: https://go-review.googlesource.com/c/148822
Reviewed-by: Herbie Ong <herbie@google.com>
2018-11-10 21:41:49 +00:00
Joe Tsai
2cdb23da01 reflect/prototype: remove TODO about custom extensions constructors
Most usages extensions are only for fields that have a singular message
as the type. In such a sitation, the Go type and constructors are determined
by the provided protoreflect.MessageType and is certainly what the user wanted.

Custom extensions constructors are mainly useful if the user wants to use their
own type to represent repeated fields. For example, we use *[]T to represent
repeated fields, while a user may want to use their own opaque type.

However, this is such a niche use-case that it is not worth supporting given
that there is a workaround:

	type MyExtensionType struct {
		protoreflect.ExtensionType
	}
	func (x MyExtensionType) GoNew() interface{} {
		... // return my custom value
	}
	func (x MyExtensionType) GoType() reflect.Type {
		... // return my custom type
	}
	func (x MyExtensionType) ValueOf(v interface{}) protoreflect.Value {
		... // convert my custom type to a protoreflect.Value
	}
	func (x MyExtensionType) InterfaceOf(v protoreflect.Value) interface{} {
		... // convert a protoreflect.Value to my custom type
	}

Change-Id: Iafc838b1c95b0ffc8583461ee9eb327bd3ce9c3c
Reviewed-on: https://go-review.googlesource.com/c/148820
Reviewed-by: Herbie Ong <herbie@google.com>
2018-11-10 00:59:49 +00:00
Joe Tsai
34eb7ef6d5 reflect/protoreflect: clarify Clear and Remove operation on missing entries
Follow the precedence of Go maps where deletion on a key without an entry in
the map is a noop. Similarly, document that the following methods are safe
to call with entries that do not exist:
* Map.Clear
* KnownFields.Clear
* ExtensionFieldTypes.Remove

Change the implementation for each of these to match the documented behavior.

Change-Id: Ifccff9b7b03baaeffdc366d05f6286ba60e14934
Reviewed-on: https://go-review.googlesource.com/c/148317
Reviewed-by: Herbie Ong <herbie@google.com>
2018-11-08 02:34:04 +00:00
Joe Tsai
1795eb16a4 internal/impl: rename legacy_extension.go as legacy_extension_hack.go
The current logic in this file is temporary and will be removed.
Rename the file as legacy_extension_hack.go so that the future
legacy_extension.go file can contain the actual legacy code that cannot be
removed anytime in the near future.

Change-Id: I7edd9d682836379c4c97d18ea2c2a2e3b15db5a2
Reviewed-on: https://go-review.googlesource.com/c/147918
Reviewed-by: Herbie Ong <herbie@google.com>
2018-11-06 21:02:36 +00:00
Joe Tsai
f0ad886d73 reflect/prototype: implement NewGoExtension
Implement the constructor for protoreflect.ExtensionType.
The constructor is more complicated than NewGoEnum and NewGoMessage because
it is responsible for providing the wrappers to present *[]T as a
protoreflect.Vector.

There are no tests since we need the follow-up logic in internal/impl to actually
make use of extensions. A subsequent CL will add that logic and comprehensively
test extensions.

Change-Id: I2d7893de299fe40be2ccedd8f39a92c40c41e59a
Reviewed-on: https://go-review.googlesource.com/c/147578
Reviewed-by: Herbie Ong <herbie@google.com>
2018-11-06 20:55:45 +00:00
Joe Tsai
88bc5a7d99 internal/value: extract Vector and Map logic as separate package
The implementation of reflect/protoreflect.NewGoExtension needs to be able to
provide a constructor for wrapping *[]T as a protoreflect.Vector.
However, it cannot depend on internal/impl since impl also depends on prototype.
Extract the common logic of Vector creation into a separate package that
has no dependencies on either impl or prototype.

Change-Id: I9295fde9b8861de11af085c91d9dfa56047d1b1e
Reviewed-on: https://go-review.googlesource.com/c/147446
Reviewed-by: Herbie Ong <herbie@google.com>
2018-11-06 01:20:32 +00:00
Joe Tsai
ce6edd3c71 internal/impl: support message and enum fields
Dynamically generate functions for handling message and enum fields,
regardless of whether they are of the v1 or v2 forms.

If a v1 message is encountered, it is automatically wrapped such that it
implements the v2 interface.

Change-Id: I457bc5286892e8fc00a61da7062dd33058daafd5
Reviewed-on: https://go-review.googlesource.com/c/143837
Reviewed-by: Damien Neil <dneil@google.com>
2018-11-05 22:25:52 +00:00
Joe Tsai
05828dba44 internal/encoding/tag: centralize logic for protobuf struct tag serialization
The bespoke text-serialization of field descriptors in protoc-gen-go is also
used in the legacy implementation of protobuf reflection to derive a
protoreflect.FieldDescriptor from legacy messages and also to convert to/from
protoreflect.ExtensionDescriptor and protoV1.ExtensionDesc.

Centralize this logic in a single place:
* to avoid reimplementing the same logic in internal/impl
* to keep the marshal and unmarshal logic co-located

Change-Id: I634c5afbb9dc6eda91d6cb6b0e68dbd724cb1ccb
Reviewed-on: https://go-review.googlesource.com/c/146758
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
2018-11-05 20:12:56 +00:00
Joe Tsai
95b0290ea8 internal/impl: support legacy unknown extension fields
The unknown fields in legacy messages is split across the XXX_unrecognized
field and also the XXX_InternalExtensions field. Implement support for
wrapping both fields and presenting it as if it were a unified set of
unknown fields.

Change-Id: If274fae2b48962520edd8a640080b6eced747684
Reviewed-on: https://go-review.googlesource.com/c/146517
Reviewed-by: Damien Neil <dneil@google.com>
2018-11-01 18:28:27 +00:00
Damien Neil
bb1d557b23 cmd/protoc-gen-go: factor out oneof field naming
A oneof is represented by a single struct field of interface type.
Pull out the decision of what to name that field into a separate
function.

The function is trivial (return oneof.GoName), but factoring out the
field name like this makes it a bit easier to experiment with changes to
the oneof implementation in the future.

Change-Id: I1114b68c85cb6608852fa1c6bf4103ff58fd5de6
Reviewed-on: https://go-review.googlesource.com/c/146397
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-10-31 20:40:31 +00:00
Joe Tsai
2d5a169f75 internal/impl: use linked-list for unknown field ordering
Unknown fields follow a policy where the latest field takes precedence when
it comes to the ordering. However, the current implementation is incorrect
as it uses a slice and simply swaps the current entry with the last entry.
While this ensures that the latest field seen remains last, it does not ensure
that the swapped out entry is second-to-last.

To provide the desired behavior, a linked-list is used.
For simplicity, we use the list package in the standard library even if it
is neither the most performant nor type safe.

Change-Id: I675145c61f6b5b624ed9e94bbe2251b5a71e2c48
Reviewed-on: https://go-review.googlesource.com/c/145241
Reviewed-by: Damien Neil <dneil@google.com>
2018-10-31 18:39:54 +00:00
Damien Neil
6238696abc cmd/protoc-gen-go: replicate v1 generator behavior for MessageSet extensions
Given:

  package foo
  extend proto2.bridge.MessageSet {
    optional Message message_set_extension = 100;
  }

Register the extension as a message set extension and give it the name
"foo.".

We really shouldn't do this in this case; the special-case treatment of
extensions to MessageSet is only for extensions nested in a parent
message. However, this is consistent with the behavior of the v1 generator.
Match that for now.

Change-Id: I919c409605a197904fd3227efc920192d484f431
Reviewed-on: https://go-review.googlesource.com/c/145957
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-10-30 18:46:20 +00:00
Damien Neil
6b54131a89 cmd/protoc-gen-go: generate forwarding decls for ExtensionDescs
When publicly importing a package with extension definitions, generate
forwarding declarations for the "E_..." ExtensionDesc var:

    var E_ExtensionField = publicimport.E_ExtensionField

Change-Id: Ifd57c487c3a44f303c2c098a42ea249b219b734f
Reviewed-on: https://go-review.googlesource.com/c/145498
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-10-29 17:01:50 +00:00
Damien Neil
40a08052f9 cmd/protoc-gen-go: don't import _ weak imports
Don't generate a blank import for unused weak imports.

Full support for weak imports would involve not importing the package at
all. This just avoids generating an import when we don't need one.

Change-Id: I7e8491f415dc8333a2837db5225256b959921be2
Reviewed-on: https://go-review.googlesource.com/c/145497
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-10-29 16:16:32 +00:00
Damien Neil
a485fbd353 reflect/protoreflect: add FieldDescriptor.DefaultEnumValue method
Default values for enums are specified by name, not number. An enum
may contain multiple values with different names but the same number.
Representing the default as a protoreflect.Value containing an EnumNumber
can discard information.

Add a method returning the EnumValueDescriptor.

Change-Id: If8beee3f81d41c4f9af45423252603b86949c7a5
Reviewed-on: https://go-review.googlesource.com/c/145158
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-10-29 15:59:23 +00:00
Joe Tsai
e2afdc27e7 internal/impl: support legacy unknown fields
Add wrapper data structures to get legacy XXX_unrecognized fields to support
the new protoreflect.UnknownFields interface. This is a challenge since the
field is a []byte, which does not give us much flexibility to work with
in terms of choice of data structures.

This implementation is relatively naive where every operation is O(n) since
it needs to strip through the entire []byte each time. The Range operation
operates slightly differently from ranging over Go maps since it presents a
stale version of RawFields should a mutation occur while ranging.
This distinction is unlikely to affect anyone in practice.

Change-Id: Ib3247cb827f9a0dd6c2192cd59830dca5eef8257
Reviewed-on: https://go-review.googlesource.com/c/144697
Reviewed-by: Damien Neil <dneil@google.com>
2018-10-26 19:55:21 +00:00
Joe Tsai
78f02a161f reflect/protoreflect: clarify Mutable semantics
The Mutable semantics are simpler than currently documented. In an older design
of the protobuf reflection API, it was intended that there could be a mutable
version of the scalar kinds. However, support for this has been dropped,
simplifying the documented semantics.

Change-Id: If66740ffa5da65a3f2c8cf46700e0c23393a16d7
Reviewed-on: https://go-review.googlesource.com/c/144857
Reviewed-by: Damien Neil <dneil@google.com>
2018-10-26 00:35:03 +00:00
Joe Tsai
620bc28bae reflect/protoreflect: clarify Range semantics
Restrict mutation operations during Range to only the current iteration key.
This simplifies the guarantees that any given implementation may need to provide.

While none of the range operations have a defined order, UnknownFields.Range
is special in that the iteration order is at least deterministic.
This exception occurs because there is no obviously correct way to re-order them
(since order of unknown fields can have semantic significance).

Change-Id: I35145e500dbc7c87be7270f0d886ef52e13d07d8
Reviewed-on: https://go-review.googlesource.com/c/144700
Reviewed-by: Damien Neil <dneil@google.com>
2018-10-25 23:23:31 +00:00
Damien Neil
44000a1bea cmd/protoc-gen-go: treat extensions fields as proto2 in some places
The v1 generator doesn't include a "proto3" tag on extension fields,
even when the field is defined in a proto3 file. Match that behavior for
consistency. We can probably change this later if we want to; it's
unlikely anyone is depending on this behavior.

The v1 generator uses pointer types for extension fields, even when the
field is defined in a proto3 file. (e.g., *FooEnum instead of FooEnum.)
Match this behavior. We can't change this without breaking compatibility
in the generated code.

Change-Id: I4072f3dd1c915bf9ab89f1d5198e0144cb4de20f
Reviewed-on: https://go-review.googlesource.com/c/144282
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-10-24 19:46:45 +00:00
Joe Tsai
ad8ee05d0e reflect/prototype: make extensions use proto2 semantics for IsPacked
Extensions are normally thought of as a proto2 feature.
However, proto3 actually allows declarations of field extensions *only*
if they extend messages declared in descriptor.proto.

In such a situation it is not well defined whether the extension field
operates under proto2 or proto3 properties.

In some ways, such declarations are proto3-like.
For example, the declaration in the proto file still follows
proto3's syntactical grammar (e.g., no "optional" label required)
and proto3 semantics (e.g., default values are not allowed).

However, in other ways, such declarations are proto2-like.
For example, a repeated field of numeric scalars is not automatically packed.
This property is determined by the following test.

Compile the following proto file:

	syntax = "proto3";
	import "google/protobuf/descriptor.proto";
	extend google.protobuf.FieldOptions {
		repeated sint32 test_field = 20181023;
	}

then compile and run a C++ program with the following snippet:

	google::protobuf::FieldOptions m;
	m.AddExtension(test_field, 1);
	m.AddExtension(test_field, 2);
	m.AddExtension(test_field, 3);
	fstream output("out", ios::out | ios::trunc | ios::binary);
	m.SerializeToOstream(&output);

which produces an "out" file with the following contents:

	$ pbdump -sints 20181023 out
	Message{
		Tag{20181023, Varint}, Svarint(1),
		Tag{20181023, Varint}, Svarint(2),
		Tag{20181023, Varint}, Svarint(3),
	}

which is indicative that packed encoding was not used by default (proto2-like).
If we repeat the above experiment and explicit set "[packed = true]",
then packed encoding is used:

	$ pbdump -sints 20181023 out
	Message{
		Tag{20181023, Bytes}, LengthPrefix{Svarint(1), Svarint(2), Svarint(3)},
	}

Note that this change does not mean that field extensions are always proto2
since the experiment above does not conclusively prove that. Thus, the Syntax
on extensions derived from the descriptor protos may still report proto3.

The behavior of packed encoding with regards to proto3 says:

	In proto3, repeated fields of scalar numeric types use packed encoding by default.

One way to interpret this is:

	In proto3 (messages), repeated fields of scalar numeric types use packed encoding by default.

In which case packedness is a property of the message's syntax
rather than the field's syntax (if such a distinction exists).
Since only proto2 messages can be extended, we can safely assume that all
extension fields use proto2 semantics for IsPacked.

Change-Id: Iae595c6d88c6e252cae7552cae083bad42f2494a
Reviewed-on: https://go-review.googlesource.com/c/144278
Reviewed-by: Damien Neil <dneil@google.com>
2018-10-24 17:43:20 +00:00
Joe Tsai
be5348c905 internal/impl: setup scaffolding for unknown and extension fields
Setup scaffolding for implementing unknown and extension fields.
Add functions to MessageType to produce a protoreflect.KnownFields or
protoreflect.UnknownFields from a message pointer.

Within the implementation of known fields, delegate the logic to the underlying
extension fields (which also implements protoreflect.KnownFields) if the field
number is not found in the set of defined fields.

Change-Id: I2c35f4cdf1c7b58727ce6a582861ef18b8d69a61
Reviewed-on: https://go-review.googlesource.com/c/144280
Reviewed-by: Damien Neil <dneil@google.com>
2018-10-24 17:40:36 +00:00
Joe Tsai
2f68590cf0 reflect/prototype: use proto2 syntax for standalone extension fields
Without resolving the question of whether an extension field is always proto2
or sometimes proto3, just make it impossible to construct a standalone extension
field of the proto3 syntax.

By dropping the syntax field from the constructor, we can just assume that the
syntax is always proto2. There is no benefit for creating a proto3 standalone
extension, so there is no loss of functionality.

In the unlikely future where there needs to be distinction, we can add the field
back into the descriptor. The zero value of protoreflect.Syntax is invalid,
so we can determine if the user did not set that field.

Change-Id: Ie93fa55de96f29553fb04ff6649bbea79a144407
Reviewed-on: https://go-review.googlesource.com/c/144279
Reviewed-by: Damien Neil <dneil@google.com>
2018-10-24 17:35:49 +00:00
Damien Neil
204f1c0ad8 reflect/protoreflect: add Descriptor.Options method
Add a method to fetch descriptor options. Since options are proto
messages (e.g., google.protobuf.FieldOptions), and proto message
packages depend on the protoreflect package, returning the actual option
type would cause a dependency cycle. Instead, we return an interface
value which can be type asserted to the appropriate concrete type.

Add options support to the prototype package.

Some of the prototype constructors included fields (such as
Field.IsPacked) which represent information from the options
(such as google.protobuf.FieldOptions.packed). To avoid confusion about
the canonical source of information, drop these fields in favor of the
options.

Drop the unimplemented Descriptor.DescriptorOptionsProto.

Change-Id: I66579b6a7d10d99eb6977402a247306a78913e74
Reviewed-on: https://go-review.googlesource.com/c/144277
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-10-23 23:44:11 +00:00
Joe Tsai
90fe996bd1 internal/impl: derive descriptors for legacy enums and messages
In order for the v2 rollout to be as seamless as possible, we need to support
the situation where a v2 message depends on some other generated v1 message that
may be stale and does not support the v2 API. In such a situation, there needs
to be some way to wrap a legacy message or enum in such a way that it satisfies
the v2 API.

This wrapping is comprised of two parts:
1) Deriving an enum or message descriptor
2) Providing an reflection implementation for messages

This CL addresses part 1 (while part 2 has already been partially implemented,
since the implementation applies to both v1 and v2).

To derive the enum and message descriptor we rely on a mixture of parsing the
raw descriptor proto and also introspection on the fields in the message.
Methods for obtaining the raw descriptor protos were added in February, 2016,
and so has not always been available. For that reason, we attempt to derive
as much information from the Go type as possible.

As part of this change, we modify prototype to be able to create multiple
standalone messages as a set. This is needed since cyclic dependencies is allowed
between messages within a single proto file.

Change-Id: I71aaf5f977faf9fba03c370b1ee17b3758ce60a6
Reviewed-on: https://go-review.googlesource.com/c/143539
Reviewed-by: Damien Neil <dneil@google.com>
2018-10-22 17:18:39 +00:00
Joe Tsai
bda671fa93 cmd/protoc-gen-go: re-escape default byte values
Historically, protoc-gen-go outputted the escaped form of bytes as provided by
protoc verbatim. This behavior is buggy, but nothing really uses this tag
since default values are properties of getters instead of serialization.
Rather than fixing it, just preserve prior behavior. Otherwise, logic depending
on the old legacy behavior will not be able to distinguish between the unescaped
or the escaped forms.

Furthermore, since protoc-gen-go historically copied the protoc output verbatim,
we will need to escape the default bytes in a way that is identical to the
CEscape function from strutil.cc of the protoc source code.

Change-Id: I0ab55e220ae430dd123ad050406e285788f6cb40
Reviewed-on: https://go-review.googlesource.com/c/143543
Reviewed-by: Damien Neil <dneil@google.com>
2018-10-22 17:14:19 +00:00
Joe Tsai
2c870bb5cc internal/impl: implement oneof fields
Dynamically generate functions for handling individual fields within an oneof.
This implementation uses Go reflection to interact with the currently generated
approach, which uses an interface that can only be set by a limited set of
wrapper structs.

Change-Id: Ic848df922d6547411a15c4a20bfbbcae362da5c0
Reviewed-on: https://go-review.googlesource.com/c/142895
Reviewed-by: Damien Neil <dneil@google.com>
2018-10-17 22:20:50 +00:00
Damien Neil
ba1159f426 protogen: move comment generation into protogen
Most plugins need to copy comments from .proto source files into the
generated code. Move this functionality into protogen to avoid
duplicating it everywhere.

Change-Id: I48a96ba794192e7ddc00281342afd4805ef6fe0f
Reviewed-on: https://go-review.googlesource.com/c/142890
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-10-17 21:03:49 +00:00
Joe Tsai
3903b218df reflect/protoreflect: remove List method
Remove List from KnownFields, UnknownFields, ExtensionFieldTypes, and Map.

Rationale:
* Each of those interfaces already have a Range method, which provides
a superset of the functionality of List. Furthermore, Range is more expressive
as it allows you to terminate iteration and provides both keys and values.
* List must always allocate a slice and populates it.
* Range is allocation free in some cases. For example, if you simply wanted to
iterate over all the populated fields to clear them, there is no need for a
closure, so a static version of the function can be directly referenced
(i.e., there is no need to create a stub function header that references the
closed-over variables).
* In the cases where a closure is needed, the allocation cost is O(1) since
there are a finite number of variables being closed over.
* In highly performance sensitive cases, the closured function could close over
a struct, such that the function and struct are stored in a sync.Pool when not
in use. For example:

	type MapLister struct {
		Entries []struct{K MapKey; V Value}
		f       func(MapKey, Value) true
	}
	func (m *MapLister) Ranger() func(MapKey, Value) bool {
		if m.f != nil {
			m.f = func(k MapKey, v Value) bool {
				m.Entries = append(m.Entries, ...)
				return true
			}
		}
		m.Entries = m.Entries[:0]
		return m.f
	}

The main benefit of List is the ease of use:

	for _, num := range knownFields.List() {
		...
	}

as opposed to:

	knownFields.Range(func(n FieldNumber, v Value) bool {
		...
		return true
	})

However, this is a marginal benefit.
Thus, remove List as it mostly inferior to Range.

Change-Id: I25586c6ea07a4706072ba06b1cf25cb6efb5e8a7
Reviewed-on: https://go-review.googlesource.com/c/142888
Reviewed-by: Damien Neil <dneil@google.com>
2018-10-17 19:30:39 +00:00
Joe Tsai
bbfaeb77e5 internal/impl: implement Map fields
Generate functions for wrapping map[K]V to implement protoreflect.Map.
This implementation uses Go reflection instead to provide a single implementation
that can handle all Go map types.

Change-Id: Idcb8069ef836614a88e5df12ef7c5044e8aa3dea
Reviewed-on: https://go-review.googlesource.com/c/142778
Reviewed-by: Damien Neil <dneil@google.com>
2018-10-17 17:45:34 +00:00
Joe Tsai
91e1466d6f internal/impl: implement Vector fields
Generate functions for wrapping []T to implement protoreflect.Vector.
This implementation uses Go reflection instead to provide a single implementation
that can handle all Go slice types.

The test harness was greatly expanded to be able to test vectors (in addition
to messages and maps in the near future).

Change-Id: I0106c175f84a1e7e0a0a5b0e02e2489b70b0d177
Reviewed-on: https://go-review.googlesource.com/c/135339
Reviewed-by: Damien Neil <dneil@google.com>
2018-10-17 17:39:36 +00:00
Damien Neil
162c12703c protogen: generate .meta file with code annotations
When the generator parameter 'annotate_code' is provided, generate a .meta
file containing a GeneratedCodeInfo message describing the generated code's
relation to the source .proto file.

Annotations are added with (*protogen.GeneratedFile).Annotate, which takes the
name of a Go identifier (e.g., "SomeMessage" or "SomeMessage.GetField") and an
associated source location. The generator examines the generated AST to
determine source offsets for the symbols.

Change the []int32 "Path" in protogen types to a "Location", which also captures
the source file name.

Change-Id: Icd2340875831f40a1f91d495e3bd7ea381475c77
Reviewed-on: https://go-review.googlesource.com/c/139759
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-10-16 17:22:41 +00:00
Damien Neil
d277b5236a protogen: produce an error if len(os.Args) > 1
Users sometimes run protoc-gen-go directly (rather than via protoc
--go_out) and are then confused when it sits and does nothing,
waiting for input from stdin. Print an error if any command-line
arguments are passed to it.

Change-Id: I10a059a82ec71b61c4eb6aeecc1e47e2446feabb
Reviewed-on: https://go-review.googlesource.com/c/139877
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-10-16 16:53:21 +00:00
Damien Neil
2193e8df3b cmd/protoc-gen-go: import public: forward default const/vars
Generate forwarders for default value const/vars defined in public
imports:

    const Default_Message_Field = pubimport.Default_Message_Field

Change-Id: Ife09e38ae6a674b4460dd6613a8264e23f30b277
Reviewed-on: https://go-review.googlesource.com/c/140897
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-10-16 16:53:07 +00:00
Joe Tsai
42fa50da34 test.bash: do not run golden tests by default
The golden tests are sensitive to the exact version used:
* Protobuf toolchain since the generated Descriptor is dependendent on protoc
(for example, default json_names were auto-populated in later versions of protoc)
* Go toolchain since the generated .pb.go files is dependent on the exact output
of gofmt and the gzip compression used.

There are other areas where it depends on unstable output, but the above are the
major causes.

Since test.bash ensures that the golden tests are run with exact versions of the
protobuf and Go toolchains, we can ensure that the tests are reproducible across
different developer workstations.

Thus, we add a "golden" build tag to disable them for normal invocations of
"go test ./..." and only run them if test.bash is run.

If test.bash is ever updated with newer versions, the golden testdata can be
updated at the same time.

Change-Id: Ia2b7b039aad2ddaef7652e332215bf9403a6d856
Reviewed-on: https://go-review.googlesource.com/c/142458
Reviewed-by: Damien Neil <dneil@google.com>
2018-10-16 16:49:18 +00:00
Joe Tsai
a85e9569ed test.bash: update to go1.10.4 and go1.11.1
Change-Id: Ib593768cb30fab41ff93e90cc12401e3b7126e6c
Reviewed-on: https://go-review.googlesource.com/c/142457
Reviewed-by: Damien Neil <dneil@google.com>
2018-10-16 16:43:44 +00:00
Damien Neil
3863ee56b5 protogen: camel-case "Foo.bar" as "FooBar" instead of "Foo_Bar".
Given:

  message Parent {
    message Child1 {}
    message child2 {}
  }

Historic behavior is to generate child messages named:

  Parent_Child1
  ParentChild2

Avoid adding an _ in the latter case.

Change-Id: I49a6732655d64967b8c7bb7ad358ae54d294f7b4
Reviewed-on: https://go-review.googlesource.com/c/140898
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-10-16 16:18:01 +00:00
Damien Neil
21f62f4d53 reflect/prototype: fix name resolution when file has no package name
A .proto source file with no 'package' statement may still contain
references to descriptors within the file.

Change-Id: I86e942c9c06e5a2915e9722162e0455ffa9ba2ab
Reviewed-on: https://go-review.googlesource.com/c/140899
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-10-16 16:16:46 +00:00
Damien Neil
bbbd38f9f9 protogen: avoid an obscure package name consistency check failure
When performing consistency checks on package names, ignore files that
are mentioned on the command line (M<filename>=<import_path>) but which
do not appear in the CodeGeneratorRequest.

Change-Id: I51f1eeda5eaaac14f4a0a975163f65d7774e7212
Reviewed-on: https://go-review.googlesource.com/c/140717
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-10-16 16:04:50 +00:00
Damien Neil
daa4fad986 cmd/protoc-gen-go: remove spurious . from enums with no package
Enums, for historical reasons, are registered with the proto package
under the name "<proto_package>.<go_type_name>". Don't include the dot
if there is no package statement in the .proto source file.

Change-Id: I6fb57d0803506668f60123a29fa06ae87fec523b
Reviewed-on: https://go-review.googlesource.com/c/140657
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-10-16 16:04:35 +00:00
Damien Neil
09f89d0246 protoc-gen-go: generate per-field comments for oneofs
Comment location and formatting matches previous generator.

Change-Id: Id51489bb8ca6e6bbd2c7bc2129a7bc170e56ed44
Reviewed-on: https://go-review.googlesource.com/c/140617
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-10-16 16:04:15 +00:00
Damien Neil
329b75ece9 protogen: rewrite _ imports as well
Call the import-rewriting function on _ imports as well as named ones.

Change-Id: I2fe2c90fe4b201e96a243f231b54387c6b65f7a5
Reviewed-on: https://go-review.googlesource.com/c/139878
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-10-16 16:03:55 +00:00
Damien Neil
a4a765351c cmd/protoc-gen-go: remove spurious space in "* T"
From:
  func (* T) isT_F() {}

To:
  func (*T) isT_F() {}

Formatting the file removes the space, but the presence or absence of
the space affects the formatter's decision on whether to split the {}
into {\n} or not.

Change-Id: I794c855a3115f9ae1b5f048728d8cad7a5f03e69
Reviewed-on: https://go-review.googlesource.com/c/140637
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-08 21:34:46 +00:00
Damien Neil
872146653d protogen: conflict resolve local package names against predefined idents
Avoid importing packages with a local name that conflicts with a predefined
identifier (e.g., "string").

Change-Id: I51164635351895e8a060355e59d718240e26ef2e
Reviewed-on: https://go-review.googlesource.com/c/140178
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-10-05 19:58:09 +00:00
Joe Tsai
c6b7561199 internal/impl: support wrapping Go structs to implement proto.Message
Given a pointer to a Go struct (that is well-formed according to the v1
struct field layout), wrap the type such that it implements the v2
protoreflect.Message interface.

Change-Id: I5987cad0d22e53970c613cdbbb1cfd4210897f69
Reviewed-on: https://go-review.googlesource.com/c/138897
Reviewed-by: Damien Neil <dneil@google.com>
2018-10-03 02:10:04 +00:00
Damien Neil
b9781cc938 cmd/protoc-gen-go: convert field number to uint64 before shifting
Avoid generating invalid tag and wire size for large oneof field numbers
which overflow int32.

Change-Id: I005fe32aba4944b33b6b6ba83ef0ddd4d6e5863b
Reviewed-on: https://go-review.googlesource.com/138519
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-09-28 22:01:48 +00:00
Damien Neil
ccf3fa6cad cmd/protoc-gen-go: don't generate default values for [default=""].
The previous generator considers a default value of "" (valid only for
string and bytes fields) to be unset, and does not generate a const or
var to hold the default value.

e.g.,

  message M {
    optional F string = 1 [default=""];
  }

does not generate this constant, even though the field has a default
value:

  const Default_M_F string = ""

Maintain consistent output.

Change-Id: Ib172b02d59c15c05e19a7056d05ce1c619a2fa40
Reviewed-on: https://go-review.googlesource.com/138518
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-09-28 21:56:28 +00:00
Damien Neil
ea7baf45f8 cmd/protoc-gen-go: annotate all well-known-types
Change-Id: I3559bb47cb8e93aadeea6224857d65e84ca4b27a
Reviewed-on: https://go-review.googlesource.com/138517
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-09-28 21:56:15 +00:00
Damien Neil
183aa3a3e6 cmd/protoc-gen-go-grpc: add deprecation comments
Change-Id: I7cf579acb11eeb4e56ec3dd4baec08758b8dc7e1
Reviewed-on: https://go-review.googlesource.com/138376
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-09-28 21:55:39 +00:00