Temporarily remove go.mod, since we can't generate an accurate one until
the corresponding v1 change is submitted.
Change-Id: I1e1ad97f2b455e33f61ffaeb8676289795e47e72
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/177000
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Added API:
FieldDescriptor.IsExtension
FieldDescriptor.IsList
FieldDescriptor.MapKey
FieldDescriptor.MapValue
FieldDescriptor.ContainingOneof
FieldDescriptor.ContainingMessage
Deprecated API (to be removed in subsequent CL):
FieldDescriptor.Oneof
FieldDescriptor.Extendee
These methods help cleanup several common usage patterns.
Change-Id: I9a3ffabc2edb2173c536509b22f330f98bba7cf3
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/176977
Reviewed-by: Damien Neil <dneil@google.com>
Added methods:
Enum.Descriptor
Message.Descriptor
EnumType.Descriptor
MessageType.Descriptor
ExtensionType.Descriptor
Message.New
All functionality is switched over to use those methods instead of
implicitly relying on the fact that {Enum,Message}Type implicitly
implement the associated descriptor interface.
This CL does not yet remove {Enum,Message}.Type or prevent
{Enum,Message,Extension}Type from implementating a descriptor.
That is a subsequent CL.
The Message.New method is also added to replace functionality
that will be lost when the Type methods are removed.
Change-Id: I7fefde1673bbd40bfdac489aca05cec9a6c98eb1
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/174918
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Herbie Ong <herbie@google.com>
The protobuf type system uses the word "descriptor" instead of "type".
We should avoid the "type" verbage when we aren't talking about Go types.
The old names are temporarily kept around for compatibility reasons.
Change-Id: Icc99c913528ead011f7a74aa8399d9c5ec6dc56e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/172238
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
Move all checks for required fields into a proto.IsInitialized function.
Initial testing makes me confident that we can provide a fast-path
implementation of IsInitialized which will perform more than
acceptably. (In the degenerate-but-common case where a message
transitively contains no required fields, this check can be nearly
zero cost.)
Unifying checks into a single function provides consistent behavior
between the wire, text, and json codecs.
Performing the check after decoding eliminates the wire decoder bug
where a split message is incorrectly seen as missing required fields.
Performing the check after decoding also provides consistent and
arguably more correct behavior when the target message was partially
prepopulated.
Change-Id: I9478b7bebb263af00c0d9f66a1f26e31ff553522
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170787
Reviewed-by: Herbie Ong <herbie@google.com>
In marshalMap, replace Map.Range and sortMap with
internal/mapsort.Range. Also, make sure to capture fatal error properly
for return.
In appendExtensions, capture fatal error properly for return. Added a
testcase that shows this was a bug.
In unmarshalAny, remove unnecessary checks and use internal/fieldnum for
Any's field numbers.
Change-Id: Id8574d5b4eb820ad961f6ad5e886f8ae2e9b90f0
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170627
Reviewed-by: Damien Neil <dneil@google.com>
Unmarshaling should fail if multiple fields in the same oneof exists in
the input.
Change-Id: I76efd88681a50c18f3eaf770c9eb48727efb412b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170517
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Provide AllowPartial option to accept messages with missing required
field during marshaling and unmarshaling.
Change-Id: Ia23783870a8125633f8ddc0b686984b4c5ca15ba
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169500
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Reviewed-by: Damien Neil <dneil@google.com>
As a goal, v2 should not depend on v1. As another step towards that end,
we move all the types that used to be in the v1 protoapi package over to v2.
For now, we place MessageV1, ExtensionRangeV1, and ExtensionDescV1
in runtime/protoiface since these are types that generated messages will
probably have to reference forever. An alternative location could be
reflect/protoreflect, but it seems unfortunate to have to dirty the
namespace of that package with these types.
We move ExtensionFieldV1, ExtensionFieldsV1, and ExtensionFieldsOf
to internal/impl, since these are related to the implementation of a
generated message.
Since moving these types from v1 to v2 implies that the v1 protoapi
package is useless, we update all usages of v1 protoapi in the v2
repository to point to the relevant v2 type or functionality.
CL/168538 is the corresponding change to alter v1.
There will be a temporary build failure as it is not possible
to submit CL/168519 and CL/168538 atomically.
Change-Id: Ide4025c1b6af5b7f0696f4b65b988b4d10a50f0b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168519
Reviewed-by: Herbie Ong <herbie@google.com>
Remove unnecessary v1 wrapping. Change most tests to operate on message
Any directly.
Change-Id: I19bbca6c1af72894f6a292aab8ebd09e0301a260
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168517
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
This makes it consistent with jsonpb.MarshalOptions. This does change
the default to be in compact form.
Change-Id: I1b07f06f282c019b30f3f1cbb43f6c8cba18f385
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168405
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Changes:
* Modify protoc-gen-go to use a helper from internal/impl
to print the message as text.
* Add a helper function to internal/impl that calls v2 textpb.
* Modify encoding/textpb to avoid depending on descriptor proto,
which would cause an import cycle.
* Modify internal/fileinit to populate a pseudo-internal
method on MessageDescriptor to check whether the message should
use the message set wire format. We avoid adding this to the
main MessageDescriptor interface since message sets are a
deprecated proto1 feature that we should avoid promoting.
Change-Id: Ibaf79a563af695756f11ddc4db69b38e25a8f1a7
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168439
Reviewed-by: Herbie Ong <herbie@google.com>
Added temporary work-around for now by always using deterninistic
marshaling so that protoV1.Equals knows how to do comparison.
Change-Id: Idec412647269d59f86ed3e485a1893c070355b3d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/167917
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Substitute protoV1.Marshal calls to use proto v2.
A few tests still need to use v1 due to missing required field checks in
v2.
Change-Id: I97ee208b81afc85c81142e70c1d7bdc47db76ce4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/167774
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
A blank line between the docstring and the package expression prevents
godoc from recognizing the comment as the package's docstring.
Change-Id: Id3c46ec6dbadb85a9b393c34fe1f0e46ed41a447
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/167766
Reviewed-by: Herbie Ong <herbie@google.com>
Unfortunately a good amount of code uses pointer comparisons on the
v1 ExtensionDesc to determine exactly which extension field is set,
rather than checking whether the extension descriptor semantically
describes the field that they are interested in.
To preserve this behavior in v1, we need a 1:1 mapping between
a v2 ExtensionType and a specific v1 ExtensionDesc.
Change-Id: I852b3cefb4585bd656e48e5adad6cc28795d02df
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/167759
Reviewed-by: Damien Neil <dneil@google.com>
Collapse Value.Float32 and Value.Float64 into single API to keep it
consistent with Value.{Int,Uint}.
Change-Id: I07737e72715fe3cc3f6bcad579cf5d6cfe3757d5
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/167317
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Create a single binary for handling generation of protos.
This replaces previous logic spread throughout the repo in:
* regenerate.bash
* cmd/protoc-gen-go/golden_test.go
* cmd/protoc-gen-go-grpc/golden_test.go
* (indirectly) internal/protogen/goldentest
One of the problems with the former approaches is that they relied on
a version of protoc that was specific to a developer's workstation.
This meant that the result of generation was not hermetic.
To address this, we rely on the hard-coded version of protobuf specified
in the test.bash script.
A summary of changes in this CL are:
* The internal_gengo.GenerateFile and internal_gengogrpc.GenerateFile
functions are unified to have consistent signatures. It seems that the
former accepted a *protogen.GeneratedFile to support v1 where gRPC code
was generated into the same file as the base .pb.go file. However, the
same functionality can be achieved by having the function return
the generated file object.
* The test.bash script patches the protobuf toolchain to have properly
specified go_package options in each proto source file.
* The test.bash script accepts a "-regenerate" argument.
* Add generation for the well-known types. Contrary to how these were
laid out in the v1 repo, all the well-known types are placed in the
same Go package.
* Add generation for the conformance proto.
* Remove regenerate.bash
* Remove internal/protogen
* Remove cmd/protoc-gen-go/golden_test.go
* Remove cmd/protoc-gen-go-grpc/golden_test.go
* Add cmd/protoc-gen-go/annotation_test.go
Change-Id: I4a1a97ae6f66e2fabcf4e4d292c95ab2a2db0248
Reviewed-on: https://go-review.googlesource.com/c/164477
Reviewed-by: Damien Neil <dneil@google.com>
These test proto definitions will be reused for encoding/jsonpb package
and hence move these one directory up.
Also, add and simplify some tests.
Change-Id: I5297546fd9b853a7fd3e72dfab2fdc7237332c9c
Reviewed-on: https://go-review.googlesource.com/c/162537
Reviewed-by: Damien Neil <dneil@google.com>
Updates golang/protobuf#798 by adding testcases to show the intention.
Also, slightly move code blocks in encode.go w/o affecting logic to make
it cleaner.
Change-Id: I14575f6e7139a0908483bd318b599339c2daf8ad
Reviewed-on: https://go-review.googlesource.com/c/161717
Reviewed-by: Damien Neil <dneil@google.com>
This only handles compliant MessageSet extension fields where the field
name has to be message_set_extension.
Current C++ lib allows for different message field names, which is a
possible bug as it makes marshal output possibly contain duplicate names
when more than one field extends the same MessageSet, and makes
unmarshaling confusing as to which field to populate.
Change-Id: Ifda828ba794fe7e058ee6004f03001b1031f6d1e
Reviewed-on: https://go-review.googlesource.com/c/156758
Reviewed-by: Damien Neil <dneil@google.com>
This CL takes a significantly different approach to generating support
for protobuf reflection. The previous approach involved generating a
large number of Go literals to represent the reflection information.
While that approach was correct, it resulted in too much binary bloat.
The approach taken here initializes the reflection information from
the raw descriptor proto, which is a relatively dense representation
of the protobuf reflection information. In order to keep initialization
cost low, several measures were taken:
* At program init, the bare minimum is parsed in order to initialize
naming information for enums, messages, extensions, and services declared
in the file. This is done because those top-level declarations are often
relevant for registration.
* Only upon first are most of the other data structures for protobuf
reflection actually initialized.
* Instead of using proto.Unmarshal, a hand-written unmarshaler is used.
This allows us to avoid a dependendency on the descriptor proto and also
because the API for the descriptor proto is fundamentally non-performant
since it requires an allocation for every primitive field.
At a high-level, the new implementation lives in internal/fileinit.
Several changes were made to other parts of the repository:
* cmd/protoc-gen-go:
* Stop compressing the raw descriptors. While compression does reduce
the size of the descriptors by approximately 2x, it is a pre-mature
optimization since the descriptors themselves are around 1% of the total
binary bloat that is due to generated protobufs.
* Seeding protobuf reflection from the raw descriptor significantly
simplifies the generator implementation since it is no longer responsible
for constructing a tree of Go literals to represent the same information.
* We remove the generation of the shadow types and instead call
protoimpl.MessageType.MessageOf. Unfortunately, this incurs an allocation
for every call to ProtoReflect since we need to allocate a tuple that wraps
a pointer to the message value, and a pointer to message type.
* internal/impl:
* We add a MessageType.GoType field and make it required that it is
set prior to first use. This is done so that we can avoid calling
MessageType.init except for when it is actually needed. The allows code
to call (*FooMessage)(nil).ProtoReflect().Type() without fearing that the
init code will run, possibly triggering a recursive deadlock (where the
init code depends on getting the Type of some dependency which may be
declared within the same file).
* internal/cmd/generate-types:
* The code to generate reflect/prototype/protofile_list_gen.go was copied
and altered to generated internal/fileinit.desc_list_gen.go.
At a high-level this CL adds significant technical complexity.
However, this is offset by several possible future changes:
* The prototype package can be drastically simplified. We can probably
reimplement internal/legacy to use internal/fileinit instead, allowing us
to drop another dependency on the prototype package. As a result, we can
probably delete most of the constructor types in that package.
* With the prototype package significantly pruned, and the fact that generated
code no longer depend on depends on that package, we can consider merging
what's left of prototype into protodesc.
Change-Id: I6090f023f2e1b6afaf62bd3ae883566242e30715
Reviewed-on: https://go-review.googlesource.com/c/158539
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
When encoding/textpb marshals out float32 values, it was previously
formatting it as float64 bitsize since both float types are stored as
float64 and internal/encoding/text only has one Float type. A
consequence of this is that the output may display a different value
than expected, e.g. 1.02 becomes 1.0199999809265137.
This CL splits Float type into Float32 and Float64 to keep track of
which bitsize to use when formatting. Values of both types are still
stored as float64 to keep the logic simple.
Decoding will always use Float64, but users can ask for a float32 value
from it.
Change-Id: Iea5b14b283fec2236a0c3946fac34d4d79b95274
Reviewed-on: https://go-review.googlesource.com/c/158497
Reviewed-by: Damien Neil <dneil@google.com>
In https://golang.org/cl/157821, I attempted to fix handling of group
field names but forgot about repeated group fields as I forgot to
properly update the marshaling tests as well for it.
Unmarshal logic for both repeated and non-repeated was already fixed in
that CL.
Change-Id: Icb4a00d8b169709ca12dfee272b2bd73e7585e6e
Reviewed-on: https://go-review.googlesource.com/c/157857
Reviewed-by: Damien Neil <dneil@google.com>
Also fix marshaling Any in expanded form to contain the correct type_url
value.
Change-Id: I4b467e74bb1d73255effd9cc4cfff9cf4558940f
Reviewed-on: https://go-review.googlesource.com/c/156342
Reviewed-by: Damien Neil <dneil@google.com>
Group field name in textproto should be the type name. Its field name
is derived from lowercasing its type name.
Change-Id: Ia12aafe934d3a59f3e07d09fe7939cfa6595a7b8
Reviewed-on: https://go-review.googlesource.com/c/157821
Reviewed-by: Damien Neil <dneil@google.com>
A "_ProtoFile" suffix can potentially conflict with a sub-message named
"ProtoFile" nested within a message that matches the camel-cased
form of the basename of the .proto source file.
Avoid unlikely conflicts and rename this to use a "_protoFile" suffix,
which can never conflict except with an enum value that is also named
"protoFile" (which is a violation of the style guide).
Change-Id: Ie9d22f9f741a63021b8f76906b20c6c2f599885b
Reviewed-on: https://go-review.googlesource.com/c/157218
Reviewed-by: Damien Neil <dneil@google.com>
Most usages of New actually prefer to interact with the reflective view
rather than the native Go type. Thus, change New to return that instead.
This parallels reflect.New, which returns the reflective view
(i.e., reflect.Value) instead of native type (i.e., interface{}).
We make the equivalent change to KnownFields.NewMessage, List.NewMessage,
and Map.NewMessage for consistency.
Since this is a subtle change where the type system will not always
catch the changed type, this change was made by both changing the type
and renaming the function to NewXXX and manually looking at every usage
of the the function to ensure that the usage correctly operates
on either the native Go type or the reflective view of the type.
After the entire codebase was cleaned up, a rename was performed to convert
NewXXX back to New.
Change-Id: I153fef627b4bf0a427e4039ce0aaec52e20c7950
Reviewed-on: https://go-review.googlesource.com/c/157077
Reviewed-by: Damien Neil <dneil@google.com>
Drop the protoreflect.ProtoEnum type (containing a single method
returning a protoreflect.Enum) and make generated enum types
directly implement protoreflect.Enum instead.
Messages have a two-level type split (ProtoMessage and Message) to
minimize conflicts between reflection methods and field names. Enums
need no such split, since enums do not have fields and therefore have
no source of conflicts.
Change-Id: I2b6222e9404253e6bfef2217859e1b760ffcd29b
Reviewed-on: https://go-review.googlesource.com/c/156902
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Reviewed-by: Damien Neil <dneil@google.com>
If there are any kind of errors in trying to expand the Any message,
always fallback to marshaling it as regular message. This makes it
consistent with V1 and C++ libs.
Change-Id: I007414c1767e682623c45d4dd8c82b9998f61781
Reviewed-on: https://go-review.googlesource.com/c/156257
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Marshal well-known type Any in expanded form by default, else fallback
to marshaling it as a regular message.
Change-Id: Ic7e9e37b47042a163941f8849dc366ffe48103ca
Reviewed-on: https://go-review.googlesource.com/c/156097
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Do not treat reserved field names as unknown. Skip over these instead
of returning error for unknown field.
Change-Id: Iba9371fd901a0690e5b7329bccf261570352b55d
Reviewed-on: https://go-review.googlesource.com/c/154178
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
This initial textproto unmarshaling implementation covers messages
without the use of extensions, Any expansion, nor weak.
Updated encoding tests. Split some testcases to keep each simpler.
Added TestRoundTrip for example messages like the well-known types.
Change-Id: Icffab02834aa004fa8409a9da70624f687f604fb
Reviewed-on: https://go-review.googlesource.com/c/153020
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The impl package currently supports wrapping legacy v1 enums and messages
so that they implement the v2 reflective APIs. This functionality is necessary
for v1 and v2 to interoperate. However, the existence of this functionality
presents several problems:
* A significant portion of the complexity in impl is for legacy wrapping.
* This complexity is linked into a Go binary even if all the other messages
in the binary natively support v2 reflection.
* It presents a cyclic dependency when trying to generate descriptor proto.
Suppose you are generating descriptor.proto. The generated code would want to
depend on the impl package because impl is the runtime implementation for
protobuf messages. However, impl currently depends depends on descriptor in
order to wrap legacy enum and messages since it needs the ability to dynamically
create new protobuf descriptor types. In the case of descriptor.proto, it would
presumably be generated with native reflection support, so the legacy wrapping
logic is unneccessary.
To break the dependency of impl on descriptor, we move the legacy support logic
to a different package and instead add hooks in impl so that legacy support could
be dynamically registered at runtime. This is dependency injection.
Change-Id: I01a582908ed5629993f6699e9bf2f4bee93857a4
Reviewed-on: https://go-review.googlesource.com/c/151877
Reviewed-by: Herbie Ong <herbie@google.com>
This initial implementation covers marshaling Message without use
of extensions, Any expansion, weak yet.
Change-Id: Ic787939c1d2a4e70e40c3a1654c6e7073052b7d3
Reviewed-on: https://go-review.googlesource.com/c/151677
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
