We define MessageState, which is essentially an atomically set *MessageInfo.
By nesting this as the first field in every generated message, we can
implement the reflective methods on a *MessageState when obtained by
unsafe casting a concrete message pointer as a *MessageState.
The MessageInfo held by MessageState provides additional Go type information
to interpret the memory that comes after the contents of the MessageState.
Since we are nesting a MessageState in every message,
the memory use of every message instance grows by 8B.
On average, the body of ProtoReflect grows from 133B to 202B (+50%).
However, this is offset by XXX_Methods, which is 108B and
will be removed in a future CL. Taking into account the eventual removal
of XXX_Methods, this is a net reduction of 25%.
name old time/op new time/op delta
Name/Value-4 70.3ns ± 2% 17.5ns ± 6% -75.08% (p=0.000 n=10+10)
Name/Nil-4 70.6ns ± 3% 33.4ns ± 2% -52.66% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
Name/Value-4 16.0B ± 0% 0.0B -100.00% (p=0.000 n=10+10)
Name/Nil-4 16.0B ± 0% 0.0B -100.00% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
Name/Value-4 1.00 ± 0% 0.00 -100.00% (p=0.000 n=10+10)
Name/Nil-4 1.00 ± 0% 0.00 -100.00% (p=0.000 n=10+10)
Change-Id: I92bd58dc681c57c92612fd5ba7fc066aea34e95a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185460
Reviewed-by: Damien Neil <dneil@google.com>
The encoding/prototext and encoding/protojson are implemented entirely
in terms of protobuf reflection, which side-steps this information.
Remove the hacks in the generator to special-case MessageSet.
Change-Id: I708c4636b77672545a103b7ab686f103b9dfc514
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185240
Reviewed-by: Herbie Ong <herbie@google.com>
We modify protoc-gen-go to stop generating exported XXX fields.
The unsafe implementation is unaffected by this change since unsafe
can access fields regardless of visibility. However, for the purego
implementation, we need to respect Go visibility rules as enforced
by the reflect package.
We work around this by generating a exporter function that given
a reference to the message and the field to export, returns a reference
to the unexported field value. This exporter function is protected by
a constant such that it is not linked into the final binary in non-purego
build environment.
Updates golang/protobuf#276
Change-Id: Idf5c1f158973fa1c61187ff41440acb21c5dac94
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185141
Reviewed-by: Damien Neil <dneil@google.com>
Associate the oneof wrapper types with a message by conveying that
information to the associated MessageInfo.
Change-Id: Iabfca593850e1d6a89498a37eacbf22dbb73bd20
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185239
Reviewed-by: Damien Neil <dneil@google.com>
This test verifies that the inialization logic does not accidentally
execute the lazy initialization logic that is intended to only evaluate
upon first use.
Change-Id: I5e9dea17ce88081c78195af0e86dd38223689f63
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/184259
Reviewed-by: Damien Neil <dneil@google.com>
If we were starting from scratch, we would not have added the enum maps.
However, they already exist and see a fair amount of usage.
The effort to remove them is not worth it. Thus, remove the deprecation
warning since they are here to stay.
Note that the generated code does not refer to the generated enum maps.
One day, the linker should be able to elide them if unused by the user.
However, https://golang.org/issue/2559 would need to be resolved first.
Change-Id: Ia8b9b1812b5d8462ca2fa1d543170e4a09ff9e4f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/183177
Reviewed-by: Damien Neil <dneil@google.com>
The internal/fileinit package is split apart into two packages:
* internal/filedesc constructs descriptors from the raw proto.
It is very similar to the previous internal/fileinit package.
* internal/filetype wraps descriptors with Go type information
Overview:
* The internal/fileinit package will be deleted in a future CL.
It is kept around since the v1 repo currently depends on it.
* The internal/prototype package is deleted. All former usages of it
are now using internal/filedesc instead. Most significantly,
the reflect/protodesc package was almost entirely re-written.
* The internal/impl package drops support for messages that do not
have a Descriptor method (pre-2016). This removes a significant amount
of technical debt.
filedesc.Builder to parse raw descriptors.
* The internal/encoding/defval package now handles enum values by name.
Change-Id: I3957bcc8588a70470fd6c7de1122216b80615ab7
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/182360
Reviewed-by: Damien Neil <dneil@google.com>
This is a breaking change in light of new methods added in CL/174918.
This removes:
Enum.Type: replacement is Descriptor
Message.Type: replacement is Descriptor and New
Change-Id: Iaa5328795407c8401ef14ed038bd5ace19d8e03b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/174938
Reviewed-by: Damien Neil <dneil@google.com>
This is a breaking change in light of API added in CL/182497.
This removes:
Files.RangeFilesByPath: replacement is Files.FindFileByPath
Change-Id: I47bf59b37c355844984661056212953853a0db51
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/182537
Reviewed-by: Damien Neil <dneil@google.com>
CL/180577 removes direct access to the Desc field in v1.
Since the only way to access the type is through {Has,Get,Set}Type,
we can now unexport the field and change the underlying type.
Change-Id: I6cc7018d4326c17228ba579d6161c5fb3f6d4127
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/180578
Reviewed-by: Damien Neil <dneil@google.com>
Generate the needed infrastructure to ensure that we can statically
enforce minimum and maximum versions. This enables us to have a policy
when we release v2 where it fails to build for:
* new generated code with really old runtimes
* new runtimes with really old generated code
Change-Id: Ib699ad62c06dff8f9285806394a741c18db00288
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/178546
Reviewed-by: Damien Neil <dneil@google.com>
The name MessageType is easily confused with protoreflect.MessageType.
Rename it as MessageInfo, which follows the pattern set by v1,
where the equivalent data structure is called InternalMessageInfo.
Change-Id: I535956e1f7c6e9b07e9585e889d5e93388d0d2ce
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/178478
Reviewed-by: Damien Neil <dneil@google.com>
A proto2 message can reference a proto3 enum. It is currently cumbersome
to set proto3 enums in proto2 messages. Add the Enum method in all situations
to support this situation. It also removes yet another point of divergence
between proto2 and proto3.
We could consider removing this method if Go ever gets generics,
but that hypothetical future is long ways away.
Change-Id: Ib83bc87e46b49f3271b90bacb2893bb8e278e4f2
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/177257
Reviewed-by: Damien Neil <dneil@google.com>
CL/177623 updates v1 to not depend on the previous import paths.
Change-Id: I46a61b8f8fa136c9c1fe367ff311732ba8841279
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/177604
Reviewed-by: Damien Neil <dneil@google.com>
Rename each generated protobuf package such that the base of the
Go package path is always equal to the Go package name to follow
proper Go package naming conventions.
The Go package name is derived from the .proto source file name by
replacing ".proto" with "pb" and stripping all underscores.
Change-Id: Iea05d1b5d94b1b2821ae10276ab771bb2df93c0e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/177380
Reviewed-by: Damien Neil <dneil@google.com>
CL/177620 modifies v1 to stop touching the Value field directly,
such that it is now unexport the Value field.
Change-Id: I0a05bbe59146862fc77c261349d7d90d6fa312e0
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/177621
Reviewed-by: Damien Neil <dneil@google.com>
Temporarily remove go.mod, since we can't generate an accurate one until
the corresponding v1 change is submitted.
Change-Id: I1e1ad97f2b455e33f61ffaeb8676289795e47e72
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/177000
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Added methods:
Enum.Descriptor
Message.Descriptor
EnumType.Descriptor
MessageType.Descriptor
ExtensionType.Descriptor
Message.New
All functionality is switched over to use those methods instead of
implicitly relying on the fact that {Enum,Message}Type implicitly
implement the associated descriptor interface.
This CL does not yet remove {Enum,Message}.Type or prevent
{Enum,Message,Extension}Type from implementating a descriptor.
That is a subsequent CL.
The Message.New method is also added to replace functionality
that will be lost when the Type methods are removed.
Change-Id: I7fefde1673bbd40bfdac489aca05cec9a6c98eb1
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/174918
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Herbie Ong <herbie@google.com>
CL/175579 switches v2 to use XXX_unrecognized exclusively.
CL/175838 switches v1 to use XXX_unrecognized exclusively.
This means that it is now safe to delete the Raw field.
Change-Id: Ic60c58147c2df4078e72c34d1202d226a7ea4bed
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/175839
Reviewed-by: Damien Neil <dneil@google.com>
This covers most of the TODO around validation. I left open the ones
that we didn't have clear consensus on yet.
Change-Id: I336c53173ee8d7447558b1e3a0c1ef945e986cd5
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/175140
Reviewed-by: Joe Tsai <joetsai@google.com>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
CL/172238 added equivalent methods without the Type suffix,
while keeping the old methods.
CL/172582 updates the v1 codebase to use the new methods.
This CL removes the methods with the Type suffix.
Change-Id: Iaaaa4fff11cac1694735657db2e5fd7cadc90afe
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/173138
Reviewed-by: Herbie Ong <herbie@google.com>
Using a named fields gives us the flexibility to change the underlying
representation of special fields without needing to regenerate user code.
We add a named type for ExtensionFields, UnknownFields, and SizeCache.
Change-Id: I107cf82899850ea76665310ce79def60f0f7ab97
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/172402
Reviewed-by: Damien Neil <dneil@google.com>
Lower-casing the FileDescriptor variable provides for shorter and
more readable global variable names. This has a very minor benefit
that the read-only data section of a binary is slightly smaller
since function names do end up in the final binary.
Change-Id: I077364905e5c9adea69873b3ea580fddc68b9eb8
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/172119
Reviewed-by: Herbie Ong <herbie@google.com>
Go1.12 on GoLLVM with LLVM 9.0.0 has a bug where the simpler but
equivalent expression results in memory corruption.
Revert to the more complex expression to avoid breaking GoLLVM and
it is not worth waiting for a GoLLVM fix.
Change-Id: I8b42965ca9bd333deb8693021e7b22f966e13521
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171463
Reviewed-by: Damien Neil <dneil@google.com>
Seperate out the generation of the oneof wrapper types from the
code block that is about getter methods.
Change-Id: Ief44ef953d0b5ad8c998a8542c830ca70468a3bf
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171029
Reviewed-by: Damien Neil <dneil@google.com>
Allow message implementations to provide optimized versions of standard
operations. Generated messages now include a ProtoReflectMethods method,
returning a protoiface.Methods struct containing pointers to assorted
optional functions.
The Methods struct also includes a Flags field indicating support for
optional features such as deterministic marshaling.
Implementation of the fast paths (and tests) will come in later CLs.
Change-Id: Idd1beed0ecf43ec5e5e7b8da2ee1e08d3ce32213
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170340
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Minor changes:
* Use x as the receiver since "e" and "m" are meaningless in the presence
of user-defined enum and message names.
* Consistently keep enum methods together, rather awkwardly split apart
by the value maps.
Change-Id: I68e5666efb56ac7a4d062fb223b9f826dc72aba9
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170357
Reviewed-by: Herbie Ong <herbie@google.com>
This reduces the init-time cost slightly since the GZIP'd
raw descriptor is constructed lazily on demand.
Change-Id: I482c6a2201b8786e425d7dee5612fdfd60ab1500
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169917
Reviewed-by: Herbie Ong <herbie@google.com>
This removes yet another set of dependencies of v2 on v1.
The only remaining dependency are in the _test.go files,
primarily for proto.Equal.
Changes made:
* cmd/protoc-gen-go no longer generates any functionality that depends
on the v1 package, and instead only depends on v2.
* internal/fileinit.FileBuilder.MessageOutputTypes is switched from
protoreflect.MessageType to protoimpl.MessageType since the
implementation must be fully inialized before registration occurs.
* The test for internal/legacy/file_test.go is switched to a legacy_test
package to avoid a cyclic dependency.
This requires Load{Enum,Message,File}Desc to be exported.
Change-Id: I43e2fe64cff4eea204258ce11e791aca5eb6e569
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169298
Reviewed-by: Damien Neil <dneil@google.com>
As a goal, v2 should not depend on v1. As another step towards that end,
we move all the types that used to be in the v1 protoapi package over to v2.
For now, we place MessageV1, ExtensionRangeV1, and ExtensionDescV1
in runtime/protoiface since these are types that generated messages will
probably have to reference forever. An alternative location could be
reflect/protoreflect, but it seems unfortunate to have to dirty the
namespace of that package with these types.
We move ExtensionFieldV1, ExtensionFieldsV1, and ExtensionFieldsOf
to internal/impl, since these are related to the implementation of a
generated message.
Since moving these types from v1 to v2 implies that the v1 protoapi
package is useless, we update all usages of v1 protoapi in the v2
repository to point to the relevant v2 type or functionality.
CL/168538 is the corresponding change to alter v1.
There will be a temporary build failure as it is not possible
to submit CL/168519 and CL/168538 atomically.
Change-Id: Ide4025c1b6af5b7f0696f4b65b988b4d10a50f0b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168519
Reviewed-by: Herbie Ong <herbie@google.com>
The v1 registration leaks the message types out to the proto package.
When doing that, it must ensure that the reflection data structures
for those types are properly initialized first. We achieve that by
doing v1 registration at the end of the reflection init function.
Change-Id: If6df18df59d05bad50ff39c2eff6beb19e7466cc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168348
Reviewed-by: Damien Neil <dneil@google.com>
This CL breaks another dependency of v2 on v1.
A missing feature in v2 is proto.Clone, but we can use a
Marshal/Unmarshal roundtrip to achieve the same effect
as a temporary stop-gap.
Change-Id: I9d0432fd3efe9fc8b34db6c5a1eabbef0c36277c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168217
Reviewed-by: Damien Neil <dneil@google.com>
The EnumName, UnmarshalJSONEnum, and CompressGZIP helpers currently live
in v1 protoapi, which would cause all generated messages to depend on v1.
In an effort to break the dependency of v2 on v1, we move these helper
functions to v2 (and re-written to take advantage of protobuf reflection).
These helpers are unfortunate, but we cannot eliminate the functionality
that they implement since they are exposed in the publicly generated API.
Since EnumName does not rely on the enum maps, it removes another dependency
on those variables. Eventually, we can get to the point where these variables
(though declared) are not linked into the binary if the user does not use them.
Also, we rely on the v1 proto package for registration instead of v1 protoapi.
This may re-introduce a cyclic dependency on descriptor proto again in the
future, but the better approach is to just start registering with v2.
Change-Id: Id755585a7a1df14e4a6a2dfa650df221a3c153fb
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/167921
Reviewed-by: Damien Neil <dneil@google.com>
Unfortunately a good amount of code uses pointer comparisons on the
v1 ExtensionDesc to determine exactly which extension field is set,
rather than checking whether the extension descriptor semantically
describes the field that they are interested in.
To preserve this behavior in v1, we need a 1:1 mapping between
a v2 ExtensionType and a specific v1 ExtensionDesc.
Change-Id: I852b3cefb4585bd656e48e5adad6cc28795d02df
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/167759
Reviewed-by: Damien Neil <dneil@google.com>
The go.sum is not particularly important for the testdata directory,
so ignore it.
Change-Id: I069613deda2af4944d09aed34008fec201df6bb4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/166879
Reviewed-by: Herbie Ong <herbie@google.com>
Running "go build ./..." does not descend into testdata directories.
However, the testdata in this repository is source code that is
intended to build properly. We could rename the directory, but that does
not test whether the generated packages can initialize properly.
Thus, we generate a trivial test that simply blank imports all packages.
Doing this reveals that some of the generated files have incorrect imports,
leading to registration conflicts.
To avoid introducing a dependency on gRPC from our go.mod file, we put
the testdata directories in their own module. Also, we avoid running
internal/testprotos through the grpc plugin because the servie and method
definitions in that directory are more for testing proto file initialization
rather than testing grpc generation.
Change-Id: Iaa6a06449787a085200e31bc7606e3ac904d3180
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/164917
Reviewed-by: Damien Neil <dneil@google.com>
Ensure that the init funcs for files within a Go package run in the
dependency order of the source .proto files. That is, if a.proto and b.proto
are part of the same Go package, and a.proto imports b.proto, then b.pb.go's
init funcs must run before a.pb.go's.
Change-Id: I0e86ff22e5c4cab9df7a73fe4805390fadd34b0d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/166419
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Reviewed-by: Herbie Ong <herbie@google.com>
Using arrays in the generated reflection information adds unnecessary
eq and hash functions being added to the package. Change to slices
to reduce bloat.
Change-Id: I1a4f6d59021644d93dd6c24679b9233141e89a75
Reviewed-on: https://go-review.googlesource.com/c/164640
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Create a single binary for handling generation of protos.
This replaces previous logic spread throughout the repo in:
* regenerate.bash
* cmd/protoc-gen-go/golden_test.go
* cmd/protoc-gen-go-grpc/golden_test.go
* (indirectly) internal/protogen/goldentest
One of the problems with the former approaches is that they relied on
a version of protoc that was specific to a developer's workstation.
This meant that the result of generation was not hermetic.
To address this, we rely on the hard-coded version of protobuf specified
in the test.bash script.
A summary of changes in this CL are:
* The internal_gengo.GenerateFile and internal_gengogrpc.GenerateFile
functions are unified to have consistent signatures. It seems that the
former accepted a *protogen.GeneratedFile to support v1 where gRPC code
was generated into the same file as the base .pb.go file. However, the
same functionality can be achieved by having the function return
the generated file object.
* The test.bash script patches the protobuf toolchain to have properly
specified go_package options in each proto source file.
* The test.bash script accepts a "-regenerate" argument.
* Add generation for the well-known types. Contrary to how these were
laid out in the v1 repo, all the well-known types are placed in the
same Go package.
* Add generation for the conformance proto.
* Remove regenerate.bash
* Remove internal/protogen
* Remove cmd/protoc-gen-go/golden_test.go
* Remove cmd/protoc-gen-go-grpc/golden_test.go
* Add cmd/protoc-gen-go/annotation_test.go
Change-Id: I4a1a97ae6f66e2fabcf4e4d292c95ab2a2db0248
Reviewed-on: https://go-review.googlesource.com/c/164477
Reviewed-by: Damien Neil <dneil@google.com>
Use the full path (including the extension) for the generation of
the per-file variable name. Several reasons for this:
* The current logic is buggy in the case where pathType == pathTypeImport
since the prefix variable will be mangled with the Go import path.
* The extension is technically part of the path.
Thus, "path/to/foo.proto" and "path/to/foo.protodevel" are two
distinctly different imports.
* Style-wise, it subjectively looks better. Rather than being a mixture
of camelCase and snake_case, it is all snake_case for the common case:
before: ProtoFile_google_protobuf_any
after: File_google_protobuf_any_proto
* Since the extension is almost always ".proto", this results in a
suffix of "_proto", which provides an additional layer of protection
against possible name conflicts. The previous approach could possibly
have a conflict between "Foo.proto" and a message named ProtoFile
with a sub-message called Foo.
Also, use the per-file variable name for the raw descriptor variables
instead of the hashed version.
Change-Id: Ic91e326b7593e5985cee6ececc60539c27fe32fe
Reviewed-on: https://go-review.googlesource.com/c/164379
Reviewed-by: Damien Neil <dneil@google.com>
Calling a helper function directly should reduce binary bloat slightly.
Change-Id: I6068dc4cd00c8d90d2e6e6d99633b81388bc8781
Reviewed-on: https://go-review.googlesource.com/c/164679
Reviewed-by: Damien Neil <dneil@google.com>
For a file "foo/bar.proto", put the FileDescriptor in "ProtoFile_foo_bar"
rather than "Bar_fileDescriptor".
Avoid name clashes when a package contains "a/foo.proto" and "b/foo.proto".
Don't camelcase the filename: These vars weren't fully camelcased to begin
with, and leaving the filename relatively unchanged is clearer and more
predictable.
Move "ProtoFile" from the end of the var name to the start, so that vars
will sort better in packages with multiple descriptors.
These changes do add a chance of name collision when the input filename
begins with an uppercase letter: Foo.proto becomes "ProtoFile_Foo", which
could be the result of camelcasing "proto_file.foo". The readability
benefits seem worth it.
Change-Id: If27d3a0d7b5bf3535aa1607a8579eb057c74d2dc
Reviewed-on: https://go-review.googlesource.com/c/163199
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Reviewed-by: Herbie Ong <herbie@google.com>
