67 Commits

Author SHA1 Message Date
Lasse Folger
f9eb6c71e7 go_features.proto: move .proto file next to generated .pb.go file
Change-Id: Ia24ca1ad2186bcc95681081d2719ee834b385728
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/565955
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
2024-02-22 08:56:15 +00:00
Lasse Folger
997075a7f5 reflect/protodesc: add editions support
Change-Id: Ie7fee936bdb480802cbaca34cf63c0dad34590b8
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/564815
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
2024-02-19 07:03:53 +00:00
Lasse Folger
e39100579e reflect/protodesc: propagate field_presence legacy_required properly
Change-Id: I3f067eb564349bc1ff867a03d9f1aa94b3428236
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/563735
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
2024-02-13 15:44:19 +00:00
Lasse Folger
020ec10f7b internal/filedesc: implement runtime editions support
This change add parsing editions specific descriptor parts to
internal/filedesc which makes it possible to load file descriptors of
protos using editions at runtime.

Change-Id: I7a0c23303acddd2cff115859d4cf82bf0102b14c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/563615
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-02-13 12:20:49 +00:00
Lasse Folger
f387b036d8 editions: add some test protos for editions
I'll add the runtime implementations and tests in a followup change to
make it easier to review.

Change-Id: I2917e0d40a99e81799d89a66584a680ed7e8dbf7
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/561939
Auto-Submit: Lasse Folger <lassefolger@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-02-12 14:14:45 +00:00
Lasse Folger
2040e8671b all: implement Go-specific GenerateLegacyUnmarshalJSON
This change adds the first Go specific editions feature and the
associated proto. The feature is used to control if the legacy
UnmarshalJSON method should be generated for enums. This change only
affects protos using editions.

More tests will be added in a followup change.

Change-Id: Ifb62454d7568bd6d90d0b93f8953adcfe46c45fd
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/561938
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
2024-02-12 13:54:51 +00:00
Lasse Folger
99e193e5dd all: update to protobuf-26.0-rc2 to implement go edition features
Change-Id: I190b1550e3e2e630392fed155716af9d651c74bc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/562615
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
2024-02-08 09:23:48 +00:00
Lasse Folger
b70f02be53 editions: minor fix to resolve editions features correctly
+ The previous commit forgot to resolve the LegacyRequired feature

Change-Id: I05a2847be3d52277c65df004363ea3c87944364e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/555976
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-01-16 15:43:50 +00:00
Lasse Folger
9e454d6514 internal/filedesc: refactor editions feature resolution
Before this CL we only used features where they were applicable, e.g.
file features were only set on the file descriptor and field features
only on the fields descriptor. After this CL all descriptors (file,
message and field) contain always all features and when initializing
either of them we merge the features specified by options with the
features of the parent descriptor and use the result as the effective
feature. This way we initialize all the descriptors the same way and the
proto spec can change on which level features can be specified without
us having to change the code (at the moment most features can only be
specified on file or field level, in the future message might also be
possible).

Change-Id: I822202bf6a2bba5167bf9622c91a65c4523e78f9
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/555975
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
2024-01-16 12:12:51 +00:00
Michael Stapelberg
26a52f39da reflect/protodesc: fix packed field conditional
Before this change, calling dynamicpb.NewMessage() on a
protoreflect.MessageDescriptor that uses Editions would fail with:

message field "[…].relatedqueries" is not packable

(While Go Protobuf does not yet support Editions, C++ programs
can save descriptors to a file that is later loaded by a Go program.)

This was an oversight in commit e8baad6b6c9e2bb1c48e4963866248d4f35d4fd7.

Change-Id: I45d2b07a85ee40cb7c028098b81c4e76bf2ff555
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/554896
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Cassondra Foesch <cfoesch@gmail.com>
Reviewed-by: Christian Höppner <hoeppi@google.com>
Auto-Submit: Michael Stapelberg <stapelberg@google.com>
2024-01-09 15:05:22 +00:00
Michael Stapelberg
7c85df2652 reflect/protodesc: fix group validation: not supported in proto3
Before this change, calling dynamicpb.NewMessage() on a
protoreflect.MessageDescriptor that uses Editions would fail with:

message field "[…].relatedqueries" is an invalid group: proto: invalid under
proto2 semantics

(While Go Protobuf does not yet support Editions, C++ programs
can save descriptors to a file that is later loaded by a Go program.)

Given that groups are only supported in proto2, but not in proto3
(see https://protobuf.dev/editions/features/#message_encoding),
I think the conditional meant to say "proto3 semantics".

Change-Id: I00368e4433ca60bb883b726e34ee6b8862cfd1e7
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/554955
Reviewed-by: Cassondra Foesch <cfoesch@gmail.com>
Reviewed-by: Christian Höppner <hoeppi@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-01-09 14:54:38 +00:00
Lasse Folger
e8baad6b6c reflect/protodesc: support editions feature resolution
This is the first of multiple changes to add protobuf editions support
to the Go implementation.
This change includes the feature resolutions for protobuf editions.
The plugin does not yet report editions support and protoc would fail
if it invoked this plugin with a proto that uses editions.

Change-Id: I7d31909366c3433b21abab74ec92263e08145434
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/547315
Auto-Submit: Lasse Folger <lassefolger@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Commit-Queue: Lasse Folger <lassefolger@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
2023-12-08 13:18:14 +00:00
Josh Humphries
a8317fb7c5 reflect/protodesc: don't generate invalid syntax values
Fixes golang/protobuf#1575

When converting a protoreflect.FileDescriptor
to a FileDescriptorProto, don't emit an invalid syntax.

This can happen for placeholder files, which represent
missing dependencies of a file.

Change-Id: I1d2ae1bd9f89c8a92e751871ce5c02727505b5e2
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/540455
Reviewed-by: Lasse Folger <lassefolger@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
2023-11-08 07:09:48 +00:00
Joe Tsai
f9212a8dfa all: modernize documentation
Modernize the documentation across the entire module
to make use of the newer ability to linkify declarations.

Change-Id: I440f9a3f025ec6fadfd9cba59b1dfb57028bf3f1
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/309430
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
2023-09-05 14:55:28 +00:00
Damien Neil
b0a944684d all: reformat with go1.19 gofmt
Apply go1.19 gofmt to non-generated files.

Generated .pb.go files are created by generate.bash using Go 1.18,
so leave them unchanged for now.

Change-Id: Ied36c83cf99704988d059bf0412e677f0fbc71b0
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/418676
Reviewed-by: Ian Lance Taylor <iant@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
2022-07-21 16:01:13 +00:00
Joe Tsai
687df2333b reflect/protodesc: fix round-tripping for package field
If a .proto file does not have a package statement,
then protoc will not populate the package field.
Emulate the same behavior for ToFileDescriptorProto.

Fixes golang/protobuf#1195

Change-Id: I617102e50c96f0fa1bac6a6bc4de1cc485750bb9
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/259899
Reviewed-by: Damien Neil <dneil@google.com>
Trust: Joe Tsai <thebrokentoaster@gmail.com>
2020-10-07 22:44:53 +00:00
Joe Tsai
42cc4c592f reflect/protoreflect: improve source information usability
Added API:
  SourceLocations.ByPath
  SourceLocations.ByDescriptor
  SourceLocation.Next
  SourcePath.String
  SourcePath.Equal

We modify compiler/protogen to use SourceLocations.ByDescriptor.
In retrospect, if this had existed during the development of protogen,
we would not have exposed protogen.Location and related fields.

Change-Id: I58f17e59f90b9ba16f0982c4b71c2542e4ff6e75
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/238000
Reviewed-by: Damien Neil <dneil@google.com>
2020-07-23 21:50:46 +00:00
Joe Tsai
e14d6b3cdc reflect/protoreflect: add FieldDescriptor.TextName
Add a new TextName accessor that returns the field name that should
be used for the text format. It is usually just the field name, except:
1) it uses the inlined message name for groups,
2) uses the full name surrounded by brackets for extensions, and
3) strips the "message_set_extension" for well-formed extensions
to the proto1 MessageSet.

We make similar adjustments to the JSONName accessor so that it applies
similar semantics for extensions.

The two changes simplifies all logic that wants the humanly readable
name for a field.

Change-Id: I524b6e017fb955146db81819270fe197f8f97980
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/239838
Reviewed-by: Herbie Ong <herbie@google.com>
2020-07-08 23:23:57 +00:00
Joe Tsai
e0b77db13b internal/genid: add new package for generated identifiers
The genid package unifies the genname, fieldnum, and detectknown
packages into a single package.

Whenever possible use the generated constants rather than
hard-coded literals. This makes it easier to search the entire
module for special logic that deal with well-known types.

Change-Id: I13beff1f4149444a0c0b9e607ebf759657f000f4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/235301
Reviewed-by: Herbie Ong <herbie@google.com>
2020-05-29 07:08:23 +00:00
Joe Tsai
387873dd53 all: implement support for proto3 optional semantics
In the upcoming 3.12.x release of protoc, the proto3 language will be
amended to support true presence for scalars. This CL adds support
to both the generator and runtime to support these semantics.

Newly added public API:
	protogen.Plugin.SupportedFeatures
	protoreflect.FieldDescriptor.HasPresence
	protoreflect.FieldDescriptor.HasOptionalKeyword
	protoreflect.OneofDescriptor.IsSynthetic

Change-Id: I7c86bf66d0ae56642109beb5f2132184593747ad
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/230698
Reviewed-by: Damien Neil <dneil@google.com>
2020-04-29 20:02:24 +00:00
Damien Neil
188e702e78 reflect/protodesc: add NewFiles
Add a function that takes a protoreflect.FileDescriptorSet and returns
a protoregistry.Files.

Updates golang/protobuf##1065.

Change-Id: I2715d042053ef7d3f1bfcee1866c20cac423e327
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/226237
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2020-03-31 19:56:16 +00:00
Joe Tsai
cd108d00a8 encoding/protowire: make package publicly available
Change-Id: I95e293c208e787a91d50e29817620535dfeaa7f2
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219838
Reviewed-by: Damien Neil <dneil@google.com>
2020-03-20 07:42:18 +00:00
Joe Tsai
6af2cee096 reflect/protodesc: improve documentation
Change-Id: Id1c0e0e702a7b0eb011c71de23cd5be76345d741
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/220341
Reviewed-by: Damien Neil <dneil@google.com>
2020-02-20 19:04:51 +00:00
Joe Tsai
25fc6fb4f2 reflect/protodesc: add FileOptions
The FileOptions type provides the ability to specify specialized options
for how a file descriptor is constructed. It follows the same optional
arguments pattern as used in the proto package.

The resolver is not an option since it almost always necessary
when constructing a file descriptor.

Change-Id: Ib98ac6289881ad8402dd615f6c895da5899cb8d9
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/218940
Reviewed-by: Damien Neil <dneil@google.com>
2020-02-11 23:14:57 +00:00
Joe Tsai
93bccf763e all: scrub all TODOs
TODOs that we do not intend to address have been deleted.
Those that are blocking v2 release are marked with "blocks".

Change-Id: I7efa9e546d0637b562101d0edc7009893d762722
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/218878
Reviewed-by: Damien Neil <dneil@google.com>
2020-02-10 19:28:48 +00:00
Damien Neil
01b51b4f96 proto, internal/errors: add Error sentinel, errors.Wrap
Add a sentinel proto.Error error which matches all errors returned by
packages in this module.

Document that protoregistry.NotFound is an exact sentinel value for
performance reasons.

Add a Wrap function to the internal/errors package and use it to wrap
errors from outside sources (resolvers). Wrapped errors match
proto.Error.

Fixes golang/protobuf#1021.

Change-Id: I45567df3fd6c8dc9a5caafdb55654827f6fb1941
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/215338
Reviewed-by: Joe Tsai <joetsai@google.com>
2020-02-07 21:09:48 +00:00
Joe Tsai
b7695fab0d proto: add Clone function and MergeOptions.Clone method
We resisted adding Clone for a while since:
* It is a function that is perfectly suited for generics.
However, generics probably still won't be available in Go for some time
and it is impractical to block addition of this function when it is very
widely used and will be necessary for the v1 to v2 migration.
* In the past, there was no protoreflect.Message.IsValid, so there was
no proper API to detect invalid top-level messages and return them as such.

Since Clone relies on certain properties about proper round-tripping
of ProtoMessage.ProtoReflect <-> Message.Interface, we add a test
in testing/prototest to check for this.

Change-Id: Ic492b68f27b8b88322a6a3fa3a5e492228db79d9
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/213297
Reviewed-by: Damien Neil <dneil@google.com>
2020-01-06 21:07:28 +00:00
Joe Tsai
6c26a04a51 internal/filedesc: use jsonName.Init method over JSONName constructor
The JSONName constructor returns a struct value which shallow copies
a sync.Once within it; this is a dubious pattern.
Instead, add a jsonName.Init method to initialize the value.

Change-Id: I190a7239b1b62a8041ee7e4e09c0fe37b64ff623
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/213237
Reviewed-by: Damien Neil <dneil@google.com>
2020-01-06 18:42:14 +00:00
Damien Neil
e9187326c3 internal/filedesc: move message options to L1 initialization
Avoid a deadlock when registering a legacy ExtensionType, caused by
initialization of the "internal/impl".ExtensionInfo calling IsMessageSet
on the MessageDescriptor of the type being extended.

We can avoid this deadlock either by initializing the ExtensionType
outside of the GlobalTypes mutex, or by moving IsMessageSet to L1
initialization of the MessageDescriptor so that it doesn't trigger lazy
init.

CL 204804 takes the former approach; this CL takes the latter.

Change-Id: Idfc1ed36a23a139839290ea32492142a17f68cf5
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/205957
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-11-08 00:29:25 +00:00
Damien Neil
c826885a2a reflect/protoregistry: add (*Types).Register{Message,Enum,Extension}
Add type-safe methods to register message, enum, and extension types.
Deprecate the NewTypes function and the (*Types).Register method.

Add (*File).RegisterFile and deprecate the NewFiles function and
the (*File).Register method.

Updates golang/protobuf#963

Change-Id: Ie89e77526e0874539e9bd929ca0ba8d758e65a6e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/199898
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-10-31 16:23:42 +00:00
Joe Tsai
84177c9bf3 all: use typed variant of protoreflect.ValueOf
Change-Id: I7479632b57e7c8efade12a2eb2b855e9c321adb1
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/196037
Reviewed-by: Damien Neil <dneil@google.com>
2019-09-17 21:33:16 +00:00
Joe Tsai
fd4c605bfb internal: simplify ExtensionInfo initialization
This CL:
* Make the meaning of impl/ExtensionInfo.goType consistent. Before,
it was sometimes a T and other times a []T depending on the current
state of initialization. Change it so that it is the constructor's
responsibility to pass in a []T if it is repeated.
* Make internal/filetype responsible for constructing a []T for
repeated extension fields.
* Makes filedesc/Extension.Cardinality one of the eagerly initialized
pieces of information since it is useful to internal/filetype.
* Unify ExtensionInfo.desc and ExtensionInfo.tdesc.ExtensionField,
which held the same information.
* Remove the internal implementation for impl.X.ExtensionDescFromType
since we are dropping support for this from v1.

Change-Id: Ie95c4de66cd674c1d886da4f63b133b7d763c7ef
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/195777
Reviewed-by: Damien Neil <dneil@google.com>
2019-09-16 22:06:06 +00:00
Joe Tsai
09217f08d2 all: make error messages unstable
Use internal/detrand in the construction of our error messages.
This alters whether there is one or two spaces following the "proto:" prefix.
While it is easy for users to still work around this mutation,
sit at least forces them to write test infrastructure to more fuzzily
match on error strings.

Change-Id: I4ddca717526ee3fc4dbb1e0b36cfca8c6e0df36d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/194038
Reviewed-by: Herbie Ong <herbie@google.com>
2019-09-07 00:39:30 +00:00
Joe Tsai
1799d1111a all: rename tag and flags for legacy support
Rename build tag "proto1_legacy" -> "protolegacy"
to be consistent with the "protoreflect" tag.

Rename flag constant "Proto1Legacy" -> "ProtoLegacy" since
it covers more than simply proto1 legacy features.
For example, it covers alpha-features of proto3 that
were eventually removed from the final proto3 release.

Change-Id: I0f4fcbadd4b5a61c87645e2e5be11d187e59157c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189345
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-08 20:49:00 +00:00
Joe Tsai
e815d6a43b all: remove dead code
Change-Id: I1344d6afca9d3348db849c2b5f387ac18b80d2ba
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189021
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-06 21:16:48 +00:00
Joe Tsai
3d8e369c4e all: implement proto1 weak fields
This implements generation of and reflection support for weak fields.
Weak fields are a proto1 feature where the "weak" option can be specified
on a singular message field. A weak reference results in generated code
that does not directly link in the dependency containing the weak message.

Weak field support is not added to any of the serialization logic.

Change-Id: I08ccfa72bc80b2ffb6af527a1677a0a81dcf33fb
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185399
Reviewed-by: Damien Neil <dneil@google.com>
2019-07-15 18:44:12 +00:00
Joe Tsai
67da21c7ab reflect/protodesc: only check enum dependency syntax if resolved
For proto3 messages with an enum field that could not be resolved,
do not check the syntax of that enum dependency.

Change-Id: I7c646539351edc35243ab950d335f4018cc4c0e9
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186001
Reviewed-by: Damien Neil <dneil@google.com>
2019-07-15 18:16:04 +00:00
Joe Tsai
e182c917f0 reflect/protoreflect: add FileDescriptor.SourceLocations
This adds minimal support for preserving the source context information.

Change-Id: I4b3cac9690b7469ecb4e5434251a809be4d7894c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/183157
Reviewed-by: Damien Neil <dneil@google.com>
2019-07-13 00:15:59 +00:00
Damien Neil
a8a2cea3e7 proto: move T->*T wrappers from internal/scalar to proto
Usage of these is pervasive in code which works with proto2, and proto2
will be with us for a long, long time to come. Move them to the proto
package.

Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-12 17:35:01 +00:00
Joe Tsai
97a87391b1 internal/strs: unify string manipulation functionality
Create a new internal/strs package that unifies common functionality:
* Since protobuf itself pseudo-specifies at least 4 different camel-case
and snake-case conversion functions, we define all variants in one place.
* We move the internal/filedesc.nameBuilder function to this package.
We simplify its implementation to not depend on a strings.Builder fork
under the hood since the semantics we desire is simpler than what
strings.Builder provides.
* We use strs.Builder in reflect/protodesc in its construction of all
the full names. This is perfect use case of strs.Builder since all
full names within a file descriptor share the same lifetime.
* Add an UnsafeString and UnsafeBytes cast function that will be useful
in the near future for optimizing encoding/prototext and encoding/protojson.

Change-Id: I2cf07cbaf6f72e5f9fd6ae3d37b0d46f6af2ad59
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185198
Reviewed-by: Damien Neil <dneil@google.com>
2019-07-09 19:57:23 +00:00
Joe Tsai
15076350e8 reflect/protodesc: enforce strict validation
Hyrum's Law dictates that if we do not prevent naughty behavior,
people will rely on it. If we do not validate that the provided
file descriptor is correct today, it will be near impossible
to add proper validation checks later on.

The logic added validates that the provided file descriptor is
correct according to the same semantics as protoc,
which was reversed engineered to derive the set of rules implemented here.
The rules are unfortunately complicated because protobuf is a language
full of many non-orthogonal features. While our logic is complicated,
it is still 1/7th the size of the equivalent C++ code!

Change-Id: I6acc5dc3bd2e4c6bea6cd9e81214f8104402602a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/184837
Reviewed-by: Damien Neil <dneil@google.com>
2019-07-03 20:46:51 +00:00
Joe Tsai
b2107fbd8d reflect/protodesc: robustify dependency resolution implementation
Overview of changes:
* Add an option that specifies whether to replace unresolvable references
with a placeholder instead of producing an error. Since the prior behavior
produced placeholders (not always), we default to that behavior for now,
but will enable strict resolving in a future CL.
* The option is not yet exported because there is concern about what the
public API should look like. This will be exposed in a future CL.
* Unlike before, we now permit placeholders for unresolvable enum values.
* We implement relative name resolution logic.
* We handle the case where the type is unknown, but type_name is specified.
In such a case, we populate both FieldDescriptor.{Enum,Message} and leave
the FieldDescriptor.Kind with the zero value. If the type_name happened
to resolve, we use that to determine the type.
* If a placeholder is used to represent a relative name,
the FullName reports an invalid full name with a "*." prefix.

Change-Id: Ifa8c750423c488fb9324eec4d033a2f251505fda
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/184317
Reviewed-by: Damien Neil <dneil@google.com>
2019-07-03 19:17:36 +00:00
Joe Tsai
a691404b5d reflect/protodesc: return deep copies of the input
There is little performance benefit to aliasing the input since we copy
every field except the options. Thus, just go all the way and copy the
options as well and document this as such.

Change-Id: If6ca5ce0ee03c9f76e528023b6056ad99d3ca209
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/184879
Reviewed-by: Damien Neil <dneil@google.com>
2019-07-03 16:36:09 +00:00
Joe Tsai
3274acc926 reflect/protodesc: fix initialization bug
This fixes a bug introduced by CL/182360.

Overview of the problem:
* CL/182360 removes the internal/prototype package, such that
protodesc was re-implemented using internal/filedesc.
* As a result of that change, resolving internal dependencies became
the responsibility of protodesc.
* Dependency resolution used the following two-pass algorithm:
	1) first pass derives the full name of all declarations
	2) second pass fully initializes each descriptor declaration,
	now being able to resolve local dependencies from the previous step.
* When the second pass looks up a local dependency, it is guaranteed to
find it, but it is not guaranteed that the dependency has been initialized
(since it may appear later on). This is problematic for default enum values
since it implies that the enum dependency may not be sufficiently
initialized to be able to query its set of values, leading to panics.
* CL/182360 recognized the problem and attempted to enforce an initialization
ordering where nested enums were always initialized before the body of the
message declaration itself.
* However, that ordering fails to enforce that that enum declarations outside
the parent tree are initialized beforehand. For example, referring to an
enum value that is declared within a sibling of the parent message.
* This CL fixes the problem with a three-pass algorithm:
	1) first pass derives the full name *and* fully initialize the
	entire descriptor *except* for dependency references (i.e., type_name).
	2) second pass only resolves dependency references,
	where we do not need to worry about initialization ordering.
	3) third pass validates the descriptors are well-formed.
	This can now depend on all information being fully initialized.
* While a lot of code moves, this change is actually very mechanical.
Other than split things apart, no new logic is introduced nor removed.

Change-Id: Ia91d4aade8f6187c19d704d43ae96b3b9d276792
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/184297
Reviewed-by: Damien Neil <dneil@google.com>
2019-07-01 22:32:12 +00:00
Joe Tsai
a7f7a409cb reflect/protodesc: fix handling of unresolved identifiers
Change-Id: I09a6384de5591d77d99e5646ccc9137f42424845
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/183779
Reviewed-by: Herbie Ong <herbie@google.com>
2019-06-25 19:57:31 +00:00
Joe Tsai
e407ee162b reflect/protoregistry: remove Files.Find{Enum,Message,Extension,Service}ByName
This is a breaking change.

The replacement is the Files.FindDescriptorByName method,
which is more flexible as it handles all descriptor types.

Change-Id: I2ccd544a7630396a2428b1d41f836c5246070912
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/183700
Reviewed-by: Damien Neil <dneil@google.com>
2019-06-24 23:47:12 +00:00
Joe Tsai
424139789a reflect/protoreflect: register all desciptors in Files
This change makes it such that Files now functionally registers all
descriptors in a file (not just enums, messages, extensions, and services),
but also including enum values, messages fields/oneofs, and service methods.

The ability to look up any descriptor by full name is needed to:
1) properly detect namespace conflicts on enum values
2) properly implement the relative name lookup logic in reflect/protodesc

The approach taken:
1) Assumes that a FileDescriptor has no internal name conflicts.
This will (in a future CL) be guaranteed by reflect/protodesc and
is guaranteed today by protoc for generated descriptors.
2) Observes that the only declarations that can possibly conflict
with another file are top-level declarations (i.e., enums, enum values,
messages, extensions, and services). Enum values are annoying
since they live in the same scope as the parent enum, rather than
being under the enum.

For the internal data structure of Files, we only register the top-level
declarations. This is the bare minimum needed to detect whether the file
being registered has any namespace conflicts with previously registered files.

We shift the effort to lookups, where we now need to peel off the end fragments
of a full name until we find a match in the internal registry. If a match
is found, we may need to descend into that declaration to find a nested
declaration by name.

For initialization, we modify internal/filedesc to initialize the
enum values for all top-level enums. This performance cost is offsetted
by the fact that Files.Register now avoids internally registering
nested enums, messages, and extensions.

For lookup, the cost has shifted from O(1) to O(N),
where N is the number of segments in the full name.
Top-level descriptors still have O(1) lookup times.
Nested descriptors have O(M) lookup times,
where M is the level of nesting within a single file.

Change-Id: I950163423431f04a503b6201ddcc20a62ccba017
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/183697
Reviewed-by: Damien Neil <dneil@google.com>
2019-06-24 22:35:47 +00:00
Joe Tsai
d888139e7b internal/filedesc, internal/filetype: initial commit
The internal/fileinit package is split apart into two packages:
* internal/filedesc constructs descriptors from the raw proto.
It is very similar to the previous internal/fileinit package.
* internal/filetype wraps descriptors with Go type information

Overview:
* The internal/fileinit package will be deleted in a future CL.
It is kept around since the v1 repo currently depends on it.
* The internal/prototype package is deleted. All former usages of it
are now using internal/filedesc instead. Most significantly,
the reflect/protodesc package was almost entirely re-written.
* The internal/impl package drops support for messages that do not
have a Descriptor method (pre-2016). This removes a significant amount
of technical debt.
filedesc.Builder to parse raw descriptors.
* The internal/encoding/defval package now handles enum values by name.

Change-Id: I3957bcc8588a70470fd6c7de1122216b80615ab7
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/182360
Reviewed-by: Damien Neil <dneil@google.com>
2019-06-20 02:06:11 +00:00
Damien Neil
3631e225f8 reflect/protodesc: don't panic on nil Resolver
The change to make protodesc.NewFile take an interface rather than a
concrete type means that NewFile(f, nil) now causes a panic. (A nil
*protoregistry.Files is valid.)

Fix this panic by using a default, empty registry when NewFile's second
parameter is nil.

Change-Id: I70a1f0759e7ea5b57fba5b6123ee85188f4d560c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/182979
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-06-19 17:32:35 +00:00
Joe Tsai
5857a5ab4f reflect/protodesc: use Resolver interface
CL/177044 switches the serialization APIs to take in a resolver interface.
This does the moral equivalent for protodesc.

This is technically a breaking change since the signature of NewFile changes.
However, it is unlikely that anything is affected by this.

Change-Id: I7b44d5c3d5570a17c052add4d229550e4a0ad163
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/182638
Reviewed-by: Damien Neil <dneil@google.com>
2019-06-18 23:12:08 +00:00