The code organization is simpler if we keep the functions encoding and
decoding a particular type (e.g., maps) together rather than split
across files. Rename various "encode" files to "codec" in preparation
for adding fast-path decoding.
Change-Id: If1e271da99d31533ffefc19b1fc847936fa9484a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185241
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The protoreflect.Descriptor.Options method is currently documented as
returning a reference to the options, where the user must not mutate
the returned message. This changes internal/filedesc to avoid returning
a copy of the options by caching the first unmarshal.
See golang/protobuf#877
Change-Id: I15701d33fbda7535b21b2add72628b02992c373f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185197
Reviewed-by: Benny Siegert <bsiegert@gmail.com>
Hyrum's Law dictates that if we do not prevent naughty behavior,
people will rely on it. If we do not validate that the provided
file descriptor is correct today, it will be near impossible
to add proper validation checks later on.
The logic added validates that the provided file descriptor is
correct according to the same semantics as protoc,
which was reversed engineered to derive the set of rules implemented here.
The rules are unfortunately complicated because protobuf is a language
full of many non-orthogonal features. While our logic is complicated,
it is still 1/7th the size of the equivalent C++ code!
Change-Id: I6acc5dc3bd2e4c6bea6cd9e81214f8104402602a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/184837
Reviewed-by: Damien Neil <dneil@google.com>
Overview of changes:
* Add an option that specifies whether to replace unresolvable references
with a placeholder instead of producing an error. Since the prior behavior
produced placeholders (not always), we default to that behavior for now,
but will enable strict resolving in a future CL.
* The option is not yet exported because there is concern about what the
public API should look like. This will be exposed in a future CL.
* Unlike before, we now permit placeholders for unresolvable enum values.
* We implement relative name resolution logic.
* We handle the case where the type is unknown, but type_name is specified.
In such a case, we populate both FieldDescriptor.{Enum,Message} and leave
the FieldDescriptor.Kind with the zero value. If the type_name happened
to resolve, we use that to determine the type.
* If a placeholder is used to represent a relative name,
the FullName reports an invalid full name with a "*." prefix.
Change-Id: Ifa8c750423c488fb9324eec4d033a2f251505fda
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/184317
Reviewed-by: Damien Neil <dneil@google.com>
This does not remove all dependencies,
but all of the cases where it can now be implemented in terms of v2.
Change-Id: Idc5b0273f0d35c284bf2141eb9cce998692ceb15
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/184878
Reviewed-by: Herbie Ong <herbie@google.com>
Previously, when aberrantLoadMessageDesc returned it was guaranteed
to have initialized the current message through the use of the done signal.
However, this does not guarantee that the descriptor for a cylic reference
has also finished initialization.
Rather than add more complicated logic to wait until all cyclic references
have finished initializing, just add a global lock for the entire
aberrantLoadMessageDesc function.
This slows down performance, but is easier to reason about.
Change-Id: I4cdae8b955f71ee40fa6979f5a8d548d9749042c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/184657
Reviewed-by: Damien Neil <dneil@google.com>
The aberrant support logic only has access to the Go type
information, and not a concrete value. However, the XXX_MessageName
method exists on some hacky dynamic proto implementations where
it is only valid to call on a concrete value, not just newly created
instance of the given type.
However, from the perspective of the support logic, it is impossible
to distinguish between dynamic messages and hand-crafted custom messages.
Thus, just drop support for XXX_MessageName. We won't get the full name
of the message right, but oh well, what can we do.
Change-Id: Icc272861e11a355639fb82a991ca2854a9edc0c7
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/184557
Reviewed-by: Damien Neil <dneil@google.com>
Aberrant messages are hand-crafted messages that happen to work because
they use the same struct tags that generated code emits.
This happens to work in v1, but is unspecified behavior and entirely outside
the compatibility promise.
Support for this was added early on in the history of the v2 implementation,
but entirely untested. It was removed in CL/182360 to reduce the
technical debt of the legacy implementation. Unfortunately, sufficient number
of targets do rely on this aberrant support, so it is being added back.
The logic being added is essentially the same thing as the previous logic,
but ported to use internal/filedesc instead of the now deleted
internal/prototype package.
Change-Id: Ib5cab3e90480825b9615db358044ce05a14b05bd
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/184517
Reviewed-by: Damien Neil <dneil@google.com>
Move data used by the fast-path implementations into a substructure of
MessageInfo and initialize it separately.
Change-Id: Ib855ee8ea5cb0379528b52ba0e191319aa5e2dff
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/184077
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Len looks like it should be O(1), but the need to check for
non-zero-length repeated fields makes it at minimum O(n) where n is
the number of repeated fields. In practice, it's O(n) where n is the
number of fields altogether.
The Len function is not especially useful, easily duplicated with Range
and a counter, and can be surprisingly inefficient. Drop it.
Change-Id: I24b27433217e131e842bd18dd58475bcdf62ef97
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/183678
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Previous code tries to do all-lowercase match all non-extension field
names to match group field names, which is incorrect. Fix to check for
only group field type and make sure that the format is correct as well.
Fixesgolang/protobuf#878.
Fix typo in text proto string in internal/impl/message_test.go that
wasn't caught before due to above issue.
Change-Id: Ief952907306435ed76a095e96e29fcc9c0027b73
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/183737
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
This is a breaking change.
The replacement is the Files.FindDescriptorByName method,
which is more flexible as it handles all descriptor types.
Change-Id: I2ccd544a7630396a2428b1d41f836c5246070912
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/183700
Reviewed-by: Damien Neil <dneil@google.com>
This change makes it such that Files now functionally registers all
descriptors in a file (not just enums, messages, extensions, and services),
but also including enum values, messages fields/oneofs, and service methods.
The ability to look up any descriptor by full name is needed to:
1) properly detect namespace conflicts on enum values
2) properly implement the relative name lookup logic in reflect/protodesc
The approach taken:
1) Assumes that a FileDescriptor has no internal name conflicts.
This will (in a future CL) be guaranteed by reflect/protodesc and
is guaranteed today by protoc for generated descriptors.
2) Observes that the only declarations that can possibly conflict
with another file are top-level declarations (i.e., enums, enum values,
messages, extensions, and services). Enum values are annoying
since they live in the same scope as the parent enum, rather than
being under the enum.
For the internal data structure of Files, we only register the top-level
declarations. This is the bare minimum needed to detect whether the file
being registered has any namespace conflicts with previously registered files.
We shift the effort to lookups, where we now need to peel off the end fragments
of a full name until we find a match in the internal registry. If a match
is found, we may need to descend into that declaration to find a nested
declaration by name.
For initialization, we modify internal/filedesc to initialize the
enum values for all top-level enums. This performance cost is offsetted
by the fact that Files.Register now avoids internally registering
nested enums, messages, and extensions.
For lookup, the cost has shifted from O(1) to O(N),
where N is the number of segments in the full name.
Top-level descriptors still have O(1) lookup times.
Nested descriptors have O(M) lookup times,
where M is the level of nesting within a single file.
Change-Id: I950163423431f04a503b6201ddcc20a62ccba017
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/183697
Reviewed-by: Damien Neil <dneil@google.com>
This currently returns uninformative errors from the fast path and then
consults the slow, reflection-based path only when an error is detected.
Perhaps it's worth going through the effort of producing better errors
directly on the fast path.
Change-Id: I68536e9438010dbd97dbaff4f47b78430221d94b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171462
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Immediately abort (un)marshal operations when encountering invalid UTF-8
data in proto3 strings. No other proto implementation supports non-UTF-8
data in proto3 strings (and many reject it in proto2 strings as well).
Producing invalid output is an interoperability threat (other
implementations won't be able to read it).
The case where existing string data is found to contain non-UTF8 data is
better handled by changing the field to the `bytes` type, which (aside
from UTF-8 validation) is wire-compatible with `string`.
Remove the errors.NonFatal type, since there are no remaining cases
where it is needed. "Non-fatal" errors which produce results and a
non-nil error are problematic because they compose poorly; the better
approach is to take an option like AllowPartial indicating which
conditions to check for.
Change-Id: I9d189ec6ffda7b5d96d094aa1b290af2e3f23736
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/183098
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The primary (cross-language) protobuf repository contains benchmark data
sets. Add benchmarks using this data. (A version of this benchmark exists
in the protobuf repository, but it uses the v1 API and isn't trivial to
get working.)
Fetch the small benchmark datasets from the
github.com/protocolbuffers/protobuf repo by default. Add a
download_benchdata.bash script which fetches the larger datasets as
well.
Generate necessary packages under internal/testprotos/benchmarks.
To run:
go run ./proto -bench=BenchmarkData
Usual caveats about benchmarking apply: While these benchmarks use
realistic data, isolated microbenchmarking of proto operations is not
necessarily representitive of performance in production systems.
Change-Id: I58d107554baf104568c86997b5ad50be8b2a5790
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/183297
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
If we were starting from scratch, we would not have added the enum maps.
However, they already exist and see a fair amount of usage.
The effort to remove them is not worth it. Thus, remove the deprecation
warning since they are here to stay.
Note that the generated code does not refer to the generated enum maps.
One day, the linker should be able to elide them if unused by the user.
However, https://golang.org/issue/2559 would need to be resolved first.
Change-Id: Ia8b9b1812b5d8462ca2fa1d543170e4a09ff9e4f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/183177
Reviewed-by: Damien Neil <dneil@google.com>
The internal/fileinit package is split apart into two packages:
* internal/filedesc constructs descriptors from the raw proto.
It is very similar to the previous internal/fileinit package.
* internal/filetype wraps descriptors with Go type information
Overview:
* The internal/fileinit package will be deleted in a future CL.
It is kept around since the v1 repo currently depends on it.
* The internal/prototype package is deleted. All former usages of it
are now using internal/filedesc instead. Most significantly,
the reflect/protodesc package was almost entirely re-written.
* The internal/impl package drops support for messages that do not
have a Descriptor method (pre-2016). This removes a significant amount
of technical debt.
filedesc.Builder to parse raw descriptors.
* The internal/encoding/defval package now handles enum values by name.
Change-Id: I3957bcc8588a70470fd6c7de1122216b80615ab7
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/182360
Reviewed-by: Damien Neil <dneil@google.com>
This is a breaking change in light of new methods added in CL/175458.
This removes:
Message.KnownFields: equivalent functionality have been hoisted up
to the Message interface itself.
Message.UnknownFields: equivalent functionality is via
the Message.{Set,Get}Unknown methods.
Change-Id: Ia08b26894d2b45033a6ad6616258ff0fb9f8b7a4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/182597
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
This is a breaking change in light of new methods added in CL/174918.
This removes:
Enum.Type: replacement is Descriptor
Message.Type: replacement is Descriptor and New
Change-Id: Iaa5328795407c8401ef14ed038bd5ace19d8e03b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/174938
Reviewed-by: Damien Neil <dneil@google.com>
This is a breaking change in light of new methods added in CL/176977.
This removes:
FieldDescriptor.Oneof: replacement is ContainingOneof
FieldDescriptor.Extendee: replacement is IsExtension and ContainingMessage
Change-Id: I82008e46fb3b80de8e8a0ac42afc54e8c4b67411
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/176942
Reviewed-by: Damien Neil <dneil@google.com>
This is a breaking change.
The equivalent replacement logic is to trivially check whether the
parent descriptor is not nil.
Change-Id: I5c89c1d9f29f9e6f721bbfbcf7774188d8f0086a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/175987
Reviewed-by: Damien Neil <dneil@google.com>
Added API:
Message.Len
Message.Range
Message.Has
Message.Clear
Message.Get
Message.Set
Message.Mutable
Message.NewMessage
Message.WhichOneof
Message.GetUnknown
Message.SetUnknown
Deprecated API (to be removed in subsequent CL):
Message.KnownFields
Message.UnknownFields
The primary difference with the new API is that the top-level
Message methods are keyed by FieldDescriptor rather than FieldNumber
with the following semantics:
* For known fields, the FieldDescriptor must exactly match the
field descriptor known by the message.
* For extension fields, the FieldDescriptor must implement ExtensionType,
where ContainingMessage.FullName matches the message name, and
the field number is within the message's extension range.
When setting an extension field, it automatically stores
the extension type information.
* Extension fields are always considered nullable,
implying that repeated extension fields are nullable.
That is, you can distinguish between a unpopulated list and an empty list.
* Message.Get always returns a valid Value even if unpopulated.
The behavior is already well-defined for scalars, but for unpopulated
composite types, it now returns an empty read-only version of it.
Change-Id: Ia120630b4db221aeaaf743d0f64160e1a61a0f61
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/175458
Reviewed-by: Damien Neil <dneil@google.com>
When encountering a type that does not have a MessageInfo, don't assume
that it's a legacy message that doesn't implement proto.Message. Add a
set of test messages exercising this case (panics prior to the
internal/impl change).
Change-Id: Ic1ec5ecfbe92278fbef44284ff52a0e0622a158c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/182477
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
CL/180577 removes direct access to the Desc field in v1.
Since the only way to access the type is through {Has,Get,Set}Type,
we can now unexport the field and change the underlying type.
Change-Id: I6cc7018d4326c17228ba579d6161c5fb3f6d4127
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/180578
Reviewed-by: Damien Neil <dneil@google.com>
Add ExtensionField.{SetType,GetType} to hide the fact that the underlying
descriptor is actually an ExtensionDescV1.
Change-Id: I1d0595484ced0a88d2df0852a732fdf0fe9aa232
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/180538
Reviewed-by: Damien Neil <dneil@google.com>
Generate the needed infrastructure to ensure that we can statically
enforce minimum and maximum versions. This enables us to have a policy
when we release v2 where it fails to build for:
* new generated code with really old runtimes
* new runtimes with really old generated code
Change-Id: Ib699ad62c06dff8f9285806394a741c18db00288
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/178546
Reviewed-by: Damien Neil <dneil@google.com>
The internal/legacy package was originally separated out from internal/impl
to avoid a cyclic dependency on descriptor proto. However, the dependency
that legacy has on descriptor has long been dropped such that we can
now merge the two packages together again.
All legacy related logic are in a file with a legacy prefix.
Change-Id: I2424fc0f50721696ad06fa7cebb9bdd0babea13c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/178542
Reviewed-by: Damien Neil <dneil@google.com>
Add the prototype package which provides constructors for
protoreflect.{Enum,Message,Extension}Type.
Switch all usages of the internal/prototype equivalent to the new package.
Switch all custom implementions of {Enum,Message}Type to the new package.
Change-Id: Ia9dae6fed4f2b90e55c123627044a7faf098c4b1
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/178438
Reviewed-by: Damien Neil <dneil@google.com>
The name MessageType is easily confused with protoreflect.MessageType.
Rename it as MessageInfo, which follows the pattern set by v1,
where the equivalent data structure is called InternalMessageInfo.
Change-Id: I535956e1f7c6e9b07e9585e889d5e93388d0d2ce
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/178478
Reviewed-by: Damien Neil <dneil@google.com>
The name descfmt more closely matches the fact that this package
provides pretty-printing for descriptors.
Change-Id: I11b0e4a0302962ee7bbf315fb259d050847cadbb
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/178479
Reviewed-by: Damien Neil <dneil@google.com>
Both legacy_unknown.go and legacy_extension.go deal with the
current data structure for unknown fields and extensions.
While the current data structure is sub-optimial,
strictly speaking, they are not "legacy" since they are very much
used by the v2 API as well.
Change-Id: I3852dba0dcebaee00bbeb94d9936cd759f774f0a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/178477
Reviewed-by: Damien Neil <dneil@google.com>
A proto2 message can reference a proto3 enum. It is currently cumbersome
to set proto3 enums in proto2 messages. Add the Enum method in all situations
to support this situation. It also removes yet another point of divergence
between proto2 and proto3.
We could consider removing this method if Go ever gets generics,
but that hypothetical future is long ways away.
Change-Id: Ib83bc87e46b49f3271b90bacb2893bb8e278e4f2
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/177257
Reviewed-by: Damien Neil <dneil@google.com>
Rename each generated protobuf package such that the base of the
Go package path is always equal to the Go package name to follow
proper Go package naming conventions.
The Go package name is derived from the .proto source file name by
replacing ".proto" with "pb" and stripping all underscores.
Change-Id: Iea05d1b5d94b1b2821ae10276ab771bb2df93c0e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/177380
Reviewed-by: Damien Neil <dneil@google.com>
CL/177620 modifies v1 to stop touching the Value field directly,
such that it is now unexport the Value field.
Change-Id: I0a05bbe59146862fc77c261349d7d90d6fa312e0
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/177621
Reviewed-by: Damien Neil <dneil@google.com>
Once the ExtensionFieldV1.Value field is unexported, it forces all
read operations to go through GetValue, which can perform some form
of lazy retrieval.
The SetEagerValue method is equivalent to setting the Value directly.
The SetLazyValue method permits setting a lazy equivalent of the Value
by simply storing a function closure returning the value.
Change-Id: I410f74924f0957f5eadc82eea3669ca335a02701
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/177619
Reviewed-by: Damien Neil <dneil@google.com>
GoExtension now supports extensions that have enum or message type that
is implemented by a Go type that can take on multiple enum or message
types (i.e. the actual enum or message type cannot be determined simply
from the zero value of its Go type). This is necessary to support
dynamic types generated at runtime from descriptors rather than at
compile-time.
Change-Id: Ia0b3b4b02332fc83c0c85e992b37ded467070472
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/177338
Reviewed-by: Joe Tsai <joetsai@google.com>
While it is general convention that the receiver being mutated
is the first argument, the standard library specifically goes against
this convention when it comes to the Unmarshal function.
Switch the ordering of the Unmarshal function to match the Go stdlib.
Change-Id: I893346680233ef9fec7104415a54a0a7ae353378
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/177258
Reviewed-by: Damien Neil <dneil@google.com>
Rename encoding/*pb to follow the convention of prefixing package names
with 'proto':
google.golang.org/protobuf/encoding/protojson
google.golang.org/protobuf/encoding/prototext
Move protogen under a compiler/ directory, just in case we ever do add
more compiler-related packages.
google.golang.org/protobuf/compiler/protogen
Change-Id: I31010cb5cabcea8274fffcac468477b58b56e8eb
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/177178
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Temporarily remove go.mod, since we can't generate an accurate one until
the corresponding v1 change is submitted.
Change-Id: I1e1ad97f2b455e33f61ffaeb8676289795e47e72
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/177000
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Added API:
FieldDescriptor.IsExtension
FieldDescriptor.IsList
FieldDescriptor.MapKey
FieldDescriptor.MapValue
FieldDescriptor.ContainingOneof
FieldDescriptor.ContainingMessage
Deprecated API (to be removed in subsequent CL):
FieldDescriptor.Oneof
FieldDescriptor.Extendee
These methods help cleanup several common usage patterns.
Change-Id: I9a3ffabc2edb2173c536509b22f330f98bba7cf3
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/176977
Reviewed-by: Damien Neil <dneil@google.com>
Querying for the parent file that contains a descriptor declaration
is a common enough operation to warrant its own first-class method.
Change-Id: I2f41e5126a5b465df23897904a6513dd3ed8dd92
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/176777
Reviewed-by: Damien Neil <dneil@google.com>
Added methods:
Enum.Descriptor
Message.Descriptor
EnumType.Descriptor
MessageType.Descriptor
ExtensionType.Descriptor
Message.New
All functionality is switched over to use those methods instead of
implicitly relying on the fact that {Enum,Message}Type implicitly
implement the associated descriptor interface.
This CL does not yet remove {Enum,Message}.Type or prevent
{Enum,Message,Extension}Type from implementating a descriptor.
That is a subsequent CL.
The Message.New method is also added to replace functionality
that will be lost when the Type methods are removed.
Change-Id: I7fefde1673bbd40bfdac489aca05cec9a6c98eb1
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/174918
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Herbie Ong <herbie@google.com>
CL/175579 switches v2 to use XXX_unrecognized exclusively.
CL/175838 switches v1 to use XXX_unrecognized exclusively.
This means that it is now safe to delete the Raw field.
Change-Id: Ic60c58147c2df4078e72c34d1202d226a7ea4bed
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/175839
Reviewed-by: Damien Neil <dneil@google.com>
Correctly report the type we were looking for when panicking.
Previously would say: "invalid type: got *T, want *T".
Change-Id: I90ea0dd1fc64f3aec37a5a5828c1832fb0ab2887
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/176258
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
