Minor refactoring of impl.MessageType initialization: Pull the
information gathered about a message struct by makeKnownFieldsFunc out
into a struct that we can pass around.
At the moment, makeKnownFieldsFunc is the only user of this struct, but
this will simplify the table (un)marshaler.
Drop the 'specialByName' map indexing XXX_ fields because it currently
isn't used anywhere.
Change-Id: I992c9f490982a05f3919d7d4e08052e2ab54d44d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/176220
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The protobuf data model makes no distinction between unknown fields
that are within the extension field ranges or not. Now that we eagerly
unmarshal extensions, there is even less need for storing unknown
fields in the extension map. Instead, use the XXX_unrecognized field
exclusively for this purpose.
Change-Id: I673a7d6259fe9fdbdc295bcfa8252ef4da415343
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/175579
Reviewed-by: Damien Neil <dneil@google.com>
Weak fields are an obsolete proto1 feature. They have been superseded
by extensions. However, some vestigial support for weak fields does
remain, mostly as Google-internal patches. (They aren't exciting;
extensions really do everything weak fields do in a cleaner and
more portable fashion.)
At the moment, the only visible impact of marking a field [weak=true]
is to exclude it from "internal/fileinit".FileBuilder.DependencyIndexes.
We want to preserve that behavior just in case we ever do add full weak
field support here.
Extend fileinit to look up message descriptors for weak fields in the
global registry. If the descriptor cannot be found, use a placeholder
instead.
Remove special-case handling of weak fields in the impl package. The
code generator doesn't do anything special for them, so they can be
treated as any other field.
Change-Id: Ifa2ee3d30d63680a0eeb59c66ebc9521f38fd660
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/175997
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
This covers most of the TODO around validation. I left open the ones
that we didn't have clear consensus on yet.
Change-Id: I336c53173ee8d7447558b1e3a0c1ef945e986cd5
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/175140
Reviewed-by: Joe Tsai <joetsai@google.com>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The protobuf documentation explicitly specifies (1<<29)-1 as the maximum
field number, but the C++ implementation itself has a special-case where it
allows field numbers up to MaxInt32 for MessageSet fields, but continues
to apply the former limit in all non-MessageSet cases.
To avoid complicated branching logic, we use the larger limit for all cases
if MessageSet is supported, otherwise, we impose the documented limit.
Change-Id: I710a2a21aa3beba161c3e6ca2f2ed9a266920a5a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/175817
Reviewed-by: Damien Neil <dneil@google.com>
Change the rewritten Go package name for the conformance protos to match
the basename of the directory (i.e., "conformance").
This is a trivial change, but avoids confusion when the package name and
import path don't match.
Change-Id: I9b091c78ce4a85f7051c55ee1a48ef6dbba68db6
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/174944
Reviewed-by: Herbie Ong <herbie@google.com>
Add support for basic equality comparison of messages.
Messages are equal if they have the same type and marshal to the
same bytes with deterministic serialization, with some exceptions:
- Messages with different registered extensions are unequal.
- NaN is not equal to itself.
Unlike the v1 Equal, a nil message is equal to an empty message of
the same type.
Change-Id: Ibabdadd8c767b801051b8241aeae1ba077e58121
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/174277
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Change use of regexp for matching literals true,false,null to simple
bytes comparison. Small gain from doing this.
Remove computing for position in Value as that is only needed in error
messages. In order to preserve ability to compute for position later,
store the original input in Value instead of just the slice containing
the value, however, need to also store the start index and size of the
parsed value.
Using benchmark in encoding/bench_test.go now shows faster time and less
memory usage than V1.
name old time/op new time/op delta
JSONEncode-4 30.3ms ± 1% 10.3ms ± 1% -66.02% (p=0.000 n=9+8)
JSONDecode-4 54.4ms ± 3% 18.9ms ± 2% -65.33% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
JSONEncode-4 10.3MB ± 0% 3.9MB ± 0% -61.74% (p=0.000 n=10+9)
JSONDecode-4 19.0MB ± 0% 3.6MB ± 0% -81.29% (p=0.000 n=10+9)
name old allocs/op new allocs/op delta
JSONEncode-4 465k ± 0% 64k ± 0% -86.30% (p=0.000 n=10+8)
JSONDecode-4 289k ± 0% 163k ± 0% -43.69% (p=0.000 n=10+9)
Change-Id: I0a3108d675d6442674facb065aaebd14051f6c5d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/172662
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
This feature seems to be used quite a bit, and the conformance tests
treat this as required, perhaps as a "required option" since the
developer guide states:
"Proto3 JSON parser should reject unknown fields by default but may
provide an option to ignore unknown fields in parsing."
Also, all invalid UTF-8 errors in skipped values are also returned as it
is similar to a parse error, except it is a non-fatal one.
Change-Id: Ia26e9a355daecdbf99af23f3061353fffa32d47d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/174017
Reviewed-by: Damien Neil <dneil@google.com>
According to IEEE-754, equality on floats reports true for both +0 and -0.
However, this definition causes us to lose information. Instead, we want
a definition that checks for exact equality with +0.
Otherwise, we would lose the ability to serialize -0.
Change-Id: I36450c24258fc4f856bfd4bc4c53a90199b216b9
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/172970
Reviewed-by: Herbie Ong <herbie@google.com>
CL/172238 added equivalent methods without the Type suffix,
while keeping the old methods.
CL/172582 updates the v1 codebase to use the new methods.
This CL removes the methods with the Type suffix.
Change-Id: Iaaaa4fff11cac1694735657db2e5fd7cadc90afe
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/173138
Reviewed-by: Herbie Ong <herbie@google.com>
The protobuf type system uses the word "descriptor" instead of "type".
We should avoid the "type" verbage when we aren't talking about Go types.
The old names are temporarily kept around for compatibility reasons.
Change-Id: Icc99c913528ead011f7a74aa8399d9c5ec6dc56e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/172238
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
Previous calls to indexNeedEscape with a type conversion from []byte
to string incurs allocation.
Make 2 different calls instead, one for string and one for bytes.
Type converting string to []byte does not incur extra allocation,
however, the benchmark results still show it to be slower by ~3% for
textpb and 6+% for jsonpb, hence decided to go with 2 separate calls
instead.
Results over current head:
name old time/op new time/op delta
TextEncode-4 18.1ms ± 2% 18.3ms ± 2% ~ (p=0.065 n=10+9)
TextDecode-4 233ms ± 3% 102ms ± 1% -56.34% (p=0.000 n=9+10)
JSONEncode-4 10.4ms ± 2% 10.5ms ± 0% +0.56% (p=0.019 n=9+9)
JSONDecode-4 870ms ± 2% 354ms ± 4% -59.33% (p=0.000 n=9+10)
name old alloc/op new alloc/op delta
TextEncode-4 28.9MB ± 0% 28.9MB ± 0% +0.00% (p=0.000 n=10+9)
TextDecode-4 1.16GB ± 0% 0.03GB ± 0% -97.44% (p=0.000 n=9+10)
JSONEncode-4 3.94MB ± 0% 3.94MB ± 0% +0.00% (p=0.000 n=10+10)
JSONDecode-4 3.35GB ± 0% 0.01GB ± 0% -99.83% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
TextEncode-4 73.5k ± 0% 73.5k ± 0% ~ (all equal)
TextDecode-4 278k ± 0% 255k ± 0% -8.26% (p=0.000 n=9+10)
JSONEncode-4 63.8k ± 0% 63.8k ± 0% ~ (all equal)
JSONDecode-4 247k ± 0% 210k ± 0% -14.92% (p=0.000 n=10+10)
Change-Id: Ibc64e9a7827ec1fffa213eb79f60497950203700
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/172239
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Using a named fields gives us the flexibility to change the underlying
representation of special fields without needing to regenerate user code.
We add a named type for ExtensionFields, UnknownFields, and SizeCache.
Change-Id: I107cf82899850ea76665310ce79def60f0f7ab97
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/172402
Reviewed-by: Damien Neil <dneil@google.com>
The v2 implementation will eagerly unmarshal extension fields,
which means that the logic to synchronize the extensions fields
is no longer necessary. Using type aliases, we can ensure
impl.ExtensionsV1 is identical to the extensions map prior to
a4ab9ec5 from May 2016. This makes it such that we only have one
legacy implementation that we have to deal with.
Change-Id: I41b977e912fbb899c5bc4f055890d474b419429c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/172240
Reviewed-by: Damien Neil <dneil@google.com>
Lower-casing the FileDescriptor variable provides for shorter and
more readable global variable names. This has a very minor benefit
that the read-only data section of a binary is slightly smaller
since function names do end up in the final binary.
Change-Id: I077364905e5c9adea69873b3ea580fddc68b9eb8
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/172119
Reviewed-by: Herbie Ong <herbie@google.com>
Drop the protoreflect.FileDescriptor.DescriptorByName method.
Descriptor lookup will always happen through a protoregistry.Files, which
is more generally useful (it's rare that you want to find a descriptor in a
specific file, as opposed to a package which may be composed of multiple files).
Split protoregistry.Files descriptor lookup into individual per-type functions
(enum, message, extension, service), matching the preg.Types API.
Drop the ability to look up enum values, message fields, and service methods
for now. This can be easily added later if needed, and is trivial to implement
in user code. (e.g., look up the service and then consult sd.Methods.ByName().)
Change-Id: I2b3d8ef888921a8464ba1434eddab20c7d3a458e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/172118
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Changes:
* Remove protoreflect.OptionsMessage and use protoreflect.ProtoMessage
now that the generated options natively implement reflection.
* Make registration of concrete option types the responsibility of
internal/fileinit since the init logic of descriptor.pb.go uses that.
* Remove equivalent logic in protoc-gen-go to special-case descriptor.
Change-Id: Id814651fafa4d888ff4532d59c6a4f9b68145157
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171465
Reviewed-by: Damien Neil <dneil@google.com>
Go1.12 on GoLLVM with LLVM 9.0.0 has a bug where the simpler but
equivalent expression results in memory corruption.
Revert to the more complex expression to avoid breaking GoLLVM and
it is not worth waiting for a GoLLVM fix.
Change-Id: I8b42965ca9bd333deb8693021e7b22f966e13521
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171463
Reviewed-by: Damien Neil <dneil@google.com>
Stop adding options to legacy MessageDescriptors and FieldDescriptors.
We would synthesize options messages containing semantic options
(FieldOptions.is_packed, FieldOptions.is_weak, MessageOptions.map_entry).
This information is already contained in the protoreflect descriptors,
so there is no real need to define a correct options message.
If we do want to include options in legacy descriptors, then we should
get the original value out of the FileDescriptorProto, since it may
include additional extensions or other information.
This change completely removes the dependency from internal/legacy to
descriptor.proto.
Change-Id: Ib6bbe4ca6e0fe7ae501f3e9b11d5fa0222808410
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171458
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
We jump through many hoops to avoid generated protos depending on
internal/legacy. Break the cycle in the other direction: Remove
the dependency on descriptor.proto from internal/legacy by
using a hand-written parser for the few descriptor fields we need.
Still to do: Remove the descriptor.proto dependency from
internal/encoding/tag.
Change-Id: I5fd99a2170470ba8530eb2679b6dde899821bf3e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171457
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Add a method that provides efficiently querying for which member field
in a oneof is actually set. This is useful when dealing with oneofs
with many member fields.
Change-Id: I918b566c432f8bdd24dcecbb5501d231ffefef29
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170580
Reviewed-by: Damien Neil <dneil@google.com>
Seperate out the generation of the oneof wrapper types from the
code block that is about getter methods.
Change-Id: Ief44ef953d0b5ad8c998a8542c830ca70468a3bf
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171029
Reviewed-by: Damien Neil <dneil@google.com>
So far only Ints is being used and it does not seem like the
other set types will ever be used. Remove them.
We can always add them back if we need them again.
Change-Id: I9a9e8ce76bd231d1fe5b726af7da690dc4019bb8
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170625
Reviewed-by: Herbie Ong <herbie@google.com>
Allow message implementations to provide optimized versions of standard
operations. Generated messages now include a ProtoReflectMethods method,
returning a protoiface.Methods struct containing pointers to assorted
optional functions.
The Methods struct also includes a Flags field indicating support for
optional features such as deterministic marshaling.
Implementation of the fast paths (and tests) will come in later CLs.
Change-Id: Idd1beed0ecf43ec5e5e7b8da2ee1e08d3ce32213
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170340
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Also added json.Decoder.Clone API for unmarshaling Any to look
ahead remaining bytes for @type field.
Change-Id: I2f803743534dfb64f9092d716805b115faa5975a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170102
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Minor changes:
* Use x as the receiver since "e" and "m" are meaningless in the presence
of user-defined enum and message names.
* Consistently keep enum methods together, rather awkwardly split apart
by the value maps.
Change-Id: I68e5666efb56ac7a4d062fb223b9f826dc72aba9
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170357
Reviewed-by: Herbie Ong <herbie@google.com>
This reduces the init-time cost slightly since the GZIP'd
raw descriptor is constructed lazily on demand.
Change-Id: I482c6a2201b8786e425d7dee5612fdfd60ab1500
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169917
Reviewed-by: Herbie Ong <herbie@google.com>
Provide AllowPartial option to accept messages with missing required
field during marshaling and unmarshaling.
Change-Id: Ia23783870a8125633f8ddc0b686984b4c5ca15ba
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169500
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Reviewed-by: Damien Neil <dneil@google.com>
This removes yet another set of dependencies of v2 on v1.
The only remaining dependency are in the _test.go files,
primarily for proto.Equal.
Changes made:
* cmd/protoc-gen-go no longer generates any functionality that depends
on the v1 package, and instead only depends on v2.
* internal/fileinit.FileBuilder.MessageOutputTypes is switched from
protoreflect.MessageType to protoimpl.MessageType since the
implementation must be fully inialized before registration occurs.
* The test for internal/legacy/file_test.go is switched to a legacy_test
package to avoid a cyclic dependency.
This requires Load{Enum,Message,File}Desc to be exported.
Change-Id: I43e2fe64cff4eea204258ce11e791aca5eb6e569
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169298
Reviewed-by: Damien Neil <dneil@google.com>
Per Joe's suggestion, remove producing numberParts when parsing a JSON
number to produce corresponding Value. This saves having to store it
inside Value as well. Only produce numberParts for calls to
Value.{Int,Uint} call.
numberParts is only used for producing integers and removing the logic to
produce numberParts improves overall decoding speed for floats, and shows no
change for integers.
name old time/op new time/op delta
Float-4 559ns ± 0% 288ns ± 0% ~ (p=1.000 n=1+1)
Int-4 471ns ± 0% 466ns ± 0% ~ (p=1.000 n=1+1)
Change-Id: I21bf304ca67dda8d41a4ea0022dcbefd51058c1c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168781
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
As a goal, v2 should not depend on v1. As another step towards that end,
we move all the types that used to be in the v1 protoapi package over to v2.
For now, we place MessageV1, ExtensionRangeV1, and ExtensionDescV1
in runtime/protoiface since these are types that generated messages will
probably have to reference forever. An alternative location could be
reflect/protoreflect, but it seems unfortunate to have to dirty the
namespace of that package with these types.
We move ExtensionFieldV1, ExtensionFieldsV1, and ExtensionFieldsOf
to internal/impl, since these are related to the implementation of a
generated message.
Since moving these types from v1 to v2 implies that the v1 protoapi
package is useless, we update all usages of v1 protoapi in the v2
repository to point to the relevant v2 type or functionality.
CL/168538 is the corresponding change to alter v1.
There will be a temporary build failure as it is not possible
to submit CL/168519 and CL/168538 atomically.
Change-Id: Ide4025c1b6af5b7f0696f4b65b988b4d10a50f0b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168519
Reviewed-by: Herbie Ong <herbie@google.com>
The protobuf type system hacks the representation of map entries into that
of a pseudo-message descriptor.
Previously, we made all message descriptors implement MessageType
where type descriptors had a GoType method that simply returned nil.
Unfortunately, this violates a nice property in the Go type system
where being able to assert to a MessageType guarantees that Go type
information is truly associated with that descriptor.
This CL makes it such that message descriptors for map entries
do not implement MessageType.
Change-Id: I23873cb71fe0ab3c0befd8052830ea6e53c97ca9
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168399
Reviewed-by: Damien Neil <dneil@google.com>
Generate field numbers for the well-known types,
so that encoding/jsonpb can benefit from them as well.
This CL fixes internal/cmd/generate-protos, which was silently failing
because the modulePath was not properly initialized. We fix this by
moving it to the start of the init function.
Change-Id: I87637176f29218cffa512b4baa49f39dae924061
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168497
Reviewed-by: Herbie Ong <herbie@google.com>
This makes it consistent with jsonpb.MarshalOptions. This does change
the default to be in compact form.
Change-Id: I1b07f06f282c019b30f3f1cbb43f6c8cba18f385
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168405
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Changes:
* Modify protoc-gen-go to use a helper from internal/impl
to print the message as text.
* Add a helper function to internal/impl that calls v2 textpb.
* Modify encoding/textpb to avoid depending on descriptor proto,
which would cause an import cycle.
* Modify internal/fileinit to populate a pseudo-internal
method on MessageDescriptor to check whether the message should
use the message set wire format. We avoid adding this to the
main MessageDescriptor interface since message sets are a
deprecated proto1 feature that we should avoid promoting.
Change-Id: Ibaf79a563af695756f11ddc4db69b38e25a8f1a7
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168439
Reviewed-by: Herbie Ong <herbie@google.com>
The v1 registration leaks the message types out to the proto package.
When doing that, it must ensure that the reflection data structures
for those types are properly initialized first. We achieve that by
doing v1 registration at the end of the reflection init function.
Change-Id: If6df18df59d05bad50ff39c2eff6beb19e7466cc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168348
Reviewed-by: Damien Neil <dneil@google.com>
When concurrent requests to wrap a type occurs, it is possible that
two different descriptors are created for the same Go type.
Add sufficient synchronization to ensure that one descriptor is ever
returned for one Go type.
Change-Id: Idbbca4c1877a70317b39900ae83bfc3085d4a9c5
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168398
Reviewed-by: Herbie Ong <herbie@google.com>
By passing the global registries to fileinit.Builder, the Build
method can register the newly constructed descriptors on our behalf.
As a result, we can stop registering with the v1 registries as well.
We only do this for descriptor proto to remove another dependency on v1.
We deliberately keep the v1 registration logic for now to make it easy
to patch away this new behavior.
Change-Id: Ic6aa8ffba3d2d0abe08a61fc5e1c9ca7668e0988
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168000
Reviewed-by: Herbie Ong <herbie@google.com>
The .pb.go.meta file is checked into the repository as testdata.
The deliberate instability of outputs breaks golden tests within
our own repository. It is reasonable for us to depend on stability
since we control the output.
Change-Id: I1f73027a08a0757732d46610f334d40840cc4cfd
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/168001
Reviewed-by: Herbie Ong <herbie@google.com>
