The following changes are made:
* Permit invalid UTF-8 in proto2. This goes against specified behavior,
but matches functional behavior in wire marshaling (not just for Go,
but also in the other major language implementations as well).
* The Format function is specified as ignoring errors since its intended
purpose is to surface information to the human user even if it's not
exactly parsible back into a message. As such, add an unexported
allowInvalidUTF8 option that is specially used by Format.
* Add an EmitASCII option that forces the formatting of
strings and bytes to always be encoded as ASCII.
This ensures that the entire output is always ASCII as well.
Note that we do not replicate this behavior for protojson since:
* The JSON format fundamentally has a stricter and well-specified
grammar for exactly what is valid/invalid, while the text format
has not had a well-specified grammar for the longest time,
leading to all sorts of weird usages due to Hyrum's law.
* This is to ease migration from the legacy implementation,
which did permit invalid UTF-8 in proto2.
* The EmitASCII option relies on the ability to always escape
Unicode characters using ASCII escape sequences, but this is not
possible in JSON since the grammar only has an escape sequence defined
for Unicode characters \u0000 to \uffff, inclusive.
However, Unicode v12.0.0 defines characters up to \U0010FFFF,
which is beyond what the JSON grammar provides escape sequences for.
Change-Id: I2b524a904e9ec59f9ed5500e299613bc27c31a14
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/233077
Reviewed-by: Herbie Ong <herbie@google.com>
Any attempt at guessing the size for a fixed coderFieldInfo array
will always get it wrong in some cases, either by under-estimating
or over-estimating the count. The former causes worse caching behavior,
while the latter causes memory waste.
As a middle ground, just pre-allocate a slice of the exact length.
Each element will have memory locality with each other, but not
be guaranteed to have memory locality with the parent coderMessageInfo.
name old time/op new time/op delta
EmptyMessage/Wire/Marshal-8 43.1ns ±11% 42.6ns ± 8% -1.32% (p=0.036 n=50+49)
EmptyMessage/Wire/Unmarshal-8 18.6ns ±10% 18.9ns ±12% ~ (p=0.054 n=50+50)
EmptyMessage/Wire/Validate-8 15.0ns ± 9% 14.7ns ±10% -2.44% (p=0.002 n=50+45)
EmptyMessage/Clone-8 163ns ±20% 149ns ±19% -8.58% (p=0.000 n=48+53)
RepeatedInt32/Wire/Marshal-8 4.27µs ±12% 4.24µs ±13% ~ (p=0.612 n=48+52)
RepeatedInt32/Wire/Unmarshal-8 3.47µs ±14% 3.50µs ±11% ~ (p=0.217 n=50+53)
RepeatedInt32/Wire/Validate-8 2.12µs ±12% 2.09µs ± 9% ~ (p=0.121 n=50+51)
RepeatedInt32/Clone-8 3.04µs ±18% 2.98µs ±36% ~ (p=0.289 n=51+54)
Required/Wire/Marshal-8 281ns ±14% 276ns ±11% ~ (p=0.059 n=48+55)
Required/Wire/Unmarshal-8 117ns ±14% 118ns ±11% ~ (p=0.358 n=49+53)
Required/Wire/Validate-8 87.6ns ± 9% 88.0ns ±12% ~ (p=0.373 n=48+53)
Required/Clone-8 533ns ±12% 507ns ±15% -4.71% (p=0.000 n=49+54)
Change-Id: I4cf3134e424130bee728b7591127e5c80f07e2db
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/232937
Reviewed-by: Damien Neil <dneil@google.com>
Simplifies some compiler plugins who want to avoid generating empty
files. With this API, generated files can be skipped by default, and
unskipped when actual code is generated.
Change-Id: I941c821646f5c4430a84e08bbad7e021434b1e71
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/232239
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
In some situations, the full Go package path cannot be determined,
in which case, we should avoid suggesting a go_package option
that is incorrect.
As a heuristic, check whether the proposed path contains at least
a dot before a slash to determine whether it is a full path or not,
which is a simple way to determine whether the first segment is a
top-level domain.
This avoids printing unhelpful warnings like:
WARNING: Missing 'go_package' option in "foo/bar.proto", please specify:
option go_package = ".;foo_package";
A future release of protoc-gen-go will require this be specified.
See https://developers.google.com/protocol-buffers/docs/reference/go-generated#package for more information.
and instead prints a warning like:
WARNING: Missing 'go_package' option in "foo/bar.proto",
please specify it with the full Go package path as
a future release of protoc-gen-go will require this be specified.
See https://developers.google.com/protocol-buffers/docs/reference/go-generated#package for more information.
We rely on the documentation on the developer website to provide
better guidance.
Change-Id: I38fc4c676d0314ba6d7ad8d5f390fb9e237f2bb1
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/232338
Reviewed-by: Damien Neil <dneil@google.com>
This CL continues to hard-code the exact commit since the protoc release
uses inconsistent naming for its download URLs.
Change-Id: I7551c4b2f9b7b89c1c85169faffc03641f21bc11
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/232097
Reviewed-by: Damien Neil <dneil@google.com>
The UnmarshalInitialized flag produced by Unmarshal and Validate are
filters such that must never have false positives (i.e., report a
partial message as initialized) otherwise it is incorrect.
It can tolerate some degree of false negatives (i.e., report an
initialized message as partial), but that leads to significant
performance degradation needing to do the full initialization check.
These should be the exception, not the norm.
Adjust the fuzzer to search for false-negative cases. For now, we only
require that the Unmarshal and Validate report initialized for any
"normalized" messages which we produce by marshaling intermediate
message again. This is to work around a known case where they cannot
determine initialization status if the wire data relies on protobuf's
merge functionality (where two or more partial messages merge
together to form an initialized message).
Change-Id: I6bb6c6594981ca08a92583bae77e5a2d44924af6
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/231577
Reviewed-by: Damien Neil <dneil@google.com>
Cleanup the generated logic by having the implementation be backed
by protoimpl rather that directly generated.
Weak fields are a deprecated feature of protobufs and
have entirely be superceded by extensions.
Unfortunately, there are still some usages of it.
Change-Id: Ie1a4b7da253e2ccf5e56627775d9b2fb4090d59a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/229717
Reviewed-by: Damien Neil <dneil@google.com>
In the upcoming 3.12.x release of protoc, the proto3 language will be
amended to support true presence for scalars. This CL adds support
to both the generator and runtime to support these semantics.
Newly added public API:
protogen.Plugin.SupportedFeatures
protoreflect.FieldDescriptor.HasPresence
protoreflect.FieldDescriptor.HasOptionalKeyword
protoreflect.OneofDescriptor.IsSynthetic
Change-Id: I7c86bf66d0ae56642109beb5f2132184593747ad
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/230698
Reviewed-by: Damien Neil <dneil@google.com>
Changes made:
* Ensure protoreflect.ExtensionType.IsValidInterface never panics,
especially if given a nil interface value.
* Have protoreflect.ExtensionType.IsValid{Interface,Value} only
perform type-checks. It does not do value checks (i.e., whether the
value itself is valid). Value validity is left to when an actual
protoreflect.Message.Set operation is performed.
* Add special-casing on proto.SetExtension to treat an invalid
message or list as functionally equivalent to Clear. This is to
be more consistent with the legacy SetExtension implementation
which never panicked when given such values.
* Add special-casing on proto.HasExtension to treat a mismatched
extension descriptor as simply not being present in the message.
This is also to be more consistent with the legacy HasExtension
implementation which did the same thing.
Change-Id: Idf0419abf27b9f85d9b92bd2ff8088e25b7990cc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/229558
Reviewed-by: Damien Neil <dneil@google.com>
This particular off-by-one can never happen in practice. We will crash
if a field has a number one greater than the maximum field number in the
dense array, but if field number N is in the array we will always put
N+1 in it as well.
Fix the off-by-one anyway.
Change-Id: I8c1304f2fc0d7b91036bde3f7ddb7115c21781ca
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/229278
Reviewed-by: Joe Tsai <joetsai@google.com>
Microbenchmarks are inconclusive/noisy, but shows a small but noticeable
improvement on internal benchmarks.
Change-Id: Ic46c6aac8a42c4dc749c4f3583d8c8c95e9548b7
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/229277
Reviewed-by: Joe Tsai <joetsai@google.com>
Two changes:
* Add RangeExtensions as a more suitable replacement for legacy
proto.ClearExtensions and proto.ExtensionDescs functions.
* Make HasExtension and GetExtension treat nil message interface
as an empty message to more consistently match legacy behavior.
Change-Id: I8eb1887a33d0737f2f80a2b80358cc296087ba3b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/229157
Reviewed-by: Damien Neil <dneil@google.com>
Expand the mention of the protodesc package to indicate that it can
construct protoreflect.Descriptors from the descriptor messages, but
also convert a protoreflect.Descriptor back to the message form.
Change-Id: I9f3d75c6050fff603655ee0a4e8e29d0a2c0cf84
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/221611
Reviewed-by: Damien Neil <dneil@google.com>
It is a sufficiently common pattern to do something like the following:
m.Proto2BytesField, ... = proto.Marshal(m2)
where the user is relying on Marshal to never return a nil byte slice,
otherwise it subtly changes the semantics of how the generated API
handles whether a proto2 "optional bytes" fields is populated or not.
Change-Id: Ie7508dbcdcc5f3295885609a91907c6eb4f04c1e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/228838
Reviewed-by: Damien Neil <dneil@google.com>
To assist users in migrating from github.com/golang/protobuf
to google.golang.org/protobuf, make it such that functiionality like
proto.Marshal doesn't panic on nil interfaces.
Similar to how the new implementation treats a typed nil message
as an empty message, we treat a nil interface as being equivalent
to an "untyped" empty message.
Change-Id: Ic037f386f855b122f732b34d370e524b7c0d76f1
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/228837
Reviewed-by: Damien Neil <dneil@google.com>
At some point in time, protoc-gen-go actually emitted an XXX_OneofFuncs
with a different signature. Adjust the logic for handling XXX_OneofFuncs
to not assume that the return arguments are in a specific order.
Change-Id: Idd9c09231c4129c655d4a635bb1ae094896a1ff4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/226980
Reviewed-by: Damien Neil <dneil@google.com>
Add a function that takes a protoreflect.FileDescriptorSet and returns
a protoregistry.Files.
Updates golang/protobuf##1065.
Change-Id: I2715d042053ef7d3f1bfcee1866c20cac423e327
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/226237
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The msgfmt formatter is intended to only for debugging purposes.
Remove unnecesary logic to detect out-of-range timestamps and durations.
Change-Id: I060149ed71aa892bbe4fdb2508b1d0b9df4b5f37
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/225258
Reviewed-by: Damien Neil <dneil@google.com>
These were originally needed by the legacy implementation,
but not anymore now that it fully wraps the new implementation.
Change-Id: I950958ebfcb7883fc4b72128d22eaba2da5cc62f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/224583
Reviewed-by: Damien Neil <dneil@google.com>
This is dead code as the logic has been moved to internal/msgfmt.
Change-Id: If65f443c9672f5fdfbb91c7cc64fb6f12a93c374
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/224658
Reviewed-by: Damien Neil <dneil@google.com>
Now we'll get a generator error if a file is missing a go_package
option.
Change-Id: I89eec716f86956e6c164a61f5531c140b74bc099
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/222378
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Add a generator option that strips a prefix from the generated
filenames.
Consider this case: We have google/protobuf/empty.proto, with a
go_package option of "google.golang.org/protobuf/types/known/emptypb".
We want to generate the code for this file, placing it into the
appropriate directory of our repository.
In the default mode used by the code generator (paths=import),
the generator outputs the file:
google.golang.org/protobuf/types/known/emptypb/empty.pb.go
This is close to what we want, but has an unnecessary
"google.golang.org/protobuf/" prefix. In the GOPATH world, we could pass
--go_out=$GOPATH to protoc and get a generated file in the desired
location, but this path is not useful in the modules world.
The 'module' option allows us to strip off the module prefix, generating
the desired filename (types/known/emptypb/empty.pb.go):
protoc --go_out=. --go_opt=module=google.golang.org/protobuf google/protobuf/empty.proto
The module name must be an exact, character-for-character match. This
matches protoc's file handling in general.
Default to and require the paths=import option when module= is
specified, since it only makes sense when combined with it.
Updates golang/protobuf#992.
Change-Id: Idbfe4826b6c0ece30d64dbc577131a4f16391936
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219298
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Changes the message Zero return type to be read-only by omitting internal known map.
Change-Id: I1c1191a125df74251be3d8bb70f4b06c1ff57070
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/223857
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Port message formatting logic in testing/protocmp to internal/msgfmt
and improve upon its output.
This formatter is optimized for humanly readable output.
It takes the best parts of both the JSON and proto text formats.
The good of prototext:
* It supports emitting unknown fields (useful for debugging).
* It is relatively concise in the common-case since keys do not
need to be represented as quoted strings (e.g., "key" vs key).
The bad of prototext:
* Requires relatively large dependency on encoding/prototext.
* Our implementation lacks support for serializing packed lists.
* Lacks support for readable maps.
* Lacks support for readable Timestamp and Duration.
The good of protojson:
* First-class support for readable maps.
* First-class support for readable Timestamp and Duration.
The bad of protojson:
* Requires relatively large dependency on encoding/protojson.
* Lacks support for emitting unknown fields.
* More verbose in the common-case as keys are quoted strings.
The msgfmt package has the benefits of both protojson and prototext,
but none of the detriments. It is a relatively simple implementation.
This output is only intended for human consumption with no associated
deserialization implementation.
To avoid any illusion that this is identical to either the proto text
or JSON formats, this always emits surrounding "{}" for
top-level messages and the keys are not quoted strings.
This CL does not use this format for generated Message.String methods
as there is concerns about being inconsistent with the String methods
as implemented in other languages. Having it be a seperate package makes
it trivial to switch over to this if desired.
Change-Id: I8b3581904d1624e84bf1b1954d2f01e5774b7f87
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/223752
Reviewed-by: Damien Neil <dneil@google.com>
This fixes the conformance test failures which occur because the
conformance protos have not been regenerated.
The generator script has been modified with a sanity check that
files do not exist outside the expected sub-tree.
Change-Id: I473efec4a016f6bc96ddf7e20d54bcf5ff9b55fe
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/223538
Reviewed-by: Damien Neil <dneil@google.com>
When using the M flags (which is functionally a mapping of
filenames to Go package paths), provide the user the ability to also
specify the package name by delimiting the package path with a ";".
Example usage:
Mpath/to/foo.proto=path/to/foo_go_proto;foopb
This uses the exact same syntax as the go_package option where a
";" delimiter can be used to specify a package name.
It brings the M flags and the go_package option closer in behavior.
Change-Id: I98e1fbb66ec2f1b70b4143b305355e5ab35ea198
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/223819
Reviewed-by: Damien Neil <dneil@google.com>
When combining multiple message fields in a MessageSet item (a case
which should never happen in practice), unmarshal could modify the input
data. Fix it to not do so. Add a general check to ensure that unmarshal
operations don't modify the input.
Change-Id: Idde46e6132a1dc96c374f9146efff81783c3bef3
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/223818
Reviewed-by: Joe Tsai <joetsai@google.com>
It's annoying for some users that they can't directly pass
*dynamicpb.Message to APIs that expect the legacy message interfaces.
The proto.MessageV1 wrappers work, but is somewhat suspect since they
were originally designed to wrap legacy generated messages.
Change-Id: I0f9900dcd1c9865c754551f8763680c9bb904813
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/223817
Reviewed-by: Damien Neil <dneil@google.com>
SortRepeated is similar to cmpopts.SortSlice where it accepts a
user-provided sort function, but only operates on repeated fields.
It pattern matches based on sort element type.
SortRepeatedFields is similar to SortRepeated, but chooses an
arbitrary sort order for the specified (by name) repeated fields.
It pattern matches based on message field name.
Change-Id: Ib6ef282e5394cf7b22522161d524f22e1b76677a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/221432
Reviewed-by: Damien Neil <dneil@google.com>
In the default .pb.go filename generation mode, we generate the filename
from the import path when a file has a go_package option and the source
.proto filename otherwise.
Change filename generation when an explicit mode of paths=import is
specified to always use the import path, no matter how it was
determiend. The practical effect is that you can override the import
path of a file with Mx.proto=import/path and have this behave
identically to setting a go_package option in the file.
Change-Id: I954b3f9d5fd17d08896629bfc77945dea1732bd6
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219597
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
This is no longer needed by the old implementation.
Change-Id: I3ba02d37f35f599ec790ec4e627258273883a308
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/223279
Reviewed-by: Damien Neil <dneil@google.com>
This introduces some conformance test failures,
which occur not because our implementation changed behavior,
but because new cases were added.
Future work will be to investigate these failuress.
Change-Id: Ifb17465883c417acd46865744572f8cd0c858383
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/222857
Reviewed-by: Damien Neil <dneil@google.com>
