10 Commits

Author SHA1 Message Date
Singee
70db1e1de2 encoding/protojson: ignore unknown enum name if DiscardUnknown=true
This makes encoding/protojson pass all protobuf conformance tests.

Other Similar CL:
  - https://go-review.googlesource.com/c/protobuf/+/350469
  - https://go-review.googlesource.com/c/protobuf/+/256677

Fixes golang/protobuf#1208

Change-Id: I9f74162b80a3c7ee4750160fc59f0313345046de
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/525535
Reviewed-by: Cassondra Foesch <cfoesch@gmail.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2023-09-08 10:59:01 +00:00
Josh Humphries
bc1253ad37 types/descriptorpb: regenerate using latest protobuf v22.0 release
This updates all generated code to match the contents of the latest
v22.0 release of Protobuf.

This involved a couple of changes to the script that does the sync'ing:
  1. The new Protobuf version no longer includes autoconf configuration
     and instead requires using bazel to build things.
  2. The new Protobuf release does not have an artifact named
     "protobuf-all-${VERSION}.tar.gz", but the one named
     "protobuf-${VERSION}.tar.gz" has all of the sources and was
     sufficient for the regenerate.bash script to complete.

This change does NOT regenerate the protos related to benchmarks.
The Protobuf repo no longer includes benchmarks. The CL removing them
says they are superceded by google/fleetbench:
    83c499de86
But that project's proto benchmark files are very different:
    https://github.com/google/fleetbench/tree/main/fleetbench/proto
So I commented out those steps in the generation code since the benchmarks
will need some work to reconcile with fleetbench.

This code adds known failing tests for conformance. New test cases were
added in https://github.com/protocolbuffers/protobuf/pull/9534, but the
Go protojson package does not behave according to the new tests. There
is an existing issue in GitHub about this:
    https://github.com/golang/protobuf/issues/1208

Change-Id: Iad796ec7889bc2a74b58db5224facf850cd1a1cd
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/469255
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
2023-02-22 09:33:03 +00:00
Joe Tsai
85f47621e0 all: update protobuf toolchain dependency
This updates the dependency on the protobuf toolchain to an unreleased
version that has the go_package option for relevant .proto files in
that repository to reference this module.

Change-Id: Ie1ac8f81a323285efbd14a17d02c7105e810af2d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/235283
Trust: Joe Tsai <joetsai@digital-static.net>
Reviewed-by: Damien Neil <dneil@google.com>
2021-03-03 01:59:11 +00:00
cybrcodr
beaa55256c encoding/protojson: add better validation to FieldMask serialization
For marshaling, apart from already existing check that each item in
paths field is reversible, also make sure that string is a valid
protobuf name.

For unmarshaling, make sure that each resulting item in paths field is
a valid protobuf name and input path item does not contain _. The latter
check satisfies the conformance test
Recommended.Proto3.JsonInput.FieldMaskInvalidCharacter.

Fixes golang/protobuf#1141.

Change-Id: Iffc278089b20e496b7216d5b8c966b21b70e782d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/236998
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2020-06-12 06:33:55 +00:00
Joe Tsai
b738ac9285 all: regenerate remote .proto files
This fixes the conformance test failures which occur because the
conformance protos have not been regenerated.
The generator script has been modified with a sanity check that
files do not exist outside the expected sub-tree.

Change-Id: I473efec4a016f6bc96ddf7e20d54bcf5ff9b55fe
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/223538
Reviewed-by: Damien Neil <dneil@google.com>
2020-03-18 20:17:11 +00:00
Joe Tsai
f92988f900 all: upgrade protobuf to v3.11.4
This introduces some conformance test failures,
which occur not because our implementation changed behavior,
but because new cases were added.

Future work will be to investigate these failuress.

Change-Id: Ifb17465883c417acd46865744572f8cd0c858383
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/222857
Reviewed-by: Damien Neil <dneil@google.com>
2020-03-13 19:49:20 +00:00
Herbie Ong
9b3d97c473 encoding/prototext: rewrite of internal/encoding/text
* Fixes golang/protobuf#842. Unmarshal can now parse singular or
  repeated message fields without the field separator.
* Fixes golang/protobuf#1011. Handles negative 0 properly.
* For unknown fields with fixed 32-bit and 64-bit wire types, output is
  now in hex format with 0x prefix similar to C++ lib output. Previous
  Go implementation simply outputs these as decimal numbers %d.
* All parsing errors, except for unexpected EOF should now contain line
  and column number info.
* Fixed following conformance-related features:
  * Parse nan,inf,-inf,infinity,-infinity as case-insensitive.
  * Interpret float32 overflows as inf or -inf.
  * Parse large int-like number as proto float.
* Discard unknown map field if DiscardUnknown=true.
* Allow whitespaces/comments in Any type URL and extension field names per spec.
* Improves performance and memory usage. It is now as fast and efficient as
  protojson, if not better on most benchmarks.

name                                     old time/op    new time/op    delta
Text/Unmarshal/google_message1_proto2-4    14.1µs ±43%     8.7µs ±12%  -38.27%  (p=0.000 n=10+10)
Text/Unmarshal/google_message1_proto3-4    11.6µs ±18%     7.7µs ± 9%  -33.69%  (p=0.000 n=10+10)
Text/Unmarshal/google_message2-4           6.20ms ±27%    4.10ms ± 5%  -33.95%  (p=0.000 n=10+10)
Text/Marshal/google_message1_proto2-4      12.8µs ± 6%    10.3µs ±23%  -19.54%  (p=0.000 n=9+10)
Text/Marshal/google_message1_proto3-4      11.9µs ±16%     8.6µs ±10%  -27.45%  (p=0.000 n=10+10)
Text/Marshal/google_message2-4             5.59ms ± 5%    5.30ms ±22%     ~     (p=0.356 n=9+10)
JSON/Unmarshal/google_message1_proto2-4    12.3µs ±61%    13.9µs ±26%     ~     (p=0.190 n=10+10)
JSON/Unmarshal/google_message1_proto3-4    7.51µs ± 6%    7.86µs ± 1%   +4.66%  (p=0.010 n=10+9)
JSON/Unmarshal/google_message2-4           3.74ms ± 2%    3.94ms ± 2%   +5.32%  (p=0.000 n=10+10)
JSON/Marshal/google_message1_proto2-4      9.90µs ±12%    9.95µs ± 4%     ~     (p=0.315 n=9+10)
JSON/Marshal/google_message1_proto3-4      7.55µs ± 4%    7.93µs ± 3%   +4.98%  (p=0.000 n=10+10)
JSON/Marshal/google_message2-4             4.29ms ± 5%    4.49ms ± 2%   +4.53%  (p=0.001 n=10+10)

name                                     old alloc/op   new alloc/op   delta
Text/Unmarshal/google_message1_proto2-4    12.5kB ± 0%     2.0kB ± 0%  -83.87%  (p=0.000 n=10+10)
Text/Unmarshal/google_message1_proto3-4    12.2kB ± 0%     1.8kB ± 0%  -85.33%  (p=0.000 n=10+10)
Text/Unmarshal/google_message2-4           5.35MB ± 0%    0.89MB ± 0%  -83.28%  (p=0.000 n=10+9)
Text/Marshal/google_message1_proto2-4      12.0kB ± 0%     1.4kB ± 0%  -88.15%  (p=0.000 n=10+10)
Text/Marshal/google_message1_proto3-4      12.4kB ± 0%     1.9kB ± 0%  -84.91%  (p=0.000 n=10+10)
Text/Marshal/google_message2-4             5.64MB ± 0%    1.02MB ± 0%  -81.85%  (p=0.000 n=10+9)
JSON/Unmarshal/google_message1_proto2-4    2.29kB ± 0%    2.29kB ± 0%     ~     (all equal)
JSON/Unmarshal/google_message1_proto3-4    2.08kB ± 0%    2.08kB ± 0%     ~     (all equal)
JSON/Unmarshal/google_message2-4            899kB ± 0%     899kB ± 0%     ~     (p=1.000 n=10+10)
JSON/Marshal/google_message1_proto2-4      1.46kB ± 0%    1.46kB ± 0%     ~     (all equal)
JSON/Marshal/google_message1_proto3-4      1.36kB ± 0%    1.36kB ± 0%     ~     (all equal)
JSON/Marshal/google_message2-4             1.19MB ± 0%    1.19MB ± 0%     ~     (p=0.197 n=10+10)

name                                     old allocs/op  new allocs/op  delta
Text/Unmarshal/google_message1_proto2-4       133 ± 0%        89 ± 0%  -33.08%  (p=0.000 n=10+10)
Text/Unmarshal/google_message1_proto3-4       108 ± 0%        67 ± 0%  -37.96%  (p=0.000 n=10+10)
Text/Unmarshal/google_message2-4            60.0k ± 0%     38.7k ± 0%  -35.52%  (p=0.000 n=10+10)
Text/Marshal/google_message1_proto2-4        65.0 ± 0%      25.0 ± 0%  -61.54%  (p=0.000 n=10+10)
Text/Marshal/google_message1_proto3-4        59.0 ± 0%      22.0 ± 0%  -62.71%  (p=0.000 n=10+10)
Text/Marshal/google_message2-4              27.4k ± 0%      7.3k ± 0%  -73.39%  (p=0.000 n=10+10)
JSON/Unmarshal/google_message1_proto2-4      95.0 ± 0%      95.0 ± 0%     ~     (all equal)
JSON/Unmarshal/google_message1_proto3-4      74.0 ± 0%      74.0 ± 0%     ~     (all equal)
JSON/Unmarshal/google_message2-4            36.3k ± 0%     36.3k ± 0%     ~     (all equal)
JSON/Marshal/google_message1_proto2-4        27.0 ± 0%      27.0 ± 0%     ~     (all equal)
JSON/Marshal/google_message1_proto3-4        30.0 ± 0%      30.0 ± 0%     ~     (all equal)
JSON/Marshal/google_message2-4              11.3k ± 0%     11.3k ± 0%     ~     (p=1.000 n=10+10)

Change-Id: I377925facde5535f06333b6f25e9c9b358dc062f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/204602
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2020-02-05 02:11:08 +00:00
Joe Tsai
cd4a31e202 encoding/prototext: add MarshalOptions.EmitUnknown
This changes text marshaling to avoid unknown fields by default
and instead adds an option so that unknown fields be emitted.
This ensures that the default marshal/unknown can round-trip.

Change-Id: I85c84ba6ab7916d538ec6bfd4e9d399a8fcba14e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/195778
Reviewed-by: Herbie Ong <herbie@google.com>
2019-09-17 02:56:29 +00:00
Damien Neil
1de09cbe21 internal/conformance: change test package name to conformance_test
Since there's no conformance package aside from the test, it's cleaner
to keep the test in the conformance_test package.

Change-Id: If9ccfdfc074fa4f390956dd17d066b8613911147
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/190981
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-20 22:30:23 +00:00
Joe Tsai
dd271b6b63 internal/conformance: make conformance test use specific go version
The use of internal/cmd/conformance/conformance.sh means that the test
is not hermetic since the script uses the system version of go,
rather than the specific versions chosen by integration_test.go.

Change the way the conformance test is invoked by turning it
into a Go unit test that integration_test.go can directly call.

Change-Id: I37d23341e1eda984f23f78757a38e862e5fac3d4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/190518
Reviewed-by: Herbie Ong <herbie@google.com>
2019-08-16 22:09:14 +00:00