mirror/protobuf-go
1
0
Fork 0
mirror of https://github.com/protocolbuffers/protobuf-go.git synced 2025-02-06 00:40:02 +00:00
Code Issues Packages Projects Releases Wiki Activity
protobuf-go/proto/decode_test.go

1822 lines
63 KiB
Go
Raw Normal View History

proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
// Copyright 2018 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style.
// license that can be found in the LICENSE file.
internal/legacy: use v2 proto.Unmarshal The v2 decoder isn't 100% complete, but it's good enough. Delete the vendored copy of the v1 Unmarshal implementation. Change-Id: Ibeabbb2e9109a1ec3df57e71f98b7aa4a583fc5b Reviewed-on: https://go-review.googlesource.com/c/154577 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-17 11:16:16 -08:00
package proto_test
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
import (
"fmt"
"reflect"
"testing"
protogen, encoding/jsonpb, encoding/textpb: rename packages Rename encoding/*pb to follow the convention of prefixing package names with 'proto': google.golang.org/protobuf/encoding/protojson google.golang.org/protobuf/encoding/prototext Move protogen under a compiler/ directory, just in case we ever do add more compiler-related packages. google.golang.org/protobuf/compiler/protogen Change-Id: I31010cb5cabcea8274fffcac468477b58b56e8eb Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/177178 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-05-14 12:44:37 -07:00
"google.golang.org/protobuf/encoding/prototext"
all: change module to google.golang.org/protobuf Temporarily remove go.mod, since we can't generate an accurate one until the corresponding v1 change is submitted. Change-Id: I1e1ad97f2b455e33f61ffaeb8676289795e47e72 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/177000 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-05-13 23:55:40 -07:00
"google.golang.org/protobuf/internal/encoding/pack"
all: support enforce_utf8 override In 2014, when proto3 was being developed, there were a number of early adopters of the new syntax. Before the finalization of proto3 when it was released in open-source in July 2016, a decision was made to strictly validate strings in proto3. However, some of the early adopters were already using invalid UTF-8 with string fields. The google.protobuf.FieldOptions.enforce_utf8 option only exists to support those grandfathered users where they can opt-out of the validation logic. Practical use of that option in open source is impossible even if a user specifies the proto1_legacy build tag since it requires a hacked variant of descriptor.proto that is not externally available. This CL supports enforce_utf8 by modifiyng internal/filedesc to expose the flag if it detects it in the raw descriptor. We add an strs.EnforceUTF8 function as a centralized place to determine whether to perform validation. Validation opt-out is supported only in builds with legacy support. We implement support for validating UTF-8 in all proto3 string fields, even if they are backed by a Go []byte. Change-Id: I9c0628b84909bc7181125f09db730c80d490e485 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186002 Reviewed-by: Damien Neil <dneil@google.com>
2019-07-13 00:44:41 -07:00
"google.golang.org/protobuf/internal/filedesc"
"google.golang.org/protobuf/internal/flags"
all: change module to google.golang.org/protobuf Temporarily remove go.mod, since we can't generate an accurate one until the corresponding v1 change is submitted. Change-Id: I1e1ad97f2b455e33f61ffaeb8676289795e47e72 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/177000 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-05-13 23:55:40 -07:00
"google.golang.org/protobuf/proto"
all: support enforce_utf8 override In 2014, when proto3 was being developed, there were a number of early adopters of the new syntax. Before the finalization of proto3 when it was released in open-source in July 2016, a decision was made to strictly validate strings in proto3. However, some of the early adopters were already using invalid UTF-8 with string fields. The google.protobuf.FieldOptions.enforce_utf8 option only exists to support those grandfathered users where they can opt-out of the validation logic. Practical use of that option in open source is impossible even if a user specifies the proto1_legacy build tag since it requires a hacked variant of descriptor.proto that is not externally available. This CL supports enforce_utf8 by modifiyng internal/filedesc to expose the flag if it detects it in the raw descriptor. We add an strs.EnforceUTF8 function as a centralized place to determine whether to perform validation. Validation opt-out is supported only in builds with legacy support. We implement support for validating UTF-8 in all proto3 string fields, even if they are backed by a Go []byte. Change-Id: I9c0628b84909bc7181125f09db730c80d490e485 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186002 Reviewed-by: Damien Neil <dneil@google.com>
2019-07-13 00:44:41 -07:00
"google.golang.org/protobuf/reflect/protodesc"
"google.golang.org/protobuf/reflect/protoreflect"
"google.golang.org/protobuf/runtime/protoimpl"
internal/cmd/generate-protos: initial commit Create a single binary for handling generation of protos. This replaces previous logic spread throughout the repo in: * regenerate.bash * cmd/protoc-gen-go/golden_test.go * cmd/protoc-gen-go-grpc/golden_test.go * (indirectly) internal/protogen/goldentest One of the problems with the former approaches is that they relied on a version of protoc that was specific to a developer's workstation. This meant that the result of generation was not hermetic. To address this, we rely on the hard-coded version of protobuf specified in the test.bash script. A summary of changes in this CL are: * The internal_gengo.GenerateFile and internal_gengogrpc.GenerateFile functions are unified to have consistent signatures. It seems that the former accepted a *protogen.GeneratedFile to support v1 where gRPC code was generated into the same file as the base .pb.go file. However, the same functionality can be achieved by having the function return the generated file object. * The test.bash script patches the protobuf toolchain to have properly specified go_package options in each proto source file. * The test.bash script accepts a "-regenerate" argument. * Add generation for the well-known types. Contrary to how these were laid out in the v1 repo, all the well-known types are placed in the same Go package. * Add generation for the conformance proto. * Remove regenerate.bash * Remove internal/protogen * Remove cmd/protoc-gen-go/golden_test.go * Remove cmd/protoc-gen-go-grpc/golden_test.go * Add cmd/protoc-gen-go/annotation_test.go Change-Id: I4a1a97ae6f66e2fabcf4e4d292c95ab2a2db0248 Reviewed-on: https://go-review.googlesource.com/c/164477 Reviewed-by: Damien Neil <dneil@google.com>
2019-02-27 21:46:29 -08:00
internal/impl: add fast-path marshal implementation This is a port of the v1 table marshaler, with some substantial cleanup and refactoring. Benchstat results from the protobuf reference benchmark data comparing the v1 package with v2, with AllowPartial:true set for the new package. This is not an apples-to-apples comparison, since v1 doesn't have a way to disable required field checks. Required field checks in v2 package currently go through reflection, which performs terribly; my initial experimentation indicates that fast-path required field checks will not add a large amount of cost; these results are incomplete but not wholly inaccurate. name old time/op new time/op delta /dataset.google_message3_1.pb/Marshal-12 219ms ± 1% 232ms ± 1% +5.85% (p=0.004 n=6+5) /dataset.google_message2.pb/Marshal-12 261µs ± 3% 248µs ± 1% -5.14% (p=0.002 n=6+6) /dataset.google_message1_proto2.pb/Marshal-12 681ns ± 2% 637ns ± 3% -6.53% (p=0.002 n=6+6) /dataset.google_message1_proto3.pb/Marshal-12 1.10µs ± 8% 0.99µs ± 3% -9.63% (p=0.002 n=6+6) /dataset.google_message3_3.pb/Marshal-12 44.2ms ± 3% 35.2ms ± 1% -20.28% (p=0.004 n=6+5) /dataset.google_message4.pb/Marshal-12 91.4ms ± 2% 94.9ms ± 2% +3.78% (p=0.002 n=6+6) /dataset.google_message3_2.pb/Marshal-12 78.7ms ± 6% 80.8ms ± 4% ~ (p=0.310 n=6+6) /dataset.google_message3_4.pb/Marshal-12 10.6ms ± 3% 10.6ms ± 8% ~ (p=0.662 n=5+6) /dataset.google_message3_5.pb/Marshal-12 675ms ± 4% 510ms ± 2% -24.40% (p=0.002 n=6+6) /dataset.google_message3_1.pb/Marshal 219ms ± 1% 236ms ± 7% +8.06% (p=0.004 n=5+6) /dataset.google_message2.pb/Marshal 257µs ± 1% 250µs ± 3% ~ (p=0.052 n=5+6) /dataset.google_message1_proto2.pb/Marshal 685ns ± 1% 628ns ± 1% -8.41% (p=0.008 n=5+5) /dataset.google_message1_proto3.pb/Marshal 1.08µs ± 1% 0.98µs ± 2% -9.31% (p=0.004 n=5+6) /dataset.google_message3_3.pb/Marshal 43.7ms ± 1% 35.1ms ± 1% -19.76% (p=0.002 n=6+6) /dataset.google_message4.pb/Marshal 93.4ms ± 4% 94.9ms ± 2% ~ (p=0.180 n=6+6) /dataset.google_message3_2.pb/Marshal 105ms ± 2% 98ms ± 7% -6.81% (p=0.009 n=5+6) /dataset.google_message3_4.pb/Marshal 16.3ms ± 6% 15.7ms ± 3% -3.44% (p=0.041 n=6+6) /dataset.google_message3_5.pb/Marshal 676ms ± 4% 504ms ± 2% -25.50% (p=0.004 n=6+5) Change-Id: I72cc4597117f4cf5d236ef505777d49dd4a5f75d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171020 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-04-01 13:49:56 -07:00
legacypb "google.golang.org/protobuf/internal/testprotos/legacy"
legacy1pb "google.golang.org/protobuf/internal/testprotos/legacy/proto2.v0.0.0-20160225-2fc053c5"
all: change module to google.golang.org/protobuf Temporarily remove go.mod, since we can't generate an accurate one until the corresponding v1 change is submitted. Change-Id: I1e1ad97f2b455e33f61ffaeb8676289795e47e72 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/177000 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-05-13 23:55:40 -07:00
testpb "google.golang.org/protobuf/internal/testprotos/test"
test3pb "google.golang.org/protobuf/internal/testprotos/test3"
all: support enforce_utf8 override In 2014, when proto3 was being developed, there were a number of early adopters of the new syntax. Before the finalization of proto3 when it was released in open-source in July 2016, a decision was made to strictly validate strings in proto3. However, some of the early adopters were already using invalid UTF-8 with string fields. The google.protobuf.FieldOptions.enforce_utf8 option only exists to support those grandfathered users where they can opt-out of the validation logic. Practical use of that option in open source is impossible even if a user specifies the proto1_legacy build tag since it requires a hacked variant of descriptor.proto that is not externally available. This CL supports enforce_utf8 by modifiyng internal/filedesc to expose the flag if it detects it in the raw descriptor. We add an strs.EnforceUTF8 function as a centralized place to determine whether to perform validation. Validation opt-out is supported only in builds with legacy support. We implement support for validating UTF-8 in all proto3 string fields, even if they are backed by a Go []byte. Change-Id: I9c0628b84909bc7181125f09db730c80d490e485 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186002 Reviewed-by: Damien Neil <dneil@google.com>
2019-07-13 00:44:41 -07:00
"google.golang.org/protobuf/types/descriptorpb"
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
)
type testProto struct {
proto: cleanup invalid extension exception The invalidExtensions flag no longer seems necessary. Tests pass without it. Change-Id: Ieb35e26912b047718ccbfcdc926625aec1cd8c87 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185937 Reviewed-by: Damien Neil <dneil@google.com>
2019-07-11 22:13:49 -07:00
desc string
decodeTo []proto.Message
wire []byte
partial bool
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
}
func TestDecode(t *testing.T) {
for _, test := range testProtos {
for _, want := range test.decodeTo {
t.Run(fmt.Sprintf("%s (%T)", test.desc, want), func(t *testing.T) {
proto: check for required fields in encoding/decoding Change-Id: I0555a92e0399782f075b1dcd248e880dd48c7d6d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170579 Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-03 12:17:24 -07:00
opts := proto.UnmarshalOptions{
AllowPartial: test.partial,
}
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
wire := append(([]byte)(nil), test.wire...)
internal/legacy: use v2 proto.Unmarshal The v2 decoder isn't 100% complete, but it's good enough. Delete the vendored copy of the v1 Unmarshal implementation. Change-Id: Ibeabbb2e9109a1ec3df57e71f98b7aa4a583fc5b Reviewed-on: https://go-review.googlesource.com/c/154577 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-17 11:16:16 -08:00
got := reflect.New(reflect.TypeOf(want).Elem()).Interface().(proto.Message)
proto: check for required fields in encoding/decoding Change-Id: I0555a92e0399782f075b1dcd248e880dd48c7d6d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170579 Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-03 12:17:24 -07:00
if err := opts.Unmarshal(wire, got); err != nil {
proto: add generic Size Change-Id: I4ed123f4a9747fb4aba392bc5b9608d294bacc4d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169697 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-27 09:23:20 -07:00
t.Errorf("Unmarshal error: %v\nMessage:\n%v", err, marshalText(want))
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
return
}
// Aliasing check: Modifying the original wire bytes shouldn't
// affect the unmarshaled message.
for i := range wire {
wire[i] = 0
}
proto: eagerly unmarshal extensions CL/172399 switches the v1 code to eagerly unmarshal extensions. This CL does the equivalent for v2. For the test, we simply switch from protoV1.Equal to protoV2.Equal, since the v2 equal does not magically unmarshal raw extensions. Change-Id: I6f64455b0a75bbc9a9a82108558641a29bd2b982 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/175838 Reviewed-by: Damien Neil <dneil@google.com>
2019-05-07 15:14:40 -07:00
if !proto.Equal(got, want) {
proto: add generic Size Change-Id: I4ed123f4a9747fb4aba392bc5b9608d294bacc4d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169697 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-27 09:23:20 -07:00
t.Errorf("Unmarshal returned unexpected result; got:\n%v\nwant:\n%v", marshalText(got), marshalText(want))
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
}
})
}
}
}
proto: check for required fields in encoding/decoding Change-Id: I0555a92e0399782f075b1dcd248e880dd48c7d6d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170579 Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-03 12:17:24 -07:00
func TestDecodeRequiredFieldChecks(t *testing.T) {
for _, test := range testProtos {
if !test.partial {
continue
}
for _, m := range test.decodeTo {
t.Run(fmt.Sprintf("%s (%T)", test.desc, m), func(t *testing.T) {
got := reflect.New(reflect.TypeOf(m).Elem()).Interface().(proto.Message)
if err := proto.Unmarshal(test.wire, got); err == nil {
t.Fatalf("Unmarshal succeeded (want error)\nMessage:\n%v", marshalText(got))
}
})
}
}
}
proto: validate UTF-8 in proto3 strings Change-Id: I6a495730c3f438e7b2c4ca86edade7d6f25aa47d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171700 Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-11 11:46:55 -07:00
func TestDecodeInvalidUTF8(t *testing.T) {
for _, test := range invalidUTF8TestProtos {
for _, want := range test.decodeTo {
t.Run(fmt.Sprintf("%s (%T)", test.desc, want), func(t *testing.T) {
got := reflect.New(reflect.TypeOf(want).Elem()).Interface().(proto.Message)
err := proto.Unmarshal(test.wire, got)
all: remove non-fatal UTF-8 validation errors (and non-fatal in general) Immediately abort (un)marshal operations when encountering invalid UTF-8 data in proto3 strings. No other proto implementation supports non-UTF-8 data in proto3 strings (and many reject it in proto2 strings as well). Producing invalid output is an interoperability threat (other implementations won't be able to read it). The case where existing string data is found to contain non-UTF8 data is better handled by changing the field to the `bytes` type, which (aside from UTF-8 validation) is wire-compatible with `string`. Remove the errors.NonFatal type, since there are no remaining cases where it is needed. "Non-fatal" errors which produce results and a non-nil error are problematic because they compose poorly; the better approach is to take an option like AllowPartial indicating which conditions to check for. Change-Id: I9d189ec6ffda7b5d96d094aa1b290af2e3f23736 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/183098 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-06-19 09:28:29 -07:00
if err == nil {
proto: validate UTF-8 in proto3 strings Change-Id: I6a495730c3f438e7b2c4ca86edade7d6f25aa47d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171700 Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-11 11:46:55 -07:00
t.Errorf("Unmarshal did not return expected error for invalid UTF8: %v\nMessage:\n%v", err, marshalText(want))
}
})
}
}
}
all: support enforce_utf8 override In 2014, when proto3 was being developed, there were a number of early adopters of the new syntax. Before the finalization of proto3 when it was released in open-source in July 2016, a decision was made to strictly validate strings in proto3. However, some of the early adopters were already using invalid UTF-8 with string fields. The google.protobuf.FieldOptions.enforce_utf8 option only exists to support those grandfathered users where they can opt-out of the validation logic. Practical use of that option in open source is impossible even if a user specifies the proto1_legacy build tag since it requires a hacked variant of descriptor.proto that is not externally available. This CL supports enforce_utf8 by modifiyng internal/filedesc to expose the flag if it detects it in the raw descriptor. We add an strs.EnforceUTF8 function as a centralized place to determine whether to perform validation. Validation opt-out is supported only in builds with legacy support. We implement support for validating UTF-8 in all proto3 string fields, even if they are backed by a Go []byte. Change-Id: I9c0628b84909bc7181125f09db730c80d490e485 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186002 Reviewed-by: Damien Neil <dneil@google.com>
2019-07-13 00:44:41 -07:00
func TestDecodeNoEnforceUTF8(t *testing.T) {
for _, test := range noEnforceUTF8TestProtos {
for _, want := range test.decodeTo {
t.Run(fmt.Sprintf("%s (%T)", test.desc, want), func(t *testing.T) {
got := reflect.New(reflect.TypeOf(want).Elem()).Interface().(proto.Message)
err := proto.Unmarshal(test.wire, got)
switch {
all: rename tag and flags for legacy support Rename build tag "proto1_legacy" -> "protolegacy" to be consistent with the "protoreflect" tag. Rename flag constant "Proto1Legacy" -> "ProtoLegacy" since it covers more than simply proto1 legacy features. For example, it covers alpha-features of proto3 that were eventually removed from the final proto3 release. Change-Id: I0f4fcbadd4b5a61c87645e2e5be11d187e59157c Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189345 Reviewed-by: Damien Neil <dneil@google.com>
2019-08-08 13:31:59 -07:00
case flags.ProtoLegacy && err != nil:
all: support enforce_utf8 override In 2014, when proto3 was being developed, there were a number of early adopters of the new syntax. Before the finalization of proto3 when it was released in open-source in July 2016, a decision was made to strictly validate strings in proto3. However, some of the early adopters were already using invalid UTF-8 with string fields. The google.protobuf.FieldOptions.enforce_utf8 option only exists to support those grandfathered users where they can opt-out of the validation logic. Practical use of that option in open source is impossible even if a user specifies the proto1_legacy build tag since it requires a hacked variant of descriptor.proto that is not externally available. This CL supports enforce_utf8 by modifiyng internal/filedesc to expose the flag if it detects it in the raw descriptor. We add an strs.EnforceUTF8 function as a centralized place to determine whether to perform validation. Validation opt-out is supported only in builds with legacy support. We implement support for validating UTF-8 in all proto3 string fields, even if they are backed by a Go []byte. Change-Id: I9c0628b84909bc7181125f09db730c80d490e485 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186002 Reviewed-by: Damien Neil <dneil@google.com>
2019-07-13 00:44:41 -07:00
t.Errorf("Unmarshal returned unexpected error: %v\nMessage:\n%v", err, marshalText(want))
all: rename tag and flags for legacy support Rename build tag "proto1_legacy" -> "protolegacy" to be consistent with the "protoreflect" tag. Rename flag constant "Proto1Legacy" -> "ProtoLegacy" since it covers more than simply proto1 legacy features. For example, it covers alpha-features of proto3 that were eventually removed from the final proto3 release. Change-Id: I0f4fcbadd4b5a61c87645e2e5be11d187e59157c Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189345 Reviewed-by: Damien Neil <dneil@google.com>
2019-08-08 13:31:59 -07:00
case !flags.ProtoLegacy && err == nil:
all: support enforce_utf8 override In 2014, when proto3 was being developed, there were a number of early adopters of the new syntax. Before the finalization of proto3 when it was released in open-source in July 2016, a decision was made to strictly validate strings in proto3. However, some of the early adopters were already using invalid UTF-8 with string fields. The google.protobuf.FieldOptions.enforce_utf8 option only exists to support those grandfathered users where they can opt-out of the validation logic. Practical use of that option in open source is impossible even if a user specifies the proto1_legacy build tag since it requires a hacked variant of descriptor.proto that is not externally available. This CL supports enforce_utf8 by modifiyng internal/filedesc to expose the flag if it detects it in the raw descriptor. We add an strs.EnforceUTF8 function as a centralized place to determine whether to perform validation. Validation opt-out is supported only in builds with legacy support. We implement support for validating UTF-8 in all proto3 string fields, even if they are backed by a Go []byte. Change-Id: I9c0628b84909bc7181125f09db730c80d490e485 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186002 Reviewed-by: Damien Neil <dneil@google.com>
2019-07-13 00:44:41 -07:00
t.Errorf("Unmarshal did not return expected error for invalid UTF8: %v\nMessage:\n%v", err, marshalText(want))
}
})
}
}
}
proto, internal/impl: zero-length proto2 bytes fields should be non-nil Fix decoding of zero-length bytes fields to produce a non-nil []byte. Change-Id: Ifb7791a47df81091700f7226523371d1386fb1ad Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/188765 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-02 15:13:00 -07:00
func TestDecodeZeroLengthBytes(t *testing.T) {
// Verify that proto3 bytes fields don't give the mistaken
// impression that they preserve presence.
wire := pack.Message{
pack.Tag{15, pack.BytesType}, pack.Bytes(nil),
}.Marshal()
m := &test3pb.TestAllTypes{}
if err := proto.Unmarshal(wire, m); err != nil {
t.Fatal(err)
}
if m.OptionalBytes != nil {
t.Errorf("unmarshal zero-length proto3 bytes field: got %v, want nil", m.OptionalBytes)
}
}
internal/impl: treat a nil oneof wrapper as if it were unset The old implementation had the behavior where a nil wrapper value: m := new(foopb.Message) m.OneofField = (*foopb.Message_OneofUint32)(nil) was functionally equivalent to it being directly set to nil: m := new(foopb.Message) m.OneofField = nil preserve this semantic in both the table-drive implementation and the reflection implementation. Change-Id: Ie44d51e044d4822e61d0e646fbc44aa8d9b90c1f Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189559 Reviewed-by: Herbie Ong <herbie@google.com>
2019-08-08 19:23:32 -07:00
func TestDecodeOneofNilWrapper(t *testing.T) {
wire := pack.Message{
pack.Tag{111, pack.VarintType}, pack.Varint(1111),
}.Marshal()
m := &testpb.TestAllTypes{OneofField: (*testpb.TestAllTypes_OneofUint32)(nil)}
if err := proto.Unmarshal(wire, m); err != nil {
t.Fatal(err)
}
if got := m.GetOneofUint32(); got != 1111 {
t.Errorf("GetOneofUint32() = %v, want %v", got, 1111)
}
}
proto: add tests for unmarshalling invalid field numbers This change adds tests for unmarshalling fields with various invalid field numbers. Our current behavior is that proto.Unmarshal will return an error when it sees zero and larger than max field numbers and return nil for reserved ones, which matches the C++ behavior. (Note: depending on which parser helper in the C++ implementation, one may need to call additional method to check the result, which we don't have in Go) Change-Id: I8791fd077f25656107556f5606d55d05c1b4a120 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/191459 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-22 18:52:43 -07:00
func TestDecodeInvalidFieldNumbers(t *testing.T) {
for _, test := range invalidFieldNumberTestProtos {
t.Run(test.desc, func(t *testing.T) {
decoded := new(testpb.TestAllTypes) // type doesn't matter since we expect errors
err := proto.Unmarshal(test.wire, decoded)
if err == nil && !test.allowed {
t.Error("unmarshal: got nil want error")
} else if err != nil && test.allowed {
t.Errorf("unmarshal: got %v want nil since %s is allowed by Unmarshal", err, test.desc)
}
})
}
}
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
var testProtos = []testProto{
{
desc: "basic scalar types",
internal/legacy: use v2 proto.Unmarshal The v2 decoder isn't 100% complete, but it's good enough. Delete the vendored copy of the v1 Unmarshal implementation. Change-Id: Ibeabbb2e9109a1ec3df57e71f98b7aa4a583fc5b Reviewed-on: https://go-review.googlesource.com/c/154577 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-17 11:16:16 -08:00
decodeTo: []proto.Message{&testpb.TestAllTypes{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
OptionalInt32: proto.Int32(1001),
OptionalInt64: proto.Int64(1002),
OptionalUint32: proto.Uint32(1003),
OptionalUint64: proto.Uint64(1004),
OptionalSint32: proto.Int32(1005),
OptionalSint64: proto.Int64(1006),
OptionalFixed32: proto.Uint32(1007),
OptionalFixed64: proto.Uint64(1008),
OptionalSfixed32: proto.Int32(1009),
OptionalSfixed64: proto.Int64(1010),
OptionalFloat: proto.Float32(1011.5),
OptionalDouble: proto.Float64(1012.5),
OptionalBool: proto.Bool(true),
OptionalString: proto.String("string"),
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
OptionalBytes: []byte("bytes"),
OptionalNestedEnum: testpb.TestAllTypes_BAR.Enum(),
proto: add proto3 encode/decode test cases Change-Id: I5f859fbd34cbd3e95a6c39a5581791dedc619e6b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169477 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-26 13:55:02 -07:00
}, &test3pb.TestAllTypes{
OptionalInt32: 1001,
OptionalInt64: 1002,
OptionalUint32: 1003,
OptionalUint64: 1004,
OptionalSint32: 1005,
OptionalSint64: 1006,
OptionalFixed32: 1007,
OptionalFixed64: 1008,
OptionalSfixed32: 1009,
OptionalSfixed64: 1010,
OptionalFloat: 1011.5,
OptionalDouble: 1012.5,
OptionalBool: true,
OptionalString: "string",
OptionalBytes: []byte("bytes"),
OptionalNestedEnum: test3pb.TestAllTypes_BAR,
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
}, build(
&testpb.TestAllExtensions{},
all: remove dependency on proto v1 This does not remove all dependencies, but all of the cases where it can now be implemented in terms of v2. Change-Id: Idc5b0273f0d35c284bf2141eb9cce998692ceb15 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/184878 Reviewed-by: Herbie Ong <herbie@google.com>
2019-05-16 15:53:25 -07:00
extend(testpb.E_OptionalInt32Extension, int32(1001)),
extend(testpb.E_OptionalInt64Extension, int64(1002)),
extend(testpb.E_OptionalUint32Extension, uint32(1003)),
extend(testpb.E_OptionalUint64Extension, uint64(1004)),
extend(testpb.E_OptionalSint32Extension, int32(1005)),
extend(testpb.E_OptionalSint64Extension, int64(1006)),
extend(testpb.E_OptionalFixed32Extension, uint32(1007)),
extend(testpb.E_OptionalFixed64Extension, uint64(1008)),
extend(testpb.E_OptionalSfixed32Extension, int32(1009)),
extend(testpb.E_OptionalSfixed64Extension, int64(1010)),
extend(testpb.E_OptionalFloatExtension, float32(1011.5)),
extend(testpb.E_OptionalDoubleExtension, float64(1012.5)),
extend(testpb.E_OptionalBoolExtension, bool(true)),
extend(testpb.E_OptionalStringExtension, string("string")),
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
extend(testpb.E_OptionalBytesExtension, []byte("bytes")),
all: remove dependency on proto v1 This does not remove all dependencies, but all of the cases where it can now be implemented in terms of v2. Change-Id: Idc5b0273f0d35c284bf2141eb9cce998692ceb15 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/184878 Reviewed-by: Herbie Ong <herbie@google.com>
2019-05-16 15:53:25 -07:00
extend(testpb.E_OptionalNestedEnumExtension, testpb.TestAllTypes_BAR),
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
)},
wire: pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(1001),
pack.Tag{2, pack.VarintType}, pack.Varint(1002),
pack.Tag{3, pack.VarintType}, pack.Uvarint(1003),
pack.Tag{4, pack.VarintType}, pack.Uvarint(1004),
pack.Tag{5, pack.VarintType}, pack.Svarint(1005),
pack.Tag{6, pack.VarintType}, pack.Svarint(1006),
pack.Tag{7, pack.Fixed32Type}, pack.Uint32(1007),
pack.Tag{8, pack.Fixed64Type}, pack.Uint64(1008),
pack.Tag{9, pack.Fixed32Type}, pack.Int32(1009),
pack.Tag{10, pack.Fixed64Type}, pack.Int64(1010),
pack.Tag{11, pack.Fixed32Type}, pack.Float32(1011.5),
pack.Tag{12, pack.Fixed64Type}, pack.Float64(1012.5),
pack.Tag{13, pack.VarintType}, pack.Bool(true),
pack.Tag{14, pack.BytesType}, pack.String("string"),
pack.Tag{15, pack.BytesType}, pack.Bytes([]byte("bytes")),
pack.Tag{21, pack.VarintType}, pack.Varint(int(testpb.TestAllTypes_BAR)),
}.Marshal(),
},
proto, internal/impl: zero-length proto2 bytes fields should be non-nil Fix decoding of zero-length bytes fields to produce a non-nil []byte. Change-Id: Ifb7791a47df81091700f7226523371d1386fb1ad Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/188765 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-02 15:13:00 -07:00
{
desc: "zero values",
decodeTo: []proto.Message{&testpb.TestAllTypes{
OptionalInt32: proto.Int32(0),
OptionalInt64: proto.Int64(0),
OptionalUint32: proto.Uint32(0),
OptionalUint64: proto.Uint64(0),
OptionalSint32: proto.Int32(0),
OptionalSint64: proto.Int64(0),
OptionalFixed32: proto.Uint32(0),
OptionalFixed64: proto.Uint64(0),
OptionalSfixed32: proto.Int32(0),
OptionalSfixed64: proto.Int64(0),
OptionalFloat: proto.Float32(0),
OptionalDouble: proto.Float64(0),
OptionalBool: proto.Bool(false),
OptionalString: proto.String(""),
OptionalBytes: []byte{},
}, &test3pb.TestAllTypes{}, build(
&testpb.TestAllExtensions{},
extend(testpb.E_OptionalInt32Extension, int32(0)),
extend(testpb.E_OptionalInt64Extension, int64(0)),
extend(testpb.E_OptionalUint32Extension, uint32(0)),
extend(testpb.E_OptionalUint64Extension, uint64(0)),
extend(testpb.E_OptionalSint32Extension, int32(0)),
extend(testpb.E_OptionalSint64Extension, int64(0)),
extend(testpb.E_OptionalFixed32Extension, uint32(0)),
extend(testpb.E_OptionalFixed64Extension, uint64(0)),
extend(testpb.E_OptionalSfixed32Extension, int32(0)),
extend(testpb.E_OptionalSfixed64Extension, int64(0)),
extend(testpb.E_OptionalFloatExtension, float32(0)),
extend(testpb.E_OptionalDoubleExtension, float64(0)),
extend(testpb.E_OptionalBoolExtension, bool(false)),
extend(testpb.E_OptionalStringExtension, string("")),
extend(testpb.E_OptionalBytesExtension, []byte{}),
)},
wire: pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(0),
pack.Tag{2, pack.VarintType}, pack.Varint(0),
pack.Tag{3, pack.VarintType}, pack.Uvarint(0),
pack.Tag{4, pack.VarintType}, pack.Uvarint(0),
pack.Tag{5, pack.VarintType}, pack.Svarint(0),
pack.Tag{6, pack.VarintType}, pack.Svarint(0),
pack.Tag{7, pack.Fixed32Type}, pack.Uint32(0),
pack.Tag{8, pack.Fixed64Type}, pack.Uint64(0),
pack.Tag{9, pack.Fixed32Type}, pack.Int32(0),
pack.Tag{10, pack.Fixed64Type}, pack.Int64(0),
pack.Tag{11, pack.Fixed32Type}, pack.Float32(0),
pack.Tag{12, pack.Fixed64Type}, pack.Float64(0),
pack.Tag{13, pack.VarintType}, pack.Bool(false),
pack.Tag{14, pack.BytesType}, pack.String(""),
pack.Tag{15, pack.BytesType}, pack.Bytes(nil),
}.Marshal(),
},
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
{
desc: "groups",
internal/legacy: use v2 proto.Unmarshal The v2 decoder isn't 100% complete, but it's good enough. Delete the vendored copy of the v1 Unmarshal implementation. Change-Id: Ibeabbb2e9109a1ec3df57e71f98b7aa4a583fc5b Reviewed-on: https://go-review.googlesource.com/c/154577 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-17 11:16:16 -08:00
decodeTo: []proto.Message{&testpb.TestAllTypes{
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
Optionalgroup: &testpb.TestAllTypes_OptionalGroup{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
A: proto.Int32(1017),
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
},
}, build(
&testpb.TestAllExtensions{},
extend(testpb.E_OptionalgroupExtension, &testpb.OptionalGroupExtension{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
A: proto.Int32(1017),
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
}),
)},
wire: pack.Message{
pack.Tag{16, pack.StartGroupType},
pack.Tag{17, pack.VarintType}, pack.Varint(1017),
pack.Tag{16, pack.EndGroupType},
}.Marshal(),
},
{
desc: "groups (field overridden)",
internal/legacy: use v2 proto.Unmarshal The v2 decoder isn't 100% complete, but it's good enough. Delete the vendored copy of the v1 Unmarshal implementation. Change-Id: Ibeabbb2e9109a1ec3df57e71f98b7aa4a583fc5b Reviewed-on: https://go-review.googlesource.com/c/154577 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-17 11:16:16 -08:00
decodeTo: []proto.Message{&testpb.TestAllTypes{
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
Optionalgroup: &testpb.TestAllTypes_OptionalGroup{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
A: proto.Int32(2),
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
},
}, build(
&testpb.TestAllExtensions{},
extend(testpb.E_OptionalgroupExtension, &testpb.OptionalGroupExtension{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
A: proto.Int32(2),
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
}),
)},
wire: pack.Message{
pack.Tag{16, pack.StartGroupType},
pack.Tag{17, pack.VarintType}, pack.Varint(1),
pack.Tag{16, pack.EndGroupType},
pack.Tag{16, pack.StartGroupType},
pack.Tag{17, pack.VarintType}, pack.Varint(2),
pack.Tag{16, pack.EndGroupType},
}.Marshal(),
},
{
desc: "messages",
internal/legacy: use v2 proto.Unmarshal The v2 decoder isn't 100% complete, but it's good enough. Delete the vendored copy of the v1 Unmarshal implementation. Change-Id: Ibeabbb2e9109a1ec3df57e71f98b7aa4a583fc5b Reviewed-on: https://go-review.googlesource.com/c/154577 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-17 11:16:16 -08:00
decodeTo: []proto.Message{&testpb.TestAllTypes{
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
OptionalNestedMessage: &testpb.TestAllTypes_NestedMessage{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
A: proto.Int32(42),
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
Corecursive: &testpb.TestAllTypes{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
OptionalInt32: proto.Int32(43),
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
},
},
proto: add proto3 encode/decode test cases Change-Id: I5f859fbd34cbd3e95a6c39a5581791dedc619e6b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169477 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-26 13:55:02 -07:00
}, &test3pb.TestAllTypes{
OptionalNestedMessage: &test3pb.TestAllTypes_NestedMessage{
A: 42,
Corecursive: &test3pb.TestAllTypes{
OptionalInt32: 43,
},
},
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
}, build(
&testpb.TestAllExtensions{},
extend(testpb.E_OptionalNestedMessageExtension, &testpb.TestAllTypes_NestedMessage{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
A: proto.Int32(42),
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
Corecursive: &testpb.TestAllTypes{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
OptionalInt32: proto.Int32(43),
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
},
}),
)},
wire: pack.Message{
pack.Tag{18, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(42),
pack.Tag{2, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(43),
}),
}),
}.Marshal(),
},
{
desc: "messages (split across multiple tags)",
internal/legacy: use v2 proto.Unmarshal The v2 decoder isn't 100% complete, but it's good enough. Delete the vendored copy of the v1 Unmarshal implementation. Change-Id: Ibeabbb2e9109a1ec3df57e71f98b7aa4a583fc5b Reviewed-on: https://go-review.googlesource.com/c/154577 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-17 11:16:16 -08:00
decodeTo: []proto.Message{&testpb.TestAllTypes{
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
OptionalNestedMessage: &testpb.TestAllTypes_NestedMessage{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
A: proto.Int32(42),
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
Corecursive: &testpb.TestAllTypes{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
OptionalInt32: proto.Int32(43),
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
},
},
proto: add proto3 encode/decode test cases Change-Id: I5f859fbd34cbd3e95a6c39a5581791dedc619e6b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169477 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-26 13:55:02 -07:00
}, &test3pb.TestAllTypes{
OptionalNestedMessage: &test3pb.TestAllTypes_NestedMessage{
A: 42,
Corecursive: &test3pb.TestAllTypes{
OptionalInt32: 43,
},
},
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
}, build(
&testpb.TestAllExtensions{},
extend(testpb.E_OptionalNestedMessageExtension, &testpb.TestAllTypes_NestedMessage{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
A: proto.Int32(42),
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
Corecursive: &testpb.TestAllTypes{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
OptionalInt32: proto.Int32(43),
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
},
}),
)},
wire: pack.Message{
pack.Tag{18, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(42),
}),
pack.Tag{18, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{2, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(43),
}),
}),
}.Marshal(),
},
{
desc: "messages (field overridden)",
internal/legacy: use v2 proto.Unmarshal The v2 decoder isn't 100% complete, but it's good enough. Delete the vendored copy of the v1 Unmarshal implementation. Change-Id: Ibeabbb2e9109a1ec3df57e71f98b7aa4a583fc5b Reviewed-on: https://go-review.googlesource.com/c/154577 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-17 11:16:16 -08:00
decodeTo: []proto.Message{&testpb.TestAllTypes{
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
OptionalNestedMessage: &testpb.TestAllTypes_NestedMessage{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
A: proto.Int32(2),
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
},
proto: add proto3 encode/decode test cases Change-Id: I5f859fbd34cbd3e95a6c39a5581791dedc619e6b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169477 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-26 13:55:02 -07:00
}, &test3pb.TestAllTypes{
OptionalNestedMessage: &test3pb.TestAllTypes_NestedMessage{
A: 2,
},
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
}, build(
&testpb.TestAllExtensions{},
extend(testpb.E_OptionalNestedMessageExtension, &testpb.TestAllTypes_NestedMessage{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
A: proto.Int32(2),
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
}),
)},
wire: pack.Message{
pack.Tag{18, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(1),
}),
pack.Tag{18, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(2),
}),
}.Marshal(),
},
{
desc: "basic repeated types",
internal/legacy: use v2 proto.Unmarshal The v2 decoder isn't 100% complete, but it's good enough. Delete the vendored copy of the v1 Unmarshal implementation. Change-Id: Ibeabbb2e9109a1ec3df57e71f98b7aa4a583fc5b Reviewed-on: https://go-review.googlesource.com/c/154577 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-17 11:16:16 -08:00
decodeTo: []proto.Message{&testpb.TestAllTypes{
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
RepeatedInt32: []int32{1001, 2001},
RepeatedInt64: []int64{1002, 2002},
RepeatedUint32: []uint32{1003, 2003},
RepeatedUint64: []uint64{1004, 2004},
RepeatedSint32: []int32{1005, 2005},
RepeatedSint64: []int64{1006, 2006},
RepeatedFixed32: []uint32{1007, 2007},
RepeatedFixed64: []uint64{1008, 2008},
RepeatedSfixed32: []int32{1009, 2009},
RepeatedSfixed64: []int64{1010, 2010},
RepeatedFloat: []float32{1011.5, 2011.5},
RepeatedDouble: []float64{1012.5, 2012.5},
RepeatedBool: []bool{true, false},
RepeatedString: []string{"foo", "bar"},
RepeatedBytes: [][]byte{[]byte("FOO"), []byte("BAR")},
RepeatedNestedEnum: []testpb.TestAllTypes_NestedEnum{
testpb.TestAllTypes_FOO,
testpb.TestAllTypes_BAR,
},
proto: add proto3 encode/decode test cases Change-Id: I5f859fbd34cbd3e95a6c39a5581791dedc619e6b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169477 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-26 13:55:02 -07:00
}, &test3pb.TestAllTypes{
RepeatedInt32: []int32{1001, 2001},
RepeatedInt64: []int64{1002, 2002},
RepeatedUint32: []uint32{1003, 2003},
RepeatedUint64: []uint64{1004, 2004},
RepeatedSint32: []int32{1005, 2005},
RepeatedSint64: []int64{1006, 2006},
RepeatedFixed32: []uint32{1007, 2007},
RepeatedFixed64: []uint64{1008, 2008},
RepeatedSfixed32: []int32{1009, 2009},
RepeatedSfixed64: []int64{1010, 2010},
RepeatedFloat: []float32{1011.5, 2011.5},
RepeatedDouble: []float64{1012.5, 2012.5},
RepeatedBool: []bool{true, false},
RepeatedString: []string{"foo", "bar"},
RepeatedBytes: [][]byte{[]byte("FOO"), []byte("BAR")},
RepeatedNestedEnum: []test3pb.TestAllTypes_NestedEnum{
test3pb.TestAllTypes_FOO,
test3pb.TestAllTypes_BAR,
},
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
}, build(
&testpb.TestAllExtensions{},
extend(testpb.E_RepeatedInt32Extension, []int32{1001, 2001}),
extend(testpb.E_RepeatedInt64Extension, []int64{1002, 2002}),
extend(testpb.E_RepeatedUint32Extension, []uint32{1003, 2003}),
extend(testpb.E_RepeatedUint64Extension, []uint64{1004, 2004}),
extend(testpb.E_RepeatedSint32Extension, []int32{1005, 2005}),
extend(testpb.E_RepeatedSint64Extension, []int64{1006, 2006}),
extend(testpb.E_RepeatedFixed32Extension, []uint32{1007, 2007}),
extend(testpb.E_RepeatedFixed64Extension, []uint64{1008, 2008}),
extend(testpb.E_RepeatedSfixed32Extension, []int32{1009, 2009}),
extend(testpb.E_RepeatedSfixed64Extension, []int64{1010, 2010}),
extend(testpb.E_RepeatedFloatExtension, []float32{1011.5, 2011.5}),
extend(testpb.E_RepeatedDoubleExtension, []float64{1012.5, 2012.5}),
extend(testpb.E_RepeatedBoolExtension, []bool{true, false}),
extend(testpb.E_RepeatedStringExtension, []string{"foo", "bar"}),
extend(testpb.E_RepeatedBytesExtension, [][]byte{[]byte("FOO"), []byte("BAR")}),
extend(testpb.E_RepeatedNestedEnumExtension, []testpb.TestAllTypes_NestedEnum{
testpb.TestAllTypes_FOO,
testpb.TestAllTypes_BAR,
}),
)},
wire: pack.Message{
pack.Tag{31, pack.VarintType}, pack.Varint(1001),
pack.Tag{31, pack.VarintType}, pack.Varint(2001),
pack.Tag{32, pack.VarintType}, pack.Varint(1002),
pack.Tag{32, pack.VarintType}, pack.Varint(2002),
pack.Tag{33, pack.VarintType}, pack.Uvarint(1003),
pack.Tag{33, pack.VarintType}, pack.Uvarint(2003),
pack.Tag{34, pack.VarintType}, pack.Uvarint(1004),
pack.Tag{34, pack.VarintType}, pack.Uvarint(2004),
pack.Tag{35, pack.VarintType}, pack.Svarint(1005),
pack.Tag{35, pack.VarintType}, pack.Svarint(2005),
pack.Tag{36, pack.VarintType}, pack.Svarint(1006),
pack.Tag{36, pack.VarintType}, pack.Svarint(2006),
pack.Tag{37, pack.Fixed32Type}, pack.Uint32(1007),
pack.Tag{37, pack.Fixed32Type}, pack.Uint32(2007),
pack.Tag{38, pack.Fixed64Type}, pack.Uint64(1008),
pack.Tag{38, pack.Fixed64Type}, pack.Uint64(2008),
pack.Tag{39, pack.Fixed32Type}, pack.Int32(1009),
pack.Tag{39, pack.Fixed32Type}, pack.Int32(2009),
pack.Tag{40, pack.Fixed64Type}, pack.Int64(1010),
pack.Tag{40, pack.Fixed64Type}, pack.Int64(2010),
pack.Tag{41, pack.Fixed32Type}, pack.Float32(1011.5),
pack.Tag{41, pack.Fixed32Type}, pack.Float32(2011.5),
pack.Tag{42, pack.Fixed64Type}, pack.Float64(1012.5),
pack.Tag{42, pack.Fixed64Type}, pack.Float64(2012.5),
pack.Tag{43, pack.VarintType}, pack.Bool(true),
pack.Tag{43, pack.VarintType}, pack.Bool(false),
pack.Tag{44, pack.BytesType}, pack.String("foo"),
pack.Tag{44, pack.BytesType}, pack.String("bar"),
pack.Tag{45, pack.BytesType}, pack.Bytes([]byte("FOO")),
pack.Tag{45, pack.BytesType}, pack.Bytes([]byte("BAR")),
pack.Tag{51, pack.VarintType}, pack.Varint(int(testpb.TestAllTypes_FOO)),
pack.Tag{51, pack.VarintType}, pack.Varint(int(testpb.TestAllTypes_BAR)),
}.Marshal(),
},
{
desc: "basic repeated types (packed encoding)",
internal/legacy: use v2 proto.Unmarshal The v2 decoder isn't 100% complete, but it's good enough. Delete the vendored copy of the v1 Unmarshal implementation. Change-Id: Ibeabbb2e9109a1ec3df57e71f98b7aa4a583fc5b Reviewed-on: https://go-review.googlesource.com/c/154577 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-17 11:16:16 -08:00
decodeTo: []proto.Message{&testpb.TestAllTypes{
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
RepeatedInt32: []int32{1001, 2001},
RepeatedInt64: []int64{1002, 2002},
RepeatedUint32: []uint32{1003, 2003},
RepeatedUint64: []uint64{1004, 2004},
RepeatedSint32: []int32{1005, 2005},
RepeatedSint64: []int64{1006, 2006},
RepeatedFixed32: []uint32{1007, 2007},
RepeatedFixed64: []uint64{1008, 2008},
RepeatedSfixed32: []int32{1009, 2009},
RepeatedSfixed64: []int64{1010, 2010},
RepeatedFloat: []float32{1011.5, 2011.5},
RepeatedDouble: []float64{1012.5, 2012.5},
RepeatedBool: []bool{true, false},
RepeatedNestedEnum: []testpb.TestAllTypes_NestedEnum{
testpb.TestAllTypes_FOO,
testpb.TestAllTypes_BAR,
},
proto: add proto3 encode/decode test cases Change-Id: I5f859fbd34cbd3e95a6c39a5581791dedc619e6b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169477 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-26 13:55:02 -07:00
}, &test3pb.TestAllTypes{
RepeatedInt32: []int32{1001, 2001},
RepeatedInt64: []int64{1002, 2002},
RepeatedUint32: []uint32{1003, 2003},
RepeatedUint64: []uint64{1004, 2004},
RepeatedSint32: []int32{1005, 2005},
RepeatedSint64: []int64{1006, 2006},
RepeatedFixed32: []uint32{1007, 2007},
RepeatedFixed64: []uint64{1008, 2008},
RepeatedSfixed32: []int32{1009, 2009},
RepeatedSfixed64: []int64{1010, 2010},
RepeatedFloat: []float32{1011.5, 2011.5},
RepeatedDouble: []float64{1012.5, 2012.5},
RepeatedBool: []bool{true, false},
RepeatedNestedEnum: []test3pb.TestAllTypes_NestedEnum{
test3pb.TestAllTypes_FOO,
test3pb.TestAllTypes_BAR,
},
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
}, build(
&testpb.TestAllExtensions{},
extend(testpb.E_RepeatedInt32Extension, []int32{1001, 2001}),
extend(testpb.E_RepeatedInt64Extension, []int64{1002, 2002}),
extend(testpb.E_RepeatedUint32Extension, []uint32{1003, 2003}),
extend(testpb.E_RepeatedUint64Extension, []uint64{1004, 2004}),
extend(testpb.E_RepeatedSint32Extension, []int32{1005, 2005}),
extend(testpb.E_RepeatedSint64Extension, []int64{1006, 2006}),
extend(testpb.E_RepeatedFixed32Extension, []uint32{1007, 2007}),
extend(testpb.E_RepeatedFixed64Extension, []uint64{1008, 2008}),
extend(testpb.E_RepeatedSfixed32Extension, []int32{1009, 2009}),
extend(testpb.E_RepeatedSfixed64Extension, []int64{1010, 2010}),
extend(testpb.E_RepeatedFloatExtension, []float32{1011.5, 2011.5}),
extend(testpb.E_RepeatedDoubleExtension, []float64{1012.5, 2012.5}),
extend(testpb.E_RepeatedBoolExtension, []bool{true, false}),
extend(testpb.E_RepeatedNestedEnumExtension, []testpb.TestAllTypes_NestedEnum{
testpb.TestAllTypes_FOO,
testpb.TestAllTypes_BAR,
}),
)},
wire: pack.Message{
pack.Tag{31, pack.BytesType}, pack.LengthPrefix{
pack.Varint(1001), pack.Varint(2001),
},
pack.Tag{32, pack.BytesType}, pack.LengthPrefix{
pack.Varint(1002), pack.Varint(2002),
},
pack.Tag{33, pack.BytesType}, pack.LengthPrefix{
pack.Uvarint(1003), pack.Uvarint(2003),
},
pack.Tag{34, pack.BytesType}, pack.LengthPrefix{
pack.Uvarint(1004), pack.Uvarint(2004),
},
pack.Tag{35, pack.BytesType}, pack.LengthPrefix{
pack.Svarint(1005), pack.Svarint(2005),
},
pack.Tag{36, pack.BytesType}, pack.LengthPrefix{
pack.Svarint(1006), pack.Svarint(2006),
},
pack.Tag{37, pack.BytesType}, pack.LengthPrefix{
pack.Uint32(1007), pack.Uint32(2007),
},
pack.Tag{38, pack.BytesType}, pack.LengthPrefix{
pack.Uint64(1008), pack.Uint64(2008),
},
pack.Tag{39, pack.BytesType}, pack.LengthPrefix{
pack.Int32(1009), pack.Int32(2009),
},
pack.Tag{40, pack.BytesType}, pack.LengthPrefix{
pack.Int64(1010), pack.Int64(2010),
},
pack.Tag{41, pack.BytesType}, pack.LengthPrefix{
pack.Float32(1011.5), pack.Float32(2011.5),
},
pack.Tag{42, pack.BytesType}, pack.LengthPrefix{
pack.Float64(1012.5), pack.Float64(2012.5),
},
pack.Tag{43, pack.BytesType}, pack.LengthPrefix{
pack.Bool(true), pack.Bool(false),
},
pack.Tag{51, pack.BytesType}, pack.LengthPrefix{
pack.Varint(int(testpb.TestAllTypes_FOO)),
pack.Varint(int(testpb.TestAllTypes_BAR)),
},
}.Marshal(),
},
reflect/protoreflect: remove nullability from repeated extension fields Remove repeated extension fields from the set of nullable fields, so that Has reports false and Range does not visit a a zero-length repeated extension field. This corrects a fuzzer-detected case where unmarshaling and remarshaling a wire-format message could result in a semantic change. For a repeated extension field in non-packed encoding, unmarshaling a packed representation of the field would result in a message which Has the extension. Remarshaling it would discard the the field. Fixes golang.org/protobuf#975 Change-Id: Ie836559c93d218db5b5201742a3b8ebbaacf54ed Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/204897 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com> Reviewed-by: Joe Tsai <joetsai@google.com>
2019-11-01 15:18:36 -07:00
{
desc: "basic repeated types (zero-length packed encoding)",
decodeTo: []proto.Message{
&testpb.TestAllTypes{},
&test3pb.TestAllTypes{},
&testpb.TestAllExtensions{},
},
wire: pack.Message{
pack.Tag{31, pack.BytesType}, pack.LengthPrefix{},
pack.Tag{32, pack.BytesType}, pack.LengthPrefix{},
pack.Tag{33, pack.BytesType}, pack.LengthPrefix{},
pack.Tag{34, pack.BytesType}, pack.LengthPrefix{},
pack.Tag{35, pack.BytesType}, pack.LengthPrefix{},
pack.Tag{36, pack.BytesType}, pack.LengthPrefix{},
pack.Tag{37, pack.BytesType}, pack.LengthPrefix{},
pack.Tag{38, pack.BytesType}, pack.LengthPrefix{},
pack.Tag{39, pack.BytesType}, pack.LengthPrefix{},
pack.Tag{40, pack.BytesType}, pack.LengthPrefix{},
pack.Tag{41, pack.BytesType}, pack.LengthPrefix{},
pack.Tag{42, pack.BytesType}, pack.LengthPrefix{},
pack.Tag{43, pack.BytesType}, pack.LengthPrefix{},
pack.Tag{51, pack.BytesType}, pack.LengthPrefix{},
}.Marshal(),
},
internal/impl: support packed extensions Change-Id: I5a9e22f1c98f5db9caae1681775017da5aa67394 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185541 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 15:23:29 -07:00
{
desc: "packed repeated types",
decodeTo: []proto.Message{&testpb.TestPackedTypes{
PackedInt32: []int32{1001, 2001},
PackedInt64: []int64{1002, 2002},
PackedUint32: []uint32{1003, 2003},
PackedUint64: []uint64{1004, 2004},
PackedSint32: []int32{1005, 2005},
PackedSint64: []int64{1006, 2006},
PackedFixed32: []uint32{1007, 2007},
PackedFixed64: []uint64{1008, 2008},
PackedSfixed32: []int32{1009, 2009},
PackedSfixed64: []int64{1010, 2010},
PackedFloat: []float32{1011.5, 2011.5},
PackedDouble: []float64{1012.5, 2012.5},
PackedBool: []bool{true, false},
PackedEnum: []testpb.ForeignEnum{
testpb.ForeignEnum_FOREIGN_FOO,
testpb.ForeignEnum_FOREIGN_BAR,
},
}, build(
&testpb.TestPackedExtensions{},
extend(testpb.E_PackedInt32Extension, []int32{1001, 2001}),
extend(testpb.E_PackedInt64Extension, []int64{1002, 2002}),
extend(testpb.E_PackedUint32Extension, []uint32{1003, 2003}),
extend(testpb.E_PackedUint64Extension, []uint64{1004, 2004}),
extend(testpb.E_PackedSint32Extension, []int32{1005, 2005}),
extend(testpb.E_PackedSint64Extension, []int64{1006, 2006}),
extend(testpb.E_PackedFixed32Extension, []uint32{1007, 2007}),
extend(testpb.E_PackedFixed64Extension, []uint64{1008, 2008}),
extend(testpb.E_PackedSfixed32Extension, []int32{1009, 2009}),
extend(testpb.E_PackedSfixed64Extension, []int64{1010, 2010}),
extend(testpb.E_PackedFloatExtension, []float32{1011.5, 2011.5}),
extend(testpb.E_PackedDoubleExtension, []float64{1012.5, 2012.5}),
extend(testpb.E_PackedBoolExtension, []bool{true, false}),
extend(testpb.E_PackedEnumExtension, []testpb.ForeignEnum{
testpb.ForeignEnum_FOREIGN_FOO,
testpb.ForeignEnum_FOREIGN_BAR,
}),
)},
wire: pack.Message{
pack.Tag{90, pack.BytesType}, pack.LengthPrefix{
pack.Varint(1001), pack.Varint(2001),
},
pack.Tag{91, pack.BytesType}, pack.LengthPrefix{
pack.Varint(1002), pack.Varint(2002),
},
pack.Tag{92, pack.BytesType}, pack.LengthPrefix{
pack.Uvarint(1003), pack.Uvarint(2003),
},
pack.Tag{93, pack.BytesType}, pack.LengthPrefix{
pack.Uvarint(1004), pack.Uvarint(2004),
},
pack.Tag{94, pack.BytesType}, pack.LengthPrefix{
pack.Svarint(1005), pack.Svarint(2005),
},
pack.Tag{95, pack.BytesType}, pack.LengthPrefix{
pack.Svarint(1006), pack.Svarint(2006),
},
pack.Tag{96, pack.BytesType}, pack.LengthPrefix{
pack.Uint32(1007), pack.Uint32(2007),
},
pack.Tag{97, pack.BytesType}, pack.LengthPrefix{
pack.Uint64(1008), pack.Uint64(2008),
},
pack.Tag{98, pack.BytesType}, pack.LengthPrefix{
pack.Int32(1009), pack.Int32(2009),
},
pack.Tag{99, pack.BytesType}, pack.LengthPrefix{
pack.Int64(1010), pack.Int64(2010),
},
pack.Tag{100, pack.BytesType}, pack.LengthPrefix{
pack.Float32(1011.5), pack.Float32(2011.5),
},
pack.Tag{101, pack.BytesType}, pack.LengthPrefix{
pack.Float64(1012.5), pack.Float64(2012.5),
},
pack.Tag{102, pack.BytesType}, pack.LengthPrefix{
pack.Bool(true), pack.Bool(false),
},
pack.Tag{103, pack.BytesType}, pack.LengthPrefix{
pack.Varint(int(testpb.ForeignEnum_FOREIGN_FOO)),
pack.Varint(int(testpb.ForeignEnum_FOREIGN_BAR)),
},
}.Marshal(),
},
reflect/protoreflect: remove nullability from repeated extension fields Remove repeated extension fields from the set of nullable fields, so that Has reports false and Range does not visit a a zero-length repeated extension field. This corrects a fuzzer-detected case where unmarshaling and remarshaling a wire-format message could result in a semantic change. For a repeated extension field in non-packed encoding, unmarshaling a packed representation of the field would result in a message which Has the extension. Remarshaling it would discard the the field. Fixes golang.org/protobuf#975 Change-Id: Ie836559c93d218db5b5201742a3b8ebbaacf54ed Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/204897 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com> Reviewed-by: Joe Tsai <joetsai@google.com>
2019-11-01 15:18:36 -07:00
{
desc: "packed repeated types (zero length)",
decodeTo: []proto.Message{
&testpb.TestPackedTypes{},
&testpb.TestPackedExtensions{},
},
wire: pack.Message{
pack.Tag{90, pack.BytesType}, pack.LengthPrefix{},
pack.Tag{91, pack.BytesType}, pack.LengthPrefix{},
pack.Tag{92, pack.BytesType}, pack.LengthPrefix{},
pack.Tag{93, pack.BytesType}, pack.LengthPrefix{},
pack.Tag{94, pack.BytesType}, pack.LengthPrefix{},
pack.Tag{95, pack.BytesType}, pack.LengthPrefix{},
pack.Tag{96, pack.BytesType}, pack.LengthPrefix{},
pack.Tag{97, pack.BytesType}, pack.LengthPrefix{},
pack.Tag{98, pack.BytesType}, pack.LengthPrefix{},
pack.Tag{99, pack.BytesType}, pack.LengthPrefix{},
pack.Tag{100, pack.BytesType}, pack.LengthPrefix{},
pack.Tag{101, pack.BytesType}, pack.LengthPrefix{},
pack.Tag{102, pack.BytesType}, pack.LengthPrefix{},
pack.Tag{103, pack.BytesType}, pack.LengthPrefix{},
}.Marshal(),
},
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
{
desc: "repeated messages",
internal/legacy: use v2 proto.Unmarshal The v2 decoder isn't 100% complete, but it's good enough. Delete the vendored copy of the v1 Unmarshal implementation. Change-Id: Ibeabbb2e9109a1ec3df57e71f98b7aa4a583fc5b Reviewed-on: https://go-review.googlesource.com/c/154577 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-17 11:16:16 -08:00
decodeTo: []proto.Message{&testpb.TestAllTypes{
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
RepeatedNestedMessage: []*testpb.TestAllTypes_NestedMessage{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
{A: proto.Int32(1)},
internal/impl: add fast-path marshal implementation This is a port of the v1 table marshaler, with some substantial cleanup and refactoring. Benchstat results from the protobuf reference benchmark data comparing the v1 package with v2, with AllowPartial:true set for the new package. This is not an apples-to-apples comparison, since v1 doesn't have a way to disable required field checks. Required field checks in v2 package currently go through reflection, which performs terribly; my initial experimentation indicates that fast-path required field checks will not add a large amount of cost; these results are incomplete but not wholly inaccurate. name old time/op new time/op delta /dataset.google_message3_1.pb/Marshal-12 219ms ± 1% 232ms ± 1% +5.85% (p=0.004 n=6+5) /dataset.google_message2.pb/Marshal-12 261µs ± 3% 248µs ± 1% -5.14% (p=0.002 n=6+6) /dataset.google_message1_proto2.pb/Marshal-12 681ns ± 2% 637ns ± 3% -6.53% (p=0.002 n=6+6) /dataset.google_message1_proto3.pb/Marshal-12 1.10µs ± 8% 0.99µs ± 3% -9.63% (p=0.002 n=6+6) /dataset.google_message3_3.pb/Marshal-12 44.2ms ± 3% 35.2ms ± 1% -20.28% (p=0.004 n=6+5) /dataset.google_message4.pb/Marshal-12 91.4ms ± 2% 94.9ms ± 2% +3.78% (p=0.002 n=6+6) /dataset.google_message3_2.pb/Marshal-12 78.7ms ± 6% 80.8ms ± 4% ~ (p=0.310 n=6+6) /dataset.google_message3_4.pb/Marshal-12 10.6ms ± 3% 10.6ms ± 8% ~ (p=0.662 n=5+6) /dataset.google_message3_5.pb/Marshal-12 675ms ± 4% 510ms ± 2% -24.40% (p=0.002 n=6+6) /dataset.google_message3_1.pb/Marshal 219ms ± 1% 236ms ± 7% +8.06% (p=0.004 n=5+6) /dataset.google_message2.pb/Marshal 257µs ± 1% 250µs ± 3% ~ (p=0.052 n=5+6) /dataset.google_message1_proto2.pb/Marshal 685ns ± 1% 628ns ± 1% -8.41% (p=0.008 n=5+5) /dataset.google_message1_proto3.pb/Marshal 1.08µs ± 1% 0.98µs ± 2% -9.31% (p=0.004 n=5+6) /dataset.google_message3_3.pb/Marshal 43.7ms ± 1% 35.1ms ± 1% -19.76% (p=0.002 n=6+6) /dataset.google_message4.pb/Marshal 93.4ms ± 4% 94.9ms ± 2% ~ (p=0.180 n=6+6) /dataset.google_message3_2.pb/Marshal 105ms ± 2% 98ms ± 7% -6.81% (p=0.009 n=5+6) /dataset.google_message3_4.pb/Marshal 16.3ms ± 6% 15.7ms ± 3% -3.44% (p=0.041 n=6+6) /dataset.google_message3_5.pb/Marshal 676ms ± 4% 504ms ± 2% -25.50% (p=0.004 n=6+5) Change-Id: I72cc4597117f4cf5d236ef505777d49dd4a5f75d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171020 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-04-01 13:49:56 -07:00
nil,
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
{A: proto.Int32(2)},
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
},
proto: add proto3 encode/decode test cases Change-Id: I5f859fbd34cbd3e95a6c39a5581791dedc619e6b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169477 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-26 13:55:02 -07:00
}, &test3pb.TestAllTypes{
RepeatedNestedMessage: []*test3pb.TestAllTypes_NestedMessage{
{A: 1},
internal/impl: add fast-path marshal implementation This is a port of the v1 table marshaler, with some substantial cleanup and refactoring. Benchstat results from the protobuf reference benchmark data comparing the v1 package with v2, with AllowPartial:true set for the new package. This is not an apples-to-apples comparison, since v1 doesn't have a way to disable required field checks. Required field checks in v2 package currently go through reflection, which performs terribly; my initial experimentation indicates that fast-path required field checks will not add a large amount of cost; these results are incomplete but not wholly inaccurate. name old time/op new time/op delta /dataset.google_message3_1.pb/Marshal-12 219ms ± 1% 232ms ± 1% +5.85% (p=0.004 n=6+5) /dataset.google_message2.pb/Marshal-12 261µs ± 3% 248µs ± 1% -5.14% (p=0.002 n=6+6) /dataset.google_message1_proto2.pb/Marshal-12 681ns ± 2% 637ns ± 3% -6.53% (p=0.002 n=6+6) /dataset.google_message1_proto3.pb/Marshal-12 1.10µs ± 8% 0.99µs ± 3% -9.63% (p=0.002 n=6+6) /dataset.google_message3_3.pb/Marshal-12 44.2ms ± 3% 35.2ms ± 1% -20.28% (p=0.004 n=6+5) /dataset.google_message4.pb/Marshal-12 91.4ms ± 2% 94.9ms ± 2% +3.78% (p=0.002 n=6+6) /dataset.google_message3_2.pb/Marshal-12 78.7ms ± 6% 80.8ms ± 4% ~ (p=0.310 n=6+6) /dataset.google_message3_4.pb/Marshal-12 10.6ms ± 3% 10.6ms ± 8% ~ (p=0.662 n=5+6) /dataset.google_message3_5.pb/Marshal-12 675ms ± 4% 510ms ± 2% -24.40% (p=0.002 n=6+6) /dataset.google_message3_1.pb/Marshal 219ms ± 1% 236ms ± 7% +8.06% (p=0.004 n=5+6) /dataset.google_message2.pb/Marshal 257µs ± 1% 250µs ± 3% ~ (p=0.052 n=5+6) /dataset.google_message1_proto2.pb/Marshal 685ns ± 1% 628ns ± 1% -8.41% (p=0.008 n=5+5) /dataset.google_message1_proto3.pb/Marshal 1.08µs ± 1% 0.98µs ± 2% -9.31% (p=0.004 n=5+6) /dataset.google_message3_3.pb/Marshal 43.7ms ± 1% 35.1ms ± 1% -19.76% (p=0.002 n=6+6) /dataset.google_message4.pb/Marshal 93.4ms ± 4% 94.9ms ± 2% ~ (p=0.180 n=6+6) /dataset.google_message3_2.pb/Marshal 105ms ± 2% 98ms ± 7% -6.81% (p=0.009 n=5+6) /dataset.google_message3_4.pb/Marshal 16.3ms ± 6% 15.7ms ± 3% -3.44% (p=0.041 n=6+6) /dataset.google_message3_5.pb/Marshal 676ms ± 4% 504ms ± 2% -25.50% (p=0.004 n=6+5) Change-Id: I72cc4597117f4cf5d236ef505777d49dd4a5f75d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171020 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-04-01 13:49:56 -07:00
nil,
proto: add proto3 encode/decode test cases Change-Id: I5f859fbd34cbd3e95a6c39a5581791dedc619e6b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169477 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-26 13:55:02 -07:00
{A: 2},
},
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
}, build(
&testpb.TestAllExtensions{},
extend(testpb.E_RepeatedNestedMessageExtension, []*testpb.TestAllTypes_NestedMessage{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
{A: proto.Int32(1)},
internal/impl: add fast-path marshal implementation This is a port of the v1 table marshaler, with some substantial cleanup and refactoring. Benchstat results from the protobuf reference benchmark data comparing the v1 package with v2, with AllowPartial:true set for the new package. This is not an apples-to-apples comparison, since v1 doesn't have a way to disable required field checks. Required field checks in v2 package currently go through reflection, which performs terribly; my initial experimentation indicates that fast-path required field checks will not add a large amount of cost; these results are incomplete but not wholly inaccurate. name old time/op new time/op delta /dataset.google_message3_1.pb/Marshal-12 219ms ± 1% 232ms ± 1% +5.85% (p=0.004 n=6+5) /dataset.google_message2.pb/Marshal-12 261µs ± 3% 248µs ± 1% -5.14% (p=0.002 n=6+6) /dataset.google_message1_proto2.pb/Marshal-12 681ns ± 2% 637ns ± 3% -6.53% (p=0.002 n=6+6) /dataset.google_message1_proto3.pb/Marshal-12 1.10µs ± 8% 0.99µs ± 3% -9.63% (p=0.002 n=6+6) /dataset.google_message3_3.pb/Marshal-12 44.2ms ± 3% 35.2ms ± 1% -20.28% (p=0.004 n=6+5) /dataset.google_message4.pb/Marshal-12 91.4ms ± 2% 94.9ms ± 2% +3.78% (p=0.002 n=6+6) /dataset.google_message3_2.pb/Marshal-12 78.7ms ± 6% 80.8ms ± 4% ~ (p=0.310 n=6+6) /dataset.google_message3_4.pb/Marshal-12 10.6ms ± 3% 10.6ms ± 8% ~ (p=0.662 n=5+6) /dataset.google_message3_5.pb/Marshal-12 675ms ± 4% 510ms ± 2% -24.40% (p=0.002 n=6+6) /dataset.google_message3_1.pb/Marshal 219ms ± 1% 236ms ± 7% +8.06% (p=0.004 n=5+6) /dataset.google_message2.pb/Marshal 257µs ± 1% 250µs ± 3% ~ (p=0.052 n=5+6) /dataset.google_message1_proto2.pb/Marshal 685ns ± 1% 628ns ± 1% -8.41% (p=0.008 n=5+5) /dataset.google_message1_proto3.pb/Marshal 1.08µs ± 1% 0.98µs ± 2% -9.31% (p=0.004 n=5+6) /dataset.google_message3_3.pb/Marshal 43.7ms ± 1% 35.1ms ± 1% -19.76% (p=0.002 n=6+6) /dataset.google_message4.pb/Marshal 93.4ms ± 4% 94.9ms ± 2% ~ (p=0.180 n=6+6) /dataset.google_message3_2.pb/Marshal 105ms ± 2% 98ms ± 7% -6.81% (p=0.009 n=5+6) /dataset.google_message3_4.pb/Marshal 16.3ms ± 6% 15.7ms ± 3% -3.44% (p=0.041 n=6+6) /dataset.google_message3_5.pb/Marshal 676ms ± 4% 504ms ± 2% -25.50% (p=0.004 n=6+5) Change-Id: I72cc4597117f4cf5d236ef505777d49dd4a5f75d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171020 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-04-01 13:49:56 -07:00
nil,
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
{A: proto.Int32(2)},
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
}),
)},
wire: pack.Message{
pack.Tag{48, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(1),
}),
internal/impl: add fast-path marshal implementation This is a port of the v1 table marshaler, with some substantial cleanup and refactoring. Benchstat results from the protobuf reference benchmark data comparing the v1 package with v2, with AllowPartial:true set for the new package. This is not an apples-to-apples comparison, since v1 doesn't have a way to disable required field checks. Required field checks in v2 package currently go through reflection, which performs terribly; my initial experimentation indicates that fast-path required field checks will not add a large amount of cost; these results are incomplete but not wholly inaccurate. name old time/op new time/op delta /dataset.google_message3_1.pb/Marshal-12 219ms ± 1% 232ms ± 1% +5.85% (p=0.004 n=6+5) /dataset.google_message2.pb/Marshal-12 261µs ± 3% 248µs ± 1% -5.14% (p=0.002 n=6+6) /dataset.google_message1_proto2.pb/Marshal-12 681ns ± 2% 637ns ± 3% -6.53% (p=0.002 n=6+6) /dataset.google_message1_proto3.pb/Marshal-12 1.10µs ± 8% 0.99µs ± 3% -9.63% (p=0.002 n=6+6) /dataset.google_message3_3.pb/Marshal-12 44.2ms ± 3% 35.2ms ± 1% -20.28% (p=0.004 n=6+5) /dataset.google_message4.pb/Marshal-12 91.4ms ± 2% 94.9ms ± 2% +3.78% (p=0.002 n=6+6) /dataset.google_message3_2.pb/Marshal-12 78.7ms ± 6% 80.8ms ± 4% ~ (p=0.310 n=6+6) /dataset.google_message3_4.pb/Marshal-12 10.6ms ± 3% 10.6ms ± 8% ~ (p=0.662 n=5+6) /dataset.google_message3_5.pb/Marshal-12 675ms ± 4% 510ms ± 2% -24.40% (p=0.002 n=6+6) /dataset.google_message3_1.pb/Marshal 219ms ± 1% 236ms ± 7% +8.06% (p=0.004 n=5+6) /dataset.google_message2.pb/Marshal 257µs ± 1% 250µs ± 3% ~ (p=0.052 n=5+6) /dataset.google_message1_proto2.pb/Marshal 685ns ± 1% 628ns ± 1% -8.41% (p=0.008 n=5+5) /dataset.google_message1_proto3.pb/Marshal 1.08µs ± 1% 0.98µs ± 2% -9.31% (p=0.004 n=5+6) /dataset.google_message3_3.pb/Marshal 43.7ms ± 1% 35.1ms ± 1% -19.76% (p=0.002 n=6+6) /dataset.google_message4.pb/Marshal 93.4ms ± 4% 94.9ms ± 2% ~ (p=0.180 n=6+6) /dataset.google_message3_2.pb/Marshal 105ms ± 2% 98ms ± 7% -6.81% (p=0.009 n=5+6) /dataset.google_message3_4.pb/Marshal 16.3ms ± 6% 15.7ms ± 3% -3.44% (p=0.041 n=6+6) /dataset.google_message3_5.pb/Marshal 676ms ± 4% 504ms ± 2% -25.50% (p=0.004 n=6+5) Change-Id: I72cc4597117f4cf5d236ef505777d49dd4a5f75d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171020 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-04-01 13:49:56 -07:00
pack.Tag{48, pack.BytesType}, pack.LengthPrefix(pack.Message{}),
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
pack.Tag{48, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(2),
}),
}.Marshal(),
},
{
desc: "repeated groups",
internal/legacy: use v2 proto.Unmarshal The v2 decoder isn't 100% complete, but it's good enough. Delete the vendored copy of the v1 Unmarshal implementation. Change-Id: Ibeabbb2e9109a1ec3df57e71f98b7aa4a583fc5b Reviewed-on: https://go-review.googlesource.com/c/154577 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-17 11:16:16 -08:00
decodeTo: []proto.Message{&testpb.TestAllTypes{
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
Repeatedgroup: []*testpb.TestAllTypes_RepeatedGroup{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
{A: proto.Int32(1017)},
internal/impl: add fast-path marshal implementation This is a port of the v1 table marshaler, with some substantial cleanup and refactoring. Benchstat results from the protobuf reference benchmark data comparing the v1 package with v2, with AllowPartial:true set for the new package. This is not an apples-to-apples comparison, since v1 doesn't have a way to disable required field checks. Required field checks in v2 package currently go through reflection, which performs terribly; my initial experimentation indicates that fast-path required field checks will not add a large amount of cost; these results are incomplete but not wholly inaccurate. name old time/op new time/op delta /dataset.google_message3_1.pb/Marshal-12 219ms ± 1% 232ms ± 1% +5.85% (p=0.004 n=6+5) /dataset.google_message2.pb/Marshal-12 261µs ± 3% 248µs ± 1% -5.14% (p=0.002 n=6+6) /dataset.google_message1_proto2.pb/Marshal-12 681ns ± 2% 637ns ± 3% -6.53% (p=0.002 n=6+6) /dataset.google_message1_proto3.pb/Marshal-12 1.10µs ± 8% 0.99µs ± 3% -9.63% (p=0.002 n=6+6) /dataset.google_message3_3.pb/Marshal-12 44.2ms ± 3% 35.2ms ± 1% -20.28% (p=0.004 n=6+5) /dataset.google_message4.pb/Marshal-12 91.4ms ± 2% 94.9ms ± 2% +3.78% (p=0.002 n=6+6) /dataset.google_message3_2.pb/Marshal-12 78.7ms ± 6% 80.8ms ± 4% ~ (p=0.310 n=6+6) /dataset.google_message3_4.pb/Marshal-12 10.6ms ± 3% 10.6ms ± 8% ~ (p=0.662 n=5+6) /dataset.google_message3_5.pb/Marshal-12 675ms ± 4% 510ms ± 2% -24.40% (p=0.002 n=6+6) /dataset.google_message3_1.pb/Marshal 219ms ± 1% 236ms ± 7% +8.06% (p=0.004 n=5+6) /dataset.google_message2.pb/Marshal 257µs ± 1% 250µs ± 3% ~ (p=0.052 n=5+6) /dataset.google_message1_proto2.pb/Marshal 685ns ± 1% 628ns ± 1% -8.41% (p=0.008 n=5+5) /dataset.google_message1_proto3.pb/Marshal 1.08µs ± 1% 0.98µs ± 2% -9.31% (p=0.004 n=5+6) /dataset.google_message3_3.pb/Marshal 43.7ms ± 1% 35.1ms ± 1% -19.76% (p=0.002 n=6+6) /dataset.google_message4.pb/Marshal 93.4ms ± 4% 94.9ms ± 2% ~ (p=0.180 n=6+6) /dataset.google_message3_2.pb/Marshal 105ms ± 2% 98ms ± 7% -6.81% (p=0.009 n=5+6) /dataset.google_message3_4.pb/Marshal 16.3ms ± 6% 15.7ms ± 3% -3.44% (p=0.041 n=6+6) /dataset.google_message3_5.pb/Marshal 676ms ± 4% 504ms ± 2% -25.50% (p=0.004 n=6+5) Change-Id: I72cc4597117f4cf5d236ef505777d49dd4a5f75d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171020 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-04-01 13:49:56 -07:00
nil,
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
{A: proto.Int32(2017)},
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
},
}, build(
&testpb.TestAllExtensions{},
extend(testpb.E_RepeatedgroupExtension, []*testpb.RepeatedGroupExtension{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
{A: proto.Int32(1017)},
internal/impl: add fast-path marshal implementation This is a port of the v1 table marshaler, with some substantial cleanup and refactoring. Benchstat results from the protobuf reference benchmark data comparing the v1 package with v2, with AllowPartial:true set for the new package. This is not an apples-to-apples comparison, since v1 doesn't have a way to disable required field checks. Required field checks in v2 package currently go through reflection, which performs terribly; my initial experimentation indicates that fast-path required field checks will not add a large amount of cost; these results are incomplete but not wholly inaccurate. name old time/op new time/op delta /dataset.google_message3_1.pb/Marshal-12 219ms ± 1% 232ms ± 1% +5.85% (p=0.004 n=6+5) /dataset.google_message2.pb/Marshal-12 261µs ± 3% 248µs ± 1% -5.14% (p=0.002 n=6+6) /dataset.google_message1_proto2.pb/Marshal-12 681ns ± 2% 637ns ± 3% -6.53% (p=0.002 n=6+6) /dataset.google_message1_proto3.pb/Marshal-12 1.10µs ± 8% 0.99µs ± 3% -9.63% (p=0.002 n=6+6) /dataset.google_message3_3.pb/Marshal-12 44.2ms ± 3% 35.2ms ± 1% -20.28% (p=0.004 n=6+5) /dataset.google_message4.pb/Marshal-12 91.4ms ± 2% 94.9ms ± 2% +3.78% (p=0.002 n=6+6) /dataset.google_message3_2.pb/Marshal-12 78.7ms ± 6% 80.8ms ± 4% ~ (p=0.310 n=6+6) /dataset.google_message3_4.pb/Marshal-12 10.6ms ± 3% 10.6ms ± 8% ~ (p=0.662 n=5+6) /dataset.google_message3_5.pb/Marshal-12 675ms ± 4% 510ms ± 2% -24.40% (p=0.002 n=6+6) /dataset.google_message3_1.pb/Marshal 219ms ± 1% 236ms ± 7% +8.06% (p=0.004 n=5+6) /dataset.google_message2.pb/Marshal 257µs ± 1% 250µs ± 3% ~ (p=0.052 n=5+6) /dataset.google_message1_proto2.pb/Marshal 685ns ± 1% 628ns ± 1% -8.41% (p=0.008 n=5+5) /dataset.google_message1_proto3.pb/Marshal 1.08µs ± 1% 0.98µs ± 2% -9.31% (p=0.004 n=5+6) /dataset.google_message3_3.pb/Marshal 43.7ms ± 1% 35.1ms ± 1% -19.76% (p=0.002 n=6+6) /dataset.google_message4.pb/Marshal 93.4ms ± 4% 94.9ms ± 2% ~ (p=0.180 n=6+6) /dataset.google_message3_2.pb/Marshal 105ms ± 2% 98ms ± 7% -6.81% (p=0.009 n=5+6) /dataset.google_message3_4.pb/Marshal 16.3ms ± 6% 15.7ms ± 3% -3.44% (p=0.041 n=6+6) /dataset.google_message3_5.pb/Marshal 676ms ± 4% 504ms ± 2% -25.50% (p=0.004 n=6+5) Change-Id: I72cc4597117f4cf5d236ef505777d49dd4a5f75d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171020 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-04-01 13:49:56 -07:00
nil,
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
{A: proto.Int32(2017)},
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
}),
)},
wire: pack.Message{
pack.Tag{46, pack.StartGroupType},
pack.Tag{47, pack.VarintType}, pack.Varint(1017),
pack.Tag{46, pack.EndGroupType},
pack.Tag{46, pack.StartGroupType},
internal/impl: add fast-path marshal implementation This is a port of the v1 table marshaler, with some substantial cleanup and refactoring. Benchstat results from the protobuf reference benchmark data comparing the v1 package with v2, with AllowPartial:true set for the new package. This is not an apples-to-apples comparison, since v1 doesn't have a way to disable required field checks. Required field checks in v2 package currently go through reflection, which performs terribly; my initial experimentation indicates that fast-path required field checks will not add a large amount of cost; these results are incomplete but not wholly inaccurate. name old time/op new time/op delta /dataset.google_message3_1.pb/Marshal-12 219ms ± 1% 232ms ± 1% +5.85% (p=0.004 n=6+5) /dataset.google_message2.pb/Marshal-12 261µs ± 3% 248µs ± 1% -5.14% (p=0.002 n=6+6) /dataset.google_message1_proto2.pb/Marshal-12 681ns ± 2% 637ns ± 3% -6.53% (p=0.002 n=6+6) /dataset.google_message1_proto3.pb/Marshal-12 1.10µs ± 8% 0.99µs ± 3% -9.63% (p=0.002 n=6+6) /dataset.google_message3_3.pb/Marshal-12 44.2ms ± 3% 35.2ms ± 1% -20.28% (p=0.004 n=6+5) /dataset.google_message4.pb/Marshal-12 91.4ms ± 2% 94.9ms ± 2% +3.78% (p=0.002 n=6+6) /dataset.google_message3_2.pb/Marshal-12 78.7ms ± 6% 80.8ms ± 4% ~ (p=0.310 n=6+6) /dataset.google_message3_4.pb/Marshal-12 10.6ms ± 3% 10.6ms ± 8% ~ (p=0.662 n=5+6) /dataset.google_message3_5.pb/Marshal-12 675ms ± 4% 510ms ± 2% -24.40% (p=0.002 n=6+6) /dataset.google_message3_1.pb/Marshal 219ms ± 1% 236ms ± 7% +8.06% (p=0.004 n=5+6) /dataset.google_message2.pb/Marshal 257µs ± 1% 250µs ± 3% ~ (p=0.052 n=5+6) /dataset.google_message1_proto2.pb/Marshal 685ns ± 1% 628ns ± 1% -8.41% (p=0.008 n=5+5) /dataset.google_message1_proto3.pb/Marshal 1.08µs ± 1% 0.98µs ± 2% -9.31% (p=0.004 n=5+6) /dataset.google_message3_3.pb/Marshal 43.7ms ± 1% 35.1ms ± 1% -19.76% (p=0.002 n=6+6) /dataset.google_message4.pb/Marshal 93.4ms ± 4% 94.9ms ± 2% ~ (p=0.180 n=6+6) /dataset.google_message3_2.pb/Marshal 105ms ± 2% 98ms ± 7% -6.81% (p=0.009 n=5+6) /dataset.google_message3_4.pb/Marshal 16.3ms ± 6% 15.7ms ± 3% -3.44% (p=0.041 n=6+6) /dataset.google_message3_5.pb/Marshal 676ms ± 4% 504ms ± 2% -25.50% (p=0.004 n=6+5) Change-Id: I72cc4597117f4cf5d236ef505777d49dd4a5f75d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171020 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-04-01 13:49:56 -07:00
pack.Tag{46, pack.EndGroupType},
pack.Tag{46, pack.StartGroupType},
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
pack.Tag{47, pack.VarintType}, pack.Varint(2017),
pack.Tag{46, pack.EndGroupType},
}.Marshal(),
},
{
desc: "maps",
internal/legacy: use v2 proto.Unmarshal The v2 decoder isn't 100% complete, but it's good enough. Delete the vendored copy of the v1 Unmarshal implementation. Change-Id: Ibeabbb2e9109a1ec3df57e71f98b7aa4a583fc5b Reviewed-on: https://go-review.googlesource.com/c/154577 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-17 11:16:16 -08:00
decodeTo: []proto.Message{&testpb.TestAllTypes{
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
MapInt32Int32: map[int32]int32{1056: 1156, 2056: 2156},
MapInt64Int64: map[int64]int64{1057: 1157, 2057: 2157},
MapUint32Uint32: map[uint32]uint32{1058: 1158, 2058: 2158},
MapUint64Uint64: map[uint64]uint64{1059: 1159, 2059: 2159},
MapSint32Sint32: map[int32]int32{1060: 1160, 2060: 2160},
MapSint64Sint64: map[int64]int64{1061: 1161, 2061: 2161},
MapFixed32Fixed32: map[uint32]uint32{1062: 1162, 2062: 2162},
MapFixed64Fixed64: map[uint64]uint64{1063: 1163, 2063: 2163},
MapSfixed32Sfixed32: map[int32]int32{1064: 1164, 2064: 2164},
MapSfixed64Sfixed64: map[int64]int64{1065: 1165, 2065: 2165},
MapInt32Float: map[int32]float32{1066: 1166.5, 2066: 2166.5},
MapInt32Double: map[int32]float64{1067: 1167.5, 2067: 2167.5},
MapBoolBool: map[bool]bool{true: false, false: true},
MapStringString: map[string]string{"69.1.key": "69.1.val", "69.2.key": "69.2.val"},
MapStringBytes: map[string][]byte{"70.1.key": []byte("70.1.val"), "70.2.key": []byte("70.2.val")},
MapStringNestedMessage: map[string]*testpb.TestAllTypes_NestedMessage{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
"71.1.key": {A: proto.Int32(1171)},
"71.2.key": {A: proto.Int32(2171)},
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
},
MapStringNestedEnum: map[string]testpb.TestAllTypes_NestedEnum{
"73.1.key": testpb.TestAllTypes_FOO,
"73.2.key": testpb.TestAllTypes_BAR,
},
proto: add proto3 encode/decode test cases Change-Id: I5f859fbd34cbd3e95a6c39a5581791dedc619e6b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169477 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-26 13:55:02 -07:00
}, &test3pb.TestAllTypes{
MapInt32Int32: map[int32]int32{1056: 1156, 2056: 2156},
MapInt64Int64: map[int64]int64{1057: 1157, 2057: 2157},
MapUint32Uint32: map[uint32]uint32{1058: 1158, 2058: 2158},
MapUint64Uint64: map[uint64]uint64{1059: 1159, 2059: 2159},
MapSint32Sint32: map[int32]int32{1060: 1160, 2060: 2160},
MapSint64Sint64: map[int64]int64{1061: 1161, 2061: 2161},
MapFixed32Fixed32: map[uint32]uint32{1062: 1162, 2062: 2162},
MapFixed64Fixed64: map[uint64]uint64{1063: 1163, 2063: 2163},
MapSfixed32Sfixed32: map[int32]int32{1064: 1164, 2064: 2164},
MapSfixed64Sfixed64: map[int64]int64{1065: 1165, 2065: 2165},
MapInt32Float: map[int32]float32{1066: 1166.5, 2066: 2166.5},
MapInt32Double: map[int32]float64{1067: 1167.5, 2067: 2167.5},
MapBoolBool: map[bool]bool{true: false, false: true},
MapStringString: map[string]string{"69.1.key": "69.1.val", "69.2.key": "69.2.val"},
MapStringBytes: map[string][]byte{"70.1.key": []byte("70.1.val"), "70.2.key": []byte("70.2.val")},
MapStringNestedMessage: map[string]*test3pb.TestAllTypes_NestedMessage{
"71.1.key": {A: 1171},
"71.2.key": {A: 2171},
},
MapStringNestedEnum: map[string]test3pb.TestAllTypes_NestedEnum{
"73.1.key": test3pb.TestAllTypes_FOO,
"73.2.key": test3pb.TestAllTypes_BAR,
},
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
}},
wire: pack.Message{
pack.Tag{56, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(1056),
pack.Tag{2, pack.VarintType}, pack.Varint(1156),
}),
pack.Tag{56, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(2056),
pack.Tag{2, pack.VarintType}, pack.Varint(2156),
}),
pack.Tag{57, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(1057),
pack.Tag{2, pack.VarintType}, pack.Varint(1157),
}),
pack.Tag{57, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(2057),
pack.Tag{2, pack.VarintType}, pack.Varint(2157),
}),
pack.Tag{58, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(1058),
pack.Tag{2, pack.VarintType}, pack.Varint(1158),
}),
pack.Tag{58, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(2058),
pack.Tag{2, pack.VarintType}, pack.Varint(2158),
}),
pack.Tag{59, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(1059),
pack.Tag{2, pack.VarintType}, pack.Varint(1159),
}),
pack.Tag{59, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(2059),
pack.Tag{2, pack.VarintType}, pack.Varint(2159),
}),
pack.Tag{60, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Svarint(1060),
pack.Tag{2, pack.VarintType}, pack.Svarint(1160),
}),
pack.Tag{60, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Svarint(2060),
pack.Tag{2, pack.VarintType}, pack.Svarint(2160),
}),
pack.Tag{61, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Svarint(1061),
pack.Tag{2, pack.VarintType}, pack.Svarint(1161),
}),
pack.Tag{61, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Svarint(2061),
pack.Tag{2, pack.VarintType}, pack.Svarint(2161),
}),
pack.Tag{62, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.Fixed32Type}, pack.Int32(1062),
pack.Tag{2, pack.Fixed32Type}, pack.Int32(1162),
}),
pack.Tag{62, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.Fixed32Type}, pack.Int32(2062),
pack.Tag{2, pack.Fixed32Type}, pack.Int32(2162),
}),
pack.Tag{63, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.Fixed64Type}, pack.Int64(1063),
pack.Tag{2, pack.Fixed64Type}, pack.Int64(1163),
}),
pack.Tag{63, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.Fixed64Type}, pack.Int64(2063),
pack.Tag{2, pack.Fixed64Type}, pack.Int64(2163),
}),
pack.Tag{64, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.Fixed32Type}, pack.Int32(1064),
pack.Tag{2, pack.Fixed32Type}, pack.Int32(1164),
}),
pack.Tag{64, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.Fixed32Type}, pack.Int32(2064),
pack.Tag{2, pack.Fixed32Type}, pack.Int32(2164),
}),
pack.Tag{65, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.Fixed64Type}, pack.Int64(1065),
pack.Tag{2, pack.Fixed64Type}, pack.Int64(1165),
}),
pack.Tag{65, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.Fixed64Type}, pack.Int64(2065),
pack.Tag{2, pack.Fixed64Type}, pack.Int64(2165),
}),
pack.Tag{66, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(1066),
pack.Tag{2, pack.Fixed32Type}, pack.Float32(1166.5),
}),
pack.Tag{66, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(2066),
pack.Tag{2, pack.Fixed32Type}, pack.Float32(2166.5),
}),
pack.Tag{67, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(1067),
pack.Tag{2, pack.Fixed64Type}, pack.Float64(1167.5),
}),
pack.Tag{67, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(2067),
pack.Tag{2, pack.Fixed64Type}, pack.Float64(2167.5),
}),
pack.Tag{68, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Bool(true),
pack.Tag{2, pack.VarintType}, pack.Bool(false),
}),
pack.Tag{68, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Bool(false),
pack.Tag{2, pack.VarintType}, pack.Bool(true),
}),
pack.Tag{69, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.BytesType}, pack.String("69.1.key"),
pack.Tag{2, pack.BytesType}, pack.String("69.1.val"),
}),
pack.Tag{69, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.BytesType}, pack.String("69.2.key"),
pack.Tag{2, pack.BytesType}, pack.String("69.2.val"),
}),
pack.Tag{70, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.BytesType}, pack.String("70.1.key"),
pack.Tag{2, pack.BytesType}, pack.String("70.1.val"),
}),
pack.Tag{70, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.BytesType}, pack.String("70.2.key"),
pack.Tag{2, pack.BytesType}, pack.String("70.2.val"),
}),
pack.Tag{71, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.BytesType}, pack.String("71.1.key"),
pack.Tag{2, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(1171),
}),
}),
pack.Tag{71, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.BytesType}, pack.String("71.2.key"),
pack.Tag{2, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(2171),
}),
}),
pack.Tag{73, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.BytesType}, pack.String("73.1.key"),
pack.Tag{2, pack.VarintType}, pack.Varint(int(testpb.TestAllTypes_FOO)),
}),
pack.Tag{73, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.BytesType}, pack.String("73.2.key"),
pack.Tag{2, pack.VarintType}, pack.Varint(int(testpb.TestAllTypes_BAR)),
}),
}.Marshal(),
},
{
proto: add proto3 encode/decode test cases Change-Id: I5f859fbd34cbd3e95a6c39a5581791dedc619e6b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169477 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-26 13:55:02 -07:00
desc: "oneof (uint32)",
decodeTo: []proto.Message{
&testpb.TestAllTypes{OneofField: &testpb.TestAllTypes_OneofUint32{1111}},
&test3pb.TestAllTypes{OneofField: &test3pb.TestAllTypes_OneofUint32{1111}},
},
wire: pack.Message{pack.Tag{111, pack.VarintType}, pack.Varint(1111)}.Marshal(),
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
},
{
desc: "oneof (message)",
proto: add proto3 encode/decode test cases Change-Id: I5f859fbd34cbd3e95a6c39a5581791dedc619e6b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169477 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-26 13:55:02 -07:00
decodeTo: []proto.Message{
&testpb.TestAllTypes{OneofField: &testpb.TestAllTypes_OneofNestedMessage{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
&testpb.TestAllTypes_NestedMessage{A: proto.Int32(1112)},
proto: add proto3 encode/decode test cases Change-Id: I5f859fbd34cbd3e95a6c39a5581791dedc619e6b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169477 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-26 13:55:02 -07:00
}}, &test3pb.TestAllTypes{OneofField: &test3pb.TestAllTypes_OneofNestedMessage{
&test3pb.TestAllTypes_NestedMessage{A: 1112},
}},
},
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
wire: pack.Message{pack.Tag{112, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Message{pack.Tag{1, pack.VarintType}, pack.Varint(1112)},
})}.Marshal(),
},
internal/impl: add fast-path marshal implementation This is a port of the v1 table marshaler, with some substantial cleanup and refactoring. Benchstat results from the protobuf reference benchmark data comparing the v1 package with v2, with AllowPartial:true set for the new package. This is not an apples-to-apples comparison, since v1 doesn't have a way to disable required field checks. Required field checks in v2 package currently go through reflection, which performs terribly; my initial experimentation indicates that fast-path required field checks will not add a large amount of cost; these results are incomplete but not wholly inaccurate. name old time/op new time/op delta /dataset.google_message3_1.pb/Marshal-12 219ms ± 1% 232ms ± 1% +5.85% (p=0.004 n=6+5) /dataset.google_message2.pb/Marshal-12 261µs ± 3% 248µs ± 1% -5.14% (p=0.002 n=6+6) /dataset.google_message1_proto2.pb/Marshal-12 681ns ± 2% 637ns ± 3% -6.53% (p=0.002 n=6+6) /dataset.google_message1_proto3.pb/Marshal-12 1.10µs ± 8% 0.99µs ± 3% -9.63% (p=0.002 n=6+6) /dataset.google_message3_3.pb/Marshal-12 44.2ms ± 3% 35.2ms ± 1% -20.28% (p=0.004 n=6+5) /dataset.google_message4.pb/Marshal-12 91.4ms ± 2% 94.9ms ± 2% +3.78% (p=0.002 n=6+6) /dataset.google_message3_2.pb/Marshal-12 78.7ms ± 6% 80.8ms ± 4% ~ (p=0.310 n=6+6) /dataset.google_message3_4.pb/Marshal-12 10.6ms ± 3% 10.6ms ± 8% ~ (p=0.662 n=5+6) /dataset.google_message3_5.pb/Marshal-12 675ms ± 4% 510ms ± 2% -24.40% (p=0.002 n=6+6) /dataset.google_message3_1.pb/Marshal 219ms ± 1% 236ms ± 7% +8.06% (p=0.004 n=5+6) /dataset.google_message2.pb/Marshal 257µs ± 1% 250µs ± 3% ~ (p=0.052 n=5+6) /dataset.google_message1_proto2.pb/Marshal 685ns ± 1% 628ns ± 1% -8.41% (p=0.008 n=5+5) /dataset.google_message1_proto3.pb/Marshal 1.08µs ± 1% 0.98µs ± 2% -9.31% (p=0.004 n=5+6) /dataset.google_message3_3.pb/Marshal 43.7ms ± 1% 35.1ms ± 1% -19.76% (p=0.002 n=6+6) /dataset.google_message4.pb/Marshal 93.4ms ± 4% 94.9ms ± 2% ~ (p=0.180 n=6+6) /dataset.google_message3_2.pb/Marshal 105ms ± 2% 98ms ± 7% -6.81% (p=0.009 n=5+6) /dataset.google_message3_4.pb/Marshal 16.3ms ± 6% 15.7ms ± 3% -3.44% (p=0.041 n=6+6) /dataset.google_message3_5.pb/Marshal 676ms ± 4% 504ms ± 2% -25.50% (p=0.004 n=6+5) Change-Id: I72cc4597117f4cf5d236ef505777d49dd4a5f75d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171020 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-04-01 13:49:56 -07:00
{
desc: "oneof (empty message)",
decodeTo: []proto.Message{
&testpb.TestAllTypes{OneofField: &testpb.TestAllTypes_OneofNestedMessage{
&testpb.TestAllTypes_NestedMessage{},
}},
&test3pb.TestAllTypes{OneofField: &test3pb.TestAllTypes_OneofNestedMessage{
&test3pb.TestAllTypes_NestedMessage{},
}},
},
wire: pack.Message{pack.Tag{112, pack.BytesType}, pack.LengthPrefix(pack.Message{})}.Marshal(),
},
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
{
proto: fix merge semantics for oneof message The proper semantics for a message field within a oneof when unmarshaling is to merge into an existing message, rather than replacing it. Change-Id: I7c08f6e4fa958c6ee6241e9083f7311515a97e15 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185957 Reviewed-by: Damien Neil <dneil@google.com>
2019-07-11 23:15:05 -07:00
desc: "oneof (merged message)",
proto: add proto3 encode/decode test cases Change-Id: I5f859fbd34cbd3e95a6c39a5581791dedc619e6b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169477 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-26 13:55:02 -07:00
decodeTo: []proto.Message{
&testpb.TestAllTypes{OneofField: &testpb.TestAllTypes_OneofNestedMessage{
&testpb.TestAllTypes_NestedMessage{
runtime/protoiface: move and rename XXX_Methods This CL moves and renames the protoreflect.ProtoMessage.XXX_Methods to protoreflect.Message.ProtoMethods. Since one needs to obtain a protoreflect.Message now to get at the fast-path methods, we modify the method signatures to take in a protoreflect.Message instead of protoreflect.ProtoMessage. Doing so also avoids the wrapper hack that was formerly done on impl.messageReflectWrapper. After this change the new protoc-gen-go no longer generates any XXX fields or methods. All internal fields and methods are truly hidden from the end-user. name old time/op new time/op delta Wire/Unmarshal/google_message1_proto2-4 1.50µs ±10% 1.50µs ± 2% ~ (p=0.483 n=10+9) Wire/Unmarshal/google_message1_proto3-4 1.06µs ± 6% 1.06µs ± 4% ~ (p=0.814 n=9+9) Wire/Unmarshal/google_message2-4 734µs ±22% 689µs ±13% ~ (p=0.133 n=10+9) Wire/Marshal/google_message1_proto2-4 790ns ±46% 652ns ± 8% ~ (p=0.590 n=10+9) Wire/Marshal/google_message1_proto3-4 872ns ± 4% 857ns ± 3% ~ (p=0.168 n=9+9) Wire/Marshal/google_message2-4 232µs ±16% 221µs ± 3% -4.75% (p=0.014 n=9+9) Wire/Size/google_message1_proto2-4 164ns ± 2% 167ns ± 4% +1.87% (p=0.046 n=9+10) Wire/Size/google_message1_proto3-4 240ns ± 9% 229ns ± 1% -4.81% (p=0.000 n=9+8) Wire/Size/google_message2-4 58.9µs ± 9% 59.6µs ± 2% +1.23% (p=0.040 n=9+9) name old alloc/op new alloc/op delta Wire/Unmarshal/google_message1_proto2-4 912B ± 0% 912B ± 0% ~ (all equal) Wire/Unmarshal/google_message1_proto3-4 688B ± 0% 688B ± 0% ~ (all equal) Wire/Unmarshal/google_message2-4 470kB ± 0% 470kB ± 0% ~ (p=0.215 n=10+10) Wire/Marshal/google_message1_proto2-4 240B ± 0% 240B ± 0% ~ (all equal) Wire/Marshal/google_message1_proto3-4 224B ± 0% 224B ± 0% ~ (all equal) Wire/Marshal/google_message2-4 90.1kB ± 0% 90.1kB ± 0% ~ (all equal) Wire/Size/google_message1_proto2-4 0.00B 0.00B ~ (all equal) Wire/Size/google_message1_proto3-4 0.00B 0.00B ~ (all equal) Wire/Size/google_message2-4 0.00B 0.00B ~ (all equal) name old allocs/op new allocs/op delta Wire/Unmarshal/google_message1_proto2-4 24.0 ± 0% 24.0 ± 0% ~ (all equal) Wire/Unmarshal/google_message1_proto3-4 6.00 ± 0% 6.00 ± 0% ~ (all equal) Wire/Unmarshal/google_message2-4 8.49k ± 0% 8.49k ± 0% ~ (all equal) Wire/Marshal/google_message1_proto2-4 1.00 ± 0% 1.00 ± 0% ~ (all equal) Wire/Marshal/google_message1_proto3-4 1.00 ± 0% 1.00 ± 0% ~ (all equal) Wire/Marshal/google_message2-4 1.00 ± 0% 1.00 ± 0% ~ (all equal) Wire/Size/google_message1_proto2-4 0.00 0.00 ~ (all equal) Wire/Size/google_message1_proto3-4 0.00 0.00 ~ (all equal) Wire/Size/google_message2-4 0.00 0.00 ~ (all equal) Change-Id: Ibf3263ad0f293326695c22020a92a6b938ef4f65 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185697 Reviewed-by: Damien Neil <dneil@google.com>
2019-07-10 23:14:31 -07:00
A: proto.Int32(1),
proto: add proto3 encode/decode test cases Change-Id: I5f859fbd34cbd3e95a6c39a5581791dedc619e6b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169477 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-26 13:55:02 -07:00
Corecursive: &testpb.TestAllTypes{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
OptionalInt32: proto.Int32(43),
proto: add proto3 encode/decode test cases Change-Id: I5f859fbd34cbd3e95a6c39a5581791dedc619e6b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169477 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-26 13:55:02 -07:00
},
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
},
proto: add proto3 encode/decode test cases Change-Id: I5f859fbd34cbd3e95a6c39a5581791dedc619e6b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169477 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-26 13:55:02 -07:00
}}, &test3pb.TestAllTypes{OneofField: &test3pb.TestAllTypes_OneofNestedMessage{
&test3pb.TestAllTypes_NestedMessage{
proto: fix merge semantics for oneof message The proper semantics for a message field within a oneof when unmarshaling is to merge into an existing message, rather than replacing it. Change-Id: I7c08f6e4fa958c6ee6241e9083f7311515a97e15 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185957 Reviewed-by: Damien Neil <dneil@google.com>
2019-07-11 23:15:05 -07:00
A: 1,
proto: add proto3 encode/decode test cases Change-Id: I5f859fbd34cbd3e95a6c39a5581791dedc619e6b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169477 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-26 13:55:02 -07:00
Corecursive: &test3pb.TestAllTypes{
OptionalInt32: 43,
},
},
}}},
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
wire: pack.Message{
pack.Tag{112, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Message{pack.Tag{1, pack.VarintType}, pack.Varint(1)},
}),
pack.Tag{112, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{2, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(43),
}),
}),
}.Marshal(),
},
{
proto: add proto3 encode/decode test cases Change-Id: I5f859fbd34cbd3e95a6c39a5581791dedc619e6b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169477 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-26 13:55:02 -07:00
desc: "oneof (string)",
decodeTo: []proto.Message{
&testpb.TestAllTypes{OneofField: &testpb.TestAllTypes_OneofString{"1113"}},
&test3pb.TestAllTypes{OneofField: &test3pb.TestAllTypes_OneofString{"1113"}},
},
wire: pack.Message{pack.Tag{113, pack.BytesType}, pack.String("1113")}.Marshal(),
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
},
{
proto: add proto3 encode/decode test cases Change-Id: I5f859fbd34cbd3e95a6c39a5581791dedc619e6b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169477 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-26 13:55:02 -07:00
desc: "oneof (bytes)",
decodeTo: []proto.Message{
&testpb.TestAllTypes{OneofField: &testpb.TestAllTypes_OneofBytes{[]byte("1114")}},
&test3pb.TestAllTypes{OneofField: &test3pb.TestAllTypes_OneofBytes{[]byte("1114")}},
},
wire: pack.Message{pack.Tag{114, pack.BytesType}, pack.String("1114")}.Marshal(),
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
},
{
proto: add proto3 encode/decode test cases Change-Id: I5f859fbd34cbd3e95a6c39a5581791dedc619e6b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169477 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-26 13:55:02 -07:00
desc: "oneof (bool)",
decodeTo: []proto.Message{
&testpb.TestAllTypes{OneofField: &testpb.TestAllTypes_OneofBool{true}},
&test3pb.TestAllTypes{OneofField: &test3pb.TestAllTypes_OneofBool{true}},
},
wire: pack.Message{pack.Tag{115, pack.VarintType}, pack.Bool(true)}.Marshal(),
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
},
{
proto: add proto3 encode/decode test cases Change-Id: I5f859fbd34cbd3e95a6c39a5581791dedc619e6b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169477 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-26 13:55:02 -07:00
desc: "oneof (uint64)",
decodeTo: []proto.Message{
&testpb.TestAllTypes{OneofField: &testpb.TestAllTypes_OneofUint64{116}},
&test3pb.TestAllTypes{OneofField: &test3pb.TestAllTypes_OneofUint64{116}},
},
wire: pack.Message{pack.Tag{116, pack.VarintType}, pack.Varint(116)}.Marshal(),
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
},
{
proto: add proto3 encode/decode test cases Change-Id: I5f859fbd34cbd3e95a6c39a5581791dedc619e6b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169477 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-26 13:55:02 -07:00
desc: "oneof (float)",
decodeTo: []proto.Message{
&testpb.TestAllTypes{OneofField: &testpb.TestAllTypes_OneofFloat{117.5}},
&test3pb.TestAllTypes{OneofField: &test3pb.TestAllTypes_OneofFloat{117.5}},
},
wire: pack.Message{pack.Tag{117, pack.Fixed32Type}, pack.Float32(117.5)}.Marshal(),
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
},
{
proto: add proto3 encode/decode test cases Change-Id: I5f859fbd34cbd3e95a6c39a5581791dedc619e6b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169477 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-26 13:55:02 -07:00
desc: "oneof (double)",
decodeTo: []proto.Message{
&testpb.TestAllTypes{OneofField: &testpb.TestAllTypes_OneofDouble{118.5}},
&test3pb.TestAllTypes{OneofField: &test3pb.TestAllTypes_OneofDouble{118.5}},
},
wire: pack.Message{pack.Tag{118, pack.Fixed64Type}, pack.Float64(118.5)}.Marshal(),
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
},
{
proto: add proto3 encode/decode test cases Change-Id: I5f859fbd34cbd3e95a6c39a5581791dedc619e6b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169477 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-26 13:55:02 -07:00
desc: "oneof (enum)",
decodeTo: []proto.Message{
&testpb.TestAllTypes{OneofField: &testpb.TestAllTypes_OneofEnum{testpb.TestAllTypes_BAR}},
&test3pb.TestAllTypes{OneofField: &test3pb.TestAllTypes_OneofEnum{test3pb.TestAllTypes_BAR}},
},
wire: pack.Message{pack.Tag{119, pack.VarintType}, pack.Varint(int(testpb.TestAllTypes_BAR))}.Marshal(),
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
},
internal/impl: add fast-path marshal implementation This is a port of the v1 table marshaler, with some substantial cleanup and refactoring. Benchstat results from the protobuf reference benchmark data comparing the v1 package with v2, with AllowPartial:true set for the new package. This is not an apples-to-apples comparison, since v1 doesn't have a way to disable required field checks. Required field checks in v2 package currently go through reflection, which performs terribly; my initial experimentation indicates that fast-path required field checks will not add a large amount of cost; these results are incomplete but not wholly inaccurate. name old time/op new time/op delta /dataset.google_message3_1.pb/Marshal-12 219ms ± 1% 232ms ± 1% +5.85% (p=0.004 n=6+5) /dataset.google_message2.pb/Marshal-12 261µs ± 3% 248µs ± 1% -5.14% (p=0.002 n=6+6) /dataset.google_message1_proto2.pb/Marshal-12 681ns ± 2% 637ns ± 3% -6.53% (p=0.002 n=6+6) /dataset.google_message1_proto3.pb/Marshal-12 1.10µs ± 8% 0.99µs ± 3% -9.63% (p=0.002 n=6+6) /dataset.google_message3_3.pb/Marshal-12 44.2ms ± 3% 35.2ms ± 1% -20.28% (p=0.004 n=6+5) /dataset.google_message4.pb/Marshal-12 91.4ms ± 2% 94.9ms ± 2% +3.78% (p=0.002 n=6+6) /dataset.google_message3_2.pb/Marshal-12 78.7ms ± 6% 80.8ms ± 4% ~ (p=0.310 n=6+6) /dataset.google_message3_4.pb/Marshal-12 10.6ms ± 3% 10.6ms ± 8% ~ (p=0.662 n=5+6) /dataset.google_message3_5.pb/Marshal-12 675ms ± 4% 510ms ± 2% -24.40% (p=0.002 n=6+6) /dataset.google_message3_1.pb/Marshal 219ms ± 1% 236ms ± 7% +8.06% (p=0.004 n=5+6) /dataset.google_message2.pb/Marshal 257µs ± 1% 250µs ± 3% ~ (p=0.052 n=5+6) /dataset.google_message1_proto2.pb/Marshal 685ns ± 1% 628ns ± 1% -8.41% (p=0.008 n=5+5) /dataset.google_message1_proto3.pb/Marshal 1.08µs ± 1% 0.98µs ± 2% -9.31% (p=0.004 n=5+6) /dataset.google_message3_3.pb/Marshal 43.7ms ± 1% 35.1ms ± 1% -19.76% (p=0.002 n=6+6) /dataset.google_message4.pb/Marshal 93.4ms ± 4% 94.9ms ± 2% ~ (p=0.180 n=6+6) /dataset.google_message3_2.pb/Marshal 105ms ± 2% 98ms ± 7% -6.81% (p=0.009 n=5+6) /dataset.google_message3_4.pb/Marshal 16.3ms ± 6% 15.7ms ± 3% -3.44% (p=0.041 n=6+6) /dataset.google_message3_5.pb/Marshal 676ms ± 4% 504ms ± 2% -25.50% (p=0.004 n=6+5) Change-Id: I72cc4597117f4cf5d236ef505777d49dd4a5f75d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171020 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-04-01 13:49:56 -07:00
{
desc: "oneof (zero)",
decodeTo: []proto.Message{
&testpb.TestAllTypes{OneofField: &testpb.TestAllTypes_OneofUint64{0}},
&test3pb.TestAllTypes{OneofField: &test3pb.TestAllTypes_OneofUint64{0}},
},
wire: pack.Message{pack.Tag{116, pack.VarintType}, pack.Varint(0)}.Marshal(),
},
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
{
proto: add proto3 encode/decode test cases Change-Id: I5f859fbd34cbd3e95a6c39a5581791dedc619e6b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169477 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-26 13:55:02 -07:00
desc: "oneof (overridden value)",
decodeTo: []proto.Message{
&testpb.TestAllTypes{OneofField: &testpb.TestAllTypes_OneofUint64{2}},
&test3pb.TestAllTypes{OneofField: &test3pb.TestAllTypes_OneofUint64{2}},
},
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
wire: pack.Message{
pack.Tag{111, pack.VarintType}, pack.Varint(1),
pack.Tag{116, pack.VarintType}, pack.Varint(2),
}.Marshal(),
},
// TODO: More unknown field tests for ordering, repeated fields, etc.
//
// It is currently impossible to produce results that the v1 Equal
// considers equivalent to those of the v1 decoder. Figure out if
// that's a problem or not.
{
desc: "unknown fields",
internal/legacy: use v2 proto.Unmarshal The v2 decoder isn't 100% complete, but it's good enough. Delete the vendored copy of the v1 Unmarshal implementation. Change-Id: Ibeabbb2e9109a1ec3df57e71f98b7aa4a583fc5b Reviewed-on: https://go-review.googlesource.com/c/154577 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-17 11:16:16 -08:00
decodeTo: []proto.Message{build(
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
&testpb.TestAllTypes{},
reflect/protoreflect: add alternative message reflection API Added API: Message.Len Message.Range Message.Has Message.Clear Message.Get Message.Set Message.Mutable Message.NewMessage Message.WhichOneof Message.GetUnknown Message.SetUnknown Deprecated API (to be removed in subsequent CL): Message.KnownFields Message.UnknownFields The primary difference with the new API is that the top-level Message methods are keyed by FieldDescriptor rather than FieldNumber with the following semantics: * For known fields, the FieldDescriptor must exactly match the field descriptor known by the message. * For extension fields, the FieldDescriptor must implement ExtensionType, where ContainingMessage.FullName matches the message name, and the field number is within the message's extension range. When setting an extension field, it automatically stores the extension type information. * Extension fields are always considered nullable, implying that repeated extension fields are nullable. That is, you can distinguish between a unpopulated list and an empty list. * Message.Get always returns a valid Value even if unpopulated. The behavior is already well-defined for scalars, but for unpopulated composite types, it now returns an empty read-only version of it. Change-Id: Ia120630b4db221aeaaf743d0f64160e1a61a0f61 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/175458 Reviewed-by: Damien Neil <dneil@google.com>
2019-04-25 23:48:08 -07:00
unknown(pack.Message{
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
pack.Tag{100000, pack.VarintType}, pack.Varint(1),
}.Marshal()),
proto: add proto3 encode/decode test cases Change-Id: I5f859fbd34cbd3e95a6c39a5581791dedc619e6b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169477 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-26 13:55:02 -07:00
), build(
&test3pb.TestAllTypes{},
reflect/protoreflect: add alternative message reflection API Added API: Message.Len Message.Range Message.Has Message.Clear Message.Get Message.Set Message.Mutable Message.NewMessage Message.WhichOneof Message.GetUnknown Message.SetUnknown Deprecated API (to be removed in subsequent CL): Message.KnownFields Message.UnknownFields The primary difference with the new API is that the top-level Message methods are keyed by FieldDescriptor rather than FieldNumber with the following semantics: * For known fields, the FieldDescriptor must exactly match the field descriptor known by the message. * For extension fields, the FieldDescriptor must implement ExtensionType, where ContainingMessage.FullName matches the message name, and the field number is within the message's extension range. When setting an extension field, it automatically stores the extension type information. * Extension fields are always considered nullable, implying that repeated extension fields are nullable. That is, you can distinguish between a unpopulated list and an empty list. * Message.Get always returns a valid Value even if unpopulated. The behavior is already well-defined for scalars, but for unpopulated composite types, it now returns an empty read-only version of it. Change-Id: Ia120630b4db221aeaaf743d0f64160e1a61a0f61 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/175458 Reviewed-by: Damien Neil <dneil@google.com>
2019-04-25 23:48:08 -07:00
unknown(pack.Message{
proto: add proto3 encode/decode test cases Change-Id: I5f859fbd34cbd3e95a6c39a5581791dedc619e6b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169477 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-26 13:55:02 -07:00
pack.Tag{100000, pack.VarintType}, pack.Varint(1),
}.Marshal()),
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
)},
wire: pack.Message{
pack.Tag{100000, pack.VarintType}, pack.Varint(1),
}.Marshal(),
},
{
desc: "field type mismatch",
internal/legacy: use v2 proto.Unmarshal The v2 decoder isn't 100% complete, but it's good enough. Delete the vendored copy of the v1 Unmarshal implementation. Change-Id: Ibeabbb2e9109a1ec3df57e71f98b7aa4a583fc5b Reviewed-on: https://go-review.googlesource.com/c/154577 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-17 11:16:16 -08:00
decodeTo: []proto.Message{build(
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
&testpb.TestAllTypes{},
reflect/protoreflect: add alternative message reflection API Added API: Message.Len Message.Range Message.Has Message.Clear Message.Get Message.Set Message.Mutable Message.NewMessage Message.WhichOneof Message.GetUnknown Message.SetUnknown Deprecated API (to be removed in subsequent CL): Message.KnownFields Message.UnknownFields The primary difference with the new API is that the top-level Message methods are keyed by FieldDescriptor rather than FieldNumber with the following semantics: * For known fields, the FieldDescriptor must exactly match the field descriptor known by the message. * For extension fields, the FieldDescriptor must implement ExtensionType, where ContainingMessage.FullName matches the message name, and the field number is within the message's extension range. When setting an extension field, it automatically stores the extension type information. * Extension fields are always considered nullable, implying that repeated extension fields are nullable. That is, you can distinguish between a unpopulated list and an empty list. * Message.Get always returns a valid Value even if unpopulated. The behavior is already well-defined for scalars, but for unpopulated composite types, it now returns an empty read-only version of it. Change-Id: Ia120630b4db221aeaaf743d0f64160e1a61a0f61 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/175458 Reviewed-by: Damien Neil <dneil@google.com>
2019-04-25 23:48:08 -07:00
unknown(pack.Message{
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
pack.Tag{1, pack.BytesType}, pack.String("string"),
}.Marshal()),
proto: add proto3 encode/decode test cases Change-Id: I5f859fbd34cbd3e95a6c39a5581791dedc619e6b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169477 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-26 13:55:02 -07:00
), build(
&test3pb.TestAllTypes{},
reflect/protoreflect: add alternative message reflection API Added API: Message.Len Message.Range Message.Has Message.Clear Message.Get Message.Set Message.Mutable Message.NewMessage Message.WhichOneof Message.GetUnknown Message.SetUnknown Deprecated API (to be removed in subsequent CL): Message.KnownFields Message.UnknownFields The primary difference with the new API is that the top-level Message methods are keyed by FieldDescriptor rather than FieldNumber with the following semantics: * For known fields, the FieldDescriptor must exactly match the field descriptor known by the message. * For extension fields, the FieldDescriptor must implement ExtensionType, where ContainingMessage.FullName matches the message name, and the field number is within the message's extension range. When setting an extension field, it automatically stores the extension type information. * Extension fields are always considered nullable, implying that repeated extension fields are nullable. That is, you can distinguish between a unpopulated list and an empty list. * Message.Get always returns a valid Value even if unpopulated. The behavior is already well-defined for scalars, but for unpopulated composite types, it now returns an empty read-only version of it. Change-Id: Ia120630b4db221aeaaf743d0f64160e1a61a0f61 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/175458 Reviewed-by: Damien Neil <dneil@google.com>
2019-04-25 23:48:08 -07:00
unknown(pack.Message{
proto: add proto3 encode/decode test cases Change-Id: I5f859fbd34cbd3e95a6c39a5581791dedc619e6b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169477 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-26 13:55:02 -07:00
pack.Tag{1, pack.BytesType}, pack.String("string"),
}.Marshal()),
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
)},
wire: pack.Message{
pack.Tag{1, pack.BytesType}, pack.String("string"),
}.Marshal(),
},
{
desc: "map field element mismatch",
internal/legacy: use v2 proto.Unmarshal The v2 decoder isn't 100% complete, but it's good enough. Delete the vendored copy of the v1 Unmarshal implementation. Change-Id: Ibeabbb2e9109a1ec3df57e71f98b7aa4a583fc5b Reviewed-on: https://go-review.googlesource.com/c/154577 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-17 11:16:16 -08:00
decodeTo: []proto.Message{
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
&testpb.TestAllTypes{
MapInt32Int32: map[int32]int32{1: 0},
proto: add proto3 encode/decode test cases Change-Id: I5f859fbd34cbd3e95a6c39a5581791dedc619e6b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169477 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-26 13:55:02 -07:00
}, &test3pb.TestAllTypes{
MapInt32Int32: map[int32]int32{1: 0},
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
},
},
wire: pack.Message{
pack.Tag{56, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(1),
pack.Tag{2, pack.BytesType}, pack.String("string"),
}),
}.Marshal(),
},
proto, internal/impl: make IsInitialized more consistent Make the fast-path and slow-path versions of IsInitialized report exactly the same errors: An errors.RequiredNotSet containing the full name of one of the unset required fields. Bugfix: Fast-path IsInitialized on a nil message reports an error only when the message directly contains required fields. Bugfix: Include fast-path IsInitialized in legacy messageIfaceWrapper. Fixes golang/protobuf#887 Change-Id: Ia5e4b386f8c23f6f855d995f4a098b1338acbae3 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185397 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-09 11:40:49 -07:00
{
desc: "required field in nil message unset",
partial: true,
decodeTo: []proto.Message{(*testpb.TestRequired)(nil)},
},
proto: check for required fields in encoding/decoding Change-Id: I0555a92e0399782f075b1dcd248e880dd48c7d6d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170579 Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-03 12:17:24 -07:00
{
desc: "required field unset",
partial: true,
decodeTo: []proto.Message{&testpb.TestRequired{}},
},
{
desc: "required field set",
decodeTo: []proto.Message{&testpb.TestRequired{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
RequiredField: proto.Int32(1),
proto: check for required fields in encoding/decoding Change-Id: I0555a92e0399782f075b1dcd248e880dd48c7d6d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170579 Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-03 12:17:24 -07:00
}},
wire: pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(1),
}.Marshal(),
},
{
desc: "required field in optional message unset",
partial: true,
decodeTo: []proto.Message{&testpb.TestRequiredForeign{
OptionalMessage: &testpb.TestRequired{},
}},
wire: pack.Message{
pack.Tag{1, pack.BytesType}, pack.LengthPrefix(pack.Message{}),
}.Marshal(),
},
{
desc: "required field in optional message set",
decodeTo: []proto.Message{&testpb.TestRequiredForeign{
OptionalMessage: &testpb.TestRequired{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
RequiredField: proto.Int32(1),
proto: check for required fields in encoding/decoding Change-Id: I0555a92e0399782f075b1dcd248e880dd48c7d6d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170579 Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-03 12:17:24 -07:00
},
}},
wire: pack.Message{
pack.Tag{1, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(1),
}),
}.Marshal(),
},
proto: add IsInitialized Move all checks for required fields into a proto.IsInitialized function. Initial testing makes me confident that we can provide a fast-path implementation of IsInitialized which will perform more than acceptably. (In the degenerate-but-common case where a message transitively contains no required fields, this check can be nearly zero cost.) Unifying checks into a single function provides consistent behavior between the wire, text, and json codecs. Performing the check after decoding eliminates the wire decoder bug where a split message is incorrectly seen as missing required fields. Performing the check after decoding also provides consistent and arguably more correct behavior when the target message was partially prepopulated. Change-Id: I9478b7bebb263af00c0d9f66a1f26e31ff553522 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170787 Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-05 13:31:40 -07:00
{
desc: "required field in optional message set (split across multiple tags)",
decodeTo: []proto.Message{&testpb.TestRequiredForeign{
OptionalMessage: &testpb.TestRequired{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
RequiredField: proto.Int32(1),
proto: add IsInitialized Move all checks for required fields into a proto.IsInitialized function. Initial testing makes me confident that we can provide a fast-path implementation of IsInitialized which will perform more than acceptably. (In the degenerate-but-common case where a message transitively contains no required fields, this check can be nearly zero cost.) Unifying checks into a single function provides consistent behavior between the wire, text, and json codecs. Performing the check after decoding eliminates the wire decoder bug where a split message is incorrectly seen as missing required fields. Performing the check after decoding also provides consistent and arguably more correct behavior when the target message was partially prepopulated. Change-Id: I9478b7bebb263af00c0d9f66a1f26e31ff553522 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170787 Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-05 13:31:40 -07:00
},
}},
wire: pack.Message{
pack.Tag{1, pack.BytesType}, pack.LengthPrefix(pack.Message{}),
pack.Tag{1, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(1),
}),
}.Marshal(),
},
proto: check for required fields in encoding/decoding Change-Id: I0555a92e0399782f075b1dcd248e880dd48c7d6d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170579 Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-03 12:17:24 -07:00
{
desc: "required field in repeated message unset",
partial: true,
decodeTo: []proto.Message{&testpb.TestRequiredForeign{
RepeatedMessage: []*testpb.TestRequired{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
{RequiredField: proto.Int32(1)},
proto: check for required fields in encoding/decoding Change-Id: I0555a92e0399782f075b1dcd248e880dd48c7d6d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170579 Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-03 12:17:24 -07:00
{},
},
}},
wire: pack.Message{
pack.Tag{2, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(1),
}),
pack.Tag{2, pack.BytesType}, pack.LengthPrefix(pack.Message{}),
}.Marshal(),
},
{
desc: "required field in repeated message set",
decodeTo: []proto.Message{&testpb.TestRequiredForeign{
RepeatedMessage: []*testpb.TestRequired{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
{RequiredField: proto.Int32(1)},
{RequiredField: proto.Int32(2)},
proto: check for required fields in encoding/decoding Change-Id: I0555a92e0399782f075b1dcd248e880dd48c7d6d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170579 Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-03 12:17:24 -07:00
},
}},
wire: pack.Message{
pack.Tag{2, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(1),
}),
pack.Tag{2, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(2),
}),
}.Marshal(),
},
{
desc: "required field in map message unset",
partial: true,
decodeTo: []proto.Message{&testpb.TestRequiredForeign{
MapMessage: map[int32]*testpb.TestRequired{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
1: {RequiredField: proto.Int32(1)},
proto: check for required fields in encoding/decoding Change-Id: I0555a92e0399782f075b1dcd248e880dd48c7d6d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170579 Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-03 12:17:24 -07:00
2: {},
},
}},
wire: pack.Message{
pack.Tag{3, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(1),
pack.Tag{2, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(1),
}),
}),
pack.Tag{3, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(2),
pack.Tag{2, pack.BytesType}, pack.LengthPrefix(pack.Message{}),
}),
}.Marshal(),
},
{
desc: "required field in map message set",
decodeTo: []proto.Message{&testpb.TestRequiredForeign{
MapMessage: map[int32]*testpb.TestRequired{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
1: {RequiredField: proto.Int32(1)},
2: {RequiredField: proto.Int32(2)},
proto: check for required fields in encoding/decoding Change-Id: I0555a92e0399782f075b1dcd248e880dd48c7d6d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170579 Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-03 12:17:24 -07:00
},
}},
wire: pack.Message{
pack.Tag{3, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(1),
pack.Tag{2, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(1),
}),
}),
pack.Tag{3, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(2),
pack.Tag{2, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(2),
}),
}),
}.Marshal(),
},
{
desc: "required field in optional group unset",
partial: true,
decodeTo: []proto.Message{&testpb.TestRequiredGroupFields{
Optionalgroup: &testpb.TestRequiredGroupFields_OptionalGroup{},
}},
wire: pack.Message{
pack.Tag{1, pack.StartGroupType},
pack.Tag{1, pack.EndGroupType},
}.Marshal(),
},
{
desc: "required field in optional group set",
decodeTo: []proto.Message{&testpb.TestRequiredGroupFields{
Optionalgroup: &testpb.TestRequiredGroupFields_OptionalGroup{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
A: proto.Int32(1),
proto: check for required fields in encoding/decoding Change-Id: I0555a92e0399782f075b1dcd248e880dd48c7d6d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170579 Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-03 12:17:24 -07:00
},
}},
wire: pack.Message{
pack.Tag{1, pack.StartGroupType},
pack.Tag{2, pack.VarintType}, pack.Varint(1),
pack.Tag{1, pack.EndGroupType},
}.Marshal(),
},
{
desc: "required field in repeated group unset",
partial: true,
decodeTo: []proto.Message{&testpb.TestRequiredGroupFields{
Repeatedgroup: []*testpb.TestRequiredGroupFields_RepeatedGroup{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
{A: proto.Int32(1)},
proto: check for required fields in encoding/decoding Change-Id: I0555a92e0399782f075b1dcd248e880dd48c7d6d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170579 Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-03 12:17:24 -07:00
{},
},
}},
wire: pack.Message{
pack.Tag{3, pack.StartGroupType},
pack.Tag{4, pack.VarintType}, pack.Varint(1),
pack.Tag{3, pack.EndGroupType},
pack.Tag{3, pack.StartGroupType},
pack.Tag{3, pack.EndGroupType},
}.Marshal(),
},
{
desc: "required field in repeated group set",
decodeTo: []proto.Message{&testpb.TestRequiredGroupFields{
Repeatedgroup: []*testpb.TestRequiredGroupFields_RepeatedGroup{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
{A: proto.Int32(1)},
{A: proto.Int32(2)},
proto: check for required fields in encoding/decoding Change-Id: I0555a92e0399782f075b1dcd248e880dd48c7d6d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170579 Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-03 12:17:24 -07:00
},
}},
wire: pack.Message{
pack.Tag{3, pack.StartGroupType},
pack.Tag{4, pack.VarintType}, pack.Varint(1),
pack.Tag{3, pack.EndGroupType},
pack.Tag{3, pack.StartGroupType},
pack.Tag{4, pack.VarintType}, pack.Varint(2),
pack.Tag{3, pack.EndGroupType},
}.Marshal(),
},
internal/impl: add fast-path for IsInitialized This currently returns uninformative errors from the fast path and then consults the slow, reflection-based path only when an error is detected. Perhaps it's worth going through the effort of producing better errors directly on the fast path. Change-Id: I68536e9438010dbd97dbaff4f47b78430221d94b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171462 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-04-09 15:57:05 -07:00
{
desc: "required field in oneof message unset",
partial: true,
decodeTo: []proto.Message{
&testpb.TestRequiredForeign{OneofField: &testpb.TestRequiredForeign_OneofMessage{
&testpb.TestRequired{},
}},
},
wire: pack.Message{pack.Tag{4, pack.BytesType}, pack.LengthPrefix(pack.Message{})}.Marshal(),
},
{
desc: "required field in oneof message set",
decodeTo: []proto.Message{
&testpb.TestRequiredForeign{OneofField: &testpb.TestRequiredForeign_OneofMessage{
&testpb.TestRequired{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
RequiredField: proto.Int32(1),
internal/impl: add fast-path for IsInitialized This currently returns uninformative errors from the fast path and then consults the slow, reflection-based path only when an error is detected. Perhaps it's worth going through the effort of producing better errors directly on the fast path. Change-Id: I68536e9438010dbd97dbaff4f47b78430221d94b Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171462 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-04-09 15:57:05 -07:00
},
}},
},
wire: pack.Message{pack.Tag{4, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(1),
})}.Marshal(),
},
proto: check for required fields in encoding/decoding Change-Id: I0555a92e0399782f075b1dcd248e880dd48c7d6d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170579 Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-03 12:17:24 -07:00
{
proto: cleanup invalid extension exception The invalidExtensions flag no longer seems necessary. Tests pass without it. Change-Id: Ieb35e26912b047718ccbfcdc926625aec1cd8c87 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185937 Reviewed-by: Damien Neil <dneil@google.com>
2019-07-11 22:13:49 -07:00
desc: "required field in extension message unset",
partial: true,
proto: check for required fields in encoding/decoding Change-Id: I0555a92e0399782f075b1dcd248e880dd48c7d6d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170579 Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-03 12:17:24 -07:00
decodeTo: []proto.Message{build(
&testpb.TestAllExtensions{},
extend(testpb.E_TestRequired_Single, &testpb.TestRequired{}),
)},
wire: pack.Message{
pack.Tag{1000, pack.BytesType}, pack.LengthPrefix(pack.Message{}),
}.Marshal(),
},
{
desc: "required field in extension message set",
decodeTo: []proto.Message{build(
&testpb.TestAllExtensions{},
extend(testpb.E_TestRequired_Single, &testpb.TestRequired{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
RequiredField: proto.Int32(1),
proto: check for required fields in encoding/decoding Change-Id: I0555a92e0399782f075b1dcd248e880dd48c7d6d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170579 Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-03 12:17:24 -07:00
}),
)},
wire: pack.Message{
pack.Tag{1000, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(1),
}),
}.Marshal(),
},
{
proto: cleanup invalid extension exception The invalidExtensions flag no longer seems necessary. Tests pass without it. Change-Id: Ieb35e26912b047718ccbfcdc926625aec1cd8c87 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185937 Reviewed-by: Damien Neil <dneil@google.com>
2019-07-11 22:13:49 -07:00
desc: "required field in repeated extension message unset",
partial: true,
proto: check for required fields in encoding/decoding Change-Id: I0555a92e0399782f075b1dcd248e880dd48c7d6d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170579 Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-03 12:17:24 -07:00
decodeTo: []proto.Message{build(
&testpb.TestAllExtensions{},
extend(testpb.E_TestRequired_Multi, []*testpb.TestRequired{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
{RequiredField: proto.Int32(1)},
proto: check for required fields in encoding/decoding Change-Id: I0555a92e0399782f075b1dcd248e880dd48c7d6d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170579 Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-03 12:17:24 -07:00
{},
}),
)},
wire: pack.Message{
pack.Tag{1001, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(1),
}),
pack.Tag{1001, pack.BytesType}, pack.LengthPrefix(pack.Message{}),
}.Marshal(),
},
{
desc: "required field in repeated extension message set",
decodeTo: []proto.Message{build(
&testpb.TestAllExtensions{},
extend(testpb.E_TestRequired_Multi, []*testpb.TestRequired{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
{RequiredField: proto.Int32(1)},
{RequiredField: proto.Int32(2)},
proto: check for required fields in encoding/decoding Change-Id: I0555a92e0399782f075b1dcd248e880dd48c7d6d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/170579 Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-03 12:17:24 -07:00
}),
)},
wire: pack.Message{
pack.Tag{1001, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(1),
}),
pack.Tag{1001, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(2),
}),
}.Marshal(),
},
proto, internal/impl: make IsInitialized more consistent Make the fast-path and slow-path versions of IsInitialized report exactly the same errors: An errors.RequiredNotSet containing the full name of one of the unset required fields. Bugfix: Fast-path IsInitialized on a nil message reports an error only when the message directly contains required fields. Bugfix: Include fast-path IsInitialized in legacy messageIfaceWrapper. Fixes golang/protobuf#887 Change-Id: Ia5e4b386f8c23f6f855d995f4a098b1338acbae3 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185397 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-09 11:40:49 -07:00
{
desc: "nil messages",
decodeTo: []proto.Message{
(*testpb.TestAllTypes)(nil),
(*test3pb.TestAllTypes)(nil),
(*testpb.TestAllExtensions)(nil),
},
},
internal/impl: add fast-path marshal implementation This is a port of the v1 table marshaler, with some substantial cleanup and refactoring. Benchstat results from the protobuf reference benchmark data comparing the v1 package with v2, with AllowPartial:true set for the new package. This is not an apples-to-apples comparison, since v1 doesn't have a way to disable required field checks. Required field checks in v2 package currently go through reflection, which performs terribly; my initial experimentation indicates that fast-path required field checks will not add a large amount of cost; these results are incomplete but not wholly inaccurate. name old time/op new time/op delta /dataset.google_message3_1.pb/Marshal-12 219ms ± 1% 232ms ± 1% +5.85% (p=0.004 n=6+5) /dataset.google_message2.pb/Marshal-12 261µs ± 3% 248µs ± 1% -5.14% (p=0.002 n=6+6) /dataset.google_message1_proto2.pb/Marshal-12 681ns ± 2% 637ns ± 3% -6.53% (p=0.002 n=6+6) /dataset.google_message1_proto3.pb/Marshal-12 1.10µs ± 8% 0.99µs ± 3% -9.63% (p=0.002 n=6+6) /dataset.google_message3_3.pb/Marshal-12 44.2ms ± 3% 35.2ms ± 1% -20.28% (p=0.004 n=6+5) /dataset.google_message4.pb/Marshal-12 91.4ms ± 2% 94.9ms ± 2% +3.78% (p=0.002 n=6+6) /dataset.google_message3_2.pb/Marshal-12 78.7ms ± 6% 80.8ms ± 4% ~ (p=0.310 n=6+6) /dataset.google_message3_4.pb/Marshal-12 10.6ms ± 3% 10.6ms ± 8% ~ (p=0.662 n=5+6) /dataset.google_message3_5.pb/Marshal-12 675ms ± 4% 510ms ± 2% -24.40% (p=0.002 n=6+6) /dataset.google_message3_1.pb/Marshal 219ms ± 1% 236ms ± 7% +8.06% (p=0.004 n=5+6) /dataset.google_message2.pb/Marshal 257µs ± 1% 250µs ± 3% ~ (p=0.052 n=5+6) /dataset.google_message1_proto2.pb/Marshal 685ns ± 1% 628ns ± 1% -8.41% (p=0.008 n=5+5) /dataset.google_message1_proto3.pb/Marshal 1.08µs ± 1% 0.98µs ± 2% -9.31% (p=0.004 n=5+6) /dataset.google_message3_3.pb/Marshal 43.7ms ± 1% 35.1ms ± 1% -19.76% (p=0.002 n=6+6) /dataset.google_message4.pb/Marshal 93.4ms ± 4% 94.9ms ± 2% ~ (p=0.180 n=6+6) /dataset.google_message3_2.pb/Marshal 105ms ± 2% 98ms ± 7% -6.81% (p=0.009 n=5+6) /dataset.google_message3_4.pb/Marshal 16.3ms ± 6% 15.7ms ± 3% -3.44% (p=0.041 n=6+6) /dataset.google_message3_5.pb/Marshal 676ms ± 4% 504ms ± 2% -25.50% (p=0.004 n=6+5) Change-Id: I72cc4597117f4cf5d236ef505777d49dd4a5f75d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171020 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-04-01 13:49:56 -07:00
{
desc: "legacy",
partial: true,
decodeTo: []proto.Message{
&legacypb.Legacy{
F1: &legacy1pb.Message{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
OptionalInt32: proto.Int32(1),
internal/impl: add fast-path marshal implementation This is a port of the v1 table marshaler, with some substantial cleanup and refactoring. Benchstat results from the protobuf reference benchmark data comparing the v1 package with v2, with AllowPartial:true set for the new package. This is not an apples-to-apples comparison, since v1 doesn't have a way to disable required field checks. Required field checks in v2 package currently go through reflection, which performs terribly; my initial experimentation indicates that fast-path required field checks will not add a large amount of cost; these results are incomplete but not wholly inaccurate. name old time/op new time/op delta /dataset.google_message3_1.pb/Marshal-12 219ms ± 1% 232ms ± 1% +5.85% (p=0.004 n=6+5) /dataset.google_message2.pb/Marshal-12 261µs ± 3% 248µs ± 1% -5.14% (p=0.002 n=6+6) /dataset.google_message1_proto2.pb/Marshal-12 681ns ± 2% 637ns ± 3% -6.53% (p=0.002 n=6+6) /dataset.google_message1_proto3.pb/Marshal-12 1.10µs ± 8% 0.99µs ± 3% -9.63% (p=0.002 n=6+6) /dataset.google_message3_3.pb/Marshal-12 44.2ms ± 3% 35.2ms ± 1% -20.28% (p=0.004 n=6+5) /dataset.google_message4.pb/Marshal-12 91.4ms ± 2% 94.9ms ± 2% +3.78% (p=0.002 n=6+6) /dataset.google_message3_2.pb/Marshal-12 78.7ms ± 6% 80.8ms ± 4% ~ (p=0.310 n=6+6) /dataset.google_message3_4.pb/Marshal-12 10.6ms ± 3% 10.6ms ± 8% ~ (p=0.662 n=5+6) /dataset.google_message3_5.pb/Marshal-12 675ms ± 4% 510ms ± 2% -24.40% (p=0.002 n=6+6) /dataset.google_message3_1.pb/Marshal 219ms ± 1% 236ms ± 7% +8.06% (p=0.004 n=5+6) /dataset.google_message2.pb/Marshal 257µs ± 1% 250µs ± 3% ~ (p=0.052 n=5+6) /dataset.google_message1_proto2.pb/Marshal 685ns ± 1% 628ns ± 1% -8.41% (p=0.008 n=5+5) /dataset.google_message1_proto3.pb/Marshal 1.08µs ± 1% 0.98µs ± 2% -9.31% (p=0.004 n=5+6) /dataset.google_message3_3.pb/Marshal 43.7ms ± 1% 35.1ms ± 1% -19.76% (p=0.002 n=6+6) /dataset.google_message4.pb/Marshal 93.4ms ± 4% 94.9ms ± 2% ~ (p=0.180 n=6+6) /dataset.google_message3_2.pb/Marshal 105ms ± 2% 98ms ± 7% -6.81% (p=0.009 n=5+6) /dataset.google_message3_4.pb/Marshal 16.3ms ± 6% 15.7ms ± 3% -3.44% (p=0.041 n=6+6) /dataset.google_message3_5.pb/Marshal 676ms ± 4% 504ms ± 2% -25.50% (p=0.004 n=6+5) Change-Id: I72cc4597117f4cf5d236ef505777d49dd4a5f75d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171020 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-04-01 13:49:56 -07:00
OptionalChildEnum: legacy1pb.Message_ALPHA.Enum(),
OptionalChildMessage: &legacy1pb.Message_ChildMessage{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
F1: proto.String("x"),
internal/impl: add fast-path marshal implementation This is a port of the v1 table marshaler, with some substantial cleanup and refactoring. Benchstat results from the protobuf reference benchmark data comparing the v1 package with v2, with AllowPartial:true set for the new package. This is not an apples-to-apples comparison, since v1 doesn't have a way to disable required field checks. Required field checks in v2 package currently go through reflection, which performs terribly; my initial experimentation indicates that fast-path required field checks will not add a large amount of cost; these results are incomplete but not wholly inaccurate. name old time/op new time/op delta /dataset.google_message3_1.pb/Marshal-12 219ms ± 1% 232ms ± 1% +5.85% (p=0.004 n=6+5) /dataset.google_message2.pb/Marshal-12 261µs ± 3% 248µs ± 1% -5.14% (p=0.002 n=6+6) /dataset.google_message1_proto2.pb/Marshal-12 681ns ± 2% 637ns ± 3% -6.53% (p=0.002 n=6+6) /dataset.google_message1_proto3.pb/Marshal-12 1.10µs ± 8% 0.99µs ± 3% -9.63% (p=0.002 n=6+6) /dataset.google_message3_3.pb/Marshal-12 44.2ms ± 3% 35.2ms ± 1% -20.28% (p=0.004 n=6+5) /dataset.google_message4.pb/Marshal-12 91.4ms ± 2% 94.9ms ± 2% +3.78% (p=0.002 n=6+6) /dataset.google_message3_2.pb/Marshal-12 78.7ms ± 6% 80.8ms ± 4% ~ (p=0.310 n=6+6) /dataset.google_message3_4.pb/Marshal-12 10.6ms ± 3% 10.6ms ± 8% ~ (p=0.662 n=5+6) /dataset.google_message3_5.pb/Marshal-12 675ms ± 4% 510ms ± 2% -24.40% (p=0.002 n=6+6) /dataset.google_message3_1.pb/Marshal 219ms ± 1% 236ms ± 7% +8.06% (p=0.004 n=5+6) /dataset.google_message2.pb/Marshal 257µs ± 1% 250µs ± 3% ~ (p=0.052 n=5+6) /dataset.google_message1_proto2.pb/Marshal 685ns ± 1% 628ns ± 1% -8.41% (p=0.008 n=5+5) /dataset.google_message1_proto3.pb/Marshal 1.08µs ± 1% 0.98µs ± 2% -9.31% (p=0.004 n=5+6) /dataset.google_message3_3.pb/Marshal 43.7ms ± 1% 35.1ms ± 1% -19.76% (p=0.002 n=6+6) /dataset.google_message4.pb/Marshal 93.4ms ± 4% 94.9ms ± 2% ~ (p=0.180 n=6+6) /dataset.google_message3_2.pb/Marshal 105ms ± 2% 98ms ± 7% -6.81% (p=0.009 n=5+6) /dataset.google_message3_4.pb/Marshal 16.3ms ± 6% 15.7ms ± 3% -3.44% (p=0.041 n=6+6) /dataset.google_message3_5.pb/Marshal 676ms ± 4% 504ms ± 2% -25.50% (p=0.004 n=6+5) Change-Id: I72cc4597117f4cf5d236ef505777d49dd4a5f75d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171020 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-04-01 13:49:56 -07:00
},
Optionalgroup: &legacy1pb.Message_OptionalGroup{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
F1: proto.String("x"),
internal/impl: add fast-path marshal implementation This is a port of the v1 table marshaler, with some substantial cleanup and refactoring. Benchstat results from the protobuf reference benchmark data comparing the v1 package with v2, with AllowPartial:true set for the new package. This is not an apples-to-apples comparison, since v1 doesn't have a way to disable required field checks. Required field checks in v2 package currently go through reflection, which performs terribly; my initial experimentation indicates that fast-path required field checks will not add a large amount of cost; these results are incomplete but not wholly inaccurate. name old time/op new time/op delta /dataset.google_message3_1.pb/Marshal-12 219ms ± 1% 232ms ± 1% +5.85% (p=0.004 n=6+5) /dataset.google_message2.pb/Marshal-12 261µs ± 3% 248µs ± 1% -5.14% (p=0.002 n=6+6) /dataset.google_message1_proto2.pb/Marshal-12 681ns ± 2% 637ns ± 3% -6.53% (p=0.002 n=6+6) /dataset.google_message1_proto3.pb/Marshal-12 1.10µs ± 8% 0.99µs ± 3% -9.63% (p=0.002 n=6+6) /dataset.google_message3_3.pb/Marshal-12 44.2ms ± 3% 35.2ms ± 1% -20.28% (p=0.004 n=6+5) /dataset.google_message4.pb/Marshal-12 91.4ms ± 2% 94.9ms ± 2% +3.78% (p=0.002 n=6+6) /dataset.google_message3_2.pb/Marshal-12 78.7ms ± 6% 80.8ms ± 4% ~ (p=0.310 n=6+6) /dataset.google_message3_4.pb/Marshal-12 10.6ms ± 3% 10.6ms ± 8% ~ (p=0.662 n=5+6) /dataset.google_message3_5.pb/Marshal-12 675ms ± 4% 510ms ± 2% -24.40% (p=0.002 n=6+6) /dataset.google_message3_1.pb/Marshal 219ms ± 1% 236ms ± 7% +8.06% (p=0.004 n=5+6) /dataset.google_message2.pb/Marshal 257µs ± 1% 250µs ± 3% ~ (p=0.052 n=5+6) /dataset.google_message1_proto2.pb/Marshal 685ns ± 1% 628ns ± 1% -8.41% (p=0.008 n=5+5) /dataset.google_message1_proto3.pb/Marshal 1.08µs ± 1% 0.98µs ± 2% -9.31% (p=0.004 n=5+6) /dataset.google_message3_3.pb/Marshal 43.7ms ± 1% 35.1ms ± 1% -19.76% (p=0.002 n=6+6) /dataset.google_message4.pb/Marshal 93.4ms ± 4% 94.9ms ± 2% ~ (p=0.180 n=6+6) /dataset.google_message3_2.pb/Marshal 105ms ± 2% 98ms ± 7% -6.81% (p=0.009 n=5+6) /dataset.google_message3_4.pb/Marshal 16.3ms ± 6% 15.7ms ± 3% -3.44% (p=0.041 n=6+6) /dataset.google_message3_5.pb/Marshal 676ms ± 4% 504ms ± 2% -25.50% (p=0.004 n=6+5) Change-Id: I72cc4597117f4cf5d236ef505777d49dd4a5f75d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171020 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-04-01 13:49:56 -07:00
},
RepeatedChildMessage: []*legacy1pb.Message_ChildMessage{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
{F1: proto.String("x")},
internal/impl: add fast-path marshal implementation This is a port of the v1 table marshaler, with some substantial cleanup and refactoring. Benchstat results from the protobuf reference benchmark data comparing the v1 package with v2, with AllowPartial:true set for the new package. This is not an apples-to-apples comparison, since v1 doesn't have a way to disable required field checks. Required field checks in v2 package currently go through reflection, which performs terribly; my initial experimentation indicates that fast-path required field checks will not add a large amount of cost; these results are incomplete but not wholly inaccurate. name old time/op new time/op delta /dataset.google_message3_1.pb/Marshal-12 219ms ± 1% 232ms ± 1% +5.85% (p=0.004 n=6+5) /dataset.google_message2.pb/Marshal-12 261µs ± 3% 248µs ± 1% -5.14% (p=0.002 n=6+6) /dataset.google_message1_proto2.pb/Marshal-12 681ns ± 2% 637ns ± 3% -6.53% (p=0.002 n=6+6) /dataset.google_message1_proto3.pb/Marshal-12 1.10µs ± 8% 0.99µs ± 3% -9.63% (p=0.002 n=6+6) /dataset.google_message3_3.pb/Marshal-12 44.2ms ± 3% 35.2ms ± 1% -20.28% (p=0.004 n=6+5) /dataset.google_message4.pb/Marshal-12 91.4ms ± 2% 94.9ms ± 2% +3.78% (p=0.002 n=6+6) /dataset.google_message3_2.pb/Marshal-12 78.7ms ± 6% 80.8ms ± 4% ~ (p=0.310 n=6+6) /dataset.google_message3_4.pb/Marshal-12 10.6ms ± 3% 10.6ms ± 8% ~ (p=0.662 n=5+6) /dataset.google_message3_5.pb/Marshal-12 675ms ± 4% 510ms ± 2% -24.40% (p=0.002 n=6+6) /dataset.google_message3_1.pb/Marshal 219ms ± 1% 236ms ± 7% +8.06% (p=0.004 n=5+6) /dataset.google_message2.pb/Marshal 257µs ± 1% 250µs ± 3% ~ (p=0.052 n=5+6) /dataset.google_message1_proto2.pb/Marshal 685ns ± 1% 628ns ± 1% -8.41% (p=0.008 n=5+5) /dataset.google_message1_proto3.pb/Marshal 1.08µs ± 1% 0.98µs ± 2% -9.31% (p=0.004 n=5+6) /dataset.google_message3_3.pb/Marshal 43.7ms ± 1% 35.1ms ± 1% -19.76% (p=0.002 n=6+6) /dataset.google_message4.pb/Marshal 93.4ms ± 4% 94.9ms ± 2% ~ (p=0.180 n=6+6) /dataset.google_message3_2.pb/Marshal 105ms ± 2% 98ms ± 7% -6.81% (p=0.009 n=5+6) /dataset.google_message3_4.pb/Marshal 16.3ms ± 6% 15.7ms ± 3% -3.44% (p=0.041 n=6+6) /dataset.google_message3_5.pb/Marshal 676ms ± 4% 504ms ± 2% -25.50% (p=0.004 n=6+5) Change-Id: I72cc4597117f4cf5d236ef505777d49dd4a5f75d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171020 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-04-01 13:49:56 -07:00
},
Repeatedgroup: []*legacy1pb.Message_RepeatedGroup{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
{F1: proto.String("x")},
internal/impl: add fast-path marshal implementation This is a port of the v1 table marshaler, with some substantial cleanup and refactoring. Benchstat results from the protobuf reference benchmark data comparing the v1 package with v2, with AllowPartial:true set for the new package. This is not an apples-to-apples comparison, since v1 doesn't have a way to disable required field checks. Required field checks in v2 package currently go through reflection, which performs terribly; my initial experimentation indicates that fast-path required field checks will not add a large amount of cost; these results are incomplete but not wholly inaccurate. name old time/op new time/op delta /dataset.google_message3_1.pb/Marshal-12 219ms ± 1% 232ms ± 1% +5.85% (p=0.004 n=6+5) /dataset.google_message2.pb/Marshal-12 261µs ± 3% 248µs ± 1% -5.14% (p=0.002 n=6+6) /dataset.google_message1_proto2.pb/Marshal-12 681ns ± 2% 637ns ± 3% -6.53% (p=0.002 n=6+6) /dataset.google_message1_proto3.pb/Marshal-12 1.10µs ± 8% 0.99µs ± 3% -9.63% (p=0.002 n=6+6) /dataset.google_message3_3.pb/Marshal-12 44.2ms ± 3% 35.2ms ± 1% -20.28% (p=0.004 n=6+5) /dataset.google_message4.pb/Marshal-12 91.4ms ± 2% 94.9ms ± 2% +3.78% (p=0.002 n=6+6) /dataset.google_message3_2.pb/Marshal-12 78.7ms ± 6% 80.8ms ± 4% ~ (p=0.310 n=6+6) /dataset.google_message3_4.pb/Marshal-12 10.6ms ± 3% 10.6ms ± 8% ~ (p=0.662 n=5+6) /dataset.google_message3_5.pb/Marshal-12 675ms ± 4% 510ms ± 2% -24.40% (p=0.002 n=6+6) /dataset.google_message3_1.pb/Marshal 219ms ± 1% 236ms ± 7% +8.06% (p=0.004 n=5+6) /dataset.google_message2.pb/Marshal 257µs ± 1% 250µs ± 3% ~ (p=0.052 n=5+6) /dataset.google_message1_proto2.pb/Marshal 685ns ± 1% 628ns ± 1% -8.41% (p=0.008 n=5+5) /dataset.google_message1_proto3.pb/Marshal 1.08µs ± 1% 0.98µs ± 2% -9.31% (p=0.004 n=5+6) /dataset.google_message3_3.pb/Marshal 43.7ms ± 1% 35.1ms ± 1% -19.76% (p=0.002 n=6+6) /dataset.google_message4.pb/Marshal 93.4ms ± 4% 94.9ms ± 2% ~ (p=0.180 n=6+6) /dataset.google_message3_2.pb/Marshal 105ms ± 2% 98ms ± 7% -6.81% (p=0.009 n=5+6) /dataset.google_message3_4.pb/Marshal 16.3ms ± 6% 15.7ms ± 3% -3.44% (p=0.041 n=6+6) /dataset.google_message3_5.pb/Marshal 676ms ± 4% 504ms ± 2% -25.50% (p=0.004 n=6+5) Change-Id: I72cc4597117f4cf5d236ef505777d49dd4a5f75d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171020 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-04-01 13:49:56 -07:00
},
MapBoolChildMessage: map[bool]*legacy1pb.Message_ChildMessage{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
true: {F1: proto.String("x")},
internal/impl: add fast-path marshal implementation This is a port of the v1 table marshaler, with some substantial cleanup and refactoring. Benchstat results from the protobuf reference benchmark data comparing the v1 package with v2, with AllowPartial:true set for the new package. This is not an apples-to-apples comparison, since v1 doesn't have a way to disable required field checks. Required field checks in v2 package currently go through reflection, which performs terribly; my initial experimentation indicates that fast-path required field checks will not add a large amount of cost; these results are incomplete but not wholly inaccurate. name old time/op new time/op delta /dataset.google_message3_1.pb/Marshal-12 219ms ± 1% 232ms ± 1% +5.85% (p=0.004 n=6+5) /dataset.google_message2.pb/Marshal-12 261µs ± 3% 248µs ± 1% -5.14% (p=0.002 n=6+6) /dataset.google_message1_proto2.pb/Marshal-12 681ns ± 2% 637ns ± 3% -6.53% (p=0.002 n=6+6) /dataset.google_message1_proto3.pb/Marshal-12 1.10µs ± 8% 0.99µs ± 3% -9.63% (p=0.002 n=6+6) /dataset.google_message3_3.pb/Marshal-12 44.2ms ± 3% 35.2ms ± 1% -20.28% (p=0.004 n=6+5) /dataset.google_message4.pb/Marshal-12 91.4ms ± 2% 94.9ms ± 2% +3.78% (p=0.002 n=6+6) /dataset.google_message3_2.pb/Marshal-12 78.7ms ± 6% 80.8ms ± 4% ~ (p=0.310 n=6+6) /dataset.google_message3_4.pb/Marshal-12 10.6ms ± 3% 10.6ms ± 8% ~ (p=0.662 n=5+6) /dataset.google_message3_5.pb/Marshal-12 675ms ± 4% 510ms ± 2% -24.40% (p=0.002 n=6+6) /dataset.google_message3_1.pb/Marshal 219ms ± 1% 236ms ± 7% +8.06% (p=0.004 n=5+6) /dataset.google_message2.pb/Marshal 257µs ± 1% 250µs ± 3% ~ (p=0.052 n=5+6) /dataset.google_message1_proto2.pb/Marshal 685ns ± 1% 628ns ± 1% -8.41% (p=0.008 n=5+5) /dataset.google_message1_proto3.pb/Marshal 1.08µs ± 1% 0.98µs ± 2% -9.31% (p=0.004 n=5+6) /dataset.google_message3_3.pb/Marshal 43.7ms ± 1% 35.1ms ± 1% -19.76% (p=0.002 n=6+6) /dataset.google_message4.pb/Marshal 93.4ms ± 4% 94.9ms ± 2% ~ (p=0.180 n=6+6) /dataset.google_message3_2.pb/Marshal 105ms ± 2% 98ms ± 7% -6.81% (p=0.009 n=5+6) /dataset.google_message3_4.pb/Marshal 16.3ms ± 6% 15.7ms ± 3% -3.44% (p=0.041 n=6+6) /dataset.google_message3_5.pb/Marshal 676ms ± 4% 504ms ± 2% -25.50% (p=0.004 n=6+5) Change-Id: I72cc4597117f4cf5d236ef505777d49dd4a5f75d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171020 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-04-01 13:49:56 -07:00
},
OneofUnion: &legacy1pb.Message_OneofChildMessage{
&legacy1pb.Message_ChildMessage{
proto: move T->*T wrappers from internal/scalar to proto Usage of these is pervasive in code which works with proto2, and proto2 will be with us for a long, long time to come. Move them to the proto package. Change-Id: I1b2e57429fd5a8f107a848a4492d20c27f304bd7 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185543 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-10 16:17:16 -07:00
F1: proto.String("x"),
internal/impl: add fast-path marshal implementation This is a port of the v1 table marshaler, with some substantial cleanup and refactoring. Benchstat results from the protobuf reference benchmark data comparing the v1 package with v2, with AllowPartial:true set for the new package. This is not an apples-to-apples comparison, since v1 doesn't have a way to disable required field checks. Required field checks in v2 package currently go through reflection, which performs terribly; my initial experimentation indicates that fast-path required field checks will not add a large amount of cost; these results are incomplete but not wholly inaccurate. name old time/op new time/op delta /dataset.google_message3_1.pb/Marshal-12 219ms ± 1% 232ms ± 1% +5.85% (p=0.004 n=6+5) /dataset.google_message2.pb/Marshal-12 261µs ± 3% 248µs ± 1% -5.14% (p=0.002 n=6+6) /dataset.google_message1_proto2.pb/Marshal-12 681ns ± 2% 637ns ± 3% -6.53% (p=0.002 n=6+6) /dataset.google_message1_proto3.pb/Marshal-12 1.10µs ± 8% 0.99µs ± 3% -9.63% (p=0.002 n=6+6) /dataset.google_message3_3.pb/Marshal-12 44.2ms ± 3% 35.2ms ± 1% -20.28% (p=0.004 n=6+5) /dataset.google_message4.pb/Marshal-12 91.4ms ± 2% 94.9ms ± 2% +3.78% (p=0.002 n=6+6) /dataset.google_message3_2.pb/Marshal-12 78.7ms ± 6% 80.8ms ± 4% ~ (p=0.310 n=6+6) /dataset.google_message3_4.pb/Marshal-12 10.6ms ± 3% 10.6ms ± 8% ~ (p=0.662 n=5+6) /dataset.google_message3_5.pb/Marshal-12 675ms ± 4% 510ms ± 2% -24.40% (p=0.002 n=6+6) /dataset.google_message3_1.pb/Marshal 219ms ± 1% 236ms ± 7% +8.06% (p=0.004 n=5+6) /dataset.google_message2.pb/Marshal 257µs ± 1% 250µs ± 3% ~ (p=0.052 n=5+6) /dataset.google_message1_proto2.pb/Marshal 685ns ± 1% 628ns ± 1% -8.41% (p=0.008 n=5+5) /dataset.google_message1_proto3.pb/Marshal 1.08µs ± 1% 0.98µs ± 2% -9.31% (p=0.004 n=5+6) /dataset.google_message3_3.pb/Marshal 43.7ms ± 1% 35.1ms ± 1% -19.76% (p=0.002 n=6+6) /dataset.google_message4.pb/Marshal 93.4ms ± 4% 94.9ms ± 2% ~ (p=0.180 n=6+6) /dataset.google_message3_2.pb/Marshal 105ms ± 2% 98ms ± 7% -6.81% (p=0.009 n=5+6) /dataset.google_message3_4.pb/Marshal 16.3ms ± 6% 15.7ms ± 3% -3.44% (p=0.041 n=6+6) /dataset.google_message3_5.pb/Marshal 676ms ± 4% 504ms ± 2% -25.50% (p=0.004 n=6+5) Change-Id: I72cc4597117f4cf5d236ef505777d49dd4a5f75d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171020 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-04-01 13:49:56 -07:00
},
},
},
},
},
wire: pack.Message{
pack.Tag{1, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{101, pack.VarintType}, pack.Varint(1),
pack.Tag{115, pack.VarintType}, pack.Varint(0),
pack.Tag{116, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.BytesType}, pack.String("x"),
}),
pack.Tag{120, pack.StartGroupType},
pack.Tag{1, pack.BytesType}, pack.String("x"),
pack.Tag{120, pack.EndGroupType},
pack.Tag{516, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.BytesType}, pack.String("x"),
}),
pack.Tag{520, pack.StartGroupType},
pack.Tag{1, pack.BytesType}, pack.String("x"),
pack.Tag{520, pack.EndGroupType},
pack.Tag{616, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.VarintType}, pack.Varint(1),
pack.Tag{2, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.BytesType}, pack.String("x"),
}),
}),
pack.Tag{716, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.BytesType}, pack.String("x"),
}),
}),
}.Marshal(),
},
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
}
proto: validate UTF-8 in proto3 strings Change-Id: I6a495730c3f438e7b2c4ca86edade7d6f25aa47d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171700 Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-11 11:46:55 -07:00
var invalidUTF8TestProtos = []testProto{
{
desc: "invalid UTF-8 in optional string field",
decodeTo: []proto.Message{&test3pb.TestAllTypes{
OptionalString: "abc\xff",
}},
wire: pack.Message{
pack.Tag{14, pack.BytesType}, pack.String("abc\xff"),
}.Marshal(),
},
{
desc: "invalid UTF-8 in repeated string field",
decodeTo: []proto.Message{&test3pb.TestAllTypes{
RepeatedString: []string{"foo", "abc\xff"},
}},
wire: pack.Message{
pack.Tag{44, pack.BytesType}, pack.String("foo"),
pack.Tag{44, pack.BytesType}, pack.String("abc\xff"),
}.Marshal(),
},
{
desc: "invalid UTF-8 in nested message",
decodeTo: []proto.Message{&test3pb.TestAllTypes{
OptionalNestedMessage: &test3pb.TestAllTypes_NestedMessage{
Corecursive: &test3pb.TestAllTypes{
OptionalString: "abc\xff",
},
},
}},
wire: pack.Message{
pack.Tag{18, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{2, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{14, pack.BytesType}, pack.String("abc\xff"),
}),
}),
}.Marshal(),
},
internal/impl: add fast-path marshal implementation This is a port of the v1 table marshaler, with some substantial cleanup and refactoring. Benchstat results from the protobuf reference benchmark data comparing the v1 package with v2, with AllowPartial:true set for the new package. This is not an apples-to-apples comparison, since v1 doesn't have a way to disable required field checks. Required field checks in v2 package currently go through reflection, which performs terribly; my initial experimentation indicates that fast-path required field checks will not add a large amount of cost; these results are incomplete but not wholly inaccurate. name old time/op new time/op delta /dataset.google_message3_1.pb/Marshal-12 219ms ± 1% 232ms ± 1% +5.85% (p=0.004 n=6+5) /dataset.google_message2.pb/Marshal-12 261µs ± 3% 248µs ± 1% -5.14% (p=0.002 n=6+6) /dataset.google_message1_proto2.pb/Marshal-12 681ns ± 2% 637ns ± 3% -6.53% (p=0.002 n=6+6) /dataset.google_message1_proto3.pb/Marshal-12 1.10µs ± 8% 0.99µs ± 3% -9.63% (p=0.002 n=6+6) /dataset.google_message3_3.pb/Marshal-12 44.2ms ± 3% 35.2ms ± 1% -20.28% (p=0.004 n=6+5) /dataset.google_message4.pb/Marshal-12 91.4ms ± 2% 94.9ms ± 2% +3.78% (p=0.002 n=6+6) /dataset.google_message3_2.pb/Marshal-12 78.7ms ± 6% 80.8ms ± 4% ~ (p=0.310 n=6+6) /dataset.google_message3_4.pb/Marshal-12 10.6ms ± 3% 10.6ms ± 8% ~ (p=0.662 n=5+6) /dataset.google_message3_5.pb/Marshal-12 675ms ± 4% 510ms ± 2% -24.40% (p=0.002 n=6+6) /dataset.google_message3_1.pb/Marshal 219ms ± 1% 236ms ± 7% +8.06% (p=0.004 n=5+6) /dataset.google_message2.pb/Marshal 257µs ± 1% 250µs ± 3% ~ (p=0.052 n=5+6) /dataset.google_message1_proto2.pb/Marshal 685ns ± 1% 628ns ± 1% -8.41% (p=0.008 n=5+5) /dataset.google_message1_proto3.pb/Marshal 1.08µs ± 1% 0.98µs ± 2% -9.31% (p=0.004 n=5+6) /dataset.google_message3_3.pb/Marshal 43.7ms ± 1% 35.1ms ± 1% -19.76% (p=0.002 n=6+6) /dataset.google_message4.pb/Marshal 93.4ms ± 4% 94.9ms ± 2% ~ (p=0.180 n=6+6) /dataset.google_message3_2.pb/Marshal 105ms ± 2% 98ms ± 7% -6.81% (p=0.009 n=5+6) /dataset.google_message3_4.pb/Marshal 16.3ms ± 6% 15.7ms ± 3% -3.44% (p=0.041 n=6+6) /dataset.google_message3_5.pb/Marshal 676ms ± 4% 504ms ± 2% -25.50% (p=0.004 n=6+5) Change-Id: I72cc4597117f4cf5d236ef505777d49dd4a5f75d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171020 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-04-01 13:49:56 -07:00
{
desc: "invalid UTF-8 in oneof field",
decodeTo: []proto.Message{
&test3pb.TestAllTypes{OneofField: &test3pb.TestAllTypes_OneofString{"abc\xff"}},
},
wire: pack.Message{pack.Tag{113, pack.BytesType}, pack.String("abc\xff")}.Marshal(),
},
proto: validate UTF-8 in proto3 strings Change-Id: I6a495730c3f438e7b2c4ca86edade7d6f25aa47d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/171700 Reviewed-by: Herbie Ong <herbie@google.com>
2019-04-11 11:46:55 -07:00
{
desc: "invalid UTF-8 in map key",
decodeTo: []proto.Message{&test3pb.TestAllTypes{
MapStringString: map[string]string{"key\xff": "val"},
}},
wire: pack.Message{
pack.Tag{69, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.BytesType}, pack.String("key\xff"),
pack.Tag{2, pack.BytesType}, pack.String("val"),
}),
}.Marshal(),
},
{
desc: "invalid UTF-8 in map value",
decodeTo: []proto.Message{&test3pb.TestAllTypes{
MapStringString: map[string]string{"key": "val\xff"},
}},
wire: pack.Message{
pack.Tag{69, pack.BytesType}, pack.LengthPrefix(pack.Message{
pack.Tag{1, pack.BytesType}, pack.String("key"),
pack.Tag{2, pack.BytesType}, pack.String("val\xff"),
}),
}.Marshal(),
},
}
all: support enforce_utf8 override In 2014, when proto3 was being developed, there were a number of early adopters of the new syntax. Before the finalization of proto3 when it was released in open-source in July 2016, a decision was made to strictly validate strings in proto3. However, some of the early adopters were already using invalid UTF-8 with string fields. The google.protobuf.FieldOptions.enforce_utf8 option only exists to support those grandfathered users where they can opt-out of the validation logic. Practical use of that option in open source is impossible even if a user specifies the proto1_legacy build tag since it requires a hacked variant of descriptor.proto that is not externally available. This CL supports enforce_utf8 by modifiyng internal/filedesc to expose the flag if it detects it in the raw descriptor. We add an strs.EnforceUTF8 function as a centralized place to determine whether to perform validation. Validation opt-out is supported only in builds with legacy support. We implement support for validating UTF-8 in all proto3 string fields, even if they are backed by a Go []byte. Change-Id: I9c0628b84909bc7181125f09db730c80d490e485 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186002 Reviewed-by: Damien Neil <dneil@google.com>
2019-07-13 00:44:41 -07:00
var noEnforceUTF8TestProtos = []testProto{
{
desc: "invalid UTF-8 in optional string field",
decodeTo: []proto.Message{&TestNoEnforceUTF8{
OptionalString: string("abc\xff"),
}},
wire: pack.Message{
pack.Tag{1, pack.BytesType}, pack.String("abc\xff"),
}.Marshal(),
},
{
desc: "invalid UTF-8 in optional string field of Go bytes",
decodeTo: []proto.Message{&TestNoEnforceUTF8{
OptionalBytes: []byte("abc\xff"),
}},
wire: pack.Message{
pack.Tag{2, pack.BytesType}, pack.String("abc\xff"),
}.Marshal(),
},
{
desc: "invalid UTF-8 in repeated string field",
decodeTo: []proto.Message{&TestNoEnforceUTF8{
RepeatedString: []string{string("foo"), string("abc\xff")},
}},
wire: pack.Message{
pack.Tag{3, pack.BytesType}, pack.String("foo"),
pack.Tag{3, pack.BytesType}, pack.String("abc\xff"),
}.Marshal(),
},
{
desc: "invalid UTF-8 in repeated string field of Go bytes",
decodeTo: []proto.Message{&TestNoEnforceUTF8{
RepeatedBytes: [][]byte{[]byte("foo"), []byte("abc\xff")},
}},
wire: pack.Message{
pack.Tag{4, pack.BytesType}, pack.String("foo"),
pack.Tag{4, pack.BytesType}, pack.String("abc\xff"),
}.Marshal(),
},
{
desc: "invalid UTF-8 in oneof string field",
decodeTo: []proto.Message{
&TestNoEnforceUTF8{OneofField: &TestNoEnforceUTF8_OneofString{string("abc\xff")}},
},
wire: pack.Message{pack.Tag{5, pack.BytesType}, pack.String("abc\xff")}.Marshal(),
},
{
desc: "invalid UTF-8 in oneof string field of Go bytes",
decodeTo: []proto.Message{
&TestNoEnforceUTF8{OneofField: &TestNoEnforceUTF8_OneofBytes{[]byte("abc\xff")}},
},
wire: pack.Message{pack.Tag{6, pack.BytesType}, pack.String("abc\xff")}.Marshal(),
},
}
type TestNoEnforceUTF8 struct {
OptionalString string `protobuf:"bytes,1,opt,name=optional_string"`
OptionalBytes []byte `protobuf:"bytes,2,opt,name=optional_bytes"`
RepeatedString []string `protobuf:"bytes,3,rep,name=repeated_string"`
RepeatedBytes [][]byte `protobuf:"bytes,4,rep,name=repeated_bytes"`
OneofField isOneofField `protobuf_oneof:"oneof_field"`
}
type isOneofField interface{ isOneofField() }
type TestNoEnforceUTF8_OneofString struct {
OneofString string `protobuf:"bytes,5,opt,name=oneof_string,oneof"`
}
type TestNoEnforceUTF8_OneofBytes struct {
OneofBytes []byte `protobuf:"bytes,6,opt,name=oneof_bytes,oneof"`
}
func (*TestNoEnforceUTF8_OneofString) isOneofField() {}
func (*TestNoEnforceUTF8_OneofBytes) isOneofField() {}
proto: remove dependency on github.com/golang/protobuf The only remaining dependencies are for benchmarks and internal/testprotos/legacy. Change-Id: I0f7f5292000ccad91bc9526e40fa4d0ec3a36e43 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186157 Reviewed-by: Damien Neil <dneil@google.com>
2019-07-14 18:51:46 -07:00
func (m *TestNoEnforceUTF8) ProtoReflect() protoreflect.Message {
all: support enforce_utf8 override In 2014, when proto3 was being developed, there were a number of early adopters of the new syntax. Before the finalization of proto3 when it was released in open-source in July 2016, a decision was made to strictly validate strings in proto3. However, some of the early adopters were already using invalid UTF-8 with string fields. The google.protobuf.FieldOptions.enforce_utf8 option only exists to support those grandfathered users where they can opt-out of the validation logic. Practical use of that option in open source is impossible even if a user specifies the proto1_legacy build tag since it requires a hacked variant of descriptor.proto that is not externally available. This CL supports enforce_utf8 by modifiyng internal/filedesc to expose the flag if it detects it in the raw descriptor. We add an strs.EnforceUTF8 function as a centralized place to determine whether to perform validation. Validation opt-out is supported only in builds with legacy support. We implement support for validating UTF-8 in all proto3 string fields, even if they are backed by a Go []byte. Change-Id: I9c0628b84909bc7181125f09db730c80d490e485 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186002 Reviewed-by: Damien Neil <dneil@google.com>
2019-07-13 00:44:41 -07:00
return messageInfo_TestNoEnforceUTF8.MessageOf(m)
}
var messageInfo_TestNoEnforceUTF8 = protoimpl.MessageInfo{
all: drop reflect/prototype package Remove the remaining uses of the prototype package. The most significant change is to impl.MessageInfo, which now directly implements the MessageType interface. This involves two notable changes to exported fields of MessageInfo: - PBType is now Desc. - GoType is now GoReflectType. (Name changed to avoid a conflict with the GoType method of the MessageType interface.) Fixes golang/protobuf#911 Change-Id: Ie2aa4766d6887ceaa9cf06b1f109aa6e6e2a208f Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189340 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-06 15:43:25 -07:00
GoReflectType: reflect.TypeOf((*TestNoEnforceUTF8)(nil)),
Desc: func() protoreflect.MessageDescriptor {
pb := new(descriptorpb.FileDescriptorProto)
if err := prototext.Unmarshal([]byte(`
all: support enforce_utf8 override In 2014, when proto3 was being developed, there were a number of early adopters of the new syntax. Before the finalization of proto3 when it was released in open-source in July 2016, a decision was made to strictly validate strings in proto3. However, some of the early adopters were already using invalid UTF-8 with string fields. The google.protobuf.FieldOptions.enforce_utf8 option only exists to support those grandfathered users where they can opt-out of the validation logic. Practical use of that option in open source is impossible even if a user specifies the proto1_legacy build tag since it requires a hacked variant of descriptor.proto that is not externally available. This CL supports enforce_utf8 by modifiyng internal/filedesc to expose the flag if it detects it in the raw descriptor. We add an strs.EnforceUTF8 function as a centralized place to determine whether to perform validation. Validation opt-out is supported only in builds with legacy support. We implement support for validating UTF-8 in all proto3 string fields, even if they are backed by a Go []byte. Change-Id: I9c0628b84909bc7181125f09db730c80d490e485 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186002 Reviewed-by: Damien Neil <dneil@google.com>
2019-07-13 00:44:41 -07:00
syntax: "proto3"
name: "test.proto"
message_type: [{
name: "TestNoEnforceUTF8"
field: [
{name:"optional_string" number:1 label:LABEL_OPTIONAL type:TYPE_STRING},
{name:"optional_bytes" number:2 label:LABEL_OPTIONAL type:TYPE_STRING},
{name:"repeated_string" number:3 label:LABEL_REPEATED type:TYPE_STRING},
{name:"repeated_bytes" number:4 label:LABEL_REPEATED type:TYPE_STRING},
{name:"oneof_string" number:5 label:LABEL_OPTIONAL type:TYPE_STRING, oneof_index:0},
{name:"oneof_bytes" number:6 label:LABEL_OPTIONAL type:TYPE_STRING, oneof_index:0}
]
oneof_decl: [{name:"oneof_field"}]
}]
`), pb); err != nil {
all: drop reflect/prototype package Remove the remaining uses of the prototype package. The most significant change is to impl.MessageInfo, which now directly implements the MessageType interface. This involves two notable changes to exported fields of MessageInfo: - PBType is now Desc. - GoType is now GoReflectType. (Name changed to avoid a conflict with the GoType method of the MessageType interface.) Fixes golang/protobuf#911 Change-Id: Ie2aa4766d6887ceaa9cf06b1f109aa6e6e2a208f Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189340 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-06 15:43:25 -07:00
panic(err)
}
fd, err := protodesc.NewFile(pb, nil)
if err != nil {
panic(err)
}
md := fd.Messages().Get(0)
for i := 0; i < md.Fields().Len(); i++ {
md.Fields().Get(i).(*filedesc.Field).L1.HasEnforceUTF8 = true
md.Fields().Get(i).(*filedesc.Field).L1.EnforceUTF8 = false
}
return md
}(),
all: support enforce_utf8 override In 2014, when proto3 was being developed, there were a number of early adopters of the new syntax. Before the finalization of proto3 when it was released in open-source in July 2016, a decision was made to strictly validate strings in proto3. However, some of the early adopters were already using invalid UTF-8 with string fields. The google.protobuf.FieldOptions.enforce_utf8 option only exists to support those grandfathered users where they can opt-out of the validation logic. Practical use of that option in open source is impossible even if a user specifies the proto1_legacy build tag since it requires a hacked variant of descriptor.proto that is not externally available. This CL supports enforce_utf8 by modifiyng internal/filedesc to expose the flag if it detects it in the raw descriptor. We add an strs.EnforceUTF8 function as a centralized place to determine whether to perform validation. Validation opt-out is supported only in builds with legacy support. We implement support for validating UTF-8 in all proto3 string fields, even if they are backed by a Go []byte. Change-Id: I9c0628b84909bc7181125f09db730c80d490e485 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186002 Reviewed-by: Damien Neil <dneil@google.com>
2019-07-13 00:44:41 -07:00
OneofWrappers: []interface{}{
(*TestNoEnforceUTF8_OneofString)(nil),
(*TestNoEnforceUTF8_OneofBytes)(nil),
},
}
proto: add tests for unmarshalling invalid field numbers This change adds tests for unmarshalling fields with various invalid field numbers. Our current behavior is that proto.Unmarshal will return an error when it sees zero and larger than max field numbers and return nil for reserved ones, which matches the C++ behavior. (Note: depending on which parser helper in the C++ implementation, one may need to call additional method to check the result, which we don't have in Go) Change-Id: I8791fd077f25656107556f5606d55d05c1b4a120 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/191459 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-22 18:52:43 -07:00
var invalidFieldNumberTestProtos = []struct {
desc string
wire []byte
allowed bool
}{
{
desc: "zero",
wire: pack.Message{
pack.Tag{pack.MinValidNumber - 1, pack.VarintType}, pack.Varint(1001),
}.Marshal(),
},
{
desc: "zero and one",
wire: pack.Message{
pack.Tag{pack.MinValidNumber - 1, pack.VarintType}, pack.Varint(1002),
pack.Tag{pack.MinValidNumber, pack.VarintType}, pack.Varint(1003),
}.Marshal(),
},
{
desc: "first reserved",
wire: pack.Message{
pack.Tag{pack.FirstReservedNumber, pack.VarintType}, pack.Varint(1004),
}.Marshal(),
allowed: true,
},
{
desc: "last reserved",
wire: pack.Message{
pack.Tag{pack.LastReservedNumber, pack.VarintType}, pack.Varint(1005),
}.Marshal(),
allowed: true,
},
{
desc: "max and max+1",
wire: pack.Message{
pack.Tag{pack.MaxValidNumber, pack.VarintType}, pack.Varint(1006),
pack.Tag{pack.MaxValidNumber + 1, pack.VarintType}, pack.Varint(1007),
}.Marshal(),
},
{
desc: "max+1",
wire: pack.Message{
pack.Tag{pack.MaxValidNumber + 1, pack.VarintType}, pack.Varint(1008),
}.Marshal(),
},
}
proto: consistently use non-nil, zero-length []bytes for empty bytes strings The fast-path decoder decodes zero-length repeated bytes values as non-nil, zero-length []bytes. Do the same in the reflection decoder. This isn't really a correctness issue, since there's no ambiguity about what a nil entry in a [][]byte means. Still a good idea for consistency, and retains v1 behavior. Change-Id: Icd2cb726d14ff1f2b9f142e65756777a359971f3 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/210257 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-12-05 14:54:35 -08:00
func TestDecodeEmptyBytes(t *testing.T) {
// There's really nothing wrong with a nil entry in a [][]byte,
// but we take care to produce non-nil []bytes for zero-length
// byte strings, so test for it.
m := &testpb.TestAllTypes{}
b := pack.Message{
pack.Tag{45, pack.BytesType}, pack.Bytes(nil),
}.Marshal()
if err := proto.Unmarshal(b, m); err != nil {
t.Fatal(err)
}
if m.RepeatedBytes[0] == nil {
t.Errorf("unmarshaling repeated bytes field containing zero-length value: Got nil bytes, want non-nil")
}
}
internal/legacy: use v2 proto.Unmarshal The v2 decoder isn't 100% complete, but it's good enough. Delete the vendored copy of the v1 Unmarshal implementation. Change-Id: Ibeabbb2e9109a1ec3df57e71f98b7aa4a583fc5b Reviewed-on: https://go-review.googlesource.com/c/154577 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-17 11:16:16 -08:00
func build(m proto.Message, opts ...buildOpt) proto.Message {
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
for _, opt := range opts {
opt(m)
}
return m
}
internal/legacy: use v2 proto.Unmarshal The v2 decoder isn't 100% complete, but it's good enough. Delete the vendored copy of the v1 Unmarshal implementation. Change-Id: Ibeabbb2e9109a1ec3df57e71f98b7aa4a583fc5b Reviewed-on: https://go-review.googlesource.com/c/154577 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-17 11:16:16 -08:00
type buildOpt func(proto.Message)
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
proto: remove dependency on github.com/golang/protobuf The only remaining dependencies are for benchmarks and internal/testprotos/legacy. Change-Id: I0f7f5292000ccad91bc9526e40fa4d0ec3a36e43 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186157 Reviewed-by: Damien Neil <dneil@google.com>
2019-07-14 18:51:46 -07:00
func unknown(raw protoreflect.RawFields) buildOpt {
internal/legacy: use v2 proto.Unmarshal The v2 decoder isn't 100% complete, but it's good enough. Delete the vendored copy of the v1 Unmarshal implementation. Change-Id: Ibeabbb2e9109a1ec3df57e71f98b7aa4a583fc5b Reviewed-on: https://go-review.googlesource.com/c/154577 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-17 11:16:16 -08:00
return func(m proto.Message) {
reflect/protoreflect: add alternative message reflection API Added API: Message.Len Message.Range Message.Has Message.Clear Message.Get Message.Set Message.Mutable Message.NewMessage Message.WhichOneof Message.GetUnknown Message.SetUnknown Deprecated API (to be removed in subsequent CL): Message.KnownFields Message.UnknownFields The primary difference with the new API is that the top-level Message methods are keyed by FieldDescriptor rather than FieldNumber with the following semantics: * For known fields, the FieldDescriptor must exactly match the field descriptor known by the message. * For extension fields, the FieldDescriptor must implement ExtensionType, where ContainingMessage.FullName matches the message name, and the field number is within the message's extension range. When setting an extension field, it automatically stores the extension type information. * Extension fields are always considered nullable, implying that repeated extension fields are nullable. That is, you can distinguish between a unpopulated list and an empty list. * Message.Get always returns a valid Value even if unpopulated. The behavior is already well-defined for scalars, but for unpopulated composite types, it now returns an empty read-only version of it. Change-Id: Ia120630b4db221aeaaf743d0f64160e1a61a0f61 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/175458 Reviewed-by: Damien Neil <dneil@google.com>
2019-04-25 23:48:08 -07:00
m.ProtoReflect().SetUnknown(raw)
proto: add Equal Add support for basic equality comparison of messages. Messages are equal if they have the same type and marshal to the same bytes with deterministic serialization, with some exceptions: - Messages with different registered extensions are unequal. - NaN is not equal to itself. Unlike the v1 Equal, a nil message is equal to an empty message of the same type. Change-Id: Ibabdadd8c767b801051b8241aeae1ba077e58121 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/174277 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-04-23 17:11:02 -07:00
}
}
all: unify protoV1.ExtensionDesc and proto.ExtensionType Change protoV1.ExtensionDesc to directly implement ExtensionType rather than delegating to one. Unify the previous types protoiface.ExtensionDescV1 and filetype.Extension in impl.ExtensionInfo. The protoV1.ExtensionDesc type becomes an alias to ExtensionInfo. This gives us: - Just one implementation of ExtensionType. - Generated foopb.E_Ext vars are canonical ExtensionTypes. - Generated foopb.E_Ext vars are also v1.ExtensionDescs for backwards compatibility. - Conversion between legacy and modern representations happens transparently when lazily initializing an ExtensionInfo. Overall, a simplification for users of generated code, since they can mostly ignore the ExtensionDesc/ExtentionType distinction and use the same value in either the old or new API. This is change 3/5 in a series of commits changing protoV1.ExtensionDesc to directly implement protoreflect.ExtensionType. 1. [v2] Add protoimpl.ExtensionInfo as an alias for protoiface.ExtensionDescV1. 2. [v1] Update references to protoimpl.ExtensionInfo to use protoiface.ExtensionInfo. 3. [v2] Create protoimpl.ExtensionInfo (an alias to a new type in the impl package) and remove protoiface.ExtensionDescV1. 4. [v1] Remove unneeded explicit conversions between ExtensionDesc and ExtensionType (since the former now directly implements the latter). 5. [v2] Remove stub conversion functions. Change-Id: I96ee890541ec11b2412e1a72c9d7b96e4d7f66b4 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189563 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-08 15:45:59 -07:00
func extend(desc protoreflect.ExtensionType, value interface{}) buildOpt {
internal/legacy: use v2 proto.Unmarshal The v2 decoder isn't 100% complete, but it's good enough. Delete the vendored copy of the v1 Unmarshal implementation. Change-Id: Ibeabbb2e9109a1ec3df57e71f98b7aa4a583fc5b Reviewed-on: https://go-review.googlesource.com/c/154577 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-17 11:16:16 -08:00
return func(m proto.Message) {
all: refactor extensions, add proto.GetExtension etc. Change protoiface.ExtensionDescV1 to implement protoreflect.ExtensionType. ExtensionDescV1's Name field conflicts with the Descriptor Name method, so change the protoreflect.{Message,Enum,Extension}Type types to no longer implement the corresponding Descriptor interface. This also leads to a clearer distinction between the two types. Introduce a protoreflect.ExtensionTypeDescriptor type which bridges between ExtensionType and ExtensionDescriptor. Add extension accessor functions to the proto package: proto.{Has,Clear,Get,Set}Extension. These functions take a protoreflect.ExtensionType parameter, which allows writing the same function call using either the old or new API: proto.GetExtension(message, somepb.E_ExtensionFoo) Fixes golang/protobuf#908 Change-Id: Ibc65d12a46666297849114fd3aefbc4a597d9f08 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189199 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-02 16:58:08 -07:00
proto.SetExtension(m, desc, value)
proto: wire decoding support Add proto.Unmarshal. Test cases all produce identical results to the v1 unmarshaller. Change-Id: I42259266018a14e88a650c5d83a043cb17a3a15d Reviewed-on: https://go-review.googlesource.com/c/153918 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-12-07 14:38:17 -08:00
}
}
proto: add generic Size Change-Id: I4ed123f4a9747fb4aba392bc5b9608d294bacc4d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169697 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-27 09:23:20 -07:00
func marshalText(m proto.Message) string {
proto/equal: equate nil Modify Equal to treat nil messages as equal iff both are nil. Of special note, a typed nil pointer to T is equal to a new(T) since they are indistinguishable from a protobuf reflection. Change-Id: Ibf90b43a982e7376e07b4159be198f06230ec194 Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/196618 Reviewed-by: Damien Neil <dneil@google.com>
2019-09-19 21:28:52 -07:00
if m == nil {
return "<nil>\n"
}
encoding/prototext: add MarshalOptions.EmitUnknown This changes text marshaling to avoid unknown fields by default and instead adds an option so that unknown fields be emitted. This ensures that the default marshal/unknown can round-trip. Change-Id: I85c84ba6ab7916d538ec6bfd4e9d399a8fcba14e Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/195778 Reviewed-by: Herbie Ong <herbie@google.com>
2019-09-14 19:14:24 -07:00
b, _ := prototext.MarshalOptions{
AllowPartial: true,
EmitUnknown: true,
Indent: "\t",
}.Marshal(m)
proto: add generic Size Change-Id: I4ed123f4a9747fb4aba392bc5b9608d294bacc4d Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/169697 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-03-27 09:23:20 -07:00
return string(b)
}
Reference in New Issue Copy Permalink