12 Commits

Author SHA1 Message Date
Damien Neil
ce8f7f6353 internal/impl: inline small tag decoding
Inline varint decoding of small (1- and 2-byte) field tags in the
fast-path unmarshaler.

name                             old time/op  new time/op  delta
EmptyMessage/Wire/Unmarshal      40.6ns ± 1%  40.2ns ± 1%   -1.02%  (p=0.000 n=37+35)
EmptyMessage/Wire/Unmarshal-12   6.77ns ± 2%  7.13ns ± 5%   +5.32%  (p=0.000 n=37+37)
RepeatedInt32/Wire/Unmarshal     9.46µs ± 1%  6.57µs ± 1%  -30.56%  (p=0.000 n=38+39)
RepeatedInt32/Wire/Unmarshal-12  1.50µs ± 2%  1.05µs ± 2%  -30.00%  (p=0.000 n=39+37)
Required/Wire/Unmarshal           371ns ± 1%   258ns ± 1%  -30.44%  (p=0.000 n=38+32)
Required/Wire/Unmarshal-12       60.3ns ± 1%  44.3ns ± 2%  -26.45%  (p=0.000 n=38+36)

Change-Id: Ie80415dea8cb6b840eafa52f0572046a1910a9b1
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/216419
Reviewed-by: Joe Tsai <joetsai@google.com>
2020-01-26 22:23:05 +00:00
Damien Neil
fe15dd4cdd all: don't allow invalid field numbers when legacy support is on
The deprecated messageset format permits extension fields with numbers
greater than the usual maximum (1<<29-1). To support this, the
internal/encoding/wire package has disabled field number validation when
legacy support is enabled.

We shouldn't skip validating all field numbers for validity just because
we support larger ones in messagesets.

This change drops range validation from the wire package (other than
checking that numbers fit in an int32) and adds it to the wire
unmarshalers instead. This gives us validation where we care
about it (when unmarshaling a wire-format message) and allows for
best-effort handling of out-of-range numbers everywhere else.

Fixes golang/protobuf#996

Change-Id: I4e11b8a8aa177dd60e89723570af074a317c2451
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/210290
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-12-09 18:35:13 +00:00
Joe Tsai
1799d1111a all: rename tag and flags for legacy support
Rename build tag "proto1_legacy" -> "protolegacy"
to be consistent with the "protoreflect" tag.

Rename flag constant "Proto1Legacy" -> "ProtoLegacy" since
it covers more than simply proto1 legacy features.
For example, it covers alpha-features of proto3 that
were eventually removed from the final proto3 release.

Change-Id: I0f4fcbadd4b5a61c87645e2e5be11d187e59157c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189345
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-08 20:49:00 +00:00
Damien Neil
e91877de26 internal/impl: add fast-path unmarshal
Benchmarks run with:
  go test ./benchmarks/ -bench=Wire  -benchtime=500ms -benchmem -count=8

Fast-path vs. parent commit:

  name                                      old time/op    new time/op    delta
  Wire/Unmarshal/google_message1_proto2-12    1.35µs ± 2%    0.45µs ± 4%  -67.01%  (p=0.000 n=8+8)
  Wire/Unmarshal/google_message1_proto3-12    1.07µs ± 1%    0.31µs ± 1%  -71.04%  (p=0.000 n=8+8)
  Wire/Unmarshal/google_message2-12            691µs ± 2%     188µs ± 2%  -72.78%  (p=0.000 n=7+8)

  name                                      old allocs/op  new allocs/op  delta
  Wire/Unmarshal/google_message1_proto2-12      60.0 ± 0%      25.0 ± 0%  -58.33%  (p=0.000 n=8+8)
  Wire/Unmarshal/google_message1_proto3-12      42.0 ± 0%       7.0 ± 0%  -83.33%  (p=0.000 n=8+8)
  Wire/Unmarshal/google_message2-12            28.6k ± 0%      8.5k ± 0%  -70.34%  (p=0.000 n=8+8)

Fast-path vs. -v1:

  name                                      old time/op    new time/op    delta
  Wire/Unmarshal/google_message1_proto2-12     702ns ± 1%     445ns ± 4%   -36.58%  (p=0.000 n=8+8)
  Wire/Unmarshal/google_message1_proto3-12     604ns ± 1%     311ns ± 1%   -48.54%  (p=0.000 n=8+8)
  Wire/Unmarshal/google_message2-12            179µs ± 3%     188µs ± 2%    +5.30%  (p=0.000 n=7+8)

  name                                      old allocs/op  new allocs/op  delta
  Wire/Unmarshal/google_message1_proto2-12      26.0 ± 0%      25.0 ± 0%    -3.85%  (p=0.000 n=8+8)
  Wire/Unmarshal/google_message1_proto3-12      8.00 ± 0%      7.00 ± 0%   -12.50%  (p=0.000 n=8+8)
  Wire/Unmarshal/google_message2-12            8.49k ± 0%     8.49k ± 0%    -0.01%  (p=0.000 n=8+8)

Change-Id: I6247ac3fd66a63d9acb902cbd192094ee3d151c3
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185147
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-07-09 19:56:42 +00:00
Joe Tsai
9aeddcdcf0 internal/encoding/wire: fix trivial misspelling
Change-Id: I71f1715145ebc58aadec1d5ecb5f1900998f65a0
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/184277
Reviewed-by: Damien Neil <dneil@google.com>
2019-07-08 20:14:46 +00:00
Damien Neil
cedb595154 proto, internal/impl: avoid string->[]byte conversions
We trust the compiler to optimize away the string->[]byte conversion in
code like:

	b = wire.AppendBytes(b, []byte(s))

In testing (go 1.12.5 linux/amd64), this optimization is not happening.
Perhaps newer versions of the compiler will optimize this, but we
shouldn't rely on it; avoid unnecessary conversions.

Benchmark differences vs https://golang.org/cl/171462:

  name                                   old time/op    new time/op    delta
  Wire/Marshal/google_message1_proto2-6     310ns ± 2%     189ns ± 3%  -39.20%  (p=0.000 n=8+8)
  Wire/Marshal/google_message1_proto3-6     389ns ± 8%     261ns ± 2%  -33.03%  (p=0.000 n=8+8)
  Wire/Marshal/google_message2-6            103µs ±11%      59µs ± 4%  -42.17%  (p=0.000 n=8+8)

  name                                   old alloc/op   new alloc/op   delta
  Wire/Marshal/google_message1_proto2-6      592B ± 0%      240B ± 0%  -59.46%  (p=0.000 n=8+8)
  Wire/Marshal/google_message1_proto3-6      576B ± 0%      224B ± 0%  -61.11%  (p=0.000 n=8+8)
  Wire/Marshal/google_message2-6            196kB ± 0%      90kB ± 0%  -54.05%  (p=0.000 n=8+8)

  name                                   old allocs/op  new allocs/op  delta
  Wire/Marshal/google_message1_proto2-6      5.00 ± 0%      1.00 ± 0%  -80.00%  (p=0.000 n=8+8)
  Wire/Marshal/google_message1_proto3-6      5.00 ± 0%      1.00 ± 0%  -80.00%  (p=0.000 n=8+8)
  Wire/Marshal/google_message2-6            1.66k ± 0%     0.00k ± 0%  -99.94%  (p=0.000 n=8+8)

Change-Id: Idab7634b8c86604dffa46895ba2e61be38c9bd9c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/183380
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-06-24 17:41:45 +00:00
Damien Neil
e89e6244e0 all: change module to google.golang.org/protobuf
Temporarily remove go.mod, since we can't generate an accurate one until
the corresponding v1 change is submitted.

Change-Id: I1e1ad97f2b455e33f61ffaeb8676289795e47e72
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/177000
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-05-14 17:28:29 +00:00
Joe Tsai
972d873b59 internal/encoding/wire: increase maximum field number
The protobuf documentation explicitly specifies (1<<29)-1 as the maximum
field number, but the C++ implementation itself has a special-case where it
allows field numbers up to MaxInt32 for MessageSet fields, but continues
to apply the former limit in all non-MessageSet cases.

To avoid complicated branching logic, we use the larger limit for all cases
if MessageSet is supported, otherwise, we impose the documented limit.

Change-Id: I710a2a21aa3beba161c3e6ca2f2ed9a266920a5a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/175817
Reviewed-by: Damien Neil <dneil@google.com>
2019-05-07 21:39:02 +00:00
Joe Tsai
01ab29648e go.mod: rename google.golang.org/proto as github.com/golang/protobuf/v2
This change was created by running:
	git ls-files | xargs sed -i "s|google.golang.org/proto|github.com/golang/protobuf/v2|g"

This change is *not* an endorsement of "github.com/golang/protobuf/v2" as the
final import path when the v2 API is eventually released as stable.
We continue to reserve the right to make breaking changes as we see fit.

This change enables us to host the v2 API on a repository that is go-gettable
(since go.googlesource.com is not a known host by the "go get" tool;
and google.golang.org/proto was just a stub URL that is not currently served).
Thus, we can start work on a forked version of the v1 API that explores
what it would take to implement v1 in terms of v2 in a backwards compatible way.

Change-Id: Ia3ebc41ac4238af62ee140200d3158b53ac9ec48
Reviewed-on: https://go-review.googlesource.com/136736
Reviewed-by: Damien Neil <dneil@google.com>
2018-09-24 16:11:50 +00:00
Joe Tsai
0e6baaa2cb internal/encoding/wire: switch to use internal errors package
Change-Id: Ibb01f8a5d4dea09c19262f807e305bbebe0e6ab8
Reviewed-on: https://go-review.googlesource.com/127825
Reviewed-by: Herbie Ong <herbie@google.com>
2018-08-04 01:02:24 +00:00
Joe Tsai
be60f99ca7 internal/encoding/wire: fix minor miscategorization in test
Some of the test cases in TestFixed64 actually belong in TestBytes.

Change-Id: I7f3efd77662881b64a96311161440fd220ae8074
Reviewed-on: https://go-review.googlesource.com/127456
Reviewed-by: Chris Manghane <cmang@golang.org>
2018-08-02 21:53:01 +00:00
Joe Tsai
22505a4929 internal/encoding/wire: initial commit
This adds package wire, which provides low-level functionality for
marshaling and unmarshaling the protobuf wire format.

High-level API:
	type Number int32
	    const MinValidNumber Number = 1 ...
	type Type int8
	    const VarintType Type = 0 ...
	func ParseError(n int) error

	func ConsumeField(b []byte) (Number, Type, int)
	func ConsumeFieldValue(num Number, typ Type, b []byte) (n int)
	func ConsumeTag(b []byte) (Number, Type, int)
	func ConsumeVarint(b []byte) (v uint64, n int)
	func ConsumeFixed32(b []byte) (v uint32, n int)
	func ConsumeFixed64(b []byte) (v uint64, n int)
	func ConsumeBytes(b []byte) (v []byte, n int)
	func ConsumeGroup(num Number, b []byte) (v []byte, n int)
	func AppendTag(b []byte, num Number, typ Type) []byte
	func AppendVarint(b []byte, v uint64) []byte
	func AppendFixed32(b []byte, v uint32) []byte
	func AppendFixed64(b []byte, v uint64) []byte
	func AppendBytes(b []byte, v []byte) []byte
	func AppendGroup(b []byte, num Number, v []byte) []byte
	func SizeTag(num Number) int
	func SizeVarint(v uint64) int
	func SizeFixed32() int
	func SizeFixed64() int
	func SizeBytes(n int) int
	func SizeGroup(num Number, n int) int

	func DecodeBool(x uint64) bool
	func DecodeTag(x uint64) (Number, Type)
	func DecodeZigZag(x uint64) int64
	func EncodeBool(x bool) uint64
	func EncodeTag(num Number, typ Type) uint64
	func EncodeZigZag(x int64) uint64

Change-Id: I052d8975414aeb182f6e9595c4736e716f1b7e9d
Reviewed-on: https://go-review.googlesource.com/127337
Reviewed-by: Chris Manghane <cmang@golang.org>
Run-TryBot: Chris Manghane <cmang@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2018-08-01 23:02:07 +00:00