Commit Graph

63 Commits

Author SHA1 Message Date
Michael Stapelberg
eb7b468655 all: Release the Opaque API
For golang/protobuf#1657

Change-Id: I7b2b0c30506706015ce278e6054439c9ad9ef727
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/634815
TryBot-Bypass: Michael Stapelberg <stapelberg@google.com>
Reviewed-by: Joseph Tsai <joetsai@digital-static.net>
Reviewed-by: Damien Neil <dneil@google.com>
2024-12-11 03:16:51 -08:00
Reno Reckling
03df6c145d proto: Implement proto.Equal fast-path
Also adds better benchmark cases for large message where some fields are
actually populated.

This change was previously done in Google internal cl/660848520.

Change-Id: I682aae0c9c2850bfe7638de29ab743ad7d7b119a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/609035
Reviewed-by: Christian Höppner <hoeppi@google.com>
Reviewed-by: Cassondra Foesch <cfoesch@gmail.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-09-06 16:39:44 +00:00
Damien Neil
b0a944684d all: reformat with go1.19 gofmt
Apply go1.19 gofmt to non-generated files.

Generated .pb.go files are created by generate.bash using Go 1.18,
so leave them unchanged for now.

Change-Id: Ied36c83cf99704988d059bf0412e677f0fbc71b0
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/418676
Reviewed-by: Ian Lance Taylor <iant@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
2022-07-21 16:01:13 +00:00
Lasse Folger
3992ea83a2 all: implement depth limit for unmarshaling
+ This change introduce a default and configurable depth limit for
  proto.Unmarshal. If a message is nested deeper than the limit,
  unmarshaling will fail. There are two ways to nest messages. Either by
  having fields which are message types itself or by using groups.
+ The default limit is 10,000 for now. This might change in the future
  to align it with other language implementation (C++ and Java use 100
  as limit).
+ If pure groups (groups that don't contain message fields) are nested
  deeper than the default limit the unmarshaling fails with:
  proto: cannot parse invalid wire-format data
+ Note: the configured limit does not apply to pure groups.
+ This change is introduced to improve security and robustness. Because
  unmarshaling is implemented using recursion it can lead to stack overflows
  for certain inputs. The introduced limit protects against this.
+ A secondary motivation for this limit is the alignment with other
  languages. Protocol buffers are a language interoperability mechanism
  and thus either all implementations should accept the input or all
  implementation should reject the input.

Change-Id: I14bdb44d06e4bd1aa90d6336c2cf6446003b2037
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/385854
Trust: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Damien Neil <dneil@google.com>
Trust: Damien Neil <dneil@google.com>
Reviewed-by: Nicolas Hillegeer <aktau@google.com>
Reviewed-by: Chressie Himpel <chressie@google.com>
2022-02-17 17:07:31 +00:00
Joe Tsai
4ab2bc9bb7 internal/version: move version information to internal package
Change-Id: I947876de5d290cf783d9ba798871725e77e16517
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/223277
Reviewed-by: Damien Neil <dneil@google.com>
2020-03-13 19:57:50 +00:00
Joe Tsai
a56bd81046 all: start v1.20.1-devel
Change-Id: Ie2940cd41fd3c449b2233e1bc3b6ba5ba91038f2
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/222699
Reviewed-by: Damien Neil <dneil@google.com>
2020-03-09 21:07:39 +00:00
Joe Tsai
290f718d07 all: release v1.20.1
Change-Id: I77c61b55d3c6aa540103ffe82c9dd226959bb753
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/222698
Reviewed-by: Damien Neil <dneil@google.com>
2020-03-09 21:07:25 +00:00
Damien Neil
969537aa29 all: start post-v1.20.0 devel
Change-Id: I811fab64a8a22288760beef04021ded63bc4ea9b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/221784
Reviewed-by: Herbie Ong <herbie@google.com>
2020-03-02 19:36:27 +00:00
Damien Neil
aa81c06513 all: release v1.20.0
Change-Id: I3bc9d46f836535393059b9a73e8364211eee9e38
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/221783
Reviewed-by: Herbie Ong <herbie@google.com>
2020-03-02 18:51:28 +00:00
Joe Tsai
fea7d5276d all: start v1.20.0-rc.1.devel
Change-Id: I7a88cf90c9b2ee61d6eae5019829362a7dce164f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/221598
Reviewed-by: Damien Neil <dneil@google.com>
2020-02-29 07:48:10 +00:00
Joe Tsai
a6bf928d57 all: release v1.20.0-rc.1
Change-Id: I383cd3e94788f80b95f3ef7f1b86922fc0ea1db3
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/221597
Reviewed-by: Damien Neil <dneil@google.com>
2020-02-29 07:47:16 +00:00
Joe Tsai
ec552f691d runtime/protoiface: trivial change to Size
Remove named input argument to be consistent with other methods.

Change-Id: I22c48abf76e007a1319bfb037bbb2b7bfb66cee7
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/220688
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2020-02-25 10:28:07 +00:00
Joe Tsai
f26a9e7e30 all: rename IsInitialized as CheckInitialized
An Is prefix implies it returns a boolean.
A Check prefix better suggests that it could return an error.

Change-Id: I6ffcb32099a944c656c07654c294a0980efb2d0e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/220338
Reviewed-by: Damien Neil <dneil@google.com>
2020-02-20 20:45:40 +00:00
Joe Tsai
3df87a0b61 runtime/protoiface: remove IsInitializedOutputFlags and IsInitialized
The IsInitialized flag is unused and unnecessary as the error returned
inherently reports whether a message is uninitialized if it is non-nil.

Also cleanup the documentation in protoiface of stale references.

Change-Id: Id75772f4f75f8d7a2a9fbe772d57f7d69ae9fb9d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/220337
Reviewed-by: Damien Neil <dneil@google.com>
2020-02-20 19:26:44 +00:00
Damien Neil
466dd77288 all: fast-path method refactoring
Move all fast-path inputs and outputs into the Input/Output structs.
Collapse all booleans into bitfields.

Change-Id: I79ebfbac9cd1d8ef5ec17c4f955311db007391ca
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219505
Reviewed-by: Joe Tsai <joetsai@google.com>
2020-02-19 22:01:50 +00:00
Damien Neil
e8e8875f94 proto, runtime/protoiface, internal/impl: add fast-path Merge
Comparing -tags=protoreflect to fast-path:

name                              old time/op    new time/op    delta
pkg:google.golang.org/protobuf/internal/benchmarks goos:linux goarch:amd64
/Clone/google_message1_proto2-12    1.70µs ± 1%    0.30µs ± 1%  -82.64%  (p=0.001 n=7+7)
/Clone/google_message1_proto3-12    1.01µs ± 1%    0.19µs ± 1%  -80.77%  (p=0.000 n=7+8)
/Clone/google_message2-12            818µs ± 8%     141µs ± 6%  -82.78%  (p=0.000 n=8+8)
pkg:google.golang.org/protobuf/internal/benchmarks/micro goos:linux goarch:amd64
EmptyMessage/Clone-12               51.1ns ± 1%    39.3ns ± 3%  -23.03%  (p=0.000 n=7+8)
RepeatedInt32/Clone-12              24.5µs ± 1%     1.1µs ± 3%  -95.64%  (p=0.000 n=8+8)
Required/Clone-12                    978ns ± 1%     132ns ± 2%  -86.46%  (p=0.000 n=8+8)

name                              old alloc/op   new alloc/op   delta
pkg:google.golang.org/protobuf/internal/benchmarks goos:linux goarch:amd64
/Clone/google_message1_proto2-12    1.08kB ± 0%    0.74kB ± 0%  -31.85%  (p=0.000 n=8+8)
/Clone/google_message1_proto3-12      872B ± 0%      544B ± 0%  -37.61%  (p=0.000 n=8+8)
/Clone/google_message2-12            602kB ± 0%     411kB ± 0%  -31.65%  (p=0.000 n=8+8)
pkg:google.golang.org/protobuf/internal/benchmarks/micro goos:linux goarch:amd64
EmptyMessage/Clone-12                96.0B ± 0%     64.0B ± 0%  -33.33%  (p=0.000 n=8+8)
RepeatedInt32/Clone-12              25.4kB ± 0%     3.2kB ± 0%  -87.33%  (p=0.000 n=8+8)
Required/Clone-12                     416B ± 0%      256B ± 0%  -38.46%  (p=0.000 n=8+8)

name                              old allocs/op  new allocs/op  delta
pkg:google.golang.org/protobuf/internal/benchmarks goos:linux goarch:amd64
/Clone/google_message1_proto2-12      52.0 ± 0%      21.0 ± 0%  -59.62%  (p=0.000 n=8+8)
/Clone/google_message1_proto3-12      33.0 ± 0%       3.0 ± 0%  -90.91%  (p=0.000 n=8+8)
/Clone/google_message2-12            22.3k ± 0%      7.5k ± 0%  -66.41%  (p=0.000 n=8+8)
pkg:google.golang.org/protobuf/internal/benchmarks/micro goos:linux goarch:amd64
EmptyMessage/Clone-12                 3.00 ± 0%      2.00 ± 0%  -33.33%  (p=0.000 n=8+8)
RepeatedInt32/Clone-12               1.51k ± 0%     0.00k ± 0%  -99.80%  (p=0.000 n=8+8)
Required/Clone-12                     51.0 ± 0%      18.0 ± 0%  -64.71%  (p=0.000 n=8+8)

Change-Id: Ife9018097c34cb025dc9c4fdd9a61b2f947853c6
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219147
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2020-02-14 21:47:10 +00:00
Damien Neil
6fb29949b8 all: tests, tweaks for lazy extension decoding
Add a test to confirm that extensions are lazily decoded when we expect.

Drop the UnmarshalDefaultResolver flag. I added it thinking for some
reason that internal/impl couldn't depend on protoregistry; since it can
(and does), it's simpler to just test if the resolver is the expected
value.

Use a default set of options when lazily unmarshaling extensions.

Change-Id: Ied7666ffdc3bf90630260a80c9568d9a945048bc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/218038
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2020-02-06 19:43:25 +00:00
Damien Neil
0ae1c9789a internal/impl: lazy extension decoding
Historically, extensions have been placed in the unknown fields section
of the unmarshaled message and decoded lazily on demand. The current
unmarshal implementation decodes extensions eagerly at unmarshal time,
permitting errors to be immediately reported and correctly detecting
unset required fields in extension values.

Add support for validated lazy extension decoding, where the extension
value is fully validated at initial unmarshal time but the fully
unmarshaled message is only created lazily.

Make this behavior conditional on the protolegacy flag for now.

Change-Id: I9d742496a4bd4dafea83fca8619cd6e8d7e65bc3
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/216764
Reviewed-by: Joe Tsai <joetsai@google.com>
2020-01-29 21:35:31 +00:00
Damien Neil
524c60670a runtime/protoiface: use more efficient options representation
Change the representation of option flags in protoiface from bools to a
bitfield. This brings the representation of options in protoiface in
sync with that in internal/impl.

This change has several benefits:

1. We will probably find that we need to add more option flags over time.
Converting to the more efficient representation of these flags as high
in the call stack as possible minimizes the performance implication of
the struct growing.

2. On a similar note, this avoids the need to convert from the compact
representation to the larger one when passing from internal/impl to
proto, since the {Marshal,Unmarshal}State methods take the compact form.

3. This removes unused options from protoiface. Instead of documenting
that AllowPartial is always set, we can just not include an AllowPartial
flag in the protoiface options.

4. Conversely, this provides a way to add option flags to protoiface
that we don't want to expose in the proto package.

name                             old time/op    new time/op    delta
EmptyMessage/Wire/Marshal-12       11.1ns ± 7%    10.1ns ± 1%   -9.35%  (p=0.000 n=8+8)
EmptyMessage/Wire/Unmarshal-12     7.07ns ± 0%    6.74ns ± 1%   -4.58%  (p=0.000 n=8+8)
EmptyMessage/Wire/Validate-12      4.30ns ± 1%    3.80ns ± 8%  -11.45%  (p=0.000 n=7+8)
RepeatedInt32/Wire/Marshal-12      1.17µs ± 1%    1.21µs ± 7%   +4.09%  (p=0.000 n=8+8)
RepeatedInt32/Wire/Unmarshal-12     938ns ± 0%     942ns ± 3%     ~     (p=0.178 n=7+8)
RepeatedInt32/Wire/Validate-12      521ns ± 4%     543ns ± 7%     ~     (p=0.157 n=7+8)
Required/Wire/Marshal-12           97.2ns ± 1%    95.3ns ± 1%   -1.98%  (p=0.001 n=7+7)
Required/Wire/Unmarshal-12         41.0ns ± 9%    38.6ns ± 3%   -5.73%  (p=0.048 n=8+8)
Required/Wire/Validate-12          25.4ns ±11%    21.4ns ± 3%  -15.62%  (p=0.000 n=8+7)

Change-Id: I3ac1b00ab36cfdf61316ec087a5dd20d9248e4f6
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/216760
Reviewed-by: Joe Tsai <joetsai@google.com>
2020-01-28 23:33:31 +00:00
Damien Neil
c600d6c086 all: do best-effort initialization check on fast path unmarshal
Add a fast check for required fields to the fast path unmarshal.
This is best-effort and will fail to detect some initialized
messages: Messages with more than 64 required fields, messages
split across multiple tags, possibly other cases.

In the cases where it works (which is most of them in practice),
this permits us to skip the IsInitialized check.

Change-Id: I6b70953a333033a5e64fb7ca37a59786cb0f75a0
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/215878
Reviewed-by: Joe Tsai <joetsai@google.com>
2020-01-22 20:57:14 +00:00
Damien Neil
d30e561d9e proto: add MarshalState, UnmarshalState
Add functions to the proto package which plumb through the fast-path state.

As a sample use case: A followup CL adds an Initialized field to
protoiface.UnmarshalOutput, permitting the unmarshaller to report back
when it can confirm that a message is fully initialized. We want to
preserve that information when an unmarshal operation threads through
the proto package (such as when unmarshaling extensions).

To allow these functions to be added as methods of MarshalOptions and
UnmarshalOptions rather than top-level functions, separate the options
from the input structs.

Also update options passed to fast-path methods to set AllowPartial and
Merge to reflect the expected behavior of those methods. (Always allow
partial, never merge.)

Change-Id: I482477b0c9340793be533e75a86d0bb88708716a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/215877
Reviewed-by: Joe Tsai <joetsai@google.com>
2020-01-22 20:52:17 +00:00
Damien Neil
61781dd92f all: abstract fast-path marshal and unmarshal inputs and outputs
We may want to make changes to the inputs and outputs of the fast-path
functions in the future. For example, we likely want to add the ability
for the fast-path unmarshal to report back whether the unmarshaled
message is known to be initialized.

Change the signatures of these functions to take in and return struct
types which can be extended with whatever fields we want in the future.

Change-Id: Idead360785df730283a4630ea405265b72482e62
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/215719
Reviewed-by: Joe Tsai <joetsai@google.com>
2020-01-22 00:22:46 +00:00
Damien Neil
f12fb45fd6 all: add ProtoMethods method to protoreflect.Message
Promote the fast-path magic ProtoMethods method to first-class citizen
of the protoreflect.Message interface.

To avoid polluting the protoreflect package with the various types
required by this method, make the necessary protoiface types unnamed and
duplicate them in protoreflect.

Updates golang/protobuf#1022.

Change-Id: I9595bae40b3bc7536d727fb6f99b3bce8f73da87
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/215718
Reviewed-by: Joe Tsai <joetsai@google.com>
2020-01-21 21:05:54 +00:00
Damien Neil
b0d217f664 proto, internal/impl: don't create fast path Size for legacy Marshalers
Implementations of the legacy Marshaler type have no way to efficiently
compute the size of the message. Rather than generating an inefficient
fast-path Size method which marshals the message and examines the
length of the result, don't generate a fast-path at all.

Drop the requirement that a fast-path MarshalAppend requires a
corresponding Size.

Avoids O(N^2) behavior when marshaling a legacy Marshaler that
recursively calls proto.Marshal.

Change-Id: I4793cf32275d08f29c8e1a1a44a193d9a5724058
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/213443
Reviewed-by: Joe Tsai <joetsai@google.com>
2020-01-06 22:47:37 +00:00
Joe Tsai
705acadcc7 proto: reset message by default in Unmarshal
We change Unmarshal to reset a message by default.
* We add a Merge option to UnmarshalOptions for explicit merging.
* We speed up Reset by checking for the Reset method.
* Remove TODOs in prototext and protojson about reset behavior.

Fixes golang/protobuf#890

Change-Id: Ibd8963c741053f564acf061fbdb846699942109c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/195457
Reviewed-by: Damien Neil <dneil@google.com>
2019-09-17 21:13:42 +00:00
Joe Tsai
cbdefb1530 internal/impl: move filetype.EnumInfo to impl
Change-Id: I5558d9d4618df80dd08969fb8af1bc16191176b5
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/195620
Reviewed-by: Damien Neil <dneil@google.com>
2019-09-16 22:07:05 +00:00
Damien Neil
f1e905b042 all: unify protoV1.ExtensionDesc and proto.ExtensionType
Change protoV1.ExtensionDesc to directly implement ExtensionType
rather than delegating to one.

Unify the previous types protoiface.ExtensionDescV1 and
filetype.Extension in impl.ExtensionInfo. The protoV1.ExtensionDesc
type becomes an alias to ExtensionInfo.

This gives us:

  - Just one implementation of ExtensionType.
  - Generated foopb.E_Ext vars are canonical ExtensionTypes.
  - Generated foopb.E_Ext vars are also v1.ExtensionDescs for backwards
    compatibility.
  - Conversion between legacy and modern representations happens
    transparently when lazily initializing an ExtensionInfo.

Overall, a simplification for users of generated code, since they can
mostly ignore the ExtensionDesc/ExtentionType distinction and use the
same value in either the old or new API.

This is change 3/5 in a series of commits changing protoV1.ExtensionDesc
to directly implement protoreflect.ExtensionType.

1. [v2] Add protoimpl.ExtensionInfo as an alias for
   protoiface.ExtensionDescV1.

2. [v1] Update references to protoimpl.ExtensionInfo to use
   protoiface.ExtensionInfo.

3. [v2] Create protoimpl.ExtensionInfo (an alias to a new type in
   the impl package) and remove protoiface.ExtensionDescV1.

4. [v1] Remove unneeded explicit conversions between ExtensionDesc and
   ExtensionType (since the former now directly implements the latter).

5. [v2] Remove stub conversion functions.

Change-Id: I96ee890541ec11b2412e1a72c9d7b96e4d7f66b4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189563
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-20 21:32:57 +00:00
Damien Neil
c0f8c0a24e runtime/protoimpl: add ExtensionInfo alias
This is change 1/5 in a series of commits changing protoV1.ExtensionDesc
to directly implement protoreflect.ExtensionType.

1. [v2] Add protoimpl.ExtensionInfo as an alias for
   protoiface.ExtensionDescV1.

2. [v1] Update references to protoimpl.ExtensionInfo to use
   protoiface.ExtensionInfo.

3. [v2] Create protoimpl.ExtensionInfo (an alias to a new type in
   the impl package) and remove protoiface.ExtensionDescV1.

4. [v1] Remove unneeded explicit conversions between ExtensionDesc and
   ExtensionType (since the former now directly implements the latter).

5. [v2] Remove stub conversion functions.

Change-Id: If6c7fd5f55364613387a05e6f8e9aa38cbfcc5b5
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189562
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-20 20:36:59 +00:00
Damien Neil
4401a0de4b cmd/protoc-gen-go, internal/filetype: clean up EnumType construction
Drop the dependency from generated files on prototype.Enum: Generated
code should only depend on runtime/proto{iface,impl}.

Drop the Enums, Messages, and Extensions returns form
filetype.Builder.Build. Of these, only Enums was used by generated code.
Change the generated init function to pass the builder a slice of values
to fill in (as is done for messages and extensions).

Remove the filetype dependency on prototype in preparation for
eventually dropping the prototype package entirely.

Change-Id: I28a3420f5dfcc13fed531a64ef07b9afddfd9d55
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189200
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-09 19:04:41 +00:00
Damien Neil
92f76189a3 all: refactor extensions, add proto.GetExtension etc.
Change protoiface.ExtensionDescV1 to implement protoreflect.ExtensionType.

ExtensionDescV1's Name field conflicts with the Descriptor Name method,
so change the protoreflect.{Message,Enum,Extension}Type types to no
longer implement the corresponding Descriptor interface. This also leads
to a clearer distinction between the two types.

Introduce a protoreflect.ExtensionTypeDescriptor type which bridges
between ExtensionType and ExtensionDescriptor.

Add extension accessor functions to the proto package:
proto.{Has,Clear,Get,Set}Extension. These functions take a
protoreflect.ExtensionType parameter, which allows writing the
same function call using either the old or new API:

  proto.GetExtension(message, somepb.E_ExtensionFoo)

Fixes golang/protobuf#908

Change-Id: Ibc65d12a46666297849114fd3aefbc4a597d9f08
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189199
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-08-08 18:20:51 +00:00
Joe Tsai
bab3d4084e runtime/protoimpl, cmd/protoc-gen-go: support release versioning
In order for protoc-gen-go to output the current version,
it needs to know what version it is currently running as.
However, we cannot rely on the git tags since the tags are not
made until *after* the commit has been submitted.
Instead, we manually encode the version into the code and
make sure that git tags match up with the version in the code.

The version.go file in runtime/protoimpl contains instructions
for how to make a release. Essentially:
* Every non-release commit has a version string with "devel" in it.
* Every release commit must not have "devel" in it and must be unique.
* The "release process" involves submitting two CLs.
The first CL creates a version string without "devel",
which is the commit that a git tag will actually reference.
The second CL follows immediately and re-introduces "devel"
into the version string.

The following example shows a possible sequence of VersionStrings
for git commits in time-ascending order:
	v1.19.0-devel      (this CL)
	v1.19.0-devel
	v1.19.0-devel
	v1.19.0-devel
	v1.20.0-rc.1       <- tagged
	v1.20.0-rc.1.devel
	v1.20.0-rc.1.devel
	v1.20.0-rc.1.devel
	v1.20.0-rc.2       <- tagged
	v1.20.0-rc.2.devel
	v1.20.0            <- tagged (future public release)
	v1.20.0-devel
	v1.20.0-devel
	v1.20.0-devel
	v1.20.0-devel
	v1.20.1            <- tagged
	v1.20.1-devel
	v1.20.1-devel
	v1.21.0            <- tagged
	v1.21.0-devel

Note that we start today with v1.19.0-devel, which means that our initial
release will be v1.20.0. This number was intentionally chosen since
1) the number 20 has some correlation to the fact that we keep calling
the new implementation the "v2" implementation, and
2) the set of tagged versions for github.com/golang/protobuf
and google.golang.org/protobuf are unlikely to ever overlap.
This way, the version of protoc-gen-go is never ambiguous which module
it was built from.

Now that we have version information, we add support for generating .pb.go
files with the version information recorded. However, we do not emit
these for .pb.go files in our own repository since they are always guaranteed
to be at the right version (enforced by integration_test.go).

Updates golang/protobuf#524

Change-Id: I25495a45042c2aa39a39cb7e7738ae8e831a9d26
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186117
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-07 22:59:30 +00:00
Joe Tsai
52ec1759a4 internal/filedesc, internal/filetype: rename {Desc,Type}Builder as Builder
Reduce the stutter in the name since the type of Builder
is obvious from the package it is from.

Change-Id: I0046a5122717536cc6bb5ebdb32b67a1560cfc23
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189020
Reviewed-by: Damien Neil <dneil@google.com>
2019-08-05 23:43:37 +00:00
Joe Tsai
3d8e369c4e all: implement proto1 weak fields
This implements generation of and reflection support for weak fields.
Weak fields are a proto1 feature where the "weak" option can be specified
on a singular message field. A weak reference results in generated code
that does not directly link in the dependency containing the weak message.

Weak field support is not added to any of the serialization logic.

Change-Id: I08ccfa72bc80b2ffb6af527a1677a0a81dcf33fb
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185399
Reviewed-by: Damien Neil <dneil@google.com>
2019-07-15 18:44:12 +00:00
Joe Tsai
f8b855d768 runtime/protoiface: API adjustments
The following adjustments were made:
* The pragma.NoUnkeyedLiterals is moved to be the first field.
This is done to keep the options struct smaller. Even if the last
field is zero-length, Go GC implementation details forces the struct
to be padded at the end.
* Methods are documented as always treating AllowPartial as true.
* Added a support flag for UnmarshalOptions.DiscardUnknown.

Change-Id: I1f75d226542ab2bb0123d9cea143c7060df226d8
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185998
Reviewed-by: Damien Neil <dneil@google.com>
2019-07-12 23:12:26 +00:00
Joe Tsai
0f81b38d61 runtime/protoiface: move and rename XXX_Methods
This CL moves and renames the protoreflect.ProtoMessage.XXX_Methods
to protoreflect.Message.ProtoMethods.

Since one needs to obtain a protoreflect.Message now to get at
the fast-path methods, we modify the method signatures to take
in a protoreflect.Message instead of protoreflect.ProtoMessage.
Doing so also avoids the wrapper hack that was formerly done on
impl.messageReflectWrapper.

After this change the new protoc-gen-go no longer generates
any XXX fields or methods. All internal fields and methods are truly
hidden from the end-user.

name                                     old time/op    new time/op    delta
Wire/Unmarshal/google_message1_proto2-4    1.50µs ±10%    1.50µs ± 2%    ~     (p=0.483 n=10+9)
Wire/Unmarshal/google_message1_proto3-4    1.06µs ± 6%    1.06µs ± 4%    ~     (p=0.814 n=9+9)
Wire/Unmarshal/google_message2-4            734µs ±22%     689µs ±13%    ~     (p=0.133 n=10+9)
Wire/Marshal/google_message1_proto2-4       790ns ±46%     652ns ± 8%    ~     (p=0.590 n=10+9)
Wire/Marshal/google_message1_proto3-4       872ns ± 4%     857ns ± 3%    ~     (p=0.168 n=9+9)
Wire/Marshal/google_message2-4              232µs ±16%     221µs ± 3%  -4.75%  (p=0.014 n=9+9)
Wire/Size/google_message1_proto2-4          164ns ± 2%     167ns ± 4%  +1.87%  (p=0.046 n=9+10)
Wire/Size/google_message1_proto3-4          240ns ± 9%     229ns ± 1%  -4.81%  (p=0.000 n=9+8)
Wire/Size/google_message2-4                58.9µs ± 9%    59.6µs ± 2%  +1.23%  (p=0.040 n=9+9)

name                                     old alloc/op   new alloc/op   delta
Wire/Unmarshal/google_message1_proto2-4      912B ± 0%      912B ± 0%    ~     (all equal)
Wire/Unmarshal/google_message1_proto3-4      688B ± 0%      688B ± 0%    ~     (all equal)
Wire/Unmarshal/google_message2-4            470kB ± 0%     470kB ± 0%    ~     (p=0.215 n=10+10)
Wire/Marshal/google_message1_proto2-4        240B ± 0%      240B ± 0%    ~     (all equal)
Wire/Marshal/google_message1_proto3-4        224B ± 0%      224B ± 0%    ~     (all equal)
Wire/Marshal/google_message2-4             90.1kB ± 0%    90.1kB ± 0%    ~     (all equal)
Wire/Size/google_message1_proto2-4          0.00B          0.00B         ~     (all equal)
Wire/Size/google_message1_proto3-4          0.00B          0.00B         ~     (all equal)
Wire/Size/google_message2-4                 0.00B          0.00B         ~     (all equal)

name                                     old allocs/op  new allocs/op  delta
Wire/Unmarshal/google_message1_proto2-4      24.0 ± 0%      24.0 ± 0%    ~     (all equal)
Wire/Unmarshal/google_message1_proto3-4      6.00 ± 0%      6.00 ± 0%    ~     (all equal)
Wire/Unmarshal/google_message2-4            8.49k ± 0%     8.49k ± 0%    ~     (all equal)
Wire/Marshal/google_message1_proto2-4        1.00 ± 0%      1.00 ± 0%    ~     (all equal)
Wire/Marshal/google_message1_proto3-4        1.00 ± 0%      1.00 ± 0%    ~     (all equal)
Wire/Marshal/google_message2-4               1.00 ± 0%      1.00 ± 0%    ~     (all equal)
Wire/Size/google_message1_proto2-4           0.00           0.00         ~     (all equal)
Wire/Size/google_message1_proto3-4           0.00           0.00         ~     (all equal)
Wire/Size/google_message2-4                  0.00           0.00         ~     (all equal)

Change-Id: Ibf3263ad0f293326695c22020a92a6b938ef4f65
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185697
Reviewed-by: Damien Neil <dneil@google.com>
2019-07-12 19:31:58 +00:00
Joe Tsai
82760ceffa internal/impl: add MessageState to every generated message
We define MessageState, which is essentially an atomically set *MessageInfo.
By nesting this as the first field in every generated message, we can
implement the reflective methods on a *MessageState when obtained by
unsafe casting a concrete message pointer as a *MessageState.
The MessageInfo held by MessageState provides additional Go type information
to interpret the memory that comes after the contents of the MessageState.

Since we are nesting a MessageState in every message,
the memory use of every message instance grows by 8B.

On average, the body of ProtoReflect grows from 133B to 202B (+50%).
However, this is offset by XXX_Methods, which is 108B and
will be removed in a future CL. Taking into account the eventual removal
of XXX_Methods, this is a net reduction of 25%.

name          old time/op    new time/op    delta
Name/Value-4    70.3ns ± 2%    17.5ns ± 6%   -75.08%  (p=0.000 n=10+10)
Name/Nil-4      70.6ns ± 3%    33.4ns ± 2%   -52.66%  (p=0.000 n=10+10)

name          old alloc/op   new alloc/op   delta
Name/Value-4     16.0B ± 0%      0.0B       -100.00%  (p=0.000 n=10+10)
Name/Nil-4       16.0B ± 0%      0.0B       -100.00%  (p=0.000 n=10+10)

name          old allocs/op  new allocs/op  delta
Name/Value-4      1.00 ± 0%      0.00       -100.00%  (p=0.000 n=10+10)
Name/Nil-4        1.00 ± 0%      0.00       -100.00%  (p=0.000 n=10+10)

Change-Id: I92bd58dc681c57c92612fd5ba7fc066aea34e95a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185460
Reviewed-by: Damien Neil <dneil@google.com>
2019-07-10 19:44:24 +00:00
Joe Tsai
c0e4bb2054 cmd/protoc-gen-go: unexport implementation-specific XXX fields
We modify protoc-gen-go to stop generating exported XXX fields.
The unsafe implementation is unaffected by this change since unsafe
can access fields regardless of visibility. However, for the purego
implementation, we need to respect Go visibility rules as enforced
by the reflect package.

We work around this by generating a exporter function that given
a reference to the message and the field to export, returns a reference
to the unexported field value. This exporter function is protected by
a constant such that it is not linked into the final binary in non-purego
build environment.

Updates golang/protobuf#276

Change-Id: Idf5c1f158973fa1c61187ff41440acb21c5dac94
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185141
Reviewed-by: Damien Neil <dneil@google.com>
2019-07-08 20:45:09 +00:00
Joe Tsai
a804450cf6 all: fix stale comments
Change-Id: I4bfef75bc74c8d876a3926635bea12bbbaf4993e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185238
Reviewed-by: Damien Neil <dneil@google.com>
2019-07-08 18:33:38 +00:00
Joe Tsai
dfe160316f internal/fileinit: remove package
This package is no longer used.

Change-Id: Ib6d50490b07d6c5005f9e7e6c4b9cdc6ce6722fb
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/183101
Reviewed-by: Damien Neil <dneil@google.com>
2019-06-20 19:35:23 +00:00
Joe Tsai
d888139e7b internal/filedesc, internal/filetype: initial commit
The internal/fileinit package is split apart into two packages:
* internal/filedesc constructs descriptors from the raw proto.
It is very similar to the previous internal/fileinit package.
* internal/filetype wraps descriptors with Go type information

Overview:
* The internal/fileinit package will be deleted in a future CL.
It is kept around since the v1 repo currently depends on it.
* The internal/prototype package is deleted. All former usages of it
are now using internal/filedesc instead. Most significantly,
the reflect/protodesc package was almost entirely re-written.
* The internal/impl package drops support for messages that do not
have a Descriptor method (pre-2016). This removes a significant amount
of technical debt.
filedesc.Builder to parse raw descriptors.
* The internal/encoding/defval package now handles enum values by name.

Change-Id: I3957bcc8588a70470fd6c7de1122216b80615ab7
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/182360
Reviewed-by: Damien Neil <dneil@google.com>
2019-06-20 02:06:11 +00:00
Joe Tsai
378c1329de reflect/protoreflect: add alternative message reflection API
Added API:
	Message.Len
	Message.Range
	Message.Has
	Message.Clear
	Message.Get
	Message.Set
	Message.Mutable
	Message.NewMessage
	Message.WhichOneof
	Message.GetUnknown
	Message.SetUnknown

Deprecated API (to be removed in subsequent CL):
	Message.KnownFields
	Message.UnknownFields

The primary difference with the new API is that the top-level
Message methods are keyed by FieldDescriptor rather than FieldNumber
with the following semantics:
* For known fields, the FieldDescriptor must exactly match the
field descriptor known by the message.
* For extension fields, the FieldDescriptor must implement ExtensionType,
where ContainingMessage.FullName matches the message name, and
the field number is within the message's extension range.
When setting an extension field, it automatically stores
the extension type information.
* Extension fields are always considered nullable,
implying that repeated extension fields are nullable.
That is, you can distinguish between a unpopulated list and an empty list.
* Message.Get always returns a valid Value even if unpopulated.
The behavior is already well-defined for scalars, but for unpopulated
composite types, it now returns an empty read-only version of it.

Change-Id: Ia120630b4db221aeaaf743d0f64160e1a61a0f61
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/175458
Reviewed-by: Damien Neil <dneil@google.com>
2019-06-17 17:33:24 +00:00
Joe Tsai
89d49632e5 internal/impl: abstract away ExtensionDescV1 as the underlying descriptor
Add ExtensionField.{SetType,GetType} to hide the fact that the underlying
descriptor is actually an ExtensionDescV1.

Change-Id: I1d0595484ced0a88d2df0852a732fdf0fe9aa232
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/180538
Reviewed-by: Damien Neil <dneil@google.com>
2019-06-05 19:53:14 +00:00
Joe Tsai
60b368fb35 runtime/protolegacy: remove package
The legacy logic has now been merged into impl.

Change-Id: I999dfa2b7f49cbd98614a54d8ec70e611a3d033a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/178544
Reviewed-by: Damien Neil <dneil@google.com>
2019-05-23 18:01:25 +00:00
Joe Tsai
58b42d8892 cmd/protoc-gen-go, runtime/protoimpl: enforce minimum and maximum versions
Generate the needed infrastructure to ensure that we can statically
enforce minimum and maximum versions. This enables us to have a policy
when we release v2 where it fails to build for:
* new generated code with really old runtimes
* new runtimes with really old generated code

Change-Id: Ib699ad62c06dff8f9285806394a741c18db00288
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/178546
Reviewed-by: Damien Neil <dneil@google.com>
2019-05-23 17:53:52 +00:00
Joe Tsai
66500913f0 runtime/protoiface: update package documentation
Contrary to the WARNING, this package should be stable.
However, it still should not be imported by the end-user.

Change-Id: I05b4b9ebb1e0d28ab626f8c51fbe827b5acf237e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/178545
Reviewed-by: Damien Neil <dneil@google.com>
2019-05-23 17:49:17 +00:00
Joe Tsai
21ade498bd internal/impl: move legacy files into impl
The internal/legacy package was originally separated out from internal/impl
to avoid a cyclic dependency on descriptor proto. However, the dependency
that legacy has on descriptor has long been dropped such that we can
now merge the two packages together again.

All legacy related logic are in a file with a legacy prefix.

Change-Id: I2424fc0f50721696ad06fa7cebb9bdd0babea13c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/178542
Reviewed-by: Damien Neil <dneil@google.com>
2019-05-22 19:40:32 +00:00
Joe Tsai
1c28304cc5 proto, encoding/protojson, encoding/prototext: use Resolver interface
Instead of accepting a concrete protoregistry.Types type,
accept an interface that provides the necessary functionality
to perform the serialization.

The advantages of this approach:
* There is no need for complex logic to allow a Parent or custom
Resolver on the protoregistry.Types type.
* Users can pass their own custom resolver implementations directly
to the serialization functions.
* This is a more principled approach to plumbing custom resolvers
than the previous approach of overloading behavior on the concrete
Types type.

The disadvantages of this approach:
* A pointer to a concrete type is 8B, while an interface is 16B.
However, the expansion of the {Marshal,Unmarshal}Options structs
should be a concern solved separately from how to plumb custom resolvers.
* The resolver interfaces as defined today may be insufficient to
provide functionality needed in the future if protobuf expands its
feature set. For example, let's suppose the Any message permits
directly representing a enum by name. This would require the ability
to lookup an enum by name. To support that hypothetical need,
we can document that the serializers type-assert the provided Resolver
to a EnumTypeResolver and use that if possible. There is some loss
of type safety with this approach, but provides a clear path forward.

Change-Id: I81ca80e59335d36be6b43d57ec8e17abfdfa3bad
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/177044
Reviewed-by: Damien Neil <dneil@google.com>
2019-05-22 19:38:45 +00:00
Joe Tsai
4fe9663f4c internal/impl: rename MessageType as MessageInfo
The name MessageType is easily confused with protoreflect.MessageType.
Rename it as MessageInfo, which follows the pattern set by v1,
where the equivalent data structure is called InternalMessageInfo.

Change-Id: I535956e1f7c6e9b07e9585e889d5e93388d0d2ce
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/178478
Reviewed-by: Damien Neil <dneil@google.com>
2019-05-22 14:54:35 +00:00
Damien Neil
e89e6244e0 all: change module to google.golang.org/protobuf
Temporarily remove go.mod, since we can't generate an accurate one until
the corresponding v1 change is submitted.

Change-Id: I1e1ad97f2b455e33f61ffaeb8676289795e47e72
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/177000
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2019-05-14 17:28:29 +00:00
Joe Tsai
0fc49f8225 reflect/protoreflect: add Descriptor specific methods
Added methods:
	Enum.Descriptor
	Message.Descriptor
	EnumType.Descriptor
	MessageType.Descriptor
	ExtensionType.Descriptor
	Message.New

All functionality is switched over to use those methods instead of
implicitly relying on the fact that {Enum,Message}Type implicitly
implement the associated descriptor interface.

This CL does not yet remove {Enum,Message}.Type or prevent
{Enum,Message,Extension}Type from implementating a descriptor.
That is a subsequent CL.

The Message.New method is also added to replace functionality
that will be lost when the Type methods are removed.

Change-Id: I7fefde1673bbd40bfdac489aca05cec9a6c98eb1
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/174918
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Herbie Ong <herbie@google.com>
2019-05-13 19:34:41 +00:00