Commit Graph

857 Commits

Author SHA1 Message Date
Joe Tsai
5f5c066b7f types/known: minor adjustments to error handling
Trivial adjustments to error handling to reduce differences between
the generated helpers and the legacy ptypes package.

Change-Id: Ib3022eb50d9a9c0906b7809fe7e8011ee9399b0a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/238009
Reviewed-by: Damien Neil <dneil@google.com>
2020-06-17 18:52:16 +00:00
Joe Tsai
a0d1c75183 types/known: handle nil in IsValid and CheckValid methods
Many usages of the legacy ptypes.Timestamp and ptypes.Duration rely
on the functions implicitly returning an error for a nil message.
In order to migrate previous usages of the ptypes constructors to
the new API, we modify CheckValid to preserve similar semantics.

There is some consistency with this change in that:
	(*M)(nil).IsValid() == (*M)(nil).ProtoReflect().IsValid()
where M is Duration, Timestamp, and FieldMask.

Change-Id: I05ae152511f9875ea034e78aefb7aab18d17a318
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/238007
Reviewed-by: Damien Neil <dneil@google.com>
2020-06-16 21:18:46 +00:00
Joe Tsai
44e4150b30 reflect/protoreflect: optimize Name.IsValid and FullName.IsValid
For simplicity, IsValid was implemented in terms of regular expressions,
which are useful for verifying a string according to a strict grammar,
but performs poorly. Implement the check in terms of hand-written code,
which provides a 20x improvement to validate the name "google.protobuf.Any".

name               old time/op  new time/op  delta
FullNameIsValid-8   683ns ± 2%    35ns ± 1%  -94.86%  (p=0.000 n=10+10)

Change-Id: I980403befca0b72cea22acd274064a46cb02644b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/238002
Reviewed-by: Damien Neil <dneil@google.com>
2020-06-16 19:13:20 +00:00
Joe Tsai
0084168af8 types/known/anypb: add New constructor
While the MarshalFrom method and function are  more flexible since
it permits the reuse of an existing Any, the most common use case
by far is to construct a new Any. Optimize for that use case
by providing a constructor that directly returns a new Any instance.
We do not provide a variant that accepts a MarshalOptions since
the more flexible MarshalFrom function can be used for that.

Change-Id: Iab0219229c7d3aaa7390a2340a4f248002995293
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/237997
Reviewed-by: Damien Neil <dneil@google.com>
2020-06-16 00:35:17 +00:00
Joe Tsai
bfc31022a5 types/known/anypb: gracefully handle nil sources
Add checks so that MarshalFrom and UnmarshalTo consistently matches
the behavior of the proto package with regard to nil messages.
In particular, using a nil message as the destination results in
a panic (which is already the current behavior), while using a
nil message as the source does not panic (it is usually treated as
an untyped empty message). Since an untyped message has no
sensible meaning in the context of Any, return an error.

Change-Id: I99e86c2cdfbd8be57cc039efd550458dc56aadbc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/237920
Reviewed-by: Damien Neil <dneil@google.com>
2020-06-15 18:37:58 +00:00
cybrcodr
beaa55256c encoding/protojson: add better validation to FieldMask serialization
For marshaling, apart from already existing check that each item in
paths field is reversible, also make sure that string is a valid
protobuf name.

For unmarshaling, make sure that each resulting item in paths field is
a valid protobuf name and input path item does not contain _. The latter
check satisfies the conformance test
Recommended.Proto3.JsonInput.FieldMaskInvalidCharacter.

Fixes golang/protobuf#1141.

Change-Id: Iffc278089b20e496b7216d5b8c966b21b70e782d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/236998
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2020-06-12 06:33:55 +00:00
Joe Tsai
596bfbf068 testing/protocmp: surface comment about unused Transform options
Change-Id: I83bdb0f3f937f745b5758c282e1bd58203a8c784
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/237221
Reviewed-by: Damien Neil <dneil@google.com>
2020-06-10 19:24:07 +00:00
Damien Neil
7e8b902d03 types/dynamicpb: add NewEnumType
Add support for creating dynamic EnumTypes.

Change-Id: Ic9f5b73630734848b29cc436f6c179549a8ea74a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/237219
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Reviewed-by: Joe Tsai <joetsai@google.com>
2020-06-10 19:22:58 +00:00
Joe Tsai
f49fd502d3 all: implement first-class WKT support
This CL introduces generation of specialized APIs directly into the
generated packages for certain well-known types. This follows the pattern
set forth by the other language implementations that have specialized
generated support for certain well-known types.

Overview of new API:
  package anypb
    func MarshalFrom(*Any, proto.Message, proto.MarshalOptions) error
    func UnmarshalTo(*Any, proto.Message, proto.UnmarshalOptions) error
    func UnmarshalNew(*Any, proto.UnmarshalOptions) (proto.Message, error)
    func (*Any) MessageIs(proto.Message) bool
    func (*Any) MessageName() protoreflect.FullName
    func (*Any) MarshalFrom(proto.Message) error
    func (*Any) UnmarshalTo(proto.Message) error
    func (*Any) UnmarshalNew() (proto.Message, error)

  package timestamppb
    func Now() *Timestamp
    func New(time.Time) *Timestamp
    func (*Timestamp) AsTime() time.Time
    func (*Timestamp) IsValid() bool
    func (*Timestamp) CheckValid() error

  package durationpb
    func New(time.Duration) *Duration
    func (*Duration) AsDuration() time.Duration
    func (*Duration) IsValid() bool
    func (*Duration) CheckValid() error

  package structpb
    func NewStruct(map[string]interface{}) (*Struct, error)
    func (*Struct) AsMap() map[string]interface{}
    func (*Struct) MarshalJSON() ([]byte, error)
    func (*Struct) UnmarshalJSON(b []byte) error

    func NewList([]interface{}) (*ListValue, error)
    func (*ListValue) AsSlice() []interface{}
    func (*ListValue) MarshalJSON() ([]byte, error)
    func (*ListValue) UnmarshalJSON(b []byte) error

    func NewValue(interface{}) (*Value, error)
    func NewNullValue() *Value
    func NewBoolValue(bool) *Value
    func NewNumberValue(float64) *Value
    func NewStringValue(string) *Value
    func NewStructValue(*Struct) *Value
    func NewListValue(*ListValue) *Value
    func (*Value) AsInterface() interface{}
    func (*Value) MarshalJSON() ([]byte, error)
    func (*Value) UnmarshalJSON(b []byte) error

  package fieldmaskpb
    func New(proto.Message, ...string) (*FieldMask, error)
    func Union(*FieldMask, *FieldMask, ...*FieldMask) *FieldMask
    func Intersect(*FieldMask, *FieldMask, ...*FieldMask) *FieldMask
    func (*FieldMask) IsValid(proto.Message) bool
    func (*FieldMask) Append(proto.Message, ...string) error
    func (*FieldMask) Normalize()

  package wrapperspb
    func Bool(bool) *BoolValue
    func Int32(int32) *Int32Value
    func Int64(int64) *Int64Value
    func UInt32(uint32) *UInt32Value
    func UInt64(uint64) *UInt64Value
    func Float(float32) *FloatValue
    func Double(float64) *DoubleValue
    func String(string) *StringValue
    func Bytes([]byte) *BytesValue

This functionality expands upon and supersedes the
older github.com/golang/protobuf/ptypes package,
which provided helpers for Any, Timestamp, and Duration.

Comparison with older ptypes package:

* ptypes.AnyMessageName is replaced by anypb.Any.MessageName.
  The former returned an error for malformed type URLs,
  while the latter simply returns an empty string.

* ptypes.Is is replaced by anypb.Any.MessageIs.

* ptypes.Empty has no direct replacement as it is equivalent to:
	mt, err := protoregistry.GlobalTypes.FindMessageByURL(any.GetTypeUrl())
	if err != nil {
		return nil, err
	}
	return mt.New().Interface(), nil
  Analysis of user code revealed that this function is seldom used.

* ptypes.MarshalAny is replaced by anypb.Any.MarshalFrom.
  The former creates a new Any message and returns it,
  while the latter is a method that modifies the receiver.

* ptypes.UnmarshalAny is replaced by anypb.Any.UnmarshalTo.

* ptypes.DynamicAny is loosely replaced by anypb.Any.UnmarshalNew.
  The DynamicAny type is a custom proto.Message that is special
  to ptypes.UnmarshalAny where it would allocate a new message
  and store it into the DynamicAny instance. The UnmarshalNew method
  accomplishes the equivalent functionality in a more direct fashion.

* ptypes.TimestampNow is replaced by timestamppb.Now.

* ptypes.TimestampProto is replaced by timestamppb.New.
  The former returned an error if the timestamp was outside the
  10000-year range recommended by timestamp.proto,
  while the latter always succeeded. To preserve the behavior of
  the former validation check, the replacement can additionally
  call the timestamppb.Timestamp.CheckValid method.

* ptypes.Timestamp is replaced by timestamppb.Timestamp.AsTime.
  The former returned an error if the timestamp was outside the
  10000-year range recommended by timestamp.proto,
  while the latter always succeeded. To preserve the behavior of
  the former validation check, the replacement can additionally
  call the timestamppb.Timestamp.CheckValid method.

* ptypes.TimestampString has no direct replacement as it is equivalent to:
	ts.AsTime().Format(time.RFC3339Nano)

* ptypes.DurationProto is replaced by durationpb.New.

* ptypes.Duration is replaced by durationpb.Duration.AsDuration.
  The former returned an error if the duration would overflow
  when converting to a time.Duration, while the latter uses
  saturation arithmetic (similiar to the time package itself).
  Underflow resulted in time.Duration(math.MinInt64), while
  overflow resulted in time.Duration(math.MaxInt64).
  To preserve the behavior of former validation checks,
  the replacement can call the durationpb.Duration.CheckValid method
  and check whether the duration is fixed to one of the overflow values.

Change-Id: Ia996b1037a1fcafced7c7e10e9408ef7fa22863a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/225298
Reviewed-by: Damien Neil <dneil@google.com>
2020-06-09 21:21:54 +00:00
Joe Tsai
69839c78c3 internal/genid: remove WhichFile
It seems safer to explicitly mention exactly which messages
have special handling, rather than special casing the .profile
that they live in. This is safer because there is no guarantee
that new messages won't be added to each of these files.

The protojson implementation is modified to no longer rely
on a isCustomType helper and instead return a marshal or unmarshal
function pointer that is non-nil if specialized serialization
exists for that message type.

Change-Id: I5e3551d66f5a4b9024e583b627c0292cb7da6803
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/235657
Reviewed-by: Herbie Ong <herbie@google.com>
2020-05-29 21:03:06 +00:00
Joe Tsai
b2f4e6269c internal/msgfmt: adjust handling of well-known types
Identify well-known types by message name instead of the .proto file
that they belong to in case more are added (however unlikely).

Change-Id: I77b8880c106960b0d9a1cb304e2990e1bcddeb39
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/235478
Reviewed-by: Herbie Ong <herbie@google.com>
2020-05-29 07:30:29 +00:00
Joe Tsai
e0b77db13b internal/genid: add new package for generated identifiers
The genid package unifies the genname, fieldnum, and detectknown
packages into a single package.

Whenever possible use the generated constants rather than
hard-coded literals. This makes it easier to search the entire
module for special logic that deal with well-known types.

Change-Id: I13beff1f4149444a0c0b9e607ebf759657f000f4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/235301
Reviewed-by: Herbie Ong <herbie@google.com>
2020-05-29 07:08:23 +00:00
Joe Tsai
74aae6a46c encoding/prototext: simplify decoder.unmarshalAny
A hasFields map that only ever contains 3 entries seems more
complex than necessary. It's simpler and more performant to
just track three discrete boolean variables for each of the cases.

Change-Id: I1ba20da130f6b560a57fe8c3a73968983e563b48
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/235477
Reviewed-by: Herbie Ong <herbie@google.com>
2020-05-27 23:07:30 +00:00
Joe Tsai
0295718ab2 all: upgrade to go@v1.13.11 and go@v1.14.3
Change-Id: I0d7ab08ef693de0199544c4bc94ebeedb87144f2
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/234938
Reviewed-by: Herbie Ong <herbie@google.com>
2020-05-27 22:53:52 +00:00
Joe Tsai
4f55bf8d21 all: start v1.24.0-devel
Change-Id: I266e227245948372355c9ca1276d328a0d031cb0
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/235300
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-26 22:26:00 +00:00
Joe Tsai
5c3dd7024a all: release v1.24.0
Change-Id: Ia64a609f82681decec4ee79f3659650045c4e6c1
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/235299
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-26 22:25:46 +00:00
Joe Tsai
27ea64ff2c reflect/protoreflect: adjust ValueOf panic message
A common mistake is to pass proto.Message to protoreflect.ValueOf
instead of a protoreflect.Message. Specialize for this case
and provide a panic message that is more helpful.

Change-Id: I5def43341aa9607182edde309dea2823e61c1fdb
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/234117
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-26 22:22:26 +00:00
Joe Tsai
0567a55b47 all: add weak dependency on google.golang.org/genproto module
This is step 4 of 6 in a multi-stage migration
to move the well-known types
from the google.golang.org/genproto module
to the google.golang.org/protobuf module.

The generated Go packages for field_mask.proto, api.proto,
type.proto, and source_context.proto are being moved over
to this module alongside all the other well-known types.

In order to move these types between two modules,
there needs to be a sequence of changes submitted in
decently rapid succession. It is impossible to atomically
make these changes, so a brief breakage is inevitable.
The steps are as follows:

Step 1: Submit a change to cloud.google.com/go/internal/gapicgen
to avoid generating the well-known types. Otherwise, the tool
will undo the changes made in step 3.
See https://code-review.googlesource.com/c/gocloud/+/56810

Step 2: Submit a change to google.golang.org/protobuf that
adds the generated well-known types being migrated to that module.
In order to prevent the situation where a user links in
too old a version of the genproto module such that
duplicate registration occurs for the well-known types,
the registry is specially modified to provide an error
message that instructs users to upgrade the genproto module.
See https://golang.org/cl/234937

Step 3: Submit a change to google.golang.org/genproto that
switches all generated well-known types to be aliases to the
ones declared in google.golang.org/protobuf from the previous step.
This will cause the genproto module to incur an dependency
on an unreleased version of the protobuf module.
See https://github.com/googleapis/go-genproto/pull/372

Step 4: Submit a change to google.golang.org/protobuf that
adds a weak module depdency on the genproto module at the
revision from the previous step.
See https://golang.org/cl/235297

Step 5: Release google.golang.org/protobuf@v1.24.0.
See https://golang.org/cl/235299

Step 6: Submit a change to google.golang.org/genproto that
updates the protobuf module dependency to v1.24.0.

Change-Id: I22d496e7496446becad742eab05b169abb735ae2
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/235297
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-26 22:06:12 +00:00
Joe Tsai
81db48ad09 all: move well-known types
This is step 2 of 6 in a multi-stage migration
to move the well-known types
from the google.golang.org/genproto module
to the google.golang.org/protobuf module.

The generated Go packages for field_mask.proto, api.proto,
type.proto, and source_context.proto are being moved over
to this module alongside all the other well-known types.

In order to move these types between two modules,
there needs to be a sequence of changes submitted in
decently rapid succession. It is impossible to atomically
make these changes, so a brief breakage is inevitable.
The steps are as follows:

Step 1: Submit a change to cloud.google.com/go/internal/gapicgen
to avoid generating the well-known types. Otherwise, the tool
will undo the changes made in step 3.
See https://code-review.googlesource.com/c/gocloud/+/56810

Step 2: Submit a change to google.golang.org/protobuf that
adds the generated well-known types being migrated to that module.
In order to prevent the situation where a user links in
too old a version of the genproto module such that
duplicate registration occurs for the well-known types,
the registry is specially modified to provide an error
message that instructs users to upgrade the genproto module.
See https://golang.org/cl/234937

Step 3: Submit a change to google.golang.org/genproto that
switches all generated well-known types to be aliases to the
ones declared in google.golang.org/protobuf from the previous step.
This will cause the genproto module to incur an dependency
on an unreleased version of the protobuf module.
See https://github.com/googleapis/go-genproto/pull/372

Step 4: Submit a change to google.golang.org/protobuf that
adds a weak module depdency on the genproto module at the
revision from the previous step.

Step 5: Release google.golang.org/protobuf@v1.24.0.

Step 6: Submit a change to google.golang.org/genproto that
updates the protobuf module dependency to v1.24.0.

Change-Id: I36a19049d2240b67a37dfad20e154505aee7c784
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/234937
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-26 19:51:55 +00:00
Joe Tsai
b10fcbc76b testing/protopack: fix format precision
64-bit floating point numbers require 16 hexadecimal digits, not 8.

Change-Id: I127b9b8e8c52de53f6fa58a58c4bf63f94196e1f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/234697
Reviewed-by: Herbie Ong <herbie@google.com>
2020-05-21 22:25:24 +00:00
Joe Tsai
118baf6390 all: funnel similar functionality through a single function
Some companies (e.g., Google) run a profiling service where they may
choose to special-case certain symbols in a binary to classify
commonly used libraries like protobufs.

This CL funnels similar functionality through a single function
so that they can be more easily identified. This is by no means a
firm statement that these identifiers will never change names,
but at least the code documents warnings to avoid changing the
name of certain identifiers.

This CL provides the following semi-stable symbol names:
	"google.golang.org/protobuf/proto".MarshalOptions.size
	"google.golang.org/protobuf/proto".MarshalOptions.marshal
	"google.golang.org/protobuf/proto".UnmarshalOptions.unmarshal
	"google.golang.org/protobuf/encoding/prototext".MarshalOptions.marshal
	"google.golang.org/protobuf/encoding/prototext".UnmarshalOptions.unmarshal
	"google.golang.org/protobuf/encoding/protojson".MarshalOptions.marshal
	"google.golang.org/protobuf/encoding/protojson".UnmarshalOptions.unmarshal

Merge and Clone are not part of the above set since there is a
possibility that MergeOptions will be added in the future.

We use an unexported method so that we have the freedom to change the
method however we want since profilers do not care about that.

Change-Id: Ia79af260d00125f48139420e1e18a86482bd1829
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/234079
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-15 18:21:49 +00:00
Joe Tsai
0fbe4ed572 all: start v1.23.0-devel
Change-Id: Ie984b488844e28b95b7565fac4df52fd378f1ebd
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/234078
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-14 20:12:40 +00:00
Joe Tsai
d165be301f all: release v1.23.0
Change-Id: Id8b69ecee5d7c5623b732bd14a8dde8da2c5084e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/234077
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-14 20:12:30 +00:00
Joe Tsai
2d80e9b3ab encoding/prototext: adjust handling of invalid UTF-8
The following changes are made:
* Permit invalid UTF-8 in proto2. This goes against specified behavior,
but matches functional behavior in wire marshaling (not just for Go,
but also in the other major language implementations as well).
* The Format function is specified as ignoring errors since its intended
purpose is to surface information to the human user even if it's not
exactly parsible back into a message. As such, add an unexported
allowInvalidUTF8 option that is specially used by Format.
* Add an EmitASCII option that forces the formatting of
strings and bytes to always be encoded as ASCII.
This ensures that the entire output is always ASCII as well.

Note that we do not replicate this behavior for protojson since:
* The JSON format fundamentally has a stricter and well-specified
grammar for exactly what is valid/invalid, while the text format
has not had a well-specified grammar for the longest time,
leading to all sorts of weird usages due to Hyrum's law.
* This is to ease migration from the legacy implementation,
which did permit invalid UTF-8 in proto2.
* The EmitASCII option relies on the ability to always escape
Unicode characters using ASCII escape sequences, but this is not
possible in JSON since the grammar only has an escape sequence defined
for Unicode characters \u0000 to \uffff, inclusive.
However, Unicode v12.0.0 defines characters up to \U0010FFFF,
which is beyond what the JSON grammar provides escape sequences for.

Change-Id: I2b524a904e9ec59f9ed5500e299613bc27c31a14
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/233077
Reviewed-by: Herbie Ong <herbie@google.com>
2020-05-13 05:25:02 +00:00
Joe Tsai
3034025d28 internal/impl: avoid inlining fixed coderFieldInfo array
Any attempt at guessing the size for a fixed coderFieldInfo array
will always get it wrong in some cases, either by under-estimating
or over-estimating the count. The former causes worse caching behavior,
while the latter causes memory waste.

As a middle ground, just pre-allocate a slice of the exact length.
Each element will have memory locality with each other, but not
be guaranteed to have memory locality with the parent coderMessageInfo.

name                            old time/op  new time/op  delta
EmptyMessage/Wire/Marshal-8     43.1ns ±11%  42.6ns ± 8%  -1.32%  (p=0.036 n=50+49)
EmptyMessage/Wire/Unmarshal-8   18.6ns ±10%  18.9ns ±12%    ~     (p=0.054 n=50+50)
EmptyMessage/Wire/Validate-8    15.0ns ± 9%  14.7ns ±10%  -2.44%  (p=0.002 n=50+45)
EmptyMessage/Clone-8             163ns ±20%   149ns ±19%  -8.58%  (p=0.000 n=48+53)
RepeatedInt32/Wire/Marshal-8    4.27µs ±12%  4.24µs ±13%    ~     (p=0.612 n=48+52)
RepeatedInt32/Wire/Unmarshal-8  3.47µs ±14%  3.50µs ±11%    ~     (p=0.217 n=50+53)
RepeatedInt32/Wire/Validate-8   2.12µs ±12%  2.09µs ± 9%    ~     (p=0.121 n=50+51)
RepeatedInt32/Clone-8           3.04µs ±18%  2.98µs ±36%    ~     (p=0.289 n=51+54)
Required/Wire/Marshal-8          281ns ±14%   276ns ±11%    ~     (p=0.059 n=48+55)
Required/Wire/Unmarshal-8        117ns ±14%   118ns ±11%    ~     (p=0.358 n=49+53)
Required/Wire/Validate-8        87.6ns ± 9%  88.0ns ±12%    ~     (p=0.373 n=48+53)
Required/Clone-8                 533ns ±12%   507ns ±15%  -4.71%  (p=0.000 n=49+54)

Change-Id: I4cf3134e424130bee728b7591127e5c80f07e2db
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/232937
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-13 03:30:14 +00:00
Oscar Söderlund
8525b20428 compiler/protogen: add (*GeneratedFile).Unskip
Simplifies some compiler plugins who want to avoid generating empty
files. With this API, generated files can be skipped by default, and
unskipped when actual code is generated.

Change-Id: I941c821646f5c4430a84e08bbad7e021434b1e71
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/232239
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2020-05-11 17:43:27 +00:00
Joe Tsai
1f5b6fe645 all: improve panic messages for better debugability
Change-Id: If3e505e715d5ce2c9a81249c868d26226a25b724
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/232339
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-05 21:10:54 +00:00
Joe Tsai
ce5d8318a0 compiler/protogen: avoid suggesting faulty go_package option
In some situations, the full Go package path cannot be determined,
in which case, we should avoid suggesting a go_package option
that is incorrect.

As a heuristic, check whether the proposed path contains at least
a dot before a slash to determine whether it is a full path or not,
which is a simple way to determine whether the first segment is a
top-level domain.

This avoids printing unhelpful warnings like:
	WARNING: Missing 'go_package' option in "foo/bar.proto", please specify:
		option go_package = ".;foo_package";
	A future release of protoc-gen-go will require this be specified.
	See https://developers.google.com/protocol-buffers/docs/reference/go-generated#package for more information.
and instead prints a warning like:
	WARNING: Missing 'go_package' option in "foo/bar.proto",
	please specify it with the full Go package path as
	a future release of protoc-gen-go will require this be specified.
	See https://developers.google.com/protocol-buffers/docs/reference/go-generated#package for more information.

We rely on the documentation on the developer website to provide
better guidance.

Change-Id: I38fc4c676d0314ba6d7ad8d5f390fb9e237f2bb1
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/232338
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-05 20:53:59 +00:00
Joe Tsai
0e19395744 all: start v1.22.0-devel
Change-Id: I7d70cbd4c9b692b5e4ffdfc576458c48d39f450d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/232118
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-04 19:17:23 +00:00
Joe Tsai
8b0d71ac93 all: release v1.22.0
Change-Id: Ia9d650390d6e0cc6f153834480748faee08fdaac
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/232117
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-04 19:17:18 +00:00
Joe Tsai
164b526074 all: update to protoc v3.12.0-rc1
This CL continues to hard-code the exact commit since the protoc release
uses inconsistent naming for its download URLs.

Change-Id: I7551c4b2f9b7b89c1c85169faffc03641f21bc11
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/232097
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-04 19:07:12 +00:00
Joe Tsai
90042a5531 reflect/protoreflect: fix typo in FieldDescriptor.HasOptionalKeyword
Change-Id: I9b1dac5d10bbac4618bda174ef04135b81c81d7e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/231777
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-04 18:59:39 +00:00
Joe Tsai
4d5be764fb internal/fuzz/wirefuzz: add test to verify initialization checks
The UnmarshalInitialized flag produced by Unmarshal and Validate are
filters such that must never have false positives (i.e., report a
partial message as initialized) otherwise it is incorrect.
It can tolerate some degree of false negatives (i.e., report an
initialized message as partial), but that leads to significant
performance degradation needing to do the full initialization check.
These should be the exception, not the norm.

Adjust the fuzzer to search for false-negative cases. For now, we only
require that the Unmarshal and Validate report initialized for any
"normalized" messages which we produce by marshaling intermediate
message again. This is to work around a known case where they cannot
determine initialization status if the wire data relies on protobuf's
merge functionality (where two or more partial messages merge
together to form an initialized message).

Change-Id: I6bb6c6594981ca08a92583bae77e5a2d44924af6
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/231577
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-01 18:43:38 +00:00
Joe Tsai
d0a499bc65 internal/impl: validate UTF-8 for proto3 optional strings
Change-Id: I090e7c5adac47818831c63d3d999cb7fea5ac696
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/231357
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-01 17:23:32 +00:00
Joe Tsai
3ebaa92e92 all: use v2 Message interface for weak fields
Cleanup the generated logic by having the implementation be backed
by protoimpl rather that directly generated.

Weak fields are a deprecated feature of protobufs and
have entirely be superceded by extensions.
Unfortunately, there are still some usages of it.

Change-Id: Ie1a4b7da253e2ccf5e56627775d9b2fb4090d59a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/229717
Reviewed-by: Damien Neil <dneil@google.com>
2020-04-29 22:34:29 +00:00
Joe Tsai
387873dd53 all: implement support for proto3 optional semantics
In the upcoming 3.12.x release of protoc, the proto3 language will be
amended to support true presence for scalars. This CL adds support
to both the generator and runtime to support these semantics.

Newly added public API:
	protogen.Plugin.SupportedFeatures
	protoreflect.FieldDescriptor.HasPresence
	protoreflect.FieldDescriptor.HasOptionalKeyword
	protoreflect.OneofDescriptor.IsSynthetic

Change-Id: I7c86bf66d0ae56642109beb5f2132184593747ad
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/230698
Reviewed-by: Damien Neil <dneil@google.com>
2020-04-29 20:02:24 +00:00
Joe Tsai
b57aae9def all: improve extension validation
Changes made:
* Ensure protoreflect.ExtensionType.IsValidInterface never panics,
especially if given a nil interface value.
* Have protoreflect.ExtensionType.IsValid{Interface,Value} only
perform type-checks. It does not do value checks (i.e., whether the
value itself is valid). Value validity is left to when an actual
protoreflect.Message.Set operation is performed.
* Add special-casing on proto.SetExtension to treat an invalid
message or list as functionally equivalent to Clear. This is to
be more consistent with the legacy SetExtension implementation
which never panicked when given such values.
* Add special-casing on proto.HasExtension to treat a mismatched
extension descriptor as simply not being present in the message.
This is also to be more consistent with the legacy HasExtension
implementation which did the same thing.

Change-Id: Idf0419abf27b9f85d9b92bd2ff8088e25b7990cc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/229558
Reviewed-by: Damien Neil <dneil@google.com>
2020-04-23 06:01:13 +00:00
Damien Neil
a5526f0129 internal/impl: fix off-by-one error in message initialization
This particular off-by-one can never happen in practice. We will crash
if a field has a number one greater than the maximum field number in the
dense array, but if field number N is in the array we will always put
N+1 in it as well.

Fix the off-by-one anyway.

Change-Id: I8c1304f2fc0d7b91036bde3f7ddb7115c21781ca
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/229278
Reviewed-by: Joe Tsai <joetsai@google.com>
2020-04-21 21:23:04 +00:00
Damien Neil
98f56d1bd5 internal/impl: inline coderInfoFields for better cache locality
Microbenchmarks are inconclusive/noisy, but shows a small but noticeable
improvement on internal benchmarks.

Change-Id: Ic46c6aac8a42c4dc749c4f3583d8c8c95e9548b7
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/229277
Reviewed-by: Joe Tsai <joetsai@google.com>
2020-04-21 21:03:31 +00:00
Joe Tsai
33f8c03eac proto: add RangeExtensions and adjust HasExtension and GetExtension
Two changes:
* Add RangeExtensions as a more suitable replacement for legacy
proto.ClearExtensions and proto.ExtensionDescs functions.
* Make HasExtension and GetExtension treat nil message interface
as an empty message to more consistently match legacy behavior.

Change-Id: I8eb1887a33d0737f2f80a2b80358cc296087ba3b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/229157
Reviewed-by: Damien Neil <dneil@google.com>
2020-04-21 00:46:21 +00:00
Joe Tsai
d824637fdf reflect/protoreflect: mention bidirectional conversion of protodesc
Expand the mention of the protodesc package to indicate that it can
construct protoreflect.Descriptors from the descriptor messages, but
also convert a protoreflect.Descriptor back to the message form.

Change-Id: I9f3d75c6050fff603655ee0a4e8e29d0a2c0cf84
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/221611
Reviewed-by: Damien Neil <dneil@google.com>
2020-04-21 00:35:05 +00:00
Joe Tsai
7650ca0905 reflect/protoreflect: cleanup wording on Message.NewField
Change-Id: Id2200e335fb5caf8e7e1574524081fed36aab268
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/227817
Reviewed-by: Damien Neil <dneil@google.com>
2020-04-21 00:34:41 +00:00
Joe Tsai
a732b3c77a proto: never return nil []byte from Marshal when successful
It is a sufficiently common pattern to do something like the following:
	m.Proto2BytesField, ... = proto.Marshal(m2)

where the user is relying on Marshal to never return a nil byte slice,
otherwise it subtly changes the semantics of how the generated API
handles whether a proto2 "optional bytes" fields is populated or not.

Change-Id: Ie7508dbcdcc5f3295885609a91907c6eb4f04c1e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/228838
Reviewed-by: Damien Neil <dneil@google.com>
2020-04-20 23:39:16 +00:00
Joe Tsai
8cfc14f022 all: consistently treat nil message interface as an empty read-only message
To assist users in migrating from github.com/golang/protobuf
to google.golang.org/protobuf, make it such that functiionality like
proto.Marshal doesn't panic on nil interfaces.

Similar to how the new implementation treats a typed nil message
as an empty message, we treat a nil interface as being equivalent
to an "untyped" empty message.

Change-Id: Ic037f386f855b122f732b34d370e524b7c0d76f1
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/228837
Reviewed-by: Damien Neil <dneil@google.com>
2020-04-20 23:12:56 +00:00
Damien Neil
d8bc21f7e1 internal/fuzz: update to use native go-fuzz
Fixes golang/protobuf#1084.

Change-Id: I2c71e9f58d09345d13f461ec78ee4e39c3a7f06d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/228277
Reviewed-by: Joe Tsai <joetsai@google.com>
2020-04-15 01:18:53 +00:00
Joe Tsai
5e85542f16 all: upgrade weak dependency to github.com/golang/protobuf@v1.4.0
Change-Id: I7e04b4fe0ef174dddc8c4938f8d6f2109581d883
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/228179
Reviewed-by: Damien Neil <dneil@google.com>
2020-04-13 20:03:07 +00:00
Joe Tsai
ba9d210a5c all: start v1.21.0-devel
Change-Id: Ib9df04de7a3f46f3a3b512e92cc7694ec0f4f927
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/228178
Reviewed-by: Damien Neil <dneil@google.com>
2020-04-13 18:57:37 +00:00
Joe Tsai
3b9eee1291 all: release v1.21.0
Change-Id: I6a91f3efd45ab0a446b220fddc8bd03aec93b388
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/228177
Reviewed-by: Damien Neil <dneil@google.com>
2020-04-13 18:57:02 +00:00
Joe Tsai
7dfcffe5a7 internal/impl: handle extremely old messages
At some point in time, protoc-gen-go actually emitted an XXX_OneofFuncs
with a different signature. Adjust the logic for handling XXX_OneofFuncs
to not assume that the return arguments are in a specific order.

Change-Id: Idd9c09231c4129c655d4a635bb1ae094896a1ff4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/226980
Reviewed-by: Damien Neil <dneil@google.com>
2020-04-02 21:12:23 +00:00
Joe Tsai
a75c9146b7 internal/impl: minor refactoring
Change-Id: I1a5f4ca31fec72a39ba6690af06f4ae8576408cc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/226897
Reviewed-by: Damien Neil <dneil@google.com>
2020-04-02 00:23:08 +00:00