638 Commits

Author SHA1 Message Date
Joe Tsai
b78321453d reflect/protoregistry: centralize MessageSet extension resolution logic
Centralize the MessageSet extension resolution logic in the registry.
This avoids needless replication of this exact logic in multiple places
(for JSON and text) and elsewhere.

Change-Id: I70bfea899e295e8c589f418965bf0dd099f93628
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/240077
Reviewed-by: Herbie Ong <herbie@google.com>
2020-07-01 17:56:02 +00:00
Joe Tsai
1726b83dc4 internal/impl: introduce instability to protoreflect.Message.Range order
Change-Id: I6f53aaf4071b4cbf596f3438a83cf3e5c93f1712
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/240017
Reviewed-by: Herbie Ong <herbie@google.com>
2020-07-01 17:34:30 +00:00
Joe Tsai
a9513ebdb8 all: start v1.25.0-devel
Change-Id: Iab5d460374c67a995a5d3058cfbc55e6aaa21b38
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/239378
Reviewed-by: Damien Neil <dneil@google.com>
2020-06-22 20:32:22 +00:00
Joe Tsai
3f7a61f89b all: release v1.25.0
Change-Id: I1f2ad6da83ea488c814ad10d6890d188ab6c2a27
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/239377
Reviewed-by: Damien Neil <dneil@google.com>
2020-06-22 20:32:15 +00:00
Joe Tsai
5a96da2901 cmd/protoc-gen-go: generate package documentation for well-known types
Change-Id: Ia079805e24d3490301355e09693255b157cabaed
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/239167
Reviewed-by: Damien Neil <dneil@google.com>
2020-06-22 19:32:21 +00:00
Joe Tsai
c9c87ce6a4 internal/filedesc: use genid constants for map entry fields
Change-Id: If551fc4f3e8da6325af7a0125f0e96fb014d3636
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/238998
Reviewed-by: Herbie Ong <herbie@google.com>
2020-06-19 21:29:54 +00:00
Joe Tsai
8de6675fa8 test.bash: add staticcheck to our integration tests
This adds the use of the staticcheck tool to check our code.
The use of staticcheck would have caught a bug in CL/211737.
Also, we adjust one minor code patterns to avoid triggering
one of the staticcheck checks.

This CL also modifies the integration script to check SHA256
checksums for archives downloaded from a third-party source.

Change-Id: Ieb69ae79464de62970448195c77cdd88e21fc48f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/212220
Reviewed-by: Damien Neil <dneil@google.com>
2020-06-17 19:14:03 +00:00
cybrcodr
beaa55256c encoding/protojson: add better validation to FieldMask serialization
For marshaling, apart from already existing check that each item in
paths field is reversible, also make sure that string is a valid
protobuf name.

For unmarshaling, make sure that each resulting item in paths field is
a valid protobuf name and input path item does not contain _. The latter
check satisfies the conformance test
Recommended.Proto3.JsonInput.FieldMaskInvalidCharacter.

Fixes golang/protobuf#1141.

Change-Id: Iffc278089b20e496b7216d5b8c966b21b70e782d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/236998
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2020-06-12 06:33:55 +00:00
Joe Tsai
f49fd502d3 all: implement first-class WKT support
This CL introduces generation of specialized APIs directly into the
generated packages for certain well-known types. This follows the pattern
set forth by the other language implementations that have specialized
generated support for certain well-known types.

Overview of new API:
  package anypb
    func MarshalFrom(*Any, proto.Message, proto.MarshalOptions) error
    func UnmarshalTo(*Any, proto.Message, proto.UnmarshalOptions) error
    func UnmarshalNew(*Any, proto.UnmarshalOptions) (proto.Message, error)
    func (*Any) MessageIs(proto.Message) bool
    func (*Any) MessageName() protoreflect.FullName
    func (*Any) MarshalFrom(proto.Message) error
    func (*Any) UnmarshalTo(proto.Message) error
    func (*Any) UnmarshalNew() (proto.Message, error)

  package timestamppb
    func Now() *Timestamp
    func New(time.Time) *Timestamp
    func (*Timestamp) AsTime() time.Time
    func (*Timestamp) IsValid() bool
    func (*Timestamp) CheckValid() error

  package durationpb
    func New(time.Duration) *Duration
    func (*Duration) AsDuration() time.Duration
    func (*Duration) IsValid() bool
    func (*Duration) CheckValid() error

  package structpb
    func NewStruct(map[string]interface{}) (*Struct, error)
    func (*Struct) AsMap() map[string]interface{}
    func (*Struct) MarshalJSON() ([]byte, error)
    func (*Struct) UnmarshalJSON(b []byte) error

    func NewList([]interface{}) (*ListValue, error)
    func (*ListValue) AsSlice() []interface{}
    func (*ListValue) MarshalJSON() ([]byte, error)
    func (*ListValue) UnmarshalJSON(b []byte) error

    func NewValue(interface{}) (*Value, error)
    func NewNullValue() *Value
    func NewBoolValue(bool) *Value
    func NewNumberValue(float64) *Value
    func NewStringValue(string) *Value
    func NewStructValue(*Struct) *Value
    func NewListValue(*ListValue) *Value
    func (*Value) AsInterface() interface{}
    func (*Value) MarshalJSON() ([]byte, error)
    func (*Value) UnmarshalJSON(b []byte) error

  package fieldmaskpb
    func New(proto.Message, ...string) (*FieldMask, error)
    func Union(*FieldMask, *FieldMask, ...*FieldMask) *FieldMask
    func Intersect(*FieldMask, *FieldMask, ...*FieldMask) *FieldMask
    func (*FieldMask) IsValid(proto.Message) bool
    func (*FieldMask) Append(proto.Message, ...string) error
    func (*FieldMask) Normalize()

  package wrapperspb
    func Bool(bool) *BoolValue
    func Int32(int32) *Int32Value
    func Int64(int64) *Int64Value
    func UInt32(uint32) *UInt32Value
    func UInt64(uint64) *UInt64Value
    func Float(float32) *FloatValue
    func Double(float64) *DoubleValue
    func String(string) *StringValue
    func Bytes([]byte) *BytesValue

This functionality expands upon and supersedes the
older github.com/golang/protobuf/ptypes package,
which provided helpers for Any, Timestamp, and Duration.

Comparison with older ptypes package:

* ptypes.AnyMessageName is replaced by anypb.Any.MessageName.
  The former returned an error for malformed type URLs,
  while the latter simply returns an empty string.

* ptypes.Is is replaced by anypb.Any.MessageIs.

* ptypes.Empty has no direct replacement as it is equivalent to:
	mt, err := protoregistry.GlobalTypes.FindMessageByURL(any.GetTypeUrl())
	if err != nil {
		return nil, err
	}
	return mt.New().Interface(), nil
  Analysis of user code revealed that this function is seldom used.

* ptypes.MarshalAny is replaced by anypb.Any.MarshalFrom.
  The former creates a new Any message and returns it,
  while the latter is a method that modifies the receiver.

* ptypes.UnmarshalAny is replaced by anypb.Any.UnmarshalTo.

* ptypes.DynamicAny is loosely replaced by anypb.Any.UnmarshalNew.
  The DynamicAny type is a custom proto.Message that is special
  to ptypes.UnmarshalAny where it would allocate a new message
  and store it into the DynamicAny instance. The UnmarshalNew method
  accomplishes the equivalent functionality in a more direct fashion.

* ptypes.TimestampNow is replaced by timestamppb.Now.

* ptypes.TimestampProto is replaced by timestamppb.New.
  The former returned an error if the timestamp was outside the
  10000-year range recommended by timestamp.proto,
  while the latter always succeeded. To preserve the behavior of
  the former validation check, the replacement can additionally
  call the timestamppb.Timestamp.CheckValid method.

* ptypes.Timestamp is replaced by timestamppb.Timestamp.AsTime.
  The former returned an error if the timestamp was outside the
  10000-year range recommended by timestamp.proto,
  while the latter always succeeded. To preserve the behavior of
  the former validation check, the replacement can additionally
  call the timestamppb.Timestamp.CheckValid method.

* ptypes.TimestampString has no direct replacement as it is equivalent to:
	ts.AsTime().Format(time.RFC3339Nano)

* ptypes.DurationProto is replaced by durationpb.New.

* ptypes.Duration is replaced by durationpb.Duration.AsDuration.
  The former returned an error if the duration would overflow
  when converting to a time.Duration, while the latter uses
  saturation arithmetic (similiar to the time package itself).
  Underflow resulted in time.Duration(math.MinInt64), while
  overflow resulted in time.Duration(math.MaxInt64).
  To preserve the behavior of former validation checks,
  the replacement can call the durationpb.Duration.CheckValid method
  and check whether the duration is fixed to one of the overflow values.

Change-Id: Ia996b1037a1fcafced7c7e10e9408ef7fa22863a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/225298
Reviewed-by: Damien Neil <dneil@google.com>
2020-06-09 21:21:54 +00:00
Joe Tsai
69839c78c3 internal/genid: remove WhichFile
It seems safer to explicitly mention exactly which messages
have special handling, rather than special casing the .profile
that they live in. This is safer because there is no guarantee
that new messages won't be added to each of these files.

The protojson implementation is modified to no longer rely
on a isCustomType helper and instead return a marshal or unmarshal
function pointer that is non-nil if specialized serialization
exists for that message type.

Change-Id: I5e3551d66f5a4b9024e583b627c0292cb7da6803
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/235657
Reviewed-by: Herbie Ong <herbie@google.com>
2020-05-29 21:03:06 +00:00
Joe Tsai
b2f4e6269c internal/msgfmt: adjust handling of well-known types
Identify well-known types by message name instead of the .proto file
that they belong to in case more are added (however unlikely).

Change-Id: I77b8880c106960b0d9a1cb304e2990e1bcddeb39
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/235478
Reviewed-by: Herbie Ong <herbie@google.com>
2020-05-29 07:30:29 +00:00
Joe Tsai
e0b77db13b internal/genid: add new package for generated identifiers
The genid package unifies the genname, fieldnum, and detectknown
packages into a single package.

Whenever possible use the generated constants rather than
hard-coded literals. This makes it easier to search the entire
module for special logic that deal with well-known types.

Change-Id: I13beff1f4149444a0c0b9e607ebf759657f000f4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/235301
Reviewed-by: Herbie Ong <herbie@google.com>
2020-05-29 07:08:23 +00:00
Joe Tsai
4f55bf8d21 all: start v1.24.0-devel
Change-Id: I266e227245948372355c9ca1276d328a0d031cb0
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/235300
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-26 22:26:00 +00:00
Joe Tsai
5c3dd7024a all: release v1.24.0
Change-Id: Ia64a609f82681decec4ee79f3659650045c4e6c1
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/235299
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-26 22:25:46 +00:00
Joe Tsai
0567a55b47 all: add weak dependency on google.golang.org/genproto module
This is step 4 of 6 in a multi-stage migration
to move the well-known types
from the google.golang.org/genproto module
to the google.golang.org/protobuf module.

The generated Go packages for field_mask.proto, api.proto,
type.proto, and source_context.proto are being moved over
to this module alongside all the other well-known types.

In order to move these types between two modules,
there needs to be a sequence of changes submitted in
decently rapid succession. It is impossible to atomically
make these changes, so a brief breakage is inevitable.
The steps are as follows:

Step 1: Submit a change to cloud.google.com/go/internal/gapicgen
to avoid generating the well-known types. Otherwise, the tool
will undo the changes made in step 3.
See https://code-review.googlesource.com/c/gocloud/+/56810

Step 2: Submit a change to google.golang.org/protobuf that
adds the generated well-known types being migrated to that module.
In order to prevent the situation where a user links in
too old a version of the genproto module such that
duplicate registration occurs for the well-known types,
the registry is specially modified to provide an error
message that instructs users to upgrade the genproto module.
See https://golang.org/cl/234937

Step 3: Submit a change to google.golang.org/genproto that
switches all generated well-known types to be aliases to the
ones declared in google.golang.org/protobuf from the previous step.
This will cause the genproto module to incur an dependency
on an unreleased version of the protobuf module.
See https://github.com/googleapis/go-genproto/pull/372

Step 4: Submit a change to google.golang.org/protobuf that
adds a weak module depdency on the genproto module at the
revision from the previous step.
See https://golang.org/cl/235297

Step 5: Release google.golang.org/protobuf@v1.24.0.
See https://golang.org/cl/235299

Step 6: Submit a change to google.golang.org/genproto that
updates the protobuf module dependency to v1.24.0.

Change-Id: I22d496e7496446becad742eab05b169abb735ae2
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/235297
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-26 22:06:12 +00:00
Joe Tsai
81db48ad09 all: move well-known types
This is step 2 of 6 in a multi-stage migration
to move the well-known types
from the google.golang.org/genproto module
to the google.golang.org/protobuf module.

The generated Go packages for field_mask.proto, api.proto,
type.proto, and source_context.proto are being moved over
to this module alongside all the other well-known types.

In order to move these types between two modules,
there needs to be a sequence of changes submitted in
decently rapid succession. It is impossible to atomically
make these changes, so a brief breakage is inevitable.
The steps are as follows:

Step 1: Submit a change to cloud.google.com/go/internal/gapicgen
to avoid generating the well-known types. Otherwise, the tool
will undo the changes made in step 3.
See https://code-review.googlesource.com/c/gocloud/+/56810

Step 2: Submit a change to google.golang.org/protobuf that
adds the generated well-known types being migrated to that module.
In order to prevent the situation where a user links in
too old a version of the genproto module such that
duplicate registration occurs for the well-known types,
the registry is specially modified to provide an error
message that instructs users to upgrade the genproto module.
See https://golang.org/cl/234937

Step 3: Submit a change to google.golang.org/genproto that
switches all generated well-known types to be aliases to the
ones declared in google.golang.org/protobuf from the previous step.
This will cause the genproto module to incur an dependency
on an unreleased version of the protobuf module.
See https://github.com/googleapis/go-genproto/pull/372

Step 4: Submit a change to google.golang.org/protobuf that
adds a weak module depdency on the genproto module at the
revision from the previous step.

Step 5: Release google.golang.org/protobuf@v1.24.0.

Step 6: Submit a change to google.golang.org/genproto that
updates the protobuf module dependency to v1.24.0.

Change-Id: I36a19049d2240b67a37dfad20e154505aee7c784
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/234937
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-26 19:51:55 +00:00
Joe Tsai
118baf6390 all: funnel similar functionality through a single function
Some companies (e.g., Google) run a profiling service where they may
choose to special-case certain symbols in a binary to classify
commonly used libraries like protobufs.

This CL funnels similar functionality through a single function
so that they can be more easily identified. This is by no means a
firm statement that these identifiers will never change names,
but at least the code documents warnings to avoid changing the
name of certain identifiers.

This CL provides the following semi-stable symbol names:
	"google.golang.org/protobuf/proto".MarshalOptions.size
	"google.golang.org/protobuf/proto".MarshalOptions.marshal
	"google.golang.org/protobuf/proto".UnmarshalOptions.unmarshal
	"google.golang.org/protobuf/encoding/prototext".MarshalOptions.marshal
	"google.golang.org/protobuf/encoding/prototext".UnmarshalOptions.unmarshal
	"google.golang.org/protobuf/encoding/protojson".MarshalOptions.marshal
	"google.golang.org/protobuf/encoding/protojson".UnmarshalOptions.unmarshal

Merge and Clone are not part of the above set since there is a
possibility that MergeOptions will be added in the future.

We use an unexported method so that we have the freedom to change the
method however we want since profilers do not care about that.

Change-Id: Ia79af260d00125f48139420e1e18a86482bd1829
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/234079
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-15 18:21:49 +00:00
Joe Tsai
0fbe4ed572 all: start v1.23.0-devel
Change-Id: Ie984b488844e28b95b7565fac4df52fd378f1ebd
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/234078
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-14 20:12:40 +00:00
Joe Tsai
d165be301f all: release v1.23.0
Change-Id: Id8b69ecee5d7c5623b732bd14a8dde8da2c5084e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/234077
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-14 20:12:30 +00:00
Joe Tsai
2d80e9b3ab encoding/prototext: adjust handling of invalid UTF-8
The following changes are made:
* Permit invalid UTF-8 in proto2. This goes against specified behavior,
but matches functional behavior in wire marshaling (not just for Go,
but also in the other major language implementations as well).
* The Format function is specified as ignoring errors since its intended
purpose is to surface information to the human user even if it's not
exactly parsible back into a message. As such, add an unexported
allowInvalidUTF8 option that is specially used by Format.
* Add an EmitASCII option that forces the formatting of
strings and bytes to always be encoded as ASCII.
This ensures that the entire output is always ASCII as well.

Note that we do not replicate this behavior for protojson since:
* The JSON format fundamentally has a stricter and well-specified
grammar for exactly what is valid/invalid, while the text format
has not had a well-specified grammar for the longest time,
leading to all sorts of weird usages due to Hyrum's law.
* This is to ease migration from the legacy implementation,
which did permit invalid UTF-8 in proto2.
* The EmitASCII option relies on the ability to always escape
Unicode characters using ASCII escape sequences, but this is not
possible in JSON since the grammar only has an escape sequence defined
for Unicode characters \u0000 to \uffff, inclusive.
However, Unicode v12.0.0 defines characters up to \U0010FFFF,
which is beyond what the JSON grammar provides escape sequences for.

Change-Id: I2b524a904e9ec59f9ed5500e299613bc27c31a14
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/233077
Reviewed-by: Herbie Ong <herbie@google.com>
2020-05-13 05:25:02 +00:00
Joe Tsai
3034025d28 internal/impl: avoid inlining fixed coderFieldInfo array
Any attempt at guessing the size for a fixed coderFieldInfo array
will always get it wrong in some cases, either by under-estimating
or over-estimating the count. The former causes worse caching behavior,
while the latter causes memory waste.

As a middle ground, just pre-allocate a slice of the exact length.
Each element will have memory locality with each other, but not
be guaranteed to have memory locality with the parent coderMessageInfo.

name                            old time/op  new time/op  delta
EmptyMessage/Wire/Marshal-8     43.1ns ±11%  42.6ns ± 8%  -1.32%  (p=0.036 n=50+49)
EmptyMessage/Wire/Unmarshal-8   18.6ns ±10%  18.9ns ±12%    ~     (p=0.054 n=50+50)
EmptyMessage/Wire/Validate-8    15.0ns ± 9%  14.7ns ±10%  -2.44%  (p=0.002 n=50+45)
EmptyMessage/Clone-8             163ns ±20%   149ns ±19%  -8.58%  (p=0.000 n=48+53)
RepeatedInt32/Wire/Marshal-8    4.27µs ±12%  4.24µs ±13%    ~     (p=0.612 n=48+52)
RepeatedInt32/Wire/Unmarshal-8  3.47µs ±14%  3.50µs ±11%    ~     (p=0.217 n=50+53)
RepeatedInt32/Wire/Validate-8   2.12µs ±12%  2.09µs ± 9%    ~     (p=0.121 n=50+51)
RepeatedInt32/Clone-8           3.04µs ±18%  2.98µs ±36%    ~     (p=0.289 n=51+54)
Required/Wire/Marshal-8          281ns ±14%   276ns ±11%    ~     (p=0.059 n=48+55)
Required/Wire/Unmarshal-8        117ns ±14%   118ns ±11%    ~     (p=0.358 n=49+53)
Required/Wire/Validate-8        87.6ns ± 9%  88.0ns ±12%    ~     (p=0.373 n=48+53)
Required/Clone-8                 533ns ±12%   507ns ±15%  -4.71%  (p=0.000 n=49+54)

Change-Id: I4cf3134e424130bee728b7591127e5c80f07e2db
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/232937
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-13 03:30:14 +00:00
Joe Tsai
1f5b6fe645 all: improve panic messages for better debugability
Change-Id: If3e505e715d5ce2c9a81249c868d26226a25b724
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/232339
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-05 21:10:54 +00:00
Joe Tsai
0e19395744 all: start v1.22.0-devel
Change-Id: I7d70cbd4c9b692b5e4ffdfc576458c48d39f450d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/232118
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-04 19:17:23 +00:00
Joe Tsai
8b0d71ac93 all: release v1.22.0
Change-Id: Ia9d650390d6e0cc6f153834480748faee08fdaac
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/232117
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-04 19:17:18 +00:00
Joe Tsai
4d5be764fb internal/fuzz/wirefuzz: add test to verify initialization checks
The UnmarshalInitialized flag produced by Unmarshal and Validate are
filters such that must never have false positives (i.e., report a
partial message as initialized) otherwise it is incorrect.
It can tolerate some degree of false negatives (i.e., report an
initialized message as partial), but that leads to significant
performance degradation needing to do the full initialization check.
These should be the exception, not the norm.

Adjust the fuzzer to search for false-negative cases. For now, we only
require that the Unmarshal and Validate report initialized for any
"normalized" messages which we produce by marshaling intermediate
message again. This is to work around a known case where they cannot
determine initialization status if the wire data relies on protobuf's
merge functionality (where two or more partial messages merge
together to form an initialized message).

Change-Id: I6bb6c6594981ca08a92583bae77e5a2d44924af6
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/231577
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-01 18:43:38 +00:00
Joe Tsai
d0a499bc65 internal/impl: validate UTF-8 for proto3 optional strings
Change-Id: I090e7c5adac47818831c63d3d999cb7fea5ac696
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/231357
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-01 17:23:32 +00:00
Joe Tsai
3ebaa92e92 all: use v2 Message interface for weak fields
Cleanup the generated logic by having the implementation be backed
by protoimpl rather that directly generated.

Weak fields are a deprecated feature of protobufs and
have entirely be superceded by extensions.
Unfortunately, there are still some usages of it.

Change-Id: Ie1a4b7da253e2ccf5e56627775d9b2fb4090d59a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/229717
Reviewed-by: Damien Neil <dneil@google.com>
2020-04-29 22:34:29 +00:00
Joe Tsai
387873dd53 all: implement support for proto3 optional semantics
In the upcoming 3.12.x release of protoc, the proto3 language will be
amended to support true presence for scalars. This CL adds support
to both the generator and runtime to support these semantics.

Newly added public API:
	protogen.Plugin.SupportedFeatures
	protoreflect.FieldDescriptor.HasPresence
	protoreflect.FieldDescriptor.HasOptionalKeyword
	protoreflect.OneofDescriptor.IsSynthetic

Change-Id: I7c86bf66d0ae56642109beb5f2132184593747ad
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/230698
Reviewed-by: Damien Neil <dneil@google.com>
2020-04-29 20:02:24 +00:00
Joe Tsai
b57aae9def all: improve extension validation
Changes made:
* Ensure protoreflect.ExtensionType.IsValidInterface never panics,
especially if given a nil interface value.
* Have protoreflect.ExtensionType.IsValid{Interface,Value} only
perform type-checks. It does not do value checks (i.e., whether the
value itself is valid). Value validity is left to when an actual
protoreflect.Message.Set operation is performed.
* Add special-casing on proto.SetExtension to treat an invalid
message or list as functionally equivalent to Clear. This is to
be more consistent with the legacy SetExtension implementation
which never panicked when given such values.
* Add special-casing on proto.HasExtension to treat a mismatched
extension descriptor as simply not being present in the message.
This is also to be more consistent with the legacy HasExtension
implementation which did the same thing.

Change-Id: Idf0419abf27b9f85d9b92bd2ff8088e25b7990cc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/229558
Reviewed-by: Damien Neil <dneil@google.com>
2020-04-23 06:01:13 +00:00
Damien Neil
a5526f0129 internal/impl: fix off-by-one error in message initialization
This particular off-by-one can never happen in practice. We will crash
if a field has a number one greater than the maximum field number in the
dense array, but if field number N is in the array we will always put
N+1 in it as well.

Fix the off-by-one anyway.

Change-Id: I8c1304f2fc0d7b91036bde3f7ddb7115c21781ca
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/229278
Reviewed-by: Joe Tsai <joetsai@google.com>
2020-04-21 21:23:04 +00:00
Damien Neil
98f56d1bd5 internal/impl: inline coderInfoFields for better cache locality
Microbenchmarks are inconclusive/noisy, but shows a small but noticeable
improvement on internal benchmarks.

Change-Id: Ic46c6aac8a42c4dc749c4f3583d8c8c95e9548b7
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/229277
Reviewed-by: Joe Tsai <joetsai@google.com>
2020-04-21 21:03:31 +00:00
Damien Neil
d8bc21f7e1 internal/fuzz: update to use native go-fuzz
Fixes golang/protobuf#1084.

Change-Id: I2c71e9f58d09345d13f461ec78ee4e39c3a7f06d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/228277
Reviewed-by: Joe Tsai <joetsai@google.com>
2020-04-15 01:18:53 +00:00
Joe Tsai
ba9d210a5c all: start v1.21.0-devel
Change-Id: Ib9df04de7a3f46f3a3b512e92cc7694ec0f4f927
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/228178
Reviewed-by: Damien Neil <dneil@google.com>
2020-04-13 18:57:37 +00:00
Joe Tsai
3b9eee1291 all: release v1.21.0
Change-Id: I6a91f3efd45ab0a446b220fddc8bd03aec93b388
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/228177
Reviewed-by: Damien Neil <dneil@google.com>
2020-04-13 18:57:02 +00:00
Joe Tsai
7dfcffe5a7 internal/impl: handle extremely old messages
At some point in time, protoc-gen-go actually emitted an XXX_OneofFuncs
with a different signature. Adjust the logic for handling XXX_OneofFuncs
to not assume that the return arguments are in a specific order.

Change-Id: Idd9c09231c4129c655d4a635bb1ae094896a1ff4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/226980
Reviewed-by: Damien Neil <dneil@google.com>
2020-04-02 21:12:23 +00:00
Joe Tsai
a75c9146b7 internal/impl: minor refactoring
Change-Id: I1a5f4ca31fec72a39ba6690af06f4ae8576408cc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/226897
Reviewed-by: Damien Neil <dneil@google.com>
2020-04-02 00:23:08 +00:00
Joe Tsai
d037755d51 internal/detectknown: add helper package to identify well-known types
Change-Id: Id54621b4b44522a350e6994074962852690b5d66
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/225257
Reviewed-by: Herbie Ong <herbie@google.com>
2020-03-24 23:06:32 +00:00
Joe Tsai
f8d77f810a internal/msgfmt: simply time formatting logic
The msgfmt formatter is intended to only for debugging purposes.
Remove unnecesary logic to detect out-of-range timestamps and durations.

Change-Id: I060149ed71aa892bbe4fdb2508b1d0b9df4b5f37
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/225258
Reviewed-by: Damien Neil <dneil@google.com>
2020-03-24 22:58:47 +00:00
Joe Tsai
28cb1e4f28 internal/impl: remove deprecated ExtensionField methods
These were originally needed by the legacy implementation,
but not anymore now that it fully wraps the new implementation.

Change-Id: I950958ebfcb7883fc4b72128d22eaba2da5cc62f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/224583
Reviewed-by: Damien Neil <dneil@google.com>
2020-03-23 18:37:32 +00:00
Joe Tsai
8cbef3ff2d all: fix golint violations
Change-Id: I35d9f6842ec2e9b36c14672a05c4381441bda87a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/224582
Reviewed-by: Herbie Ong <herbie@google.com>
2020-03-21 00:04:20 +00:00
Damien Neil
c92333953e internal/cmd/generate-protos: use module= generator option
Now we'll get a generator error if a file is missing a go_package
option.

Change-Id: I89eec716f86956e6c164a61f5531c140b74bc099
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/222378
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2020-03-20 21:35:56 +00:00
Joe Tsai
2ce1ca9e3f internal/msgfmt: use msgfmt package to format messages
Port message formatting logic in testing/protocmp to internal/msgfmt
and improve upon its output.

This formatter is optimized for humanly readable output.
It takes the best parts of both the JSON and proto text formats.

The good of prototext:
	* It supports emitting unknown fields (useful for debugging).
	* It is relatively concise in the common-case since keys do not
	need to be represented as quoted strings (e.g., "key" vs key).

The bad of prototext:
	* Requires relatively large dependency on encoding/prototext.
	* Our implementation lacks support for serializing packed lists.
	* Lacks support for readable maps.
	* Lacks support for readable Timestamp and Duration.

The good of protojson:
	* First-class support for readable maps.
	* First-class support for readable Timestamp and Duration.

The bad of protojson:
	* Requires relatively large dependency on encoding/protojson.
	* Lacks support for emitting unknown fields.
	* More verbose in the common-case as keys are quoted strings.

The msgfmt package has the benefits of both protojson and prototext,
but none of the detriments. It is a relatively simple implementation.

This output is only intended for human consumption with no associated
deserialization implementation.
To avoid any illusion that this is identical to either the proto text
or JSON formats, this always emits surrounding "{}" for
top-level messages and the keys are not quoted strings.

This CL does not use this format for generated Message.String methods
as there is concerns about being inconsistent with the String methods
as implemented in other languages. Having it be a seperate package makes
it trivial to switch over to this if desired.

Change-Id: I8b3581904d1624e84bf1b1954d2f01e5774b7f87
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/223752
Reviewed-by: Damien Neil <dneil@google.com>
2020-03-20 19:25:02 +00:00
Joe Tsai
cfd80493c5 testing/protopack: make package publicly available
Change-Id: I342ed27df17867f18c58e60880bcac5a31a3096b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219837
Reviewed-by: Damien Neil <dneil@google.com>
2020-03-20 18:05:51 +00:00
Joe Tsai
cd108d00a8 encoding/protowire: make package publicly available
Change-Id: I95e293c208e787a91d50e29817620535dfeaa7f2
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219838
Reviewed-by: Damien Neil <dneil@google.com>
2020-03-20 07:42:18 +00:00
Joe Tsai
b738ac9285 all: regenerate remote .proto files
This fixes the conformance test failures which occur because the
conformance protos have not been regenerated.
The generator script has been modified with a sanity check that
files do not exist outside the expected sub-tree.

Change-Id: I473efec4a016f6bc96ddf7e20d54bcf5ff9b55fe
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/223538
Reviewed-by: Damien Neil <dneil@google.com>
2020-03-18 20:17:11 +00:00
Damien Neil
d3874051d7 internal/encoding/messageset: don't modify input data when unmarshaling
When combining multiple message fields in a MessageSet item (a case
which should never happen in practice), unmarshal could modify the input
data. Fix it to not do so. Add a general check to ensure that unmarshal
operations don't modify the input.

Change-Id: Idde46e6132a1dc96c374f9146efff81783c3bef3
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/223818
Reviewed-by: Joe Tsai <joetsai@google.com>
2020-03-18 05:27:34 +00:00
Joe Tsai
92af527de9 internal/impl: remove Export.ExtensionDescFromType
This is no longer needed by the old implementation.

Change-Id: I3ba02d37f35f599ec790ec4e627258273883a308
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/223279
Reviewed-by: Damien Neil <dneil@google.com>
2020-03-13 23:34:19 +00:00
Joe Tsai
4ab2bc9bb7 internal/version: move version information to internal package
Change-Id: I947876de5d290cf783d9ba798871725e77e16517
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/223277
Reviewed-by: Damien Neil <dneil@google.com>
2020-03-13 19:57:50 +00:00
Joe Tsai
f92988f900 all: upgrade protobuf to v3.11.4
This introduces some conformance test failures,
which occur not because our implementation changed behavior,
but because new cases were added.

Future work will be to investigate these failuress.

Change-Id: Ifb17465883c417acd46865744572f8cd0c858383
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/222857
Reviewed-by: Damien Neil <dneil@google.com>
2020-03-13 19:49:20 +00:00
Joe Tsai
e05f789c09 proto: fix signature of UnmarshalState and MarshalState
The pseudo-internal MarshalState and UnmarshalState method should
not have a seperate Message argument since it is passed in through
the extensible MarshalInput and UnmarshalInput values.

Change-Id: I838aadaee30e91cdf888ab024e65348c73c1cd7e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/222678
Reviewed-by: Damien Neil <dneil@google.com>
2020-03-09 20:02:17 +00:00