Commit Graph

212 Commits

Author SHA1 Message Date
Michael Stapelberg
eb7b468655 all: Release the Opaque API
For golang/protobuf#1657

Change-Id: I7b2b0c30506706015ce278e6054439c9ad9ef727
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/634815
TryBot-Bypass: Michael Stapelberg <stapelberg@google.com>
Reviewed-by: Joseph Tsai <joetsai@digital-static.net>
Reviewed-by: Damien Neil <dneil@google.com>
2024-12-11 03:16:51 -08:00
Michael Stapelberg
5c14d72191 encoding/prototext: use testmessages_test.go approach, too
With this change, we establish a similar mechanism in encoding/prototext as
already exists in the proto package: a testmessages_test.go which serves as an
extension point where one can inject more variants into the corpus of test
messages.

Change-Id: I9f83735a44e5fd2e4736f888aaca428e4cc8ed42
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/623116
Reviewed-by: Chressie Himpel <chressie@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-10-29 14:35:35 +00:00
justinsb
fb995f184a encoding/protojson: allow missing value for Any of type Empty
Some other implementations do not send the value field when encoding
an Any representing a message of the Empty Well-Known-Type.

Fixes golang/protobuf#1620

Change-Id: I89bffd5f92656ba3ac462c0b6365ed4b49e6189d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/618395
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
2024-10-09 07:32:47 +00:00
Michael Stapelberg
cf4f382b27 encoding/prototext: capture current reserved field behavior
This change was originally done in Google-internal cl/512993970.

Change-Id: I66ea95cd975b160892ed9f931312f1767cbc9fc9
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/608315
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Michael Stapelberg <stapelberg@google.com>
Reviewed-by: Christian Höppner <hoeppi@google.com>
2024-08-27 06:11:03 +00:00
Michael Stapelberg
b643888146 encoding/protojson: use fd.HasPresence() for editions compatibility
This change was originally done by Mike Kruskal in Google-internal cl/515091827.

Change-Id: I5d1588193663249964cf8bc9b9fe9f61d29c98a6
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/606756
Reviewed-by: Christian Höppner <hoeppi@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-08-20 08:10:26 +00:00
Thalita Oliveira
d4621760ea protojson: changes error message returned by unmarshal
The current error message does not return the field that triggered the error on JSON unmarshal, so the error message was changed to include the field name.

Resolves golang/protobuf#1504

Change-Id: I4ae098d2fc39bf68ede3560c36d50f630db5f6b4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/587536
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Than McIntosh <thanm@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
2024-06-14 06:16:58 +00:00
Michael Stapelberg
cbc3dd69c1 all: replace interface{} by any now that we are on Go 1.21
I generated this change using:

  % sed -i 's,interface{},any,g' **/*.go
  % git checkout -- **/*.pb.go
  % $EDITOR cmd/protoc-gen-go/internal_gengo/well_known_types.go
  % ./regenerate.bash

Change-Id: I728f4b69c87ffc8f3b19bf807bf9bf1479bdbab4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/585735
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
2024-05-15 12:42:15 +00:00
Michael Stapelberg
15d7b138c5 all: remove Go 1.17 build tags / workarounds
related to golang/protobuf#1613

Change-Id: Ie4255c24c1b79b13aab763a75125836191088d26
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/585096
Reviewed-by: Lasse Folger <lassefolger@google.com>
Auto-Submit: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-05-13 14:03:25 +00:00
Lasse Folger
9d9d8d3ef5 encoding/proto[json|text]: accept lower case names for group-like fields
This is a result of the discussion in [1]. Before editions, a group defined a multiple things:

* a type
* a field
* an encoding scheme

With editions this has changed and groups no longer exist and the different parts have to be defined individually. Most importantly, the field and the type also had the same name (usually and CamelCase name). To keep compatibility with proto2 groups, [2] introduced a concept of group-like fields and adjusted the Text/JSON parsers to accept the type name instead of the field name for such fields. This means you can convert from proto2 groups to editions without changing the semantics.
Furthermore, to avoid suprises with group-like fields (e.g. when a user by coincident specified a field that is group-like) protobuf decided that group-like fields should always accept the type and the field name for group like fields. This also allows us to eventually emit the field name rather than the type name for group like fields in the future.

This change implements this decision in Go.


[1] https://github.com/protocolbuffers/protobuf/issues/16239
[2] https://go.dev/cl/575916

Change-Id: I701c4cd228d2e0867b2a87771b6c6331459c4910
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/582755
Reviewed-by: Lasse Folger <lassefolger@google.com>
Reviewed-by: Mike Kruskal <mkruskal@google.com>
Commit-Queue: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Auto-Submit: Michael Stapelberg <stapelberg@google.com>
2024-05-06 08:56:34 +00:00
Andrew Gerrand
c2a26e757e encoding/{protojson,prototext}: strengthen wording on stability
Because these encoders use internal/detrand to make their output
nondeterministic across builds, use stronger wording to warn users of
their instability.

Updates golang/protobuf#1121

Change-Id: Ia809e5c26ce24d17f602e7fbae26d9df2b57d05b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/579895
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
2024-04-18 14:49:38 +00:00
Mike Kruskal
a18684df74 protobuf: fix delimited fields under editions in go
This brings go into conformance with other implementations.  Group-like message fields with delimited encoding will continue to use the type name for text-format, but everything else will use the field name.

Change-Id: Ib6d07f19ccfa853ce0370392c89fd24fb7148793
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/575896
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Commit-Queue: Michael Stapelberg <stapelberg@google.com>
2024-04-04 07:31:06 +00:00
Damien Neil
f01a588e58 encoding/protojson, internal/encoding/json: handle missing object values
In internal/encoding/json, report an error when encountering a }
when we are expecting an object field value. For example, the input
`{"":}` now correctly results in an error at the closing } token.

In encoding/protojson, check for an unexpected EOF token in
skipJSONValue. This is redundant with the check in internal/encoding/json,
but adds a bit more defense against any other similar bugs that
might exist.

Fixes CVE-2024-24786

Change-Id: I03d52512acb5091c8549e31ca74541d57e56c99d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/569356
TryBot-Bypass: Damien Neil <dneil@google.com>
Reviewed-by: Roland Shoemaker <roland@golang.org>
Commit-Queue: Damien Neil <dneil@google.com>
2024-03-05 19:00:02 +00:00
Lasse Folger
055c812a4f encoding/prototext: add proto editions and fuzz tests
Change-Id: I2afc5ae83bf68600def3568e1d3ad51ef00e7671
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/566395
Auto-Submit: Lasse Folger <lassefolger@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-02-23 12:14:08 +00:00
Lasse Folger
f2cb7f136e encoding/protojson: add protojson editions tests including fuzztests
Change-Id: I478bf6a945cb2c86c71fd20b64dedb4b2e585f1d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/566035
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
2024-02-23 08:35:31 +00:00
Josh Humphries
bfcd6476a3 protojson: configurable recursion limit when unmarshalling
Fixes golang/protobuf#1583 and golang/protobuf#1584

Limits the level of recursion when parsing JSON to avoid
fatal stack overflow errors if input uses pathologically
deep nesting. This is already a feature of the binary
format, and this adds that feature to the JSON format.

This also re-implements how JSON values are discarded
to be more efficient (and not use recursion).

Change-Id: I4026b739abe0335387209a43645f65e4b6e43409
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/552255
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: David Chase <drchase@google.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
2023-12-21 22:43:23 +00:00
Oliver Newman
6352deccdb prototext: Fix parsing of unknown repeated message fields
If `DiscardUnknown` was enabled, previously this syntax was rejected
with an error "unexpected token: ]":

  unknown_field: [
    {}
  ]

This is because the parser short-circuited after parsing the {}, and
didn't handle the closing square brace. Now it no longer short-circuits.

Change-Id: I6230b56766632752a5cc8822b17ed8262873d6cc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/530616
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
2023-09-27 16:15:44 +00:00
Sam Eiderman
8088bf85b8 encoding: Add EmitDefaultValues option
Introduce the EmitDefaultValues in addition to the existing
EmitUnpopulated option.

EmitDefaultValues is added to emit json messages more compatible with
the `always_print_primitive_fields` option of the cpp protobuf library.

EmitUnpopulated overrides EmitDefaultValues since the former generates
a strict superset of the latter.

See descussion:
https://github.com/golang/protobuf/issues/1536

Change-Id: Ib29b69d630fa3e8d8fdeb0de43b5683f30152151
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/521215
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Cassondra Foesch <cfoesch@gmail.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
2023-09-20 13:23:09 +00:00
Singee
70db1e1de2 encoding/protojson: ignore unknown enum name if DiscardUnknown=true
This makes encoding/protojson pass all protobuf conformance tests.

Other Similar CL:
  - https://go-review.googlesource.com/c/protobuf/+/350469
  - https://go-review.googlesource.com/c/protobuf/+/256677

Fixes golang/protobuf#1208

Change-Id: I9f74162b80a3c7ee4750160fc59f0313345046de
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/525535
Reviewed-by: Cassondra Foesch <cfoesch@gmail.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2023-09-08 10:59:01 +00:00
Joe Tsai
f9212a8dfa all: modernize documentation
Modernize the documentation across the entire module
to make use of the newer ability to linkify declarations.

Change-Id: I440f9a3f025ec6fadfd9cba59b1dfb57028bf3f1
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/309430
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
2023-09-05 14:55:28 +00:00
Damien Neil
4396dd4cd0 encoding/protodelim: drop TestUnmarshalFromBufioAllocations
This test is failing on at least one architecture.
It's probably too fragile to be worth figuring out;
drop the test entirely.

Fixes golang/protobuf#1554

Change-Id: I9106dbdd12af8719761fd7257b38d4f8584279a6
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/512677
Run-TryBot: Damien Neil <dneil@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
2023-07-25 06:14:34 +00:00
Michael Stapelberg
59a8581684 encoding/protodelim: fix handling of io.EOF
Before this change, when encountering an io.EOF after reading at least
one byte, the zero value byte was — incorrectly — appended to sizeBuf,
and the io.EOF was ignored, resulting in a complete varint (0 has no
continuation bit), which in turn resulted in incorrect unmarshalling.

Change-Id: If06d45039d998eaddf91d0864814bb31d4cb7ae0
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/505555
Reviewed-by: Lasse Folger <lassefolger@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
2023-06-26 07:38:07 +00:00
Vanja Pejovic
b8fc770601 encoding/protodelim: If UnmarshalFrom gets a bufio.Reader, try to reuse its buffer instead of creating a new one
When unmarshalling many messages, this reduces the amount of memory allocated and saves CPU time.

Change-Id: I440b8b223319ba2ed31ce559c125b1d640d5880c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/491596
Reviewed-by: Joseph Tsai <joetsai@digital-static.net>
Reviewed-by: Lasse Folger <lassefolger@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
2023-05-08 20:37:08 +00:00
Edward McFarlane
05cbe34333 encoding: add MarshalAppend to protojson and prototext
Adds MarshalAppend methods to allow for byte slices to be reused.
Copies signature from the binary encoding.

Small changes to internal json and text libraries to use strconv
AppendInt and AppendUint for number encoding.

Change-Id: Ife7c8979c1c153a0a0bf9b70b296b8158d38dffc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/489615
Reviewed-by: Edward McFarlane <emcfarlane000@gmail.com>
Reviewed-by: Joseph Tsai <joetsai@digital-static.net>
Reviewed-by: Lasse Folger <lassefolger@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
2023-05-01 15:10:15 +00:00
Damien Neil
edaf511a7a internal/encoding/text: fix parsing of incomplete numbers
Fix a panic when parsing the incomplete negative number "- ".

Fixes golang/protobuf#1530

Change-Id: Iba5e8ee68d5f7255c28f1a74f31beee36c9ed847
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/475995
Reviewed-by: Lasse Folger <lassefolger@google.com>
Run-TryBot: Damien Neil <dneil@google.com>
2023-03-14 07:09:30 +00:00
Josh Humphries
fcf5f6cb72 encoding/prototext: allow whitespace and comments between minus sign and number in negative numeric literal
The text format specification[1] indicates that whitespace and comments
may appear after a minus sign and before the subsequent numeric component
in negative number literals. But the Go implementation does not allow
this.

This brings the Go implementation info conformance with this aspect.

Fixes golang/protobuf#1526

[1] https://protobuf.dev/reference/protobuf/textformat-spec/#parsing

Change-Id: I3996c89ee9d37cf2b7502fc6736d6e2ed6dbcf43
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/473015
Reviewed-by: Lasse Folger <lassefolger@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
2023-03-06 07:33:04 +00:00
Oleksandr Redko
49eaa78c6c all: update links to Protocol Buffer documentation
https://developers.google.com/protocol-buffers/ states that
this site will be deprecated January 2023 and provides
https://protobuf.dev/ as a link to the new location.

Generated .pb.go files are created by generate.bash using Go 1.18,
so leave them unchanged for now.

Fixes golang/protobuf#1520.

Change-Id: I1ddf6f329c96ba05d7c7d8562ad2945f479ee09b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/466375
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
2023-02-08 13:52:20 +00:00
Samuel Benzaquen
31a5920811 protobuf: remove the check for reserved field numbers.
Field numbers in the reserved range are semantically valid,
although the protocol buffer compiler may reject attempts to
define fields in this range.

Change-Id: Ib939ed5f4fef09b29047b509829b2f01494bfb6a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/461238
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
2023-01-12 08:45:15 +00:00
Joe Tsai
f930b1dc76 encoding/protojson: fix parsing of google.protobuf.Timestamp
The Timestamp message uses a subset of RFC 3339.
RFC 3339, section 5.6 specifies that the subsecond field can have
any non-zero number of digits.
On the other hand, the protobuf documentation specifies that Timestamp
uses RFC 3339 with a few restrictions. In other words,
protobuf does NOT use RFC 3339, but rather a subset of it.

An upstream change https://go.dev/cl/425037 modifies the time package
to be internally consistent about handling of extra subsecond digits
allowing it to be more in line with RFC 3339.

Make a corresponding change here to ensure we remain compliant
with protobuf's restricted use of RFC 3339.

Change-Id: Ic145c68492fb41a5f7b79b653f3246dd9091d5d8
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/425554
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
2022-08-31 09:28:52 +00:00
Sachin Padmanabhan
fb0abd9158 encoding: add protodelim package
Fixes golang/protobuf#1382

Change-Id: I30dc9bf9aa44e35cde8fb472c3b8b116d459714e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/419254
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Joseph Tsai <joetsai@digital-static.net>
2022-08-02 22:27:24 +00:00
Damien Neil
b0a944684d all: reformat with go1.19 gofmt
Apply go1.19 gofmt to non-generated files.

Generated .pb.go files are created by generate.bash using Go 1.18,
so leave them unchanged for now.

Change-Id: Ied36c83cf99704988d059bf0412e677f0fbc71b0
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/418676
Reviewed-by: Ian Lance Taylor <iant@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
2022-07-21 16:01:13 +00:00
Koichi Shiraishi
784c482554 all: remove shorthand import aliases
Change-Id: I62ec79419354a2c65ac69e4bcc1cc864a6a40a1a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/407934
Reviewed-by: Lasse Folger <lassefolger@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
2022-05-24 20:05:50 +00:00
Koichi Shiraishi
e62d8edb75 all: remove duplicate imports
protoreflect (as pref) and protoiface (as piface) are imported
duplicates in some files.
Respect package name, remove unnecessary aliased import statements.

Change-Id: Ie9897f17a50d19a462035964e366af72afed0e4d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/405694
Reviewed-by: Kirill Kolyshkin <kolyshkin@gmail.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
2022-05-23 17:24:26 +00:00
Kir Kolyshkin
bf9455640d all: fix typos
Brought to you by codespell v2.1.0, using the command

	codespell -S .cache,vendor -L ot,ba,fo,unparseable -w

Note that the misspelled "unparseable" comes from the
github.com/protocolbuffers/protobuf, where it is explicitly ignored
(see [1] and some explanation at [2]), so we ignore it here, too.

[1] https://github.com/protocolbuffers/protobuf/pull/7752
[2] https://github.com/protocolbuffers/protobuf/pull/7751#discussion_r460170422

Change-Id: Ie1ca705db4f11df8ec8b22fdc22b6a6ee667ae5b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/406845
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
2022-05-19 09:32:38 +00:00
Lasse Folger
3992ea83a2 all: implement depth limit for unmarshaling
+ This change introduce a default and configurable depth limit for
  proto.Unmarshal. If a message is nested deeper than the limit,
  unmarshaling will fail. There are two ways to nest messages. Either by
  having fields which are message types itself or by using groups.
+ The default limit is 10,000 for now. This might change in the future
  to align it with other language implementation (C++ and Java use 100
  as limit).
+ If pure groups (groups that don't contain message fields) are nested
  deeper than the default limit the unmarshaling fails with:
  proto: cannot parse invalid wire-format data
+ Note: the configured limit does not apply to pure groups.
+ This change is introduced to improve security and robustness. Because
  unmarshaling is implemented using recursion it can lead to stack overflows
  for certain inputs. The introduced limit protects against this.
+ A secondary motivation for this limit is the alignment with other
  languages. Protocol buffers are a language interoperability mechanism
  and thus either all implementations should accept the input or all
  implementation should reject the input.

Change-Id: I14bdb44d06e4bd1aa90d6336c2cf6446003b2037
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/385854
Trust: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Damien Neil <dneil@google.com>
Trust: Damien Neil <dneil@google.com>
Reviewed-by: Nicolas Hillegeer <aktau@google.com>
Reviewed-by: Chressie Himpel <chressie@google.com>
2022-02-17 17:07:31 +00:00
Joe Tsai
c30776bb3b encoding/prototext: fix skipping of unknown fields
Inside decoder.skipValue we should not be calling skipValue again
since we had already read the value earlier. The only possible
composite type in the context of a list is another message,
which is already handled in the case above.

Change-Id: If40da2d369e0a64a64ba9b961377331231158fe2
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/317430
Trust: Joe Tsai <joetsai@digital-static.net>
Trust: Herbie Ong <herbie@google.com>
Reviewed-by: Herbie Ong <herbie@google.com>
2021-05-06 20:07:56 +00:00
Joe Tsai
174b9ecfe3 all: document that Unmarshal must be a mutable message
Fixes #937

Change-Id: I40b2678eba0195ed01676167f8e01e2fedea293b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/302329
Trust: Joe Tsai <joetsai@digital-static.net>
Reviewed-by: Damien Neil <dneil@google.com>
2021-03-16 20:06:34 +00:00
Joe Tsai
0fd4f3a506 encoding/protojson: restrict valid values for google.protobuf.Value.number_value
The purpose of struct.proto is to be an exact mapping of JSON in protobufs.
Since JSON doesn't support NaN and Inf, we should reject serialization
of such values. Prior to this CL, they would be serialzed as a JSON string,
which would change the interpretation of the value when round-tripped.

Fixes golang/protobuf#1182

Change-Id: I6dba9973b1c24d99e5688b509611c0a952c00022
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/247737
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Herbie Ong <herbie@google.com>
2020-09-04 00:11:24 +00:00
Joe Tsai
f2423bb714 encoding/protojson: simplify Duration formatting
Simplify the implementation by reducing the number of branches.

Change-Id: I6e2ffee0fc2d77f7e2a70f76e03d081f4fc0e99d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/247459
Reviewed-by: Herbie Ong <herbie@google.com>
2020-08-08 00:59:21 +00:00
Joe Tsai
28b807b56e encoding/protojson: use synthetic @type field for Any messages
In order for the synthetic @type field to potentially get reordered,
we implement insertion of that synthetic field by adding it
as a synthetic field that Range may iterate over.

This change sets up this code to more readily support a
hypothetical serialization mode for canonical serialization.

Change-Id: Ia0015a1a0804c15805dc5f3a3511fcf0f8513418
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/243817
Reviewed-by: Herbie Ong <herbie@google.com>
2020-07-22 17:37:26 +00:00
Joe Tsai
e14d6b3cdc reflect/protoreflect: add FieldDescriptor.TextName
Add a new TextName accessor that returns the field name that should
be used for the text format. It is usually just the field name, except:
1) it uses the inlined message name for groups,
2) uses the full name surrounded by brackets for extensions, and
3) strips the "message_set_extension" for well-formed extensions
to the proto1 MessageSet.

We make similar adjustments to the JSONName accessor so that it applies
similar semantics for extensions.

The two changes simplifies all logic that wants the humanly readable
name for a field.

Change-Id: I524b6e017fb955146db81819270fe197f8f97980
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/239838
Reviewed-by: Herbie Ong <herbie@google.com>
2020-07-08 23:23:57 +00:00
Joe Tsai
92679665d7 internal/order: add a package for ordered iteration over messages and maps
The order package replaces the mapsort and fieldsort packages.
It presents a common API for ordered iteration over message fields
and map fields.

It has a number of pre-defined orderings.

Change-Id: Ie6cd423da30b4757864c352cb04454f21fe07ee2
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/239837
Reviewed-by: Herbie Ong <herbie@google.com>
2020-07-01 18:57:38 +00:00
Joe Tsai
b78321453d reflect/protoregistry: centralize MessageSet extension resolution logic
Centralize the MessageSet extension resolution logic in the registry.
This avoids needless replication of this exact logic in multiple places
(for JSON and text) and elsewhere.

Change-Id: I70bfea899e295e8c589f418965bf0dd099f93628
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/240077
Reviewed-by: Herbie Ong <herbie@google.com>
2020-07-01 17:56:02 +00:00
cybrcodr
beaa55256c encoding/protojson: add better validation to FieldMask serialization
For marshaling, apart from already existing check that each item in
paths field is reversible, also make sure that string is a valid
protobuf name.

For unmarshaling, make sure that each resulting item in paths field is
a valid protobuf name and input path item does not contain _. The latter
check satisfies the conformance test
Recommended.Proto3.JsonInput.FieldMaskInvalidCharacter.

Fixes golang/protobuf#1141.

Change-Id: Iffc278089b20e496b7216d5b8c966b21b70e782d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/236998
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2020-06-12 06:33:55 +00:00
Joe Tsai
69839c78c3 internal/genid: remove WhichFile
It seems safer to explicitly mention exactly which messages
have special handling, rather than special casing the .profile
that they live in. This is safer because there is no guarantee
that new messages won't be added to each of these files.

The protojson implementation is modified to no longer rely
on a isCustomType helper and instead return a marshal or unmarshal
function pointer that is non-nil if specialized serialization
exists for that message type.

Change-Id: I5e3551d66f5a4b9024e583b627c0292cb7da6803
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/235657
Reviewed-by: Herbie Ong <herbie@google.com>
2020-05-29 21:03:06 +00:00
Joe Tsai
e0b77db13b internal/genid: add new package for generated identifiers
The genid package unifies the genname, fieldnum, and detectknown
packages into a single package.

Whenever possible use the generated constants rather than
hard-coded literals. This makes it easier to search the entire
module for special logic that deal with well-known types.

Change-Id: I13beff1f4149444a0c0b9e607ebf759657f000f4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/235301
Reviewed-by: Herbie Ong <herbie@google.com>
2020-05-29 07:08:23 +00:00
Joe Tsai
74aae6a46c encoding/prototext: simplify decoder.unmarshalAny
A hasFields map that only ever contains 3 entries seems more
complex than necessary. It's simpler and more performant to
just track three discrete boolean variables for each of the cases.

Change-Id: I1ba20da130f6b560a57fe8c3a73968983e563b48
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/235477
Reviewed-by: Herbie Ong <herbie@google.com>
2020-05-27 23:07:30 +00:00
Joe Tsai
81db48ad09 all: move well-known types
This is step 2 of 6 in a multi-stage migration
to move the well-known types
from the google.golang.org/genproto module
to the google.golang.org/protobuf module.

The generated Go packages for field_mask.proto, api.proto,
type.proto, and source_context.proto are being moved over
to this module alongside all the other well-known types.

In order to move these types between two modules,
there needs to be a sequence of changes submitted in
decently rapid succession. It is impossible to atomically
make these changes, so a brief breakage is inevitable.
The steps are as follows:

Step 1: Submit a change to cloud.google.com/go/internal/gapicgen
to avoid generating the well-known types. Otherwise, the tool
will undo the changes made in step 3.
See https://code-review.googlesource.com/c/gocloud/+/56810

Step 2: Submit a change to google.golang.org/protobuf that
adds the generated well-known types being migrated to that module.
In order to prevent the situation where a user links in
too old a version of the genproto module such that
duplicate registration occurs for the well-known types,
the registry is specially modified to provide an error
message that instructs users to upgrade the genproto module.
See https://golang.org/cl/234937

Step 3: Submit a change to google.golang.org/genproto that
switches all generated well-known types to be aliases to the
ones declared in google.golang.org/protobuf from the previous step.
This will cause the genproto module to incur an dependency
on an unreleased version of the protobuf module.
See https://github.com/googleapis/go-genproto/pull/372

Step 4: Submit a change to google.golang.org/protobuf that
adds a weak module depdency on the genproto module at the
revision from the previous step.

Step 5: Release google.golang.org/protobuf@v1.24.0.

Step 6: Submit a change to google.golang.org/genproto that
updates the protobuf module dependency to v1.24.0.

Change-Id: I36a19049d2240b67a37dfad20e154505aee7c784
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/234937
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-26 19:51:55 +00:00
Joe Tsai
118baf6390 all: funnel similar functionality through a single function
Some companies (e.g., Google) run a profiling service where they may
choose to special-case certain symbols in a binary to classify
commonly used libraries like protobufs.

This CL funnels similar functionality through a single function
so that they can be more easily identified. This is by no means a
firm statement that these identifiers will never change names,
but at least the code documents warnings to avoid changing the
name of certain identifiers.

This CL provides the following semi-stable symbol names:
	"google.golang.org/protobuf/proto".MarshalOptions.size
	"google.golang.org/protobuf/proto".MarshalOptions.marshal
	"google.golang.org/protobuf/proto".UnmarshalOptions.unmarshal
	"google.golang.org/protobuf/encoding/prototext".MarshalOptions.marshal
	"google.golang.org/protobuf/encoding/prototext".UnmarshalOptions.unmarshal
	"google.golang.org/protobuf/encoding/protojson".MarshalOptions.marshal
	"google.golang.org/protobuf/encoding/protojson".UnmarshalOptions.unmarshal

Merge and Clone are not part of the above set since there is a
possibility that MergeOptions will be added in the future.

We use an unexported method so that we have the freedom to change the
method however we want since profilers do not care about that.

Change-Id: Ia79af260d00125f48139420e1e18a86482bd1829
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/234079
Reviewed-by: Damien Neil <dneil@google.com>
2020-05-15 18:21:49 +00:00
Joe Tsai
2d80e9b3ab encoding/prototext: adjust handling of invalid UTF-8
The following changes are made:
* Permit invalid UTF-8 in proto2. This goes against specified behavior,
but matches functional behavior in wire marshaling (not just for Go,
but also in the other major language implementations as well).
* The Format function is specified as ignoring errors since its intended
purpose is to surface information to the human user even if it's not
exactly parsible back into a message. As such, add an unexported
allowInvalidUTF8 option that is specially used by Format.
* Add an EmitASCII option that forces the formatting of
strings and bytes to always be encoded as ASCII.
This ensures that the entire output is always ASCII as well.

Note that we do not replicate this behavior for protojson since:
* The JSON format fundamentally has a stricter and well-specified
grammar for exactly what is valid/invalid, while the text format
has not had a well-specified grammar for the longest time,
leading to all sorts of weird usages due to Hyrum's law.
* This is to ease migration from the legacy implementation,
which did permit invalid UTF-8 in proto2.
* The EmitASCII option relies on the ability to always escape
Unicode characters using ASCII escape sequences, but this is not
possible in JSON since the grammar only has an escape sequence defined
for Unicode characters \u0000 to \uffff, inclusive.
However, Unicode v12.0.0 defines characters up to \U0010FFFF,
which is beyond what the JSON grammar provides escape sequences for.

Change-Id: I2b524a904e9ec59f9ed5500e299613bc27c31a14
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/233077
Reviewed-by: Herbie Ong <herbie@google.com>
2020-05-13 05:25:02 +00:00
Joe Tsai
387873dd53 all: implement support for proto3 optional semantics
In the upcoming 3.12.x release of protoc, the proto3 language will be
amended to support true presence for scalars. This CL adds support
to both the generator and runtime to support these semantics.

Newly added public API:
	protogen.Plugin.SupportedFeatures
	protoreflect.FieldDescriptor.HasPresence
	protoreflect.FieldDescriptor.HasOptionalKeyword
	protoreflect.OneofDescriptor.IsSynthetic

Change-Id: I7c86bf66d0ae56642109beb5f2132184593747ad
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/230698
Reviewed-by: Damien Neil <dneil@google.com>
2020-04-29 20:02:24 +00:00