With this change, we establish a similar mechanism in encoding/prototext as
already exists in the proto package: a testmessages_test.go which serves as an
extension point where one can inject more variants into the corpus of test
messages.
Change-Id: I9f83735a44e5fd2e4736f888aaca428e4cc8ed42
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/623116
Reviewed-by: Chressie Himpel <chressie@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Some other implementations do not send the value field when encoding
an Any representing a message of the Empty Well-Known-Type.
Fixesgolang/protobuf#1620
Change-Id: I89bffd5f92656ba3ac462c0b6365ed4b49e6189d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/618395
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
This change was originally done by Mike Kruskal in Google-internal cl/515091827.
Change-Id: I5d1588193663249964cf8bc9b9fe9f61d29c98a6
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/606756
Reviewed-by: Christian Höppner <hoeppi@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
The current error message does not return the field that triggered the error on JSON unmarshal, so the error message was changed to include the field name.
Resolvesgolang/protobuf#1504
Change-Id: I4ae098d2fc39bf68ede3560c36d50f630db5f6b4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/587536
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Than McIntosh <thanm@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
related to golang/protobuf#1613
Change-Id: Ie4255c24c1b79b13aab763a75125836191088d26
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/585096
Reviewed-by: Lasse Folger <lassefolger@google.com>
Auto-Submit: Michael Stapelberg <stapelberg@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Because these encoders use internal/detrand to make their output
nondeterministic across builds, use stronger wording to warn users of
their instability.
Updates golang/protobuf#1121
Change-Id: Ia809e5c26ce24d17f602e7fbae26d9df2b57d05b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/579895
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
In internal/encoding/json, report an error when encountering a }
when we are expecting an object field value. For example, the input
`{"":}` now correctly results in an error at the closing } token.
In encoding/protojson, check for an unexpected EOF token in
skipJSONValue. This is redundant with the check in internal/encoding/json,
but adds a bit more defense against any other similar bugs that
might exist.
Fixes CVE-2024-24786
Change-Id: I03d52512acb5091c8549e31ca74541d57e56c99d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/569356
TryBot-Bypass: Damien Neil <dneil@google.com>
Reviewed-by: Roland Shoemaker <roland@golang.org>
Commit-Queue: Damien Neil <dneil@google.com>
Fixesgolang/protobuf#1583 and golang/protobuf#1584
Limits the level of recursion when parsing JSON to avoid
fatal stack overflow errors if input uses pathologically
deep nesting. This is already a feature of the binary
format, and this adds that feature to the JSON format.
This also re-implements how JSON values are discarded
to be more efficient (and not use recursion).
Change-Id: I4026b739abe0335387209a43645f65e4b6e43409
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/552255
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: David Chase <drchase@google.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
Introduce the EmitDefaultValues in addition to the existing
EmitUnpopulated option.
EmitDefaultValues is added to emit json messages more compatible with
the `always_print_primitive_fields` option of the cpp protobuf library.
EmitUnpopulated overrides EmitDefaultValues since the former generates
a strict superset of the latter.
See descussion:
https://github.com/golang/protobuf/issues/1536
Change-Id: Ib29b69d630fa3e8d8fdeb0de43b5683f30152151
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/521215
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Cassondra Foesch <cfoesch@gmail.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
Modernize the documentation across the entire module
to make use of the newer ability to linkify declarations.
Change-Id: I440f9a3f025ec6fadfd9cba59b1dfb57028bf3f1
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/309430
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
Adds MarshalAppend methods to allow for byte slices to be reused.
Copies signature from the binary encoding.
Small changes to internal json and text libraries to use strconv
AppendInt and AppendUint for number encoding.
Change-Id: Ife7c8979c1c153a0a0bf9b70b296b8158d38dffc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/489615
Reviewed-by: Edward McFarlane <emcfarlane000@gmail.com>
Reviewed-by: Joseph Tsai <joetsai@digital-static.net>
Reviewed-by: Lasse Folger <lassefolger@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
The Timestamp message uses a subset of RFC 3339.
RFC 3339, section 5.6 specifies that the subsecond field can have
any non-zero number of digits.
On the other hand, the protobuf documentation specifies that Timestamp
uses RFC 3339 with a few restrictions. In other words,
protobuf does NOT use RFC 3339, but rather a subset of it.
An upstream change https://go.dev/cl/425037 modifies the time package
to be internally consistent about handling of extra subsecond digits
allowing it to be more in line with RFC 3339.
Make a corresponding change here to ensure we remain compliant
with protobuf's restricted use of RFC 3339.
Change-Id: Ic145c68492fb41a5f7b79b653f3246dd9091d5d8
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/425554
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
The purpose of struct.proto is to be an exact mapping of JSON in protobufs.
Since JSON doesn't support NaN and Inf, we should reject serialization
of such values. Prior to this CL, they would be serialzed as a JSON string,
which would change the interpretation of the value when round-tripped.
Fixesgolang/protobuf#1182
Change-Id: I6dba9973b1c24d99e5688b509611c0a952c00022
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/247737
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Herbie Ong <herbie@google.com>
Simplify the implementation by reducing the number of branches.
Change-Id: I6e2ffee0fc2d77f7e2a70f76e03d081f4fc0e99d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/247459
Reviewed-by: Herbie Ong <herbie@google.com>
In order for the synthetic @type field to potentially get reordered,
we implement insertion of that synthetic field by adding it
as a synthetic field that Range may iterate over.
This change sets up this code to more readily support a
hypothetical serialization mode for canonical serialization.
Change-Id: Ia0015a1a0804c15805dc5f3a3511fcf0f8513418
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/243817
Reviewed-by: Herbie Ong <herbie@google.com>
Add a new TextName accessor that returns the field name that should
be used for the text format. It is usually just the field name, except:
1) it uses the inlined message name for groups,
2) uses the full name surrounded by brackets for extensions, and
3) strips the "message_set_extension" for well-formed extensions
to the proto1 MessageSet.
We make similar adjustments to the JSONName accessor so that it applies
similar semantics for extensions.
The two changes simplifies all logic that wants the humanly readable
name for a field.
Change-Id: I524b6e017fb955146db81819270fe197f8f97980
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/239838
Reviewed-by: Herbie Ong <herbie@google.com>
The order package replaces the mapsort and fieldsort packages.
It presents a common API for ordered iteration over message fields
and map fields.
It has a number of pre-defined orderings.
Change-Id: Ie6cd423da30b4757864c352cb04454f21fe07ee2
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/239837
Reviewed-by: Herbie Ong <herbie@google.com>
Centralize the MessageSet extension resolution logic in the registry.
This avoids needless replication of this exact logic in multiple places
(for JSON and text) and elsewhere.
Change-Id: I70bfea899e295e8c589f418965bf0dd099f93628
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/240077
Reviewed-by: Herbie Ong <herbie@google.com>
For marshaling, apart from already existing check that each item in
paths field is reversible, also make sure that string is a valid
protobuf name.
For unmarshaling, make sure that each resulting item in paths field is
a valid protobuf name and input path item does not contain _. The latter
check satisfies the conformance test
Recommended.Proto3.JsonInput.FieldMaskInvalidCharacter.
Fixesgolang/protobuf#1141.
Change-Id: Iffc278089b20e496b7216d5b8c966b21b70e782d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/236998
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
It seems safer to explicitly mention exactly which messages
have special handling, rather than special casing the .profile
that they live in. This is safer because there is no guarantee
that new messages won't be added to each of these files.
The protojson implementation is modified to no longer rely
on a isCustomType helper and instead return a marshal or unmarshal
function pointer that is non-nil if specialized serialization
exists for that message type.
Change-Id: I5e3551d66f5a4b9024e583b627c0292cb7da6803
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/235657
Reviewed-by: Herbie Ong <herbie@google.com>
The genid package unifies the genname, fieldnum, and detectknown
packages into a single package.
Whenever possible use the generated constants rather than
hard-coded literals. This makes it easier to search the entire
module for special logic that deal with well-known types.
Change-Id: I13beff1f4149444a0c0b9e607ebf759657f000f4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/235301
Reviewed-by: Herbie Ong <herbie@google.com>
This is step 2 of 6 in a multi-stage migration
to move the well-known types
from the google.golang.org/genproto module
to the google.golang.org/protobuf module.
The generated Go packages for field_mask.proto, api.proto,
type.proto, and source_context.proto are being moved over
to this module alongside all the other well-known types.
In order to move these types between two modules,
there needs to be a sequence of changes submitted in
decently rapid succession. It is impossible to atomically
make these changes, so a brief breakage is inevitable.
The steps are as follows:
Step 1: Submit a change to cloud.google.com/go/internal/gapicgen
to avoid generating the well-known types. Otherwise, the tool
will undo the changes made in step 3.
See https://code-review.googlesource.com/c/gocloud/+/56810
Step 2: Submit a change to google.golang.org/protobuf that
adds the generated well-known types being migrated to that module.
In order to prevent the situation where a user links in
too old a version of the genproto module such that
duplicate registration occurs for the well-known types,
the registry is specially modified to provide an error
message that instructs users to upgrade the genproto module.
See https://golang.org/cl/234937
Step 3: Submit a change to google.golang.org/genproto that
switches all generated well-known types to be aliases to the
ones declared in google.golang.org/protobuf from the previous step.
This will cause the genproto module to incur an dependency
on an unreleased version of the protobuf module.
See https://github.com/googleapis/go-genproto/pull/372
Step 4: Submit a change to google.golang.org/protobuf that
adds a weak module depdency on the genproto module at the
revision from the previous step.
Step 5: Release google.golang.org/protobuf@v1.24.0.
Step 6: Submit a change to google.golang.org/genproto that
updates the protobuf module dependency to v1.24.0.
Change-Id: I36a19049d2240b67a37dfad20e154505aee7c784
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/234937
Reviewed-by: Damien Neil <dneil@google.com>
Some companies (e.g., Google) run a profiling service where they may
choose to special-case certain symbols in a binary to classify
commonly used libraries like protobufs.
This CL funnels similar functionality through a single function
so that they can be more easily identified. This is by no means a
firm statement that these identifiers will never change names,
but at least the code documents warnings to avoid changing the
name of certain identifiers.
This CL provides the following semi-stable symbol names:
"google.golang.org/protobuf/proto".MarshalOptions.size
"google.golang.org/protobuf/proto".MarshalOptions.marshal
"google.golang.org/protobuf/proto".UnmarshalOptions.unmarshal
"google.golang.org/protobuf/encoding/prototext".MarshalOptions.marshal
"google.golang.org/protobuf/encoding/prototext".UnmarshalOptions.unmarshal
"google.golang.org/protobuf/encoding/protojson".MarshalOptions.marshal
"google.golang.org/protobuf/encoding/protojson".UnmarshalOptions.unmarshal
Merge and Clone are not part of the above set since there is a
possibility that MergeOptions will be added in the future.
We use an unexported method so that we have the freedom to change the
method however we want since profilers do not care about that.
Change-Id: Ia79af260d00125f48139420e1e18a86482bd1829
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/234079
Reviewed-by: Damien Neil <dneil@google.com>
In the upcoming 3.12.x release of protoc, the proto3 language will be
amended to support true presence for scalars. This CL adds support
to both the generator and runtime to support these semantics.
Newly added public API:
protogen.Plugin.SupportedFeatures
protoreflect.FieldDescriptor.HasPresence
protoreflect.FieldDescriptor.HasOptionalKeyword
protoreflect.OneofDescriptor.IsSynthetic
Change-Id: I7c86bf66d0ae56642109beb5f2132184593747ad
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/230698
Reviewed-by: Damien Neil <dneil@google.com>
To assist users in migrating from github.com/golang/protobuf
to google.golang.org/protobuf, make it such that functiionality like
proto.Marshal doesn't panic on nil interfaces.
Similar to how the new implementation treats a typed nil message
as an empty message, we treat a nil interface as being equivalent
to an "untyped" empty message.
Change-Id: Ic037f386f855b122f732b34d370e524b7c0d76f1
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/228837
Reviewed-by: Damien Neil <dneil@google.com>
An Is prefix implies it returns a boolean.
A Check prefix better suggests that it could return an error.
Change-Id: I6ffcb32099a944c656c07654c294a0980efb2d0e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/220338
Reviewed-by: Damien Neil <dneil@google.com>
Remove the generated proto packages that already exist in
google.golang.org/genproto. We want to eventually move these
packages here, but it doesn't need to happen yet.
Add a local copy of fieldmaskpb for use in tests.
Refactor proto generation to override import paths using the
M<source>=<import_path> compiler option instead of by patching the
source files.
Change-Id: I8d31f67e931d70140182f19f3e0106111f71c4b4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219598
Reviewed-by: Joe Tsai <joetsai@google.com>
Move Multiline and Indent to the top so that there is a separation
between options with semantic significance and those that are merely
for aesthetic purposes.
Change-Id: Icd5ee94ec010db8139a5e720f5b9842274fb3755
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219500
Reviewed-by: Damien Neil <dneil@google.com>
All unmarshaling error messages now contain line number and column
information, except for the following errors:
- `unexpected EOF`
- `no support for proto1 MessageSets`
- `required fields X not set`
Changes to internal/encoding/json:
- Moved encoding funcs in string.go and number.go into encode.go.
- Separated out encoding kind constants from decoding ones.
- Renamed file string.go to decode_string.go.
- Renamed file number.go to decode_number.go.
- Renamed Type struct to Kind.
- Renamed Value struct to Token.
- Token accessor methods no longer return error.
Name, Bool, ParsedString will panic if called on the wrong kind.
Float, Int, Uint has ok bool result to check against.
- Changed Peek to return Token and error.
Changes to encoding/protojson:
- Updated internal/encoding/json API calls.
- Added line info on most unmarshaling error messages and kept
description simple and consistent.
Change-Id: Ie50456694f2214c5c4fafd2c9b9239680da0deec
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/218978
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The Format function and MarshalOptions.Format method are helper
functions for directly obtaining the formatted string for a message
without having to deal with errors or convert a []byte to string.
It is only intended for human consumption (e.g., debugging or logging).
We also add a MarshalOptions.Multiline option to specify that the output
should use some default indentation in a multiline output.
This assists in the v1 to v2 migration where:
protoV1.CompactTextString(m) => prototext.MarshalOptions{}.Format(m)
protoV1.MarshalTextString(m) => prototext.Format(m)
At Google, there are approximately 10x more usages of MarshalTextString than
CompactTextString, so it makes sense that the top-level Format function
does multiline expansion by default.
Fixes#850
Change-Id: I149c9e190a6d99b985d3884df675499a3313e9b3
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/213460
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Herbie Ong <herbie@google.com>
The encoding/testprotos and reflect/protoregistry/testprotos are
accessible by other modules. Move them under internal/testprotos
to dissuade programmers who are too lazy to use their own test protos
when they need one.
Change-Id: I3dbfbce74e68ef033ec252bed076861cb47dd21e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/214341
Reviewed-by: Damien Neil <dneil@google.com>
Change tests which use private types registries to use the global one.
Except in cases where we want to explicitly test that the private
registry is used, it's simpler to use the global registry.
Change-Id: I998fb463b6beef91c7f5ce2ca2083251ae24d1db
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/199897
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Reviewed-by: Herbie Ong <herbie@google.com>
Suppose a oneof has N fields, the previous marshaling logic
would traverse every field checking for presence. This is O(N).
Using the protoreflect.Message.WhichOneof method, we can reduce
this to O(1). This optimization is exceptionally useful for oneofs
with a large number of fields.
Change-Id: I5f4aa8b1a899930f5c95e9cf1d68bac4b0b7884d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/196121
Reviewed-by: Herbie Ong <herbie@google.com>
Ironically, the "real" protobuf name of a group is not the name
of the field descriptor, but the message descriptor.
Change-Id: I26ab546a94e934766fa6af6252cacd294442a221
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/195780
Reviewed-by: Herbie Ong <herbie@google.com>
We change Unmarshal to reset a message by default.
* We add a Merge option to UnmarshalOptions for explicit merging.
* We speed up Reset by checking for the Reset method.
* Remove TODOs in prototext and protojson about reset behavior.
Fixesgolang/protobuf#890
Change-Id: Ibd8963c741053f564acf061fbdb846699942109c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/195457
Reviewed-by: Damien Neil <dneil@google.com>
This is meant to deter users from doing byte for byte comparison.
Change-Id: If005d2dc1eba45eaa4254171d2f247820db109e4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/194037
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
UseProtoNames=true uses proto field name in JSON field names.
Change-Id: I23249dc1787d9735bef780b1ef8d294a9c55c043
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/193998
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
