This change removes most usages of Syntax() from the repository and uses
edition features for instead. The appropriate edition feature defaults are
loaded for proto2/proto3 when the initialization of the descriptors
start.
All of these changes were tested on the Google corpus.
Change-Id: Ieca076a2b38ca8e50e084cd32e725b7b3dcb4171
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/572435
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
In internal/encoding/json, report an error when encountering a }
when we are expecting an object field value. For example, the input
`{"":}` now correctly results in an error at the closing } token.
In encoding/protojson, check for an unexpected EOF token in
skipJSONValue. This is redundant with the check in internal/encoding/json,
but adds a bit more defense against any other similar bugs that
might exist.
Fixes CVE-2024-24786
Change-Id: I03d52512acb5091c8549e31ca74541d57e56c99d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/569356
TryBot-Bypass: Damien Neil <dneil@google.com>
Reviewed-by: Roland Shoemaker <roland@golang.org>
Commit-Queue: Damien Neil <dneil@google.com>
Adds MarshalAppend methods to allow for byte slices to be reused.
Copies signature from the binary encoding.
Small changes to internal json and text libraries to use strconv
AppendInt and AppendUint for number encoding.
Change-Id: Ife7c8979c1c153a0a0bf9b70b296b8158d38dffc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/489615
Reviewed-by: Edward McFarlane <emcfarlane000@gmail.com>
Reviewed-by: Joseph Tsai <joetsai@digital-static.net>
Reviewed-by: Lasse Folger <lassefolger@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
Fix a panic when parsing the incomplete negative number "- ".
Fixesgolang/protobuf#1530
Change-Id: Iba5e8ee68d5f7255c28f1a74f31beee36c9ed847
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/475995
Reviewed-by: Lasse Folger <lassefolger@google.com>
Run-TryBot: Damien Neil <dneil@google.com>
The text format specification[1] indicates that whitespace and comments
may appear after a minus sign and before the subsequent numeric component
in negative number literals. But the Go implementation does not allow
this.
This brings the Go implementation info conformance with this aspect.
Fixesgolang/protobuf#1526
[1] https://protobuf.dev/reference/protobuf/textformat-spec/#parsing
Change-Id: I3996c89ee9d37cf2b7502fc6736d6e2ed6dbcf43
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/473015
Reviewed-by: Lasse Folger <lassefolger@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
Fix typos in comments and unexported function name.
Change-Id: I239fff85bc68d7f0ef6f8853acdc9bbe19d86304
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/466535
Reviewed-by: Lasse Folger <lassefolger@google.com>
Reviewed-by: Michael Stapelberg <stapelberg@google.com>
Reviewed-by: Joseph Tsai <joetsai@digital-static.net>
Apply go1.19 gofmt to non-generated files.
Generated .pb.go files are created by generate.bash using Go 1.18,
so leave them unchanged for now.
Change-Id: Ied36c83cf99704988d059bf0412e677f0fbc71b0
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/418676
Reviewed-by: Ian Lance Taylor <iant@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
Correctly distinguish between 0 and -0 in tests.
Change-Id: I41f6702a903ca0cf9edf9e95be812a3a29930978
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/406917
Reviewed-by: Lasse Folger <lassefolger@google.com>
Avoid very long errors returned by limiting the length of what errId
returns to 32 bytes (the value is chosen so that the error will not
be too long yet useful).
Append ellipsis to the returned value to denote that it was truncated.
Change-Id: I232d5192a2d9ad675daa0be0fe0c8518489c2953
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/406694
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
This eliminates the last user of the regexp package, which should save
about 130K from the resulting stripped binary importing this package
(unless, of course, regexp is brought in directly of via another
dependency).
Added some new cases to TestDecoder to test the new function.
Benchmark (not included) shows the following results, comparing to
old implementation using regexp.Find:
name old time/op new time/op delta
ErrId-4 1.93µs ± 1% 0.21µs ± 1% -89.20% (p=0.002 n=6+6)
name old alloc/op new alloc/op delta
ErrId-4 128B ± 0% 0B -100.00% (p=0.002 n=6+6)
name old allocs/op new allocs/op delta
ErrId-4 13.0 ± 0% 0.0 -100.00% (p=0.002 n=6+6)
Change-Id: I5569a47580f41cc60f92c444e8d43bb3f26faa4e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/402774
Reviewed-by: Cassondra Foesch <cfoesch@gmail.com>
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
The protopath package provides a means to programmatically represent
a sequence of protobuf reflection operations.
The protorange package traverses through a message and
calls a user-provided function as it iterates.
This feature sets the groundwork for the often requested feature
of being able to exclude certain fields when merging or serializing.
package protopath
type Path []Step
type Step struct{ ... }
func Root(protoreflect.MessageDescriptor) Step
func FieldAccess(protoreflect.FieldDescriptor) Step
func UnknownAccess() Step
func ListIndex(int) Step
func MapIndex(protoreflect.MapKey) Step
func AnyExpand(protoreflect.MessageDescriptor) Step
func (Step) Kind() StepKind
func (Step) FieldDescriptor() protoreflect.FieldDescriptor
func (Step) MessageDescriptor() protoreflect.MessageDescriptor
func (Step) ListIndex() int
func (Step) MapIndex() protoreflect.MapKey
func (Step) String() string
type StepKind int
const RootStep StepKind
const FieldAccessStep StepKind
const UnknownAccessStep StepKind
const ListIndexStep StepKind
const MapIndexStep StepKind
const AnyExpandStep StepKind
type Values struct {
Path Path
Values []protoreflect.Value
}
func (Values) Index(int) (out struct{ ... })
func (Values) Len() int
func (Values) String() string
package protorange
var Break error
var Terminate error
func Range(protoreflect.Message, func(protopath.Values) error) error
type Options struct {
Stable bool
Resolver interface { ... }
}
func (Options) Range(m protoreflect.Message, push, pop func(protopath.Values) error) error
Change-Id: I29cbd5142fe169d78367d54a95d37801888b64f4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/236540
Trust: Joe Tsai <joetsai@digital-static.net>
Reviewed-by: Damien Neil <dneil@google.com>
Escape not only ASCII control characters, but Unicode as well.
Change-Id: I5f5791ae51fc5624599f66ce012ecef364e7ea97
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/262682
Trust: Damien Neil <dneil@google.com>
Run-TryBot: Damien Neil <dneil@google.com>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Reviewed-by: Joe Tsai <joetsai@google.com>
We already escape ASCII control characters in the range [0x00,0x20).
Escape the one control character outside this range as well.
Change-Id: If954da0d4a178b36128d1a53d25397d1b3fd2e17
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/262681
Trust: Damien Neil <dneil@google.com>
Run-TryBot: Damien Neil <dneil@google.com>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Add a new TextName accessor that returns the field name that should
be used for the text format. It is usually just the field name, except:
1) it uses the inlined message name for groups,
2) uses the full name surrounded by brackets for extensions, and
3) strips the "message_set_extension" for well-formed extensions
to the proto1 MessageSet.
We make similar adjustments to the JSONName accessor so that it applies
similar semantics for extensions.
The two changes simplifies all logic that wants the humanly readable
name for a field.
Change-Id: I524b6e017fb955146db81819270fe197f8f97980
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/239838
Reviewed-by: Herbie Ong <herbie@google.com>
Fix the tests to work on GOARCH=386.
Also add another test suite that tests on GOARCH=386.
Change-Id: I41290998053dc059347adb8af70012dc5cc31811
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/241537
Reviewed-by: Herbie Ong <herbie@google.com>
Centralize the MessageSet extension resolution logic in the registry.
This avoids needless replication of this exact logic in multiple places
(for JSON and text) and elsewhere.
Change-Id: I70bfea899e295e8c589f418965bf0dd099f93628
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/240077
Reviewed-by: Herbie Ong <herbie@google.com>
When combining multiple message fields in a MessageSet item (a case
which should never happen in practice), unmarshal could modify the input
data. Fix it to not do so. Add a general check to ensure that unmarshal
operations don't modify the input.
Change-Id: Idde46e6132a1dc96c374f9146efff81783c3bef3
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/223818
Reviewed-by: Joe Tsai <joetsai@google.com>
For historical reasons, MessageSets items are allowed to have field
numbers outside the usual valid range. Detect the case where the field
number cannot fit in an int32 and report an error. Also check for
a field number of 0 (always invalid).
Handle the case where a MessageSet item includes an unknown field.
We have no place to put the contents of the field, so drop it. This is,
I believe, consistent with other implementations.
Change-Id: Ic403427e1c276cbfa232ca577e7a799cce706bc7
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/221939
Reviewed-by: Herbie Ong <herbie@google.com>
Remove a stray bit of punctuation that crept into one of the license
headers and got copied around everywhere.
Change-Id: Iebe4e882650ab6dab28f132b5e324e2ab0b99a73
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/220339
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
All unmarshaling error messages now contain line number and column
information, except for the following errors:
- `unexpected EOF`
- `no support for proto1 MessageSets`
- `required fields X not set`
Changes to internal/encoding/json:
- Moved encoding funcs in string.go and number.go into encode.go.
- Separated out encoding kind constants from decoding ones.
- Renamed file string.go to decode_string.go.
- Renamed file number.go to decode_number.go.
- Renamed Type struct to Kind.
- Renamed Value struct to Token.
- Token accessor methods no longer return error.
Name, Bool, ParsedString will panic if called on the wrong kind.
Float, Int, Uint has ok bool result to check against.
- Changed Peek to return Token and error.
Changes to encoding/protojson:
- Updated internal/encoding/json API calls.
- Added line info on most unmarshaling error messages and kept
description simple and consistent.
Change-Id: Ie50456694f2214c5c4fafd2c9b9239680da0deec
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/218978
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Add a sentinel proto.Error error which matches all errors returned by
packages in this module.
Document that protoregistry.NotFound is an exact sentinel value for
performance reasons.
Add a Wrap function to the internal/errors package and use it to wrap
errors from outside sources (resolvers). Wrapped errors match
proto.Error.
Fixesgolang/protobuf#1021.
Change-Id: I45567df3fd6c8dc9a5caafdb55654827f6fb1941
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/215338
Reviewed-by: Joe Tsai <joetsai@google.com>
For large positive integers, add check for number of decimal digits
before converting number to plain integer w/o exponent.
If exponent value is large, previous implementation may end up
constructing a large string with lots of zeroes that is not useful as it
will fail later on when called with strconv.Parse{Uint,Int} anyways.
Fixesgolang/protobuf#1002.
Change-Id: I65bfad304401e076743853d7501786b7231b083b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/213717
Reviewed-by: Damien Neil <dneil@google.com>
The JSONName constructor returns a struct value which shallow copies
a sync.Once within it; this is a dubious pattern.
Instead, add a jsonName.Init method to initialize the value.
Change-Id: I190a7239b1b62a8041ee7e4e09c0fe37b64ff623
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/213237
Reviewed-by: Damien Neil <dneil@google.com>
The deprecated messageset format permits extension fields with numbers
greater than the usual maximum (1<<29-1). To support this, the
internal/encoding/wire package has disabled field number validation when
legacy support is enabled.
We shouldn't skip validating all field numbers for validity just because
we support larger ones in messagesets.
This change drops range validation from the wire package (other than
checking that numbers fit in an int32) and adds it to the wire
unmarshalers instead. This gives us validation where we care
about it (when unmarshaling a wire-format message) and allows for
best-effort handling of out-of-range numbers everywhere else.
Fixesgolang/protobuf#996
Change-Id: I4e11b8a8aa177dd60e89723570af074a317c2451
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/210290
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
In the v1 implementation, unknown MessageSet items are stored in a
message's unknown fields section in non-MessageSet format. For example,
consider a MessageSet containing an item with type_id T and value V.
If the type_id is not resolvable, the item will be placed in the unknown
fields as a bytes-valued field with number T and contents V. This
conversion is then reversed when marshaling a MessageSet containing
unknown fields.
Preserve this behavior in v2.
One consequence of this change is that actual unknown fields in a
MessageSet (any field other than 1) are now discarded. This matches
the previous behavior.
Change-Id: I3d913613f84e0ae82481078dbc91cb25628651cc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/205697
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
This is more consistent with the indent documentation:
If indent is a non-empty string, it causes every entry in a List or Message
to be preceded by the indent and trailed by a newline.
Since an empty message has no entries, there should be no newlines.
Change-Id: I5d57165aaf94ca6b184bb35bf05d5d68f5ee9dd5
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/194877
Reviewed-by: Herbie Ong <herbie@google.com>
This is meant to deter users from doing byte for byte comparison.
Change-Id: If005d2dc1eba45eaa4254171d2f247820db109e4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/194037
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Simply move logic into similar code block. Maintains the same logic.
Change-Id: I7b5a3f3d57f6102c7919cdc03dd105f08d21aca3
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/194039
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Descriptor methods generally return a Descriptor with no Go type
information. ExtensionType's Descriptor is an exception, returning an
ExtensionTypeDescriptor containing both the proto descriptor and a
reference back to the ExtensionType. The pure descriptor is accessed
by xt.Descriptor().Descriptor().
Rename ExtensionType's Descriptor method to TypeDescriptor to make it
clear that it behaves a bit differently.
Change 1/2: Add the TypeDescriptor method and deprecate Descriptor.
Change-Id: I1806095044d35a474d60f94d2a28bdf528f12238
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/192139
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
