Unmarshaling of scalar, messages, repeated, and maps.
Need to further improve on error messages for consistency, some error
messages contain the position info while some currently do not. There
are cases where position info is wrong as well when a value is decoded
in another pass, e.g. numbers in string value, or map keys.
Change-Id: I6f9e903c499b5e87fb258dbdada7434389fc7522
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/166338
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The prototype package was initially used by generated reflection support,
but has now been replaced by internal/fileinit.
Eventually, this functionality should be deleted and re-written in terms
of other components in the repo.
Usages that prototype currently provides (but should be moved) are:
* Constructing standalone messages and enums, which is behavior we should
provide in reflect/protodesc. The google.protobuf.{Enum,Type} are well-known
proto messages designed for this purpose.
* Constructing placeholder files, enums, and messages.
* Consructing protoreflect.{Message,Enum,Extension}Types, which are protobuf
descriptors with associated Go type information.
Change-Id: Id7dbefff952682781b439aa555508c59b2629f9e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/167383
Reviewed-by: Damien Neil <dneil@google.com>
Collapse Value.Float32 and Value.Float64 into single API to keep it
consistent with Value.{Int,Uint}.
Change-Id: I07737e72715fe3cc3f6bcad579cf5d6cfe3757d5
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/167317
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Previous decoder decodes a JSON number into a float64, which lacks
64-bit integer precision.
I attempted to retrofit it with storing the raw bytes and parsed out
number parts, see golang.org/cl/164377. While that is possible, the
encoding logic for Value is not symmetrical with the decoding logic and
can be confusing since both utilizes the same Value struct.
Joe and I decided that it would be better to rewrite the JSON encoder
and decoder to be token-based instead, removing the need for sharing a
model type plus making it more efficient.
Change-Id: Ic0601428a824be4e20141623409ab4d92b6167c7
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/165677
Reviewed-by: Damien Neil <dneil@google.com>
We're rewriting internal/encoding/json. So, make a copy of it first in
order not to break encoding/jsonpb package.
Change-Id: I8b63c468d3f432102d2af4db22a7549998ce3876
Reviewed-on: https://go-review.googlesource.com/c/164642
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
This test examines the result of converting math.NaN() to a fixed byte
string. Change it to use a specific NaN value instead, since the value
returned by math.NaN is specified only as being a NaN, not a specific
one.
Use specific float32 and float64 NaN values, since the result of
converting a float64 NaN to a float32 can and does vary.
Fixes test failure on ARM.
Change-Id: Ia1517fdba768cdd88e5ee5f5af37f0b481e651b4
Reviewed-on: https://go-review.googlesource.com/c/162117
Reviewed-by: Herbie Ong <herbie@google.com>
When encoding/textpb marshals out float32 values, it was previously
formatting it as float64 bitsize since both float types are stored as
float64 and internal/encoding/text only has one Float type. A
consequence of this is that the output may display a different value
than expected, e.g. 1.02 becomes 1.0199999809265137.
This CL splits Float type into Float32 and Float64 to keep track of
which bitsize to use when formatting. Values of both types are still
stored as float64 to keep the logic simple.
Decoding will always use Float64, but users can ask for a float32 value
from it.
Change-Id: Iea5b14b283fec2236a0c3946fac34d4d79b95274
Reviewed-on: https://go-review.googlesource.com/c/158497
Reviewed-by: Damien Neil <dneil@google.com>
Add fields to the Message and Field builder structs which hold the value
of MessageOptions.map_entry, FieldOptions.packed, and FieldOptions.weak
options. Remove all access to the contents of options messages from the
prototype package.
Change IsPacked to always return false for unpackable field types,
which is consistent with the equivalent C++ API.
This change helps avoid dependency cycles between prototype and the
options messages. (Previously this was resolved by accessing options
with reflection, but just breaking the dependency from prototype to the
options message is cleaner and simpler.)
Change-Id: I756aefe2e04cfa8fea31eaaaa0b5a99d4ac9e851
Reviewed-on: https://go-review.googlesource.com/c/153517
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Logic for serializing the default value in textual form exists in
multiple places in trivially similar forms. Centralize that logic.
Change-Id: I4408ddfeef2c0dfa5c7468e01a4d4df5654ae57f
Reviewed-on: https://go-review.googlesource.com/c/153022
Reviewed-by: Herbie Ong <herbie@google.com>
In order to generate descriptor.proto, the generated code would want to depend
on the prototype package to construct the reflection data structures.
However, this is a problem since descriptor itself is one of the dependencies
for prototype. To break this dependency, we do the following:
* Avoid using concrete *descriptorpb.XOptions messages in the public API, and
instead just use protoreflect.ProtoMessage. We do lose some type safety here
as a result.
* Use protobuf reflection to interpret the Options message.
* Split out NewFileFromDescriptorProto into a separate protodesc package since
constructing protobuf reflection from the descriptor proto obviously depends
on the descriptor protos themselves.
As part of this CL, we check in a pre-generated version of descriptor and plugin
that supports protobuf reflection natively and switchover all usages of those
protos to the new definitions. These files were generated by protoc-gen-go
from CL/150074, but hand-modified to remove dependencies on the v1 proto runtime.
Change-Id: I81e03c42eeab480b03764e2fcbe1aae0e058fc57
Reviewed-on: https://go-review.googlesource.com/c/152020
Reviewed-by: Damien Neil <dneil@google.com>
Since detrand is an internal package, we can safely provide a function
that can be called to disable its functionality for testing purposes.
Change-Id: I26383e12a5832eb5af01952898a4c73f627d7aa5
Reviewed-on: https://go-review.googlesource.com/c/151678
Reviewed-by: Herbie Ong <herbie@google.com>
Add the scalar package to reduce dependencies on the v1 proto runtime package.
It may very well be the case that these functions should be exposed in the
public API of v2, but that is not a decision we need to make now.
Change-Id: Ifbc6d15311ba5837909ac72af47c630a80a142ef
Reviewed-on: https://go-review.googlesource.com/c/151402
Reviewed-by: Herbie Ong <herbie@google.com>
The use of math/rand in serialization is to provide some form of instability
to the output to provide a clear signal to the user that the should not
depend on the the property of stability. However, it is reasonable that users
expect the output for these to be deterministic.
As such, add a detrand package that provides deterministic, yet unstable
randomization functionality.
Since this package hashes the binary, it does impose a small initialization cost:
Benchmark 100000 20712 ns/op 480 B/op 6 allocs/op
Change-Id: I232d0fea1789a4278079837a67ee2f63474a4364
Reviewed-on: https://go-review.googlesource.com/c/151340
Reviewed-by: Herbie Ong <herbie@google.com>
Make output deliberately unstable so users don't rely on exactness.
For multi-line output, add another extra random space after <key>: for
at most one field per message.
-- example --
key1: field1
key2: {
foo: bar
}
For single-line output, add another extra random space after a field per
message.
-- example --
key1:field1 key2:{foo:bar}
Change-Id: I3ab25d4d970fdebb88bbd9dd8fa6d73af84338ea
Reviewed-on: https://go-review.googlesource.com/c/150977
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Add a corpus of generated protobuf messages generated at specific versions
of protoc-gen-go to ensure that we continue to support for generated messages
that have may never be updated.
Change-Id: I04a1b74306f471d7c99f5daf52399a5bd9adcbbc
Reviewed-on: https://go-review.googlesource.com/c/148831
Reviewed-by: Herbie Ong <herbie@google.com>
The bespoke text-serialization of field descriptors in protoc-gen-go is also
used in the legacy implementation of protobuf reflection to derive a
protoreflect.FieldDescriptor from legacy messages and also to convert to/from
protoreflect.ExtensionDescriptor and protoV1.ExtensionDesc.
Centralize this logic in a single place:
* to avoid reimplementing the same logic in internal/impl
* to keep the marshal and unmarshal logic co-located
Change-Id: I634c5afbb9dc6eda91d6cb6b0e68dbd724cb1ccb
Reviewed-on: https://go-review.googlesource.com/c/146758
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
Add a method to fetch descriptor options. Since options are proto
messages (e.g., google.protobuf.FieldOptions), and proto message
packages depend on the protoreflect package, returning the actual option
type would cause a dependency cycle. Instead, we return an interface
value which can be type asserted to the appropriate concrete type.
Add options support to the prototype package.
Some of the prototype constructors included fields (such as
Field.IsPacked) which represent information from the options
(such as google.protobuf.FieldOptions.packed). To avoid confusion about
the canonical source of information, drop these fields in favor of the
options.
Drop the unimplemented Descriptor.DescriptorOptionsProto.
Change-Id: I66579b6a7d10d99eb6977402a247306a78913e74
Reviewed-on: https://go-review.googlesource.com/c/144277
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
This change was created by running:
git ls-files | xargs sed -i "s|google.golang.org/proto|github.com/golang/protobuf/v2|g"
This change is *not* an endorsement of "github.com/golang/protobuf/v2" as the
final import path when the v2 API is eventually released as stable.
We continue to reserve the right to make breaking changes as we see fit.
This change enables us to host the v2 API on a repository that is go-gettable
(since go.googlesource.com is not a known host by the "go get" tool;
and google.golang.org/proto was just a stub URL that is not currently served).
Thus, we can start work on a forked version of the v1 API that explores
what it would take to implement v1 in terms of v2 in a backwards compatible way.
Change-Id: Ia3ebc41ac4238af62ee140200d3158b53ac9ec48
Reviewed-on: https://go-review.googlesource.com/136736
Reviewed-by: Damien Neil <dneil@google.com>
According to linguistics, this is actually a concrete syntax tree, rather
than an abstract syntax tree since it perfectly represents the grammatical
structure of the original raw input.
On the other hand, an abstract syntax tree (AST) loses some
grammatical structure and is only concerned with preserving syntax.
See https://eli.thegreenplace.net/2009/02/16/abstract-vs-concrete-syntax-trees/
Change-Id: Ia3fdb407d2b15c5431984956b7d74921891c2ad9
Reviewed-on: https://go-review.googlesource.com/133995
Reviewed-by: Herbie Ong <herbie@google.com>
Package pack enables manual encoding and decoding of protobuf wire data.
This package is intended only for testing and debugging purposes.
Message.Marshal is useful for hand-crafting raw wire testdata in tests
in a readable form.
Message.Unmarshal is useful for parsing raw wire data for debugging.
For that reason, effort was put into trying to get its string formatted
output look humanly readable.
High-level API:
type Number = wire.Number
const MinValidNumber Number = wire.MinValidNumber ...
type Type = wire.Type
const VarintType Type = wire.VarintType ...
type Token token
type Tag struct { ... }
type Bool bool
type Varint int64
type Svarint int64
type Uvarint uint64
type Int32 int32
type Uint32 uint32
type Float32 float32
type Int64 int64
type Uint64 uint64
type Float64 float64
type String string
type Bytes []byte
type LengthPrefix Message
type Denormalized struct { ... }
type Raw []byte
type Message []Token
func (Message) Size() int
func (Message) Marshal() []byte
func (*Message) Unmarshal(in []byte)
func (*Message) UnmarshalDescriptor(in []byte, desc protoreflect.MessageDescriptor)
func (Message) Format(s fmt.State, r rune)
Change-Id: Id99b340971a09c8a040838b155782a5d32b548bc
Reviewed-on: https://go-review.googlesource.com/129404
Reviewed-by: Herbie Ong <herbie@google.com>
Package text provides a parser and serializer for the proto text format.
This focuses on the grammar of the format and is agnostic towards specific
semantics of protobuf types.
High-level API:
func Marshal(v Value, indent string, delims [2]byte, outputASCII bool) ([]byte, error)
func Unmarshal(b []byte) (Value, error)
type Type uint8
const Bool Type ...
type Value struct{ ... }
func ValueOf(v interface{}) Value
func (v Value) Type() Type
func (v Value) Bool() (x bool, ok bool)
func (v Value) Int(b64 bool) (x int64, ok bool)
func (v Value) Uint(b64 bool) (x uint64, ok bool)
func (v Value) Float(b64 bool) (x float64, ok bool)
func (v Value) Name() (protoreflect.Name, bool)
func (v Value) String() string
func (v Value) List() []Value
func (v Value) Message() [][2]Value
func (v Value) Raw() []byte
Change-Id: I4a78ec4474c160d0de4d32120651edd931ea2c1e
Reviewed-on: https://go-review.googlesource.com/127455
Reviewed-by: Herbie Ong <herbie@google.com>
Package json provides a parser and serializer for the JSON format.
This focuses on the grammar of the format and is agnostic towards specific
semantics of protobuf types.
High-level API:
func Marshal(v Value, indent string) ([]byte, error)
func Unmarshal(b []byte) (Value, error)
type Type uint8
const Null Type ...
type Value struct{ ... }
func ValueOf(v interface{}) Value
func (v Value) Type() Type
func (v Value) Bool() bool
func (v Value) Number() float64
func (v Value) String() string
func (v Value) Array() []Value
func (v Value) Object() [][2]Value
func (v Value) Raw() []byte
Change-Id: I26422f6b3881ef1a11b8aa95160645b1384b27b8
Reviewed-on: https://go-review.googlesource.com/127824
Reviewed-by: Herbie Ong <herbie@google.com>
Some of the test cases in TestFixed64 actually belong in TestBytes.
Change-Id: I7f3efd77662881b64a96311161440fd220ae8074
Reviewed-on: https://go-review.googlesource.com/127456
Reviewed-by: Chris Manghane <cmang@golang.org>
This adds package wire, which provides low-level functionality for
marshaling and unmarshaling the protobuf wire format.
High-level API:
type Number int32
const MinValidNumber Number = 1 ...
type Type int8
const VarintType Type = 0 ...
func ParseError(n int) error
func ConsumeField(b []byte) (Number, Type, int)
func ConsumeFieldValue(num Number, typ Type, b []byte) (n int)
func ConsumeTag(b []byte) (Number, Type, int)
func ConsumeVarint(b []byte) (v uint64, n int)
func ConsumeFixed32(b []byte) (v uint32, n int)
func ConsumeFixed64(b []byte) (v uint64, n int)
func ConsumeBytes(b []byte) (v []byte, n int)
func ConsumeGroup(num Number, b []byte) (v []byte, n int)
func AppendTag(b []byte, num Number, typ Type) []byte
func AppendVarint(b []byte, v uint64) []byte
func AppendFixed32(b []byte, v uint32) []byte
func AppendFixed64(b []byte, v uint64) []byte
func AppendBytes(b []byte, v []byte) []byte
func AppendGroup(b []byte, num Number, v []byte) []byte
func SizeTag(num Number) int
func SizeVarint(v uint64) int
func SizeFixed32() int
func SizeFixed64() int
func SizeBytes(n int) int
func SizeGroup(num Number, n int) int
func DecodeBool(x uint64) bool
func DecodeTag(x uint64) (Number, Type)
func DecodeZigZag(x uint64) int64
func EncodeBool(x bool) uint64
func EncodeTag(num Number, typ Type) uint64
func EncodeZigZag(x int64) uint64
Change-Id: I052d8975414aeb182f6e9595c4736e716f1b7e9d
Reviewed-on: https://go-review.googlesource.com/127337
Reviewed-by: Chris Manghane <cmang@golang.org>
Run-TryBot: Chris Manghane <cmang@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
