Commit Graph

18 Commits

Author SHA1 Message Date
Joe Tsai
01ab29648e go.mod: rename google.golang.org/proto as github.com/golang/protobuf/v2
This change was created by running:
	git ls-files | xargs sed -i "s|google.golang.org/proto|github.com/golang/protobuf/v2|g"

This change is *not* an endorsement of "github.com/golang/protobuf/v2" as the
final import path when the v2 API is eventually released as stable.
We continue to reserve the right to make breaking changes as we see fit.

This change enables us to host the v2 API on a repository that is go-gettable
(since go.googlesource.com is not a known host by the "go get" tool;
and google.golang.org/proto was just a stub URL that is not currently served).
Thus, we can start work on a forked version of the v1 API that explores
what it would take to implement v1 in terms of v2 in a backwards compatible way.

Change-Id: Ia3ebc41ac4238af62ee140200d3158b53ac9ec48
Reviewed-on: https://go-review.googlesource.com/136736
Reviewed-by: Damien Neil <dneil@google.com>
2018-09-24 16:11:50 +00:00
Joe Tsai
ded92f8033 reflect/prototype: lower minimum support to Go1.9
The minimum supported version is currently Go1.10 since we use strings.Builder
for a fairly significant optimization when constructing all of the descriptor
full names.

The strings.Builder implementation is not particularly complicated,
so just fork it into our code. The golang/protobuf and the golang/go
projects share the same authors and copyright license.

Change-Id: Ibb9519dbe756327a07369f10f80c15761002b5e7
Reviewed-on: https://go-review.googlesource.com/136735
Reviewed-by: Damien Neil <dneil@google.com>
2018-09-21 19:30:05 +00:00
Joe Tsai
fa02f4eaa6 internal/impl: initial commit
This provides an implementation of the has, get, set, clear methods for each
field in a message. The approach taken here is similar to the table-driven
implementation in the current v1 proto package.

The pointer_reflect.go and pointer_unsafe.go files are a simplified version of
the same files in the v1 implementation. They provide a pointer abstraction
that enables a high-efficiency approach in a non-purego environment.
The unsafe fast-path is not implemented in this commit.

This commit only implements the accessor methods for scalars using pure
Go reflection.

Change-Id: Icdf707e9d4e3385e55434f93b30a341a7680ae11
Reviewed-on: https://go-review.googlesource.com/135136
Reviewed-by: Damien Neil <dneil@google.com>
2018-09-13 20:23:15 +00:00
Joe Tsai
8cb9190264 internal/encoding/pack: replace AST with CST in documentation
According to linguistics, this is actually a concrete syntax tree, rather
than an abstract syntax tree since it perfectly represents the grammatical
structure of the original raw input.

On the other hand, an abstract syntax tree (AST) loses some
grammatical structure and is only concerned with preserving syntax.

See https://eli.thegreenplace.net/2009/02/16/abstract-vs-concrete-syntax-trees/

Change-Id: Ia3fdb407d2b15c5431984956b7d74921891c2ad9
Reviewed-on: https://go-review.googlesource.com/133995
Reviewed-by: Herbie Ong <herbie@google.com>
2018-09-11 00:29:57 +00:00
Damien Neil
3de06c0868 reflect: add Index method to descriptor
Useful for dealing with SourceCodeInfo location paths, which identify
entities by their index.

Change-Id: I2034fc06b14c9b29b26e356fad21e106f63fbd14
Reviewed-on: https://go-review.googlesource.com/134115
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-09-07 17:57:37 +00:00
Joe Tsai
769b9e720b internal/cmd/pbdump: simplify and cleanup
Minor changes:
* Use spaces instead of tabs in help printout since each system may have
a different tab width.
* Reduce repetition for field flags.
* Print to stdout instead of stderr for flag usages as is customary for
most other Unix command line tools.

Change-Id: I4544dbf46ee150d552a8fdfe72683d0a1aa7f0ba
Reviewed-on: https://go-review.googlesource.com/133755
Reviewed-by: Herbie Ong <herbie@google.com>
2018-09-06 18:36:20 +00:00
Joe Tsai
b4e370ef3a internal/cmd/pbdump: initial commit
The pbdump binary is an internal-only command line tool that is similar
to the hexdump tool in Linux, but is tailored towards the protocol buffer
wire format. This tool is not necessary for the new API, but is very useful
for debugging the wire format.

Change-Id: Ie688338d33b01aee5dd88a63606ec0ffce57741d
Reviewed-on: https://go-review.googlesource.com/129405
Reviewed-by: Herbie Ong <herbie@google.com>
2018-09-04 03:30:39 +00:00
Joe Tsai
3ab648c9a8 internal/encoding/pack: initial commit
Package pack enables manual encoding and decoding of protobuf wire data.
This package is intended only for testing and debugging purposes.
Message.Marshal is useful for hand-crafting raw wire testdata in tests
in a readable form.
Message.Unmarshal is useful for parsing raw wire data for debugging.
For that reason, effort was put into trying to get its string formatted
output look humanly readable.

High-level API:
	type Number = wire.Number
		const MinValidNumber Number = wire.MinValidNumber ...
	type Type = wire.Type
		const VarintType Type = wire.VarintType ...

	type Token token
	type Tag struct { ... }
	type Bool bool
	type Varint int64
	type Svarint int64
	type Uvarint uint64
	type Int32 int32
	type Uint32 uint32
	type Float32 float32
	type Int64 int64
	type Uint64 uint64
	type Float64 float64
	type String string
	type Bytes []byte
	type LengthPrefix Message
	type Denormalized struct { ... }
	type Raw []byte

	type Message []Token
		func (Message) Size() int
		func (Message) Marshal() []byte
		func (*Message) Unmarshal(in []byte)
		func (*Message) UnmarshalDescriptor(in []byte, desc protoreflect.MessageDescriptor)
		func (Message) Format(s fmt.State, r rune)

Change-Id: Id99b340971a09c8a040838b155782a5d32b548bc
Reviewed-on: https://go-review.googlesource.com/129404
Reviewed-by: Herbie Ong <herbie@google.com>
2018-08-30 21:46:37 +00:00
Joe Tsai
bfda014ecd reflect/prototype: initial commit
The prototype package provides constructors to create protobuf types that
implement the interfaces defined in the protoreflect package.

High-level API:
	func NewFile(t *File) (protoreflect.FileDescriptor, error)
	type File struct{ ... }
	type Message struct{ ... }
	type Field struct{ ... }
	type Oneof struct{ ... }
	type Enum struct{ ... }
	type EnumValue struct{ ... }
	type Extension struct{ ... }
	type Service struct{ ... }
	type Method struct{ ... }

	func NewEnum(t *StandaloneEnum) (protoreflect.EnumDescriptor, error)
	func NewMessage(t *StandaloneMessage) (protoreflect.MessageDescriptor, error)
	func NewExtension(t *StandaloneExtension) (protoreflect.ExtensionDescriptor, error)
	type StandaloneEnum struct{ ... }
	type StandaloneMessage struct{ ... }
	type StandaloneExtension struct{ ... }

	func PlaceholderFile(path string, pkg protoreflect.FullName) protoreflect.FileDescriptor
	func PlaceholderEnum(name protoreflect.FullName) protoreflect.EnumDescriptor
	func PlaceholderMessage(name protoreflect.FullName) protoreflect.MessageDescriptor

This CL is missing some features that are to be added later:
* The stringer methods are not implemented, providing no way to print the
descriptors in a humanly readable manner.
* There is no support for proto options or retrieving the raw descriptor.
* There are constructors for Go specific types (e.g., protoreflect.MessageType).

We drop go1.9 support since we use strings.Builder.
We switch to go.11rc1 to obtain some bug fixes for modules.

Change-Id: Ieac9a2530afc81e5a5bb9ab5816804372f652b18
Reviewed-on: https://go-review.googlesource.com/129057
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Herbie Ong <herbie@google.com>
2018-08-15 18:44:04 +00:00
Joe Tsai
d55639e713 internal/set: add set package for set data structures
Package set provides simple set data structures for uint64 and string types.

High-level API:
	type Set(T {}) xxx
	func (Set) Len() int
	func (Set) Has(T) bool
	func (Set) Set(T)
	func (Set) Clear(T)

These data structures are useful for implementing required fields efficiently
or ensuring that protobuf identifiers do not conflict.

Change-Id: If846630a9034909a43121b3e0f6720275f4b7aaf
Reviewed-on: https://go-review.googlesource.com/128898
Reviewed-by: Chris Manghane <cmang@golang.org>
2018-08-09 21:46:35 +00:00
Joe Tsai
27c2a76c85 internal/encoding/text: initial commit of proto text format parser/serializer
Package text provides a parser and serializer for the proto text format.
This focuses on the grammar of the format and is agnostic towards specific
semantics of protobuf types.

High-level API:
	func Marshal(v Value, indent string, delims [2]byte, outputASCII bool) ([]byte, error)
	func Unmarshal(b []byte) (Value, error)
	type Type uint8
		const Bool Type ...
	type Value struct{ ... }
		func ValueOf(v interface{}) Value
		func (v Value) Type() Type
		func (v Value) Bool() (x bool, ok bool)
		func (v Value) Int(b64 bool) (x int64, ok bool)
		func (v Value) Uint(b64 bool) (x uint64, ok bool)
		func (v Value) Float(b64 bool) (x float64, ok bool)
		func (v Value) Name() (protoreflect.Name, bool)
		func (v Value) String() string
		func (v Value) List() []Value
		func (v Value) Message() [][2]Value
		func (v Value) Raw() []byte

Change-Id: I4a78ec4474c160d0de4d32120651edd931ea2c1e
Reviewed-on: https://go-review.googlesource.com/127455
Reviewed-by: Herbie Ong <herbie@google.com>
2018-08-07 22:44:06 +00:00
Joe Tsai
879b18d902 internal/encoding/json: initial commit of JSON parser/serializer
Package json provides a parser and serializer for the JSON format.
This focuses on the grammar of the format and is agnostic towards specific
semantics of protobuf types.

High-level API:
	func Marshal(v Value, indent string) ([]byte, error)
	func Unmarshal(b []byte) (Value, error)
	type Type uint8
	    const Null Type ...
	type Value struct{ ... }
	    func ValueOf(v interface{}) Value
		func (v Value) Type() Type
		func (v Value) Bool() bool
		func (v Value) Number() float64
		func (v Value) String() string
		func (v Value) Array() []Value
		func (v Value) Object() [][2]Value
		func (v Value) Raw() []byte

Change-Id: I26422f6b3881ef1a11b8aa95160645b1384b27b8
Reviewed-on: https://go-review.googlesource.com/127824
Reviewed-by: Herbie Ong <herbie@google.com>
2018-08-07 22:40:28 +00:00
Joe Tsai
ce91da5f05 internal/pragma: package to statically enforce certain properties
Package pragma provides certain constructs that can be used to statically enforce
certain language properties.

Change-Id: I58a87bf7429f3be1b5aae418595fadda32f9d039
Reviewed-on: https://go-review.googlesource.com/127822
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Herbie Ong <herbie@google.com>
2018-08-06 21:47:01 +00:00
Joe Tsai
0e6baaa2cb internal/encoding/wire: switch to use internal errors package
Change-Id: Ibb01f8a5d4dea09c19262f807e305bbebe0e6ab8
Reviewed-on: https://go-review.googlesource.com/127825
Reviewed-by: Herbie Ong <herbie@google.com>
2018-08-04 01:02:24 +00:00
Joe Tsai
025d81dac6 internal/flags: add package to control specialized functionality
Package flags defines a set of constants to control support for certain features.

Other flags that may be added in the future may include:
	PureGo: to control whether protobuf code should avoid using the unsafe
		package and only use Go reflection.
	PureProto: to control whether the proto package ignores the protoiface
		package and only uses protobuf reflection. This is useful for testing.
	Proto2ValidateUTF8: whether to validate UTF-8 in proto2 strings.
	Proto3ValidateUTF8: whether to validate UTF-8 in proto3 strings.

Change-Id: Ibcb8dd8b3e977633b8a4e4a22a0617f2eebcc325
Reviewed-on: https://go-review.googlesource.com/127820
Reviewed-by: Damien Neil <dneil@google.com>
2018-08-03 22:59:53 +00:00
Joe Tsai
be60f99ca7 internal/encoding/wire: fix minor miscategorization in test
Some of the test cases in TestFixed64 actually belong in TestBytes.

Change-Id: I7f3efd77662881b64a96311161440fd220ae8074
Reviewed-on: https://go-review.googlesource.com/127456
Reviewed-by: Chris Manghane <cmang@golang.org>
2018-08-02 21:53:01 +00:00
Joe Tsai
9834a7df51 internal/errors: initial commit
This adds package errors, which provides error handling constructs that are
specific to the protobuf repo. In particular, it provides a NonFatal type
which is useful for capturing non-fatal errors like invalid UTF-8 and required
fields not set.

High-level API:
	type NonFatalErrors []error
		func (NonFatalErrors) Error() string
	type NonFatal struct{ ... }
		func (*NonFatal) Merge(err error) (ok bool)
		func (*NonFatal) AppendInvalidUTF8(field string)
		func (*NonFatal) AppendRequiredNotSet(field string)
	func New(string, ...interface{}) error

Change-Id: I9448c586008240e8987573fe79e0ffb024e7629d
Reviewed-on: https://go-review.googlesource.com/127338
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Herbie Ong <herbie@google.com>
2018-08-02 18:28:47 +00:00
Joe Tsai
22505a4929 internal/encoding/wire: initial commit
This adds package wire, which provides low-level functionality for
marshaling and unmarshaling the protobuf wire format.

High-level API:
	type Number int32
	    const MinValidNumber Number = 1 ...
	type Type int8
	    const VarintType Type = 0 ...
	func ParseError(n int) error

	func ConsumeField(b []byte) (Number, Type, int)
	func ConsumeFieldValue(num Number, typ Type, b []byte) (n int)
	func ConsumeTag(b []byte) (Number, Type, int)
	func ConsumeVarint(b []byte) (v uint64, n int)
	func ConsumeFixed32(b []byte) (v uint32, n int)
	func ConsumeFixed64(b []byte) (v uint64, n int)
	func ConsumeBytes(b []byte) (v []byte, n int)
	func ConsumeGroup(num Number, b []byte) (v []byte, n int)
	func AppendTag(b []byte, num Number, typ Type) []byte
	func AppendVarint(b []byte, v uint64) []byte
	func AppendFixed32(b []byte, v uint32) []byte
	func AppendFixed64(b []byte, v uint64) []byte
	func AppendBytes(b []byte, v []byte) []byte
	func AppendGroup(b []byte, num Number, v []byte) []byte
	func SizeTag(num Number) int
	func SizeVarint(v uint64) int
	func SizeFixed32() int
	func SizeFixed64() int
	func SizeBytes(n int) int
	func SizeGroup(num Number, n int) int

	func DecodeBool(x uint64) bool
	func DecodeTag(x uint64) (Number, Type)
	func DecodeZigZag(x uint64) int64
	func EncodeBool(x bool) uint64
	func EncodeTag(num Number, typ Type) uint64
	func EncodeZigZag(x int64) uint64

Change-Id: I052d8975414aeb182f6e9595c4736e716f1b7e9d
Reviewed-on: https://go-review.googlesource.com/127337
Reviewed-by: Chris Manghane <cmang@golang.org>
Run-TryBot: Chris Manghane <cmang@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2018-08-01 23:02:07 +00:00