This is step 2 of 6 in a multi-stage migration
to move the well-known types
from the google.golang.org/genproto module
to the google.golang.org/protobuf module.
The generated Go packages for field_mask.proto, api.proto,
type.proto, and source_context.proto are being moved over
to this module alongside all the other well-known types.
In order to move these types between two modules,
there needs to be a sequence of changes submitted in
decently rapid succession. It is impossible to atomically
make these changes, so a brief breakage is inevitable.
The steps are as follows:
Step 1: Submit a change to cloud.google.com/go/internal/gapicgen
to avoid generating the well-known types. Otherwise, the tool
will undo the changes made in step 3.
See https://code-review.googlesource.com/c/gocloud/+/56810
Step 2: Submit a change to google.golang.org/protobuf that
adds the generated well-known types being migrated to that module.
In order to prevent the situation where a user links in
too old a version of the genproto module such that
duplicate registration occurs for the well-known types,
the registry is specially modified to provide an error
message that instructs users to upgrade the genproto module.
See https://golang.org/cl/234937
Step 3: Submit a change to google.golang.org/genproto that
switches all generated well-known types to be aliases to the
ones declared in google.golang.org/protobuf from the previous step.
This will cause the genproto module to incur an dependency
on an unreleased version of the protobuf module.
See https://github.com/googleapis/go-genproto/pull/372
Step 4: Submit a change to google.golang.org/protobuf that
adds a weak module depdency on the genproto module at the
revision from the previous step.
Step 5: Release google.golang.org/protobuf@v1.24.0.
Step 6: Submit a change to google.golang.org/genproto that
updates the protobuf module dependency to v1.24.0.
Change-Id: I36a19049d2240b67a37dfad20e154505aee7c784
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/234937
Reviewed-by: Damien Neil <dneil@google.com>
The following changes are made:
* Permit invalid UTF-8 in proto2. This goes against specified behavior,
but matches functional behavior in wire marshaling (not just for Go,
but also in the other major language implementations as well).
* The Format function is specified as ignoring errors since its intended
purpose is to surface information to the human user even if it's not
exactly parsible back into a message. As such, add an unexported
allowInvalidUTF8 option that is specially used by Format.
* Add an EmitASCII option that forces the formatting of
strings and bytes to always be encoded as ASCII.
This ensures that the entire output is always ASCII as well.
Note that we do not replicate this behavior for protojson since:
* The JSON format fundamentally has a stricter and well-specified
grammar for exactly what is valid/invalid, while the text format
has not had a well-specified grammar for the longest time,
leading to all sorts of weird usages due to Hyrum's law.
* This is to ease migration from the legacy implementation,
which did permit invalid UTF-8 in proto2.
* The EmitASCII option relies on the ability to always escape
Unicode characters using ASCII escape sequences, but this is not
possible in JSON since the grammar only has an escape sequence defined
for Unicode characters \u0000 to \uffff, inclusive.
However, Unicode v12.0.0 defines characters up to \U0010FFFF,
which is beyond what the JSON grammar provides escape sequences for.
Change-Id: I2b524a904e9ec59f9ed5500e299613bc27c31a14
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/233077
Reviewed-by: Herbie Ong <herbie@google.com>
Cleanup the generated logic by having the implementation be backed
by protoimpl rather that directly generated.
Weak fields are a deprecated feature of protobufs and
have entirely be superceded by extensions.
Unfortunately, there are still some usages of it.
Change-Id: Ie1a4b7da253e2ccf5e56627775d9b2fb4090d59a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/229717
Reviewed-by: Damien Neil <dneil@google.com>
In the upcoming 3.12.x release of protoc, the proto3 language will be
amended to support true presence for scalars. This CL adds support
to both the generator and runtime to support these semantics.
Newly added public API:
protogen.Plugin.SupportedFeatures
protoreflect.FieldDescriptor.HasPresence
protoreflect.FieldDescriptor.HasOptionalKeyword
protoreflect.OneofDescriptor.IsSynthetic
Change-Id: I7c86bf66d0ae56642109beb5f2132184593747ad
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/230698
Reviewed-by: Damien Neil <dneil@google.com>
This fixes the conformance test failures which occur because the
conformance protos have not been regenerated.
The generator script has been modified with a sanity check that
files do not exist outside the expected sub-tree.
Change-Id: I473efec4a016f6bc96ddf7e20d54bcf5ff9b55fe
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/223538
Reviewed-by: Damien Neil <dneil@google.com>
This introduces some conformance test failures,
which occur not because our implementation changed behavior,
but because new cases were added.
Future work will be to investigate these failuress.
Change-Id: Ifb17465883c417acd46865744572f8cd0c858383
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/222857
Reviewed-by: Damien Neil <dneil@google.com>
Under some rare circumstances registration can deadlock
when lazy descriptor initialization consults the registry.
Move the call triggering the lazy init out of the critical section.
Fixesgolang/protobuf#1052.
Change-Id: Ic266e06b0db99fea65e797b879ce53e5342fff95
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/204804
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Fix a few files with no or unusual copyright headers.
Manually add copyright headers to the legacy testprotos, which will
(probably) never be regenerated.
Change-Id: Ifb52fa0eb59bf9d2de40b8df7e581a3c9731c883
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/220498
Reviewed-by: Joe Tsai <joetsai@google.com>
Remove the generated proto packages that already exist in
google.golang.org/genproto. We want to eventually move these
packages here, but it doesn't need to happen yet.
Add a local copy of fieldmaskpb for use in tests.
Refactor proto generation to override import paths using the
M<source>=<import_path> compiler option instead of by patching the
source files.
Change-Id: I8d31f67e931d70140182f19f3e0106111f71c4b4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/219598
Reviewed-by: Joe Tsai <joetsai@google.com>
Change the protoc flags such that when one of our test .proto files
imports another, the filename is consistently specified relative to the
module root.
Change-Id: I690282795cef23347c8794c1c6357e4fe9560d8a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/217762
Reviewed-by: Joe Tsai <joetsai@google.com>
The proto package tests often test several variations of messages with a
similar shape. For example, most tests are performed with a proto2
message with a regular field, a proto2 message with an extension field,
and a proto3 message.
Add a protobuild package which can initialize all these variations from
a single template. For example, these three messages:
&testpb.TestAllTypes{OptionalInt32: proto.Int32(1)}
&test3pb.TestAllTypes{OptionalInt32: 1}
m := &testpb.TestAllExtensions{}
proto.SetExtension(m, &testpb.E_OptionalInt32, 1)
can all be constructed from the template:
protobuild.Message{"optional_int32": 1}
This reduces redundancy in tests and will make it more practical to
test alternative code generators.
Change-Id: I3245a4bf74ee1bce957bc772fed513d427720677
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/217457
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Tweak the test message to allow creating messages with extensions that
contain extensions that contain extensions, etc.
Change-Id: I41844ae699c88ab96bf0d30db3a3fbaf09616161
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/216761
Reviewed-by: Joe Tsai <joetsai@google.com>
Add a place to put microbenchmarks used to justify performance-related changes.
Change-Id: I6e90a3500594b3f6297cee0b8e321a50d0a556ca
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/216480
Reviewed-by: Joe Tsai <joetsai@google.com>
Recognize messagesets in the validator. Currently, this just gives
up and reports an unknown validity rather than trying to descend
into the messageset.
Plumb fast-path initialization checks through messageset decoding.
Change-Id: Ice55f28e8555764e4ce2720251830e8cf475c133
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/216245
Reviewed-by: Joe Tsai <joetsai@google.com>
Promote the fast-path magic ProtoMethods method to first-class citizen
of the protoreflect.Message interface.
To avoid polluting the protoreflect package with the various types
required by this method, make the necessary protoiface types unnamed and
duplicate them in protoreflect.
Updates golang/protobuf#1022.
Change-Id: I9595bae40b3bc7536d727fb6f99b3bce8f73da87
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/215718
Reviewed-by: Joe Tsai <joetsai@google.com>
Add a missed case in validation so we correctly validate bytes fields.
Fixes a case where we would report required bytes fields as potentially
missing.
Change-Id: I3dc4196d6995942d32a795a64214b3679d60ab6c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/215000
Reviewed-by: Joe Tsai <joetsai@google.com>
The fast-path unmarshal was getting confused when parsing a group
containing a field with a number the same as the group's own field
number. Separate the handling of EndGroup tags.
Change-Id: I637702b42c94a26102e693ee29a55e80b37d7f28
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/214737
Reviewed-by: Joe Tsai <joetsai@google.com>
Fix a typo in legacy_enum.go.
Rename package in ancient legacy proto so that it doesn't confuse
tooling that assume that the package and directory names match.
Change-Id: I0b896045e74b0a7f998d3e5693b853eb3aa3839c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/214182
Reviewed-by: Joe Tsai <joetsai@google.com>
The encoding/testprotos and reflect/protoregistry/testprotos are
accessible by other modules. Move them under internal/testprotos
to dissuade programmers who are too lazy to use their own test protos
when they need one.
Change-Id: I3dbfbce74e68ef033ec252bed076861cb47dd21e
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/214341
Reviewed-by: Damien Neil <dneil@google.com>
Avoid dots and dashes in the directory to avoid issues on
build systems that cannot support them well.
Change-Id: I7ea5e6ce0b16c7158c7e53bcf5c3c1a334fe4718
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/214342
Reviewed-by: Damien Neil <dneil@google.com>
Refactor the internal logic of protoc-gen-go to better plumb local
settings and parameters down the call tree.
Change-Id: I09fec188d7359f2b66be584aa8f10e682a7b6796
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/214357
Reviewed-by: Patrik Nyblom <pnyb@google.com>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
This adds a experimental function to the internal/impl package which
validates a wire-format message against a message type. The validator
reports whether the message can be successfully unmarshaled, and whether
the result is initialized (all required fields are set). In some cases,
the validator returns ambiguous results when full validation would be
expensive.
The validator is unused outside of tests. In the future, it may be used
to permit lazy unmarshaling of some data. It is being added now for
testing; in particular, the wire fuzzer now checks the validator output
for consistency with the unmarshaler.
The validator adds a small amount of unused per-MessageType state. If
this becomes a concern, we could conditionalize it with a build tag.
Change-Id: I4216ef81d6a9ed975302eed189b02d08608858b4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/212302
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Add a new Fuzz message containing all the message types we want to make
available to fuzzers. Previously, testing (for example) required fields
would require modifying the fuzzer; now, it's just a matter of adding a
message with required fields as a field of the top-level Fuzz message.
Add internal/cmd/generate-corpus to codify where the fuzz seed corpus
comes from. This will simplify adding text and json fuzzers.
Rename internal/fuzz/wire to internal/fuzz/wirefuzz to minimize package
name ambiguity. Also, the addition of the Fuzz container message
invalidates the existing corpus, so using a new name seems like a good
idea.
Change-Id: I94f8f64ba93596c8e8cecb4d42bcc5b98c17d838
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/212218
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
These are not necessary now that weak fields are unexported.
Change-Id: Ida18b984abedfdf52fd3d5f3cb2f4ca580659a5c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/210745
Reviewed-by: Damien Neil <dneil@google.com>
Both the generator and the runtime need to agree upon the names of
specialized Go struct fields. Centralize that information in an
internal genname package.
In the mean time, also change the XXX_weak field name to match
the name used internally at Google.
Change-Id: I026bf354418c363482e5902f21aa5e0cacae24b0
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/207080
Reviewed-by: Damien Neil <dneil@google.com>
The deprecated messageset format permits extension fields with numbers
greater than the usual maximum (1<<29-1). To support this, the
internal/encoding/wire package has disabled field number validation when
legacy support is enabled.
We shouldn't skip validating all field numbers for validity just because
we support larger ones in messagesets.
This change drops range validation from the wire package (other than
checking that numbers fit in an int32) and adds it to the wire
unmarshalers instead. This gives us validation where we care
about it (when unmarshaling a wire-format message) and allows for
best-effort handling of out-of-range numbers everywhere else.
Fixesgolang/protobuf#996
Change-Id: I4e11b8a8aa177dd60e89723570af074a317c2451
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/210290
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Various protoreflect methods can return an "empty, read-only" message,
list, or map value. Provide a method to test if a value is one of these.
Fixesgolang/protobuf#966
Change-Id: I793d8426d6e2201755983c06f024412a7e09bc4c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/209018
Reviewed-by: Joe Tsai <joetsai@google.com>
The v1 wire marshaler sorts fields as follows:
- All extensions, sorted by field number.
- All non-oneof fields, sorted by field number.
- All oneof fields, in indeterminate order.
We already make some steps toward supporting this ordering: The
fast path encoder places extensions in sorted order at the start
of the message.
This commit moves oneof fields to the end of the message, makes the
reflection-based encoder use this ordering when deterministic marshaling
is enabled, and adds a test to catch unintentional changes to the
ordering.
Users SHOULD NOT depend on stability of the marshal output. It is
subject to change over time. Without deterministic marshaling enabled,
it is subject to change over calls to Marshal.
Change-Id: I6cfd89090d790a3bb50785f32b94d2781d7d08db
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/206800
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
In the v1 implementation, unknown MessageSet items are stored in a
message's unknown fields section in non-MessageSet format. For example,
consider a MessageSet containing an item with type_id T and value V.
If the type_id is not resolvable, the item will be placed in the unknown
fields as a bytes-valued field with number T and contents V. This
conversion is then reversed when marshaling a MessageSet containing
unknown fields.
Preserve this behavior in v2.
One consequence of this change is that actual unknown fields in a
MessageSet (any field other than 1) are now discarded. This matches
the previous behavior.
Change-Id: I3d913613f84e0ae82481078dbc91cb25628651cc
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/205697
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The MessageInfo cache, once set, must not be cleared, otherwise
there exists a *messageState value where the MessageInfo value is nil.
Fix the generation of the Reset method to avoid clearing this value.
Change-Id: Ic84ca8b2640a43e967c36993da1ccd3f2b7096c4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/201478
Reviewed-by: Damien Neil <dneil@google.com>
These methods are difficult or impossible to use correctly; types
created by the dynamicpb package, for example, all have the same GoType.
Fixesgolang/protobuf#938
Change-Id: I33d4ef381579ff18569b11df501d0ba7f38a6b5c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/199060
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The protolegacy package is a minimal version of the v1 proto package.
This allows us to use this stub version as the dependency for
internal/testprotos/legacy packages and avoid a dependency
on the real v1 proto package.
The implementation of most v1 functionality will panic if called.
This way, we know if we the v2 code depends on one of those
unimplemented functions.
Updates golang/protobuf#962
Change-Id: I20b4091706fd456e4b01ae0931cce30a872639b0
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/199297
Reviewed-by: Damien Neil <dneil@google.com>
Support, to some limited degree, types which implement protoV1.Message
but which are not struct pointers. Our ability to work with these types
is largely limited to calling Marshal or Unmarshal methods, when
present.
Change-Id: Ie1b851d9e753e2b2cb189b17ffeefebe2d8b3a8f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/198237
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Field-tracking is an experimental feature in the Go linker
where it statically tracks whether a Go struct field is used.
This CL modifies the generator to emit special markers that
tells the linker to know which fields to track.
Change-Id: I235da3f3d6c0ef2021b5fe1c16106ebb8fd6f557
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189558
Reviewed-by: Damien Neil <dneil@google.com>
Too much code inside Google rely on ExtensionDescV1.Name being the
target message name rather than the real name of the extension.
Double down on this assumption consistently across our codebase.
Change-Id: If0b7cd16dd1f2f7acfb2e562dbf1a2b5487162b5
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186980
Reviewed-by: Herbie Ong <herbie@google.com>
Change protoV1.ExtensionDesc to directly implement ExtensionType
rather than delegating to one.
Unify the previous types protoiface.ExtensionDescV1 and
filetype.Extension in impl.ExtensionInfo. The protoV1.ExtensionDesc
type becomes an alias to ExtensionInfo.
This gives us:
- Just one implementation of ExtensionType.
- Generated foopb.E_Ext vars are canonical ExtensionTypes.
- Generated foopb.E_Ext vars are also v1.ExtensionDescs for backwards
compatibility.
- Conversion between legacy and modern representations happens
transparently when lazily initializing an ExtensionInfo.
Overall, a simplification for users of generated code, since they can
mostly ignore the ExtensionDesc/ExtentionType distinction and use the
same value in either the old or new API.
This is change 3/5 in a series of commits changing protoV1.ExtensionDesc
to directly implement protoreflect.ExtensionType.
1. [v2] Add protoimpl.ExtensionInfo as an alias for
protoiface.ExtensionDescV1.
2. [v1] Update references to protoimpl.ExtensionInfo to use
protoiface.ExtensionInfo.
3. [v2] Create protoimpl.ExtensionInfo (an alias to a new type in
the impl package) and remove protoiface.ExtensionDescV1.
4. [v1] Remove unneeded explicit conversions between ExtensionDesc and
ExtensionType (since the former now directly implements the latter).
5. [v2] Remove stub conversion functions.
Change-Id: I96ee890541ec11b2412e1a72c9d7b96e4d7f66b4
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189563
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
A conformance test has been added for the text format.
Update our conformance runner to handle that format and
update with the list of current failures.
Change-Id: I36a271fcfbe328c4ee8c870caff4661659ad27ef
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/190517
Reviewed-by: Herbie Ong <herbie@google.com>
The old implementation had the behavior where a nil wrapper value:
m := new(foopb.Message)
m.OneofField = (*foopb.Message_OneofUint32)(nil)
was functionally equivalent to it being directly set to nil:
m := new(foopb.Message)
m.OneofField = nil
preserve this semantic in both the table-drive implementation
and the reflection implementation.
Change-Id: Ie44d51e044d4822e61d0e646fbc44aa8d9b90c1f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189559
Reviewed-by: Herbie Ong <herbie@google.com>
Generate the current index into depIdxs for easier human debugging.
Change-Id: Ida42aa95137b2044a4dc267c31cebec5023bdfb1
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/190278
Reviewed-by: Herbie Ong <herbie@google.com>
If a string default value contains a backtick, it breaks the
struct tag formatting logic which assumes such characters never exist.
Fix this by adding a structTags type, whose String method knows how
to properly escape the value.
Change-Id: I96fe5a6630387eac89eef429da5f917125570e4c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189557
Reviewed-by: Herbie Ong <herbie@google.com>
Remove the remaining uses of the prototype package.
The most significant change is to impl.MessageInfo, which now directly
implements the MessageType interface. This involves two notable changes
to exported fields of MessageInfo:
- PBType is now Desc.
- GoType is now GoReflectType. (Name changed to avoid a conflict with
the GoType method of the MessageType interface.)
Fixesgolang/protobuf#911
Change-Id: Ie2aa4766d6887ceaa9cf06b1f109aa6e6e2a208f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189340
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Drop the dependency from generated files on prototype.Enum: Generated
code should only depend on runtime/proto{iface,impl}.
Drop the Enums, Messages, and Extensions returns form
filetype.Builder.Build. Of these, only Enums was used by generated code.
Change the generated init function to pass the builder a slice of values
to fill in (as is done for messages and extensions).
Remove the filetype dependency on prototype in preparation for
eventually dropping the prototype package entirely.
Change-Id: I28a3420f5dfcc13fed531a64ef07b9afddfd9d55
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189200
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
