Drop the dependency from generated files on prototype.Enum: Generated
code should only depend on runtime/proto{iface,impl}.
Drop the Enums, Messages, and Extensions returns form
filetype.Builder.Build. Of these, only Enums was used by generated code.
Change the generated init function to pass the builder a slice of values
to fill in (as is done for messages and extensions).
Remove the filetype dependency on prototype in preparation for
eventually dropping the prototype package entirely.
Change-Id: I28a3420f5dfcc13fed531a64ef07b9afddfd9d55
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189200
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Rename build tag "proto1_legacy" -> "protolegacy"
to be consistent with the "protoreflect" tag.
Rename flag constant "Proto1Legacy" -> "ProtoLegacy" since
it covers more than simply proto1 legacy features.
For example, it covers alpha-features of proto3 that
were eventually removed from the final proto3 release.
Change-Id: I0f4fcbadd4b5a61c87645e2e5be11d187e59157c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189345
Reviewed-by: Damien Neil <dneil@google.com>
Refactor the command execution logic to be more configurable.
Change-Id: I6d21f6c3f38691a8ffc992a170ff221c8a06f990
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189341
Reviewed-by: Damien Neil <dneil@google.com>
Change protoiface.ExtensionDescV1 to implement protoreflect.ExtensionType.
ExtensionDescV1's Name field conflicts with the Descriptor Name method,
so change the protoreflect.{Message,Enum,Extension}Type types to no
longer implement the corresponding Descriptor interface. This also leads
to a clearer distinction between the two types.
Introduce a protoreflect.ExtensionTypeDescriptor type which bridges
between ExtensionType and ExtensionDescriptor.
Add extension accessor functions to the proto package:
proto.{Has,Clear,Get,Set}Extension. These functions take a
protoreflect.ExtensionType parameter, which allows writing the
same function call using either the old or new API:
proto.GetExtension(message, somepb.E_ExtensionFoo)
Fixesgolang/protobuf#908
Change-Id: Ibc65d12a46666297849114fd3aefbc4a597d9f08
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189199
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
We occasionally need to work with immutable, empty lists, maps, and
messages. Notably, Message.Get on an empty repeated field will return a
"frozen" empty value.
Move handling of these immutable, zero-length composites into Converter,
to unify the behavior of regular and extension fields.
Add a Zero method to Converter, MessageType, and ExtensionType, to
provide a consistent way to get an empty, frozen value of a composite
type. Adding this method to the public {Message,Extension}Type
interfaces does increase our API surface, but lets us (for example)
cleanly represent an empty map as a nil map rather than a non-nil
one wrapped in a frozenMap type.
Drop the frozen{List,Map,Message} types as no longer necessary.
(These types did have support for creating a read-only view of a
non-empty value, but we are not currently using that feature.)
Change-Id: Ia76f149d591da07b40ce75b7404a7ab8a60cb9d8
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189339
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
In order for protoc-gen-go to output the current version,
it needs to know what version it is currently running as.
However, we cannot rely on the git tags since the tags are not
made until *after* the commit has been submitted.
Instead, we manually encode the version into the code and
make sure that git tags match up with the version in the code.
The version.go file in runtime/protoimpl contains instructions
for how to make a release. Essentially:
* Every non-release commit has a version string with "devel" in it.
* Every release commit must not have "devel" in it and must be unique.
* The "release process" involves submitting two CLs.
The first CL creates a version string without "devel",
which is the commit that a git tag will actually reference.
The second CL follows immediately and re-introduces "devel"
into the version string.
The following example shows a possible sequence of VersionStrings
for git commits in time-ascending order:
v1.19.0-devel (this CL)
v1.19.0-devel
v1.19.0-devel
v1.19.0-devel
v1.20.0-rc.1 <- tagged
v1.20.0-rc.1.devel
v1.20.0-rc.1.devel
v1.20.0-rc.1.devel
v1.20.0-rc.2 <- tagged
v1.20.0-rc.2.devel
v1.20.0 <- tagged (future public release)
v1.20.0-devel
v1.20.0-devel
v1.20.0-devel
v1.20.0-devel
v1.20.1 <- tagged
v1.20.1-devel
v1.20.1-devel
v1.21.0 <- tagged
v1.21.0-devel
Note that we start today with v1.19.0-devel, which means that our initial
release will be v1.20.0. This number was intentionally chosen since
1) the number 20 has some correlation to the fact that we keep calling
the new implementation the "v2" implementation, and
2) the set of tagged versions for github.com/golang/protobuf
and google.golang.org/protobuf are unlikely to ever overlap.
This way, the version of protoc-gen-go is never ambiguous which module
it was built from.
Now that we have version information, we add support for generating .pb.go
files with the version information recorded. However, we do not emit
these for .pb.go files in our own repository since they are always guaranteed
to be at the right version (enforced by integration_test.go).
Updates golang/protobuf#524
Change-Id: I25495a45042c2aa39a39cb7e7738ae8e831a9d26
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186117
Reviewed-by: Damien Neil <dneil@google.com>
The following improvements were made:
* All standalone comments above the "syntax" marker are preserved
similar to Java and some other generators.
* All standalone comments above the "package" marker are preserved
to be consistent with our former behavior.
* Leading comments are now generated for enums and extension fields.
* Single-line trailing comments are now generated for
enum values, message fields, and extension fields.
* The leading comments for each field that is part of a oneof are now
generated with the wrapper types rather than being shoved into the
comment for the oneof itself in an unreadable way.
* The deprecation marker is always generated as being above the declaration
rather than sometimes being an inlined comment.
* The deprecation marker is now properly generated for weak field setters.
Updates golang/protobuf#666
Change-Id: I7fd832dd4f86d15bfff70d7c22c6ba4934c05fcf
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189238
Reviewed-by: Damien Neil <dneil@google.com>
This is a breaking change. High-level protogen API changes:
* remove GeneratedFile.PrintLeadingComments method
* add {Message,Field,Oneof,Enum,EnumValue,Service,Method}.Comments field
* add CommentSet and Comments type
CL/183157 added protoreflect.SourceLocations and it was discovered
that there can actually be duplicate locations for certain paths.
For that reason, we decided not to expose any helper methods
for looking up locations by path since it is unclear which location
to return if multiple matches.
The protogen.GeneratedFile.PrintLeadingComments has a similar dilemma
where it also needs to figure out what to do when duplicates exist.
Previously, it just chooses the first one with comments,
which may not be the right choice in a given context.
Analysis of current PrintLeadingComments usage shows that it is only
ever used (except once) for descriptor declarations.
In the case of descriptor declarations, they are guaranteed by protoc
to have only location.
Thus, we avoid the duplicate location problem by:
* Providing a CommentSet for every descriptor. The CommentSet contains
a set of leading and trailing comments of the Comments type.
* The Comments.String method knows how to interpret the comments
as provided by protoc and format them as // prefixed line comments.
* Values of the Comments type can be passed to the P method.
We drop direct support printing leading comments for non-descriptor locations,
but the exposure of the Comments type makes it easy for users to manually
handle other types of comments themselves.
Change-Id: Id4851456dc4e64d76bd6a30e8ad6137408dfb27a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189198
Reviewed-by: Damien Neil <dneil@google.com>
For better readability in godoc, group extension fields by the
target message that they are extending.
Change-Id: Icc0a247b37639e3dbf7a107810923b8ca8294724
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189257
Reviewed-by: Damien Neil <dneil@google.com>
Add methods to protoreflect.{Message,List,Map} to constrict values
assignable to a message field, list element, or map value. These
methods return the default value for scalar fields, the zero value for
scalar list elements and map values, and an empty, mutable value for
messages, lists, and maps.
Deprecate the NewMessage methods on these types, which are superseded.
Updates golang/protobuf#879
Change-Id: I0f064f60c89a239330ccea81523f559f14fd2c4f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/188997
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Since the enum maps are here to stay, group the declarations together
in a var block for better readability in godoc.
Change-Id: I9a313266539e9a60781f98b80a5293379f82607b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189077
Reviewed-by: Damien Neil <dneil@google.com>
Group the default constant and variable declarations together in a block
for better readability in godoc.
Change-Id: I6b62f5374f0302d0f7cb224cbe34102359c8c51d
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189057
Reviewed-by: Damien Neil <dneil@google.com>
Reduce the stutter in the name since the type of Builder
is obvious from the package it is from.
Change-Id: I0046a5122717536cc6bb5ebdb32b67a1560cfc23
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189020
Reviewed-by: Damien Neil <dneil@google.com>
A Converter converts between reflect.Values and protoreflect.Values.
The existing usage of Converter is somewhat confusing: The
internal/value package creates Converters for scalar types only, the
internal/impl package creates Converters for legacy messages and enums,
and the reflect/prototype package creates Converters for repeated fields.
Change the Converter type to an interface. The constructor for
Converter takes a FieldDescriptor and reflect.Type, and directly
handles conversions for all field types: Scalars, lists, maps, and
legacy types.
Move Converter into the internal/impl package, since that package
contains the necessary support for dealing with legacy messages and
enums. Drop the internal/value package.
Replace two uses of prototype.Extension with more focused
implementations, since the implementation is trivial with the
refactored Converter. Drop prototype.Extension for the moment since
it is now unused.
Change-Id: If0c570fefac002cc5925b3d56281b6eb17e90d5f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/187857
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Rather than waiting for all tests for all Go versions to finish
only to find that there was some minor breakage,
fail fast so that we stop wasting time.
Change-Id: Ie255ceb5ac2cbdc598a074c6da281f5e49eb1326
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189019
Reviewed-by: Damien Neil <dneil@google.com>
Fix decoding of zero-length bytes fields to produce a non-nil []byte.
Change-Id: Ifb7791a47df81091700f7226523371d1386fb1ad
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/188765
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Suppose some init logic performs protobuf reflection.
If so, it will cause the table-driven logic for protobuf reflection
to be initialized. This is problematic for weak fields since we
can not be certain that all weak references have been registered
at this point in time. This is a fundamental issue with with weak
dependencies, since it means that we cannot enforce any ordering
constraints on the weak dependency unless we directly import the weakly
referenced package (which would defeat the point of weak imports).
Alleviate the problem by pushing evaluation of weak reference to
actual usage. This does not completely fix the problem,
but signifcantly reduces the probability of it being problematic.
In general, people should avoid interacting with weak fields at init time.
Change-Id: Iebaefddde8cf07b5cd7dee49b7015b05b5428618
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/188980
Reviewed-by: Damien Neil <dneil@google.com>
Suppose some init logic performs protobuf reflection.
If so, it will cause the lazy descriptor init logic to be executed.
This is problematic for weak fields since we can not be certain that
all weak references have been registered at this point in time.
This is a fundamental issue with with weak dependencies,
since it means that we cannot enforce any ordering constraints
on the weak dependency unless we directly import the weakly referenced package
(which would defeat the point of weak imports).
Alleviate the problem by pushing evaluation of weak reference to
actual usage. This does not completely fix the problem,
but signifcantly reduces the probability of it being problematic.
In general, people should avoid interacting with weak fields at init time.
Change-Id: Ie5957ddedd61333e72ee9a1bba0c53dace65547c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/188982
Reviewed-by: Damien Neil <dneil@google.com>
The messageState.mi field is atomically checked and set
in generated code to the *MessageInfo associated with that message.
However, the messageState type accesses the mi field without
any atomic loads, thus being a potential race.
We fix this by always calling a messageInfo method that performs
a atomic.LoadPointer on the *MessageInfo.
There is no performance effect from this change on x86 since
an atomic.LoadPointer is identical to a MOV instruction.
From an assembly perspective, there was no memory race previously.
However, the lack of an atomic.LoadPointer meant that the compiler
could in theory reorder the "normal" load to produce truly racy code.
Change-Id: I8afefaf35c1916872781abc0239cbb63d62edf16
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/189017
Reviewed-by: Damien Neil <dneil@google.com>
These were originally kept separate to assist Google-internal patches,
but it turns out that Google-internal patches do not use the
genMessageInternalFields function.
Change-Id: Idfa962b943d3bede9982b5b0875ba90c86c6d181
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/188979
Reviewed-by: Damien Neil <dneil@google.com>
It is possible for filedesc to construct a lazy options decoder before
the descriptor package has been imported. For example, top-level enum
values are eagerly decoded, so a generated proto package can construct a
lazy options decoder for an enum value at init time.
Don't close over the variables in descopts. Instead, close over a pointer
to the variable.
Panic with an informative message in the case where options are decoded
before the descriptor package has been initialized.
Change-Id: I277a57602b083cb7bbf92c8114c50b467e59521f
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/188820
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The problem is that atomicNilMessage.m.mi is accessed both by atomic and
non-atomic operations. (Init uses an atomic read to verify that m.mi is
non-nil, but then returns a non-atomic m.)
Race condition is demonstrated by this test with
"go test -race -count=1000":
func TestPointer(t *testing.T) {
var m atomicNilMessage
var mi MessageInfo
ch := make(chan *MessageInfo)
for i := 0; i < 20; i++ {
go func() {
r := m.Init(&mi)
if &mi != r.mi {
// This conditional exists just
// ensure r.mi is touched.
t.Error("mismatch")
}
ch <- r.mi
}()
}
for i := 0; i < 20; i++ {
<-ch
}
}
I chose not to add the test since it seems a bit overfit to the specific
situation.
Change-Id: Id4664ef3cd5b29515ed310851b9aeb7561be30d0
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/188337
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Our specific protoreflect.Message implementations have a special
ProtoMessageInfo method to obtain the *MessageInfo for v1 compatibility.
Use that instead to implement getMessageInfo.
Change-Id: I6cab9aeaa93714be73bd812c3d9a3be0ec86dd52
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/187777
Reviewed-by: Damien Neil <dneil@google.com>
Golden test output doesn't match when math.NaN() has different bits from
the test's NaNs. Drop the NaN-related tests as too fiddly to be worth
keeping.
Change-Id: I89cf961273c2afab3b6b9f6c63878816314e9f43
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186639
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
This CL makes no feature changes except to move code around.
The only change to the actual generated code is the placement of
the default constants and variables. They move because the new logic
generates all methods together, while previously the constants
were interspersed in-between.
Change-Id: I45932d5aeec5ba45180fb255ea17898beb6c3bd2
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186878
Reviewed-by: Herbie Ong <herbie@google.com>
The ignoreConflict function provides the ability to ignore certain conflicts.
By default, all conflicts are ignored with a log message produced instead.
Change-Id: I67fe56eef492e12421e5c8cb8d618dc2a46c82ed
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186658
Reviewed-by: Damien Neil <dneil@google.com>
Declare the internal protobuf dependencies as variables of interface
type so that they can be more easily replaced by custom implementations.
Change-Id: I7fff885fd79ee0117c1c62654b2fd4b1877708da
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186659
Reviewed-by: Damien Neil <dneil@google.com>
Rather than panicking at init time due to registration failures,
print a warning to stderr. Historically, the Go protobuf implementation
has not been strict about registration conflicts, which has led users
to unknowningly tolerating conflicts that may or may not expose
themselvs as a bug.
Registration conlicts now produce a log message:
<<<
2019/07/17 17:36:42 WARNING: proto: file "path/to/example.proto" is already registered
previously from: "example.com/company/example_proto"
currently from: "example.com/user/example_proto"
A future release will panic on registration conflicts.
>>>
Change-Id: I2d583f04977c8bc8cb6bbd33d239277690bbec54
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186181
Reviewed-by: Damien Neil <dneil@google.com>
The v2 MessageInfo is needed by v1 to be able to access the OneofWrappers
and Exporter function. The ProtoMessageInfo method can be deleted
once v1 is entirely implemented in terms of v2.
Change-Id: Iabb1b429af5210faffc6477f52b5020b3aa1fb50
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186577
Reviewed-by: Damien Neil <dneil@google.com>
The only remaining dependencies are for benchmarks and
internal/testprotos/legacy.
Change-Id: I0f7f5292000ccad91bc9526e40fa4d0ec3a36e43
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186157
Reviewed-by: Damien Neil <dneil@google.com>
Treating NaNs as inequal has benefits in mathmetical operations,
but is almost certainly never desired behavior in tests.
Making them equal allows us to document that Equal reports true
if the encoded bytes are also equal (under deterministic marshaling).
Change-Id: Id11c9c1681d8785bcc52f0f42064339194065649
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186179
Reviewed-by: Damien Neil <dneil@google.com>
This CL unifies common MessageSet logic in prototext and protojson
into the messageset package. While we are at it, also enable
MessageSet support only if the proto1_legacy build flag is enabled.
Change-Id: I1a7d475e8bb1dad61ecd286df45e4239e5bef072
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185898
Reviewed-by: Damien Neil <dneil@google.com>
The v2 implementation strictly enforces that there are no conflicts at
all in the protobuf namespace unlike the prior v1 implementation.
This change is almost certainly going to cause loud failures for users
that were unknowingly tolerating registration conflicts.
We modify internal/filedesc to be able to record the Go package path
that the file descriptor is declared within. This information is used
by reflect/protoregistry to print both the previous Go package that
registered some declaration, and current Go package that is attempting
to register some declaration.
Change-Id: Ib5eb21c1c98495afc51aa08bd4404bd9d64b5b57
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186177
Reviewed-by: Damien Neil <dneil@google.com>
In 2014, when proto3 was being developed, there were a number of early
adopters of the new syntax. Before the finalization of proto3 when
it was released in open-source in July 2016, a decision was made to
strictly validate strings in proto3. However, some of the early adopters
were already using invalid UTF-8 with string fields.
The google.protobuf.FieldOptions.enforce_utf8 option only exists to support
those grandfathered users where they can opt-out of the validation logic.
Practical use of that option in open source is impossible even if a user
specifies the proto1_legacy build tag since it requires a hacked
variant of descriptor.proto that is not externally available.
This CL supports enforce_utf8 by modifiyng internal/filedesc to
expose the flag if it detects it in the raw descriptor.
We add an strs.EnforceUTF8 function as a centralized place to determine
whether to perform validation. Validation opt-out is supported
only in builds with legacy support.
We implement support for validating UTF-8 in all proto3 string fields,
even if they are backed by a Go []byte.
Change-Id: I9c0628b84909bc7181125f09db730c80d490e485
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186002
Reviewed-by: Damien Neil <dneil@google.com>
MessageSets are a deprecated proto1 feature, long since superseded by
extensions. Add disabled-by-default support behind flags.Proto1Legacy.
Change-Id: I7d3ace07f3b0efd59673034f3dc633b908345a88
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185538
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
If the message for a weak field is linked in,
we treat it as if it were identical to a normal known field.
However, if the weak field is not linked in,
we treat it as if the field were not known.
Change-Id: I576d911deec98e13211304024a6353734d055465
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185457
Reviewed-by: Herbie Ong <herbie@google.com>
This implements generation of and reflection support for weak fields.
Weak fields are a proto1 feature where the "weak" option can be specified
on a singular message field. A weak reference results in generated code
that does not directly link in the dependency containing the weak message.
Weak field support is not added to any of the serialization logic.
Change-Id: I08ccfa72bc80b2ffb6af527a1677a0a81dcf33fb
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/185399
Reviewed-by: Damien Neil <dneil@google.com>
For proto3 messages with an enum field that could not be resolved,
do not check the syntax of that enum dependency.
Change-Id: I7c646539351edc35243ab950d335f4018cc4c0e9
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/186001
Reviewed-by: Damien Neil <dneil@google.com>
