Repository navigation
gengorums: support several services per proto file and Go package - #380
Conversation
4d76daf to
5e7eaac
Compare
ba4e8a8 to
8fa54fc
Compare
8fa54fc to
e9dbc53
Compare
e9dbc53 to
b56da97
Compare
b56da97 to
a4338e1
Compare
a4338e1 to
0c6972e
Compare
a2fa5f4 to
743a8d7
Compare
|
Restacked on the new #379 tip, which now generates two client interfaces per service (see the #308 design update). The branch was force-pushed.
|
743a8d7 to
722fea2
Compare
|
Restacked on the updated #379 (review fixes: unexported client names such as
|
meling
left a comment
There was a problem hiding this comment.
Reviewed the current tip, 722fea2, against #379's ef8e0df.
I found no additional actionable defects in this PR's incremental changes.
One inherited issue remains: a protobuf oneof wrapper can collide with the new unexported client struct, and generation succeeds before the Go package fails to compile.
The reproducer and finding are in the review on #379, so I have not duplicated the inline finding here.
The separate client/Correctable-alias collision reported there is already rejected by this PR's packageIdents check.
The earlier response-type alias collision and mixed gRPC package concerns are addressed in the current source and covered by passing generator tests.
Validation: go test ./... passes in the root, examples, and benchkit modules.
Each service now has its own client types, so services in one file no longer collide on RPC names. Remove the one-service check and validate the methods of every service, not only the first. Check that no two services in the Go package generate the same name, such as NewFooClient from both NewFoo and Foo. Collect the generated names from every file of the package in the protoc run, and include the server names. Name the file of each colliding declaration.
Every generated _gorums.pb.go file declared the static aliases Config, ID, and Node, so two proto files with Gorums services in one Go package redeclared them. The aliases are useful to callers, so keep them, but emit the static code only in the generated file of the first proto file of the package with Gorums methods, in protoc's file order. In that order, imported files come first. The other files of the package use the same aliases. A package's proto files must therefore be compiled in one protoc run.
Each generated file declared a CorrectableX alias for every quorum call response type of its services. Two proto files of one Go package whose quorum calls return the same message both declared the alias, so the package did not compile. The aliases are useful to callers, so keep them, but declare each alias only in the generated file of the first proto file of the package, in protoc's file order, whose quorum calls produce it. This is the order that places the Config and Node aliases. The aliases are package-level names, so add them to the package-wide name check: a message or enum must not take an alias name, and an alias must not equal a client or server identifier of any service. Services may share an alias.
A proto file, or a Go package, may now hold several services. Gorums declares the static aliases Config, ID, and Node, and each Correctable alias, in only one generated file per Go package. Add a user guide section on several services, describe the once-per-package aliases and the name checks in the developer guide, and note the new generated names in the migration guide.
Regenerate with make dev and make genproto. The new multiservice test package gets its generated code. The other generated files do not change.
The static aliases (Config, ID, and Node) and each CorrectableX alias belong to one file per Go package. The generator picked that file only among the files that a protoc run generates. Thus, two proto files of one package compiled in separate runs both declared the aliases, and the package failed with "Config redeclared". The generator now picks the owner among all files of the package in the request, including files that are only imports of the run. A run emits an alias only if it generates the owner; otherwise the run of the owner emits it. So if b.proto imports a.proto from the same package, the two files may be compiled in separate runs, in either order. Two files that never import each other must still share one run. The name collision checks also include the services of imported files of the package, because those services generate their names into the same package in their own run. Every service of a Go package with Gorums code counts as a Gorums service, so an imported file with a plain gRPC service in the same package can own the aliases. The docs say to put such services in another Go package.
The alias name CorrectableX uses only the Go name of the response type. Two quorum calls of one Go package can return different types with the same Go name, such as left.Response and right.Response. The generator treated the matching names as one alias, so it declared the alias for the first type only, and code that used it for the second type did not compile. Which type won depended on the file order. Record the response type of each alias, including its import path. Share an alias only between quorum calls that return the same type, and report two different types with the same alias name as an error that names both types and the quorum calls that return them.
The Several Services section shows only configuration clients. Show that a node client of a service reaches one node of the shared configuration, built from a node of that configuration.
A Go package may hold a file whose service is plain gRPC. Such a file is generated in a protoc run without the Gorums plugin, since Gorums would generate its unannotated methods as remote calls. Tests need to reproduce that run, so add Options.NoGorums, which runs only protoc-gen-go.
The static aliases Config, ID, and Node went into the _gorums.pb.go file of the first proto file of the Go package with Gorums methods, counting imported files. A package that imports a same-package file whose service is plain gRPC lost its aliases: the imported file owned them, but no Gorums run generates that file. Separate runs of files that do not import each other declared the aliases twice. Each run that generates Gorums code for a Go package now also writes gorums.pb.go next to the first such generated file. Its content depends only on the package name and the tool versions, so every run writes the same file. The per-file outputs no longer hold the static code, and the static alias owner rule is gone. Correctable aliases keep the owner rule: only a quorum call produces one, and a file with a quorum call always has a Gorums run. Dev mode keeps dev/aliases.go and writes no gorums.pb.go.
The static aliases Config, ID, and Node now live in one gorums.pb.go file per Go package, which every protoc run writes with the same content. Describe the file in the user, developer, and migration guides. The separate-runs rule now applies only to Correctable aliases, and a plain gRPC service may share a Go package with Gorums services if its file is compiled without --gorums_out.
Regenerate with the generator that writes the static aliases into one gorums.pb.go file per Go package. The _gorums.pb.go files lose the alias block, and each generated package gains its gorums.pb.go file.
protoc-gen-go writes gorums.pb.go for a proto file named gorums.proto. If that file shares the Go package and output directory of the Gorums package file, the two outputs collide: protoc rejects the second write, and separate runs overwrite each other. Report the proto file and ask to rename it, whether it is a Gorums file or only an import. Also rewrap the TestBundleReservedIdentifiers doc comment.
The NoGorums change left one doc comment line at about 110 columns. Rewrap the comment to the usual width.
Only runs that generate Gorums code for a package write its gorums.pb.go, and no run removes a stale one, so say to delete it after removing the last Gorums proto file. Mention the new error for a proto file named gorums.proto. The release guide said GenVersion is in every generated file; gorums.pb.go has version header lines but no version check.
The multiservice fixture and the inline test protos use the internal.testprotos proto package, like the other test protos under internal/testprotos.
The package doc describes the clients of one service. State that a Go package may hold several services, that each gets its own clients, that the services may share one configuration and one server, and that their generated names must not collide.
722fea2 to
65f941a
Compare
One function now computes the Correctable aliases that a file declares, and the template ranges over them. This replaces four helpers and the Declared template field. packageIdents uses two plain loops instead of a closure with a side map; its error messages are unchanged. The generated code is unchanged.
The generator declared one CorrectableX alias per quorum call response type X. Files of a Go package shared these aliases, so one file had to own each alias. This needed an owner rule across protoc runs, an error for two response types named X from different Go packages, and care when compiling files that do not import each other in separate runs. The static code now declares Correctable[Resp any] once per package in gorums.pb.go, next to Config, ID, and Node. The _gorums.pb.go files declare only the names of their own services, so the owner rule, the types template, and the alias checks are gone. packageIdents now only checks service names, with helpers for the names of a service and the collision error. Correctable becomes a reserved identifier.
Closes #349. Stacked on #379 (generated service clients) in a
gh stack; the base moves when #379 merges.Important
Part of the v0.13 changes with #379; merge after #379, or together with it.
Problem
A proto file could hold only one service, and two proto files could not share a Go package.
The generator rejected the first case.
In the second case, every generated file repeated the
Config,Node, andCorrectableXdeclarations, so the package did not compile.#379 already moved each RPC onto per-service client interfaces, so two services with the same RPC name no longer collide.
This PR removes the remaining blockers.
Changes
The names of a service are the interface, constructor, and struct of each client that it has (
SClient,NewSClient,sClient, andSNodeClient,NewSNodeClient,sNodeClient), andSServerandRegisterSServer.An error names the file that declares the message and the service that generates the name, for example:
other.proto: contains message ReservedClient, which is the Gorums identifier generated for service Reserved (failing.proto).For a plain gRPC service in an imported file, the check reserves the node client names, not the names that
protoc-gen-go-grpcgenerates; the doc comment ofcheckReservedIdentssays so.Config,ID,Node,Correctable) in onegorums.pb.goper Go package (B2, option (ii) in feat: support multiple services per proto file and per Go package #349): every protoc run that generates Gorums code for a package writes the samegorums.pb.gonext to itsX_gorums.pb.gofiles, so separate runs and mixed packages with plain gRPC services work.pb.Config,pb.ID, andpb.Nodestay as they are, and these names stay reserved.The file content depends only on the Go package name, so the runs never disagree.
A proto file named
gorums.protoin the same package is rejected, because protoc-gen-go would write the same file.Correctablealias (B3;masterno longer generatesAsyncXaliases):type Correctable[Resp any] = gorums.Correctable[Resp]ingorums.pb.goreplaces the per-typeCorrectableXaliases.The generator no longer needs to decide which file declares an alias, and separate protoc runs work in every case.
Correctableis now a reserved name, likeConfig,ID, andNode.gorums.pb.gofile in the user and dev guides, and a migration note for theCorrectableXaliases.Breaking changes
CorrectableXaliases are gone.Replace
pb.CorrectableXwith*pb.Correctable[*pb.X].doc/migration-v2-to-v3.mddescribes the change.Correctableis now a generator error.pb.Config,pb.ID, andpb.Nodekeep their names.Regenerating moves these aliases from each
X_gorums.pb.gointo the newgorums.pb.go.Some proto files whose generated code did not compile now get a clear generator error instead.
Notes
gorums.pb.go.go test ./..., becauseinternal/tests/multiservicegets its generated code later. Some commits also come before the commit that regenerates their output. The tip passes.Testing
Heartbeat); failing cases for duplicate names within a file, across files, against server names, against a message in another file, against a message namedCorrectable(in the same file and in an imported file), for a client name shared by two services (FooNodewith a quorum call andFoowith a remote call), and for options of a service that is not the first.TestGeneratePackageFilescompiles two proto files in one Go package, also with a shared response type and the same RPC name in both, checks that both outputs exist, and checks which Go packages get agorums.pb.gofile (also with a messages-only first file, an imported file first, and two Go packages in one run).TestGeneratePackageFilesSeparateRunscompiles the output of two separate runs, in both orders, as one package, also without an import relation and with a plain gRPC import.TestMultiServiceSharedConfigregisters two services on one server and calls both through onegorums.Config(and one node client); each reply shows that it reached its own service.go test ./...in the root,examples, andbenchkitmodules, with and without-tags=integration;make goplscheck,make modernize, andgolangci-lintare clean.