Skip to content

gengorums: support several services per proto file and Go package - #380

Merged
meling merged 23 commits into
feature/308/service-clientsfrom
feature/349/multi-service
Oct 30, 2026
Merged

meling merged 23 commits into
feature/308/service-clientsfrom
feature/349/multi-service

Conversation

@meling

@meling meling commented Oct 24, 2026 •

Copy link
Copy Markdown
Member

Closes #349. Stacked on #379 (generated service clients) in a gh stack; the base moves when #379 merges.

Important

Part of the v0.13 changes with #379; merge after #379, or together with it.

Problem

A proto file could hold only one service, and two proto files could not share a Go package.
The generator rejected the first case.
In the second case, every generated file repeated the Config, Node, and CorrectableX declarations, so the package did not compile.

#379 already moved each RPC onto per-service client interfaces, so two services with the same RPC name no longer collide.
This PR removes the remaining blockers.

Changes

  1. Several services per file (B1 in feat: support multiple services per proto file and per Go package #349): the one-service check is gone, and the options of every service are validated.
  2. Name checks across the package: the generated names of each service are checked against the messages, enums, enum values, and oneof wrappers of every file in the Go package, and against the names of every other service in the same protoc run.
    The names of a service are the interface, constructor, and struct of each client that it has (SClient, NewSClient, sClient, and SNodeClient, NewSNodeClient, sNodeClient), and SServer and RegisterSServer.
    An error names the file that declares the message and the service that generates the name, for example: other.proto: contains message ReservedClient, which is the Gorums identifier generated for service Reserved (failing.proto).
    For a plain gRPC service in an imported file, the check reserves the node client names, not the names that protoc-gen-go-grpc generates; the doc comment of checkReservedIdents says so.
  3. Static aliases (Config, ID, Node, Correctable) in one gorums.pb.go per Go package (B2, option (ii) in feat: support multiple services per proto file and per Go package #349): every protoc run that generates Gorums code for a package writes the same gorums.pb.go next to its X_gorums.pb.go files, so separate runs and mixed packages with plain gRPC services work.
    pb.Config, pb.ID, and pb.Node stay as they are, and these names stay reserved.
    The file content depends only on the Go package name, so the runs never disagree.
    A proto file named gorums.proto in the same package is rejected, because protoc-gen-go would write the same file.
  4. One generic Correctable alias (B3; master no longer generates AsyncX aliases): type Correctable[Resp any] = gorums.Correctable[Resp] in gorums.pb.go replaces the per-type CorrectableX aliases.
    The generator no longer needs to decide which file declares an alias, and separate protoc runs work in every case.
    Correctable is now a reserved name, like Config, ID, and Node.
  5. Docs: a "Several Services" section in the user guide, a paragraph on several services in the package doc, the gorums.pb.go file in the user and dev guides, and a migration note for the CorrectableX aliases.
  6. Generated code, in the last commit.
flowchart LR
  cfg["gorums.Config (one, shared)"]
  alpha["pb.NewAlphaClient(cfg)<br/>.Heartbeat()"]
  beta["pb.NewBetaClient(cfg)<br/>.Heartbeat() .Write()"]
  node["pb.NewAlphaNodeClient(node)<br/>.Read()"]
  srv["gorums.Server<br/>RegisterAlphaServer + RegisterBetaServer"]
  alpha --> cfg
  beta --> cfg
  node -. "one node of cfg" .-> cfg
  cfg -- "pkg.Alpha.Heartbeat / pkg.Beta.Heartbeat" --> srv
Loading

Breaking changes

  • The per-type CorrectableX aliases are gone.
    Replace pb.CorrectableX with *pb.Correctable[*pb.X].
    doc/migration-v2-to-v3.md describes the change.
  • A proto message or enum named Correctable is now a generator error.

pb.Config, pb.ID, and pb.Node keep their names.
Regenerating moves these aliases from each X_gorums.pb.go into the new gorums.pb.go.
Some proto files whose generated code did not compile now get a clear generator error instead.

Notes

  • Proto files of one Go package may be compiled in separate protoc runs, in any order.
  • After removing the last Gorums file of a package, delete its stale gorums.pb.go.
  • The first four commits do not pass go test ./..., because internal/tests/multiservice gets its generated code later. Some commits also come before the commit that regenerates their output. The tip passes.

Testing

  • Generator: a multi-service fixture with golden output (two services, both with Heartbeat); failing cases for duplicate names within a file, across files, against server names, against a message in another file, against a message named Correctable (in the same file and in an imported file), for a client name shared by two services (FooNode with a quorum call and Foo with a remote call), and for options of a service that is not the first.
  • TestGeneratePackageFiles compiles two proto files in one Go package, also with a shared response type and the same RPC name in both, checks that both outputs exist, and checks which Go packages get a gorums.pb.go file (also with a messages-only first file, an imported file first, and two Go packages in one run). TestGeneratePackageFilesSeparateRuns compiles the output of two separate runs, in both orders, as one package, also without an import relation and with a plain gRPC import.
  • TestMultiServiceSharedConfig registers two services on one server and calls both through one gorums.Config (and one node client); each reply shows that it reached its own service.
  • go test ./... in the root, examples, and benchkit modules, with and without -tags=integration; make goplscheck, make modernize, and golangci-lint are clean.

@meling
meling force-pushed the feature/349/multi-service branch 2 times, most recently from 4d76daf to 5e7eaac Compare October 24, 2026 21:20
@meling
meling force-pushed the feature/349/multi-service branch from ba4e8a8 to 8fa54fc Compare October 25, 2026 09:26
@meling
meling force-pushed the feature/349/multi-service branch from 8fa54fc to e9dbc53 Compare October 25, 2026 20:57
@meling meling mentioned this pull request Oct 25, 2026
@meling
meling marked this pull request as ready for review October 25, 2026 23:35
Copilot AI lite review requested due to automatic review settings October 25, 2026 23:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

@meling
meling force-pushed the feature/349/multi-service branch from e9dbc53 to b56da97 Compare October 25, 2026 23:51
@meling
meling added this pull request to stack #391 October 25, 2026 23:51

@meling meling left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One alias-deduplication issue reproduced; details inline. The existing test suites pass in the root, examples, and benchkit modules. The nested-type naming issue reported on #379 also persists at this tip.

@meling
meling force-pushed the feature/349/multi-service branch from b56da97 to a4338e1 Compare October 26, 2026 23:27
@meling
meling force-pushed the feature/349/multi-service branch from a4338e1 to 0c6972e Compare October 27, 2026 00:08
@meling
meling force-pushed the feature/349/multi-service branch 3 times, most recently from a2fa5f4 to 743a8d7 Compare October 28, 2026 22:35
@meling

meling commented Oct 28, 2026

Copy link
Copy Markdown
Member Author

Restacked on the new #379 tip, which now generates two client interfaces per service (see the #308 design update). The branch was force-pushed.

  • The name checks cover the interface, constructor, and unexported struct of each client that a service generates, plus its server names.
  • Six commits that only made sense with node-argument calls and the helper methods are dropped.
  • The docs describe several services with the two-interface design.
  • One open point: for a plain gRPC service in an imported file, Gorums reserves the node client names (SNodeClient, NewSNodeClient, sNodeClient), but protoc-gen-go-grpc generates SClient, NewSClient, and sClient. A clash with those shows up only when the package compiles. The docs say this; the code is unchanged. Checking the gRPC names too would reserve more names, so I left it for you to decide.

@meling
meling force-pushed the feature/349/multi-service branch from 743a8d7 to 722fea2 Compare October 29, 2026 22:37
@meling

meling commented Oct 29, 2026

Copy link
Copy Markdown
Member Author

Restacked on the updated #379 (review fixes: unexported client names such as tlsNodeClient, InvalidArgument for ErrNoNode, shorter dev guide). The branch was force-pushed.

  • The detailed name-check paragraphs of the dev guide are removed, as in the gengorums: generate service clients #379 review. The rules are now in the doc comment of checkReservedIdents, including the plain gRPC service note.
  • Generated files are fresh at the tip, and the tests pass with and without -tags=integration.

@meling meling left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the current tip, 722fea2, against #379's ef8e0df.
I found no additional actionable defects in this PR's incremental changes.

One inherited issue remains: a protobuf oneof wrapper can collide with the new unexported client struct, and generation succeeds before the Go package fails to compile.
The reproducer and finding are in the review on #379, so I have not duplicated the inline finding here.
The separate client/Correctable-alias collision reported there is already rejected by this PR's packageIdents check.

The earlier response-type alias collision and mixed gRPC package concerns are addressed in the current source and covered by passing generator tests.
Validation: go test ./... passes in the root, examples, and benchkit modules.

Each service now has its own client types, so services in one file no
longer collide on RPC names. Remove the one-service check and validate
the methods of every service, not only the first.

Check that no two services in the Go package generate the same name,
such as NewFooClient from both NewFoo and Foo. Collect the generated
names from every file of the package in the protoc run, and include the
server names. Name the file of each colliding declaration.
Every generated _gorums.pb.go file declared the static aliases Config,
ID, and Node, so two proto files with Gorums services in one Go package
redeclared them. The aliases are useful to callers, so keep them, but
emit the static code only in the generated file of the first proto file
of the package with Gorums methods, in protoc's file order. In that
order, imported files come first. The other files of the package use
the same aliases.

A package's proto files must therefore be compiled in one protoc run.
Each generated file declared a CorrectableX alias for every quorum call
response type of its services. Two proto files of one Go package whose
quorum calls return the same message both declared the alias, so the
package did not compile. The aliases are useful to callers, so keep
them, but declare each alias only in the generated file of the first
proto file of the package, in protoc's file order, whose quorum calls
produce it. This is the order that places the Config and Node aliases.

The aliases are package-level names, so add them to the package-wide
name check: a message or enum must not take an alias name, and an alias
must not equal a client or server identifier of any service.
Services may share an alias.
meling added 16 commits October 30, 2026 00:19
A proto file, or a Go package, may now hold several services. Gorums
declares the static aliases Config, ID, and Node, and each Correctable
alias, in only one generated file per Go package. Add a user guide
section on several services, describe the once-per-package aliases and
the name checks in the developer guide, and note the new generated
names in the migration guide.
Regenerate with make dev and make genproto. The new multiservice test
package gets its generated code. The other generated files do not
change.
The static aliases (Config, ID, and Node) and each CorrectableX alias
belong to one file per Go package. The generator picked that file only
among the files that a protoc run generates. Thus, two proto files of
one package compiled in separate runs both declared the aliases, and
the package failed with "Config redeclared".

The generator now picks the owner among all files of the package in
the request, including files that are only imports of the run. A run
emits an alias only if it generates the owner; otherwise the run of the
owner emits it. So if b.proto imports a.proto from the same package,
the two files may be compiled in separate runs, in either order. Two
files that never import each other must still share one run.

The name collision checks also include the services of imported files
of the package, because those services generate their names into the
same package in their own run.

Every service of a Go package with Gorums code counts as a Gorums
service, so an imported file with a plain gRPC service in the same
package can own the aliases. The docs say to put such services in
another Go package.
The alias name CorrectableX uses only the Go name of the response type.
Two quorum calls of one Go package can return different types with the
same Go name, such as left.Response and right.Response. The generator
treated the matching names as one alias, so it declared the alias for
the first type only, and code that used it for the second type did not
compile. Which type won depended on the file order.

Record the response type of each alias, including its import path.
Share an alias only between quorum calls that return the same type,
and report two different types with the same alias name as an error
that names both types and the quorum calls that return them.
The Several Services section shows only configuration clients. Show
that a node client of a service reaches one node of the shared
configuration, built from a node of that configuration.
A Go package may hold a file whose service is plain gRPC. Such a file
is generated in a protoc run without the Gorums plugin, since Gorums
would generate its unannotated methods as remote calls. Tests need to
reproduce that run, so add Options.NoGorums, which runs only
protoc-gen-go.
The static aliases Config, ID, and Node went into the _gorums.pb.go
file of the first proto file of the Go package with Gorums methods,
counting imported files. A package that imports a same-package file
whose service is plain gRPC lost its aliases: the imported file owned
them, but no Gorums run generates that file. Separate runs of files
that do not import each other declared the aliases twice.

Each run that generates Gorums code for a Go package now also writes
gorums.pb.go next to the first such generated file. Its content
depends only on the package name and the tool versions, so every run
writes the same file. The per-file outputs no longer hold the static
code, and the static alias owner rule is gone. Correctable aliases
keep the owner rule: only a quorum call produces one, and a file with
a quorum call always has a Gorums run. Dev mode keeps dev/aliases.go
and writes no gorums.pb.go.
The static aliases Config, ID, and Node now live in one gorums.pb.go
file per Go package, which every protoc run writes with the same
content. Describe the file in the user, developer, and migration
guides. The separate-runs rule now applies only to Correctable
aliases, and a plain gRPC service may share a Go package with Gorums
services if its file is compiled without --gorums_out.
Regenerate with the generator that writes the static aliases into one
gorums.pb.go file per Go package. The _gorums.pb.go files lose the
alias block, and each generated package gains its gorums.pb.go file.
protoc-gen-go writes gorums.pb.go for a proto file named gorums.proto.
If that file shares the Go package and output directory of the Gorums
package file, the two outputs collide: protoc rejects the second write,
and separate runs overwrite each other. Report the proto file and ask
to rename it, whether it is a Gorums file or only an import.

Also rewrap the TestBundleReservedIdentifiers doc comment.
The NoGorums change left one doc comment line at about 110 columns.
Rewrap the comment to the usual width.
Only runs that generate Gorums code for a package write its
gorums.pb.go, and no run removes a stale one, so say to delete it
after removing the last Gorums proto file. Mention the new error for
a proto file named gorums.proto. The release guide said GenVersion is
in every generated file; gorums.pb.go has version header lines but no
version check.
The multiservice fixture and the inline test protos use the
internal.testprotos proto package, like the other test protos under
internal/testprotos.
The package doc describes the clients of one service. State that a Go
package may hold several services, that each gets its own clients, that
the services may share one configuration and one server, and that their
generated names must not collide.
One function now computes the Correctable aliases that a file declares,
and the template ranges over them. This replaces four helpers and the
Declared template field. packageIdents uses two plain loops instead of
a closure with a side map; its error messages are unchanged. The
generated code is unchanged.
The generator declared one CorrectableX alias per quorum call response
type X. Files of a Go package shared these aliases, so one file had to
own each alias. This needed an owner rule across protoc runs, an error
for two response types named X from different Go packages, and care
when compiling files that do not import each other in separate runs.

The static code now declares Correctable[Resp any] once per package in
gorums.pb.go, next to Config, ID, and Node. The _gorums.pb.go files
declare only the names of their own services, so the owner rule, the
types template, and the alias checks are gone. packageIdents now only
checks service names, with helpers for the names of a service and the
collision error. Correctable becomes a reserved identifier.
@meling
meling merged commit f576e4e into master Oct 30, 2026
3 checks passed
@meling
meling deleted the feature/349/multi-service branch October 30, 2026 19:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: support multiple services per proto file and per Go package

2 participants