Wiring Go at scale across many teams
A few years ago I worked on a Go service that translated a free-text DSL into structured SQL. Each query was assembled from a half-dozen inputs scattered across the codebase. The pieces moved through the call stack as function arguments, and whether they actually ran was decided by whichever package finally invoked them. Channels were threaded through the same call paths to handle timeouts and async fetching. To understand what a single request did, you had to follow a closure through four or five packages and keep track of which channel was carrying which signal at each point. The translation logic itself was small. The wiring around it was where most of the difficulty lived.
This was the predictable result of ten developers across three teams writing into the same Go service with no enforced boundary between their work. Every time someone needed a value from another team's package, the path of least resistance was to add another parameter, another channel, or another package-level singleton. After a year the wiring was the application, and the actual business logic was a few thousand lines buried inside it.
This post is about a Go library I wrote called graft and the specific class of multi-team rot it is designed to remove.
The shape of the rot#
Most large Go services I've seen accumulate the same patterns. They're not anyone's fault individually. They're the local minimum that any single PR falls into when there is no boundary telling it not to.
Some of the usual ones:
- Implicit ordering in
main. A long sequence ofNew...calls where the order matters and nobody documents why. Move two lines and a different package'sinit()panics. The team that owns the database doesn't know that the team that owns the rate limiter relies on the database being up before its first request. - Channels passed through too many packages. Each handoff loses information about the contract. Who owns close? Is it buffered? What happens on a slow consumer? By the fourth package the channel is just an opaque pipe and the assumptions live in three different teams' heads.
- Cross-team plumbing in shared types. Team A needs a value that team B
computes, so a field gets added to a shared
Contextstruct or a middleware. Team C now also depends on that field existing. Removing it is a multi-week coordination project. - Hidden fan-out in request handlers. A handler grows from one fetch to
five over the course of a year. If the fetches run concurrently, the
errgroupblock grows alongside it, and after enough additions the request path has nineteen subtly-different errgroup blocks and a third of them have subtle bugs. (I counted, in a real service.) If the fetches run sequentially, they get extracted into their own packages to keep the handler readable, and now the handler hands a state object off to each one in turn. After enough rounds of that, you can read the handler end-to-end and not realize it has five distinct dependencies, because the state struct that gets shunted between them looks like one thing. The plumbing hides the shape of the work. sync.Onceladders. Each team adds memoization for the value they own. After a while the cache topology is undocumented and the invalidation story is "restart the pod."
Every one of these is the same underlying problem. There is no enforced boundary between what one team produces and what another team consumes. Without that boundary, every shortcut is locally rational and the application as a whole becomes harder to change every week.
There are of course ways to prevent this. A strong engineering culture with empowered tech leads can push back on poor design and prioritize paying the debt down before it compounds. But most engineering orgs aren't structured for that kind of long-horizon investment, and the local pressure on any given PR is always for the smaller fix that ships this sprint.
What a real boundary needs to look like#
A boundary that holds across teams has to satisfy a few things:
- It has to be a typed contract. A team should not be able to depend on another team's internal representation, only on a value the other team has explicitly agreed to expose.
- It has to be enforced by the compiler or by CI, not by code review or documentation. Anything that depends on humans noticing will fail at three teams and ten developers.
- It has to eliminate the plumbing, not just rename it. If the only way to use the boundary is to thread a value through five packages, the boundary will get bypassed.
- It has to allow independent ownership. A team should be able to change how they produce their value without coordinating with every team that consumes it, as long as the typed output stays the same.
Go's existing tools get partway. Packages are typed. Interfaces are typed.
But neither of them addresses the wiring problem. A package interface tells
you what foo.Service looks like. It doesn't tell you how to construct one,
when to construct it, what it depends on, or how its lifecycle relates to
everything else in the binary. That's where the rot lives.
What graft does#
Graft is a small Go library that turns the wiring into a declared graph.
Each unit of work is a node. A node is a package owned by exactly one
team. It declares an ID, a typed output, the IDs of nodes it depends on, and
a Run function that produces the output.
package user
const ID graft.ID = "user"
type Output struct{ User User }
func init() {
graft.Register(graft.Node[Output]{
ID: ID,
DependsOn: []graft.ID{tenant.ID},
Run: run,
})
}
func run(ctx context.Context) (Output, error) {
t, err := graft.Dep[tenant.Output](ctx)
if err != nil {
return Output{}, err
}
return Output{User: fetchUser(ctx, t.TenantID)}, nil
}
The team that owns user data owns this package. Nobody else writes in it. The output type is the only thing other teams can see. Internal types, helpers, and intermediate values stay inside the package and stop being part of the cross-team API surface.
When the binary calls graft.Execute(ctx), the engine assembles every
registered node into a graph, runs nodes whose dependencies are satisfied,
and feeds the typed outputs forward. Independent nodes run concurrently. A
node that returns an error cancels the rest. There is no main doing
twenty New... calls in a careful order. The order is in the graph definition.
The boundaries this gives you#
Each team gets its own package, with no shared state#
A node has no way to reach into another node except through the typed
output that node declares. There is no shared mutable struct. There is no
package-level variable that a downstream team can mutate. The user team only
exposes user.Output.
The path of least resistance for sharing a value between teams is no longer "add another field to the shared context" or "pass another channel through five packages." It is "the team that produces this value declares it on their node's output type." The boundary sits at the point where one team's output becomes another team's input, which is the only place it can hold.
The wiring disappears#
The DSL-to-SQL service from the start of this post would, in a graft
codebase, be a set of nodes for the inputs that contributed to the query
and a node for the assembled SQL that depended on them. The downstream
code that needed each input would declare it in DependsOn and call
graft.Dep[...] to read it. There would be no closures threaded through
five packages, no channels carrying signals between layers, no shared
state object passed hand-to-hand, and no manual ordering. The graph runs
the producers in parallel where it can, hands the typed outputs to the
consumers, and cancels everything on the first error. The plumbing that
made the original service hard to reason about would not be refactored
into something better; it would not be written in the first place.
Compile-time and CI enforcement#
If a node's DependsOn references an ID that doesn't exist, the build
fails. If a node depends on another node's output type that has changed
shape, the build fails. If two teams introduce a cycle, it shows up at
runtime today and there is a check (graft.AssertDepsValid) that catches
it in CI.
func TestDeps(t *testing.T) {
graft.AssertDepsValid(t, ".")
}
That single test, run on the merged repo, catches a class of multi-team bugs that would otherwise show up at deploy time. A team can't accidentally break another team's node. They can fail to satisfy its declared input type, but that becomes a compile error in the team that introduced the problem, not a 3am page for the on-call.
Fan-out and per-execution caching come for free#
Two of the patterns in the rot list, hand-rolled errgroups and sync.Once
ladders, go away because the engine handles them. Independent nodes run
concurrently because the topology says they can. A node marked
Cacheable: true is computed once per execution and reused by every
downstream consumer. The fetch happens once for this request and the next
request gets a fresh fetch.
Each team can ship independently#
Because a team's API is a typed output, a team can change everything about how they produce that output without coordinating with anyone. They can swap the database, add caching, or rewrite the function in a different style. As long as the output type is the same, downstream teams don't need to know.
This is the part that is hard to feel in a small codebase and load-bearing in a large one. The thing that makes a 10-team service slow to change is that no team can move without checking in with the others. Graft's typed boundary moves that coordination from "every PR" to "the rare PR that changes the output shape."
A note on what graft isn't#
Graft could be compared to dependency injection containers like Wire and
Fx, and there's substantial overlap. Most of what people use Wire or Fx
for can be done with graft. Register your database, logger, and config
as nodes, mark them Cacheable: true with a process-wide cache, and let
your request-handler nodes depend on them via graft.Dep[...]. The
first request that triggers an ExecuteFor walks back through those
dependencies and constructs them; the cache holds them for the life of
the process; every subsequent request reuses them. That's the same
shape of problem Wire and Fx solve, just without a separate startup
wiring phase.
Where the libraries diverge is in what each one is opinionated about. Fx ships an explicit lifecycle model with start and stop hooks, ordered graceful shutdown, and a runtime invocation model. Wire ships codegen that turns your provider set into a generated wiring file with no runtime overhead. Graft doesn't ship either of those. If you want managed lifecycle hooks, Fx is going to have a better story for you. If you want zero-runtime-cost wiring, Wire is going to have a better story for you.
Where graft has an opinion the DI containers don't is in the
per-execution case. You can use Wire for per-request injection by
writing an injector that takes the request and returns whatever value
tree you ask for, and Wire will generate the code for it. What Wire
doesn't give you is parallel execution between independent branches,
per-request memoization of shared intermediates, or partial-graph
execution as a primary API. Those aren't features it ships, and you'd
hand-roll them. Fx is mostly built around application bootstrapping
with explicit lifecycle hooks, and per-request work is possible but
awkward to model in it. Graft puts the per-execution case at the center
of the model. The Cacheable flag handles per-request memoization,
independent nodes run in parallel without any errgroup boilerplate, and
ExecuteFor[T] walks back from a desired output to compute only what's
needed. The same graph definition holds your long-lived singletons and
your per-request derivations, and the engine treats them uniformly.
It is also not a workflow engine. There is no durability, no distribution, no retries across processes. The graph describes a single in-process execution. If you need long-running work across machines, reach for Temporal or something like it.
When this is worth it#
Graft works fine on a small service. The registration through init() is
one extra line per node compared to writing a constructor, and the
concurrency, per-execution caching, and typed boundaries between packages
all still apply. However the value compounds with team count.
The argument is strongest when one of these is true:
- Multiple teams contribute to the same binary and the wiring is becoming the integration surface.
- The wiring across packages has gotten bad enough that the org is reaching for Wire, Fx, or some other DI system to manage it.
- The request or batch path has enough independent work that hand-rolled errgroups or custom abstractions are accumulating.
- There are enough optional outputs that subgraph execution starts to matter (handlers with conditional fields, CLIs with many subcommands, reports that pull a slice of metrics).
- Multiple consumers are computing the same intermediate value and ad-hoc caches are starting to appear.
A short case study#
To make the multi-team argument concrete, consider a fictional product I'll call Atlas. Atlas is a B2B analytics dashboard. The main render endpoint returns a workspace home page that pulls from six teams' worth of data:
- The identity team owns the user, their roles, and their tenant.
- The billing team owns the workspace's plan, seat count, and any current overages.
- The catalog team owns the list of data sources the workspace has connected.
- The metrics team owns aggregate query results across those sources.
- The alerts team owns the active and recently-fired alerts for the workspace.
- The growth team owns the in-product nudges, banners, and onboarding prompts the user might see today.
Six teams collaborating on one rendered endpoint, each with their own internal abstractions and dependencies. The rot grows in the seams between them, in the wiring code that has to thread state from one team's package to another's.
The home endpoint has to be wired with the entry point of every team's
subtree, and each of those entry points pulls a half-dozen of its own
dependencies through its constructor. By the time the endpoint is
composed, the constructor takes twenty arguments, and a third of them are
themselves objects holding ten more. The wiring lives in
cmd/server/main.go and gets longer every quarter.
Then someone needs to thread a new value through. It's owned by the identity team and needed by the alerts team. The natural path runs it through the home endpoint constructor so the alerts subtree can be given it, and then down through the alerts team's own internal abstractions to the function that uses it. The change ends up being three PRs across two teams and a coordination thread on Slack, even though the underlying work is one new field being read in one place.
Execution order is its own problem. Each team owns the order inside its
own subtree, but no one owns the global picture. When billing's startup
gets slower, identity has to be told because identity reads from billing
during request handling and assumes its values are warm. None of that is
documented. It's encoded in the order of constructor calls in main and
the lifecycle of a few singletons, and it changes every time someone
reshuffles main to add a new dependency.
In a graft codebase, the same endpoint is a single ExecuteFor[home.Output]
call. Each team owns a node:
// teams/identity/identity.go
graft.Register(graft.Node[Output]{
ID: ID,
DependsOn: []graft.ID{request.ID},
Run: run,
})
// teams/billing/billing.go
graft.Register(graft.Node[Output]{
ID: ID,
DependsOn: []graft.ID{identity.ID},
Run: run,
})
// teams/catalog/catalog.go
graft.Register(graft.Node[Output]{
ID: ID,
DependsOn: []graft.ID{identity.ID, billing.ID},
Run: run,
})
// teams/home/home.go
graft.Register(graft.Node[Output]{
ID: ID,
DependsOn: []graft.ID{
identity.ID, billing.ID, catalog.ID,
metrics.ID, alerts.ID, growth.ID,
},
Run: assemble,
})
The graph engine runs identity first because everyone depends on it,
billing next, then catalog, metrics, alerts, and growth in parallel because
none of them depend on each other. The home node assembles the typed
outputs into the final response. There is no errgroup written by hand,
because the topology declares the order of execution. There is no shared
RenderContext, because each team's surface is its own Output type.
You also get the kind of encapsulation that usually only comes from
splitting into separate services. When the metrics team wants to swap
their backing store from Redshift to BigQuery, they change their node's
Run function and ship. The home team
doesn't know. When the growth team wants to add a new nudge condition,
they change their node's Output type and the only callers that have to
update are the ones that read the new field. When a new endpoint needs the
same data minus the growth nudges, it's a one-line ExecuteFor against a
node that depends on five of the six instead of all six.
A disciplined team can build this same shape with constructor injection,
a careful main, and rigorous code review. The point of graft is that you
get this shape by default, not by fighting for it on every PR.
Try it#
The code is at github.com/grindlemire/graft.
The examples/ directory has the multi-team pattern with each node in its
own package, the AssertDepsValid check wired into a test, and a Mermaid
diagram generator so you can show a team what the graph looks like
when their nodes are added.
If you've worked in a large Go service that grew the kind of rot I described at the top of this post, I'd love to hear what the failure mode looked like in your codebase, and whether the boundary I'm describing here would have helped.