Case studies · Tutorial 07
Run a Multi-Agent Coding Team with Herdr
Install Herdr, keep coding agents in persistent terminals, and coordinate isolated workers from one orchestrator without confusing status with proof.

0 of 12 complete
Last tested and updated: August 17, 2026, against Herdr 0.8.0. Herdr, Grok, and worker-agent CLIs change quickly; check the linked official documentation before using this on a production repository.
Herdr is the terminal runtime our team prefers when one coding agent is no longer enough. It keeps each agent in a real terminal, preserves the session when you detach, shows which workers are active or blocked, and exposes the same controls to people, scripts, and an orchestrator agent.
Our working pattern has three distinct layers:
| Layer | Job | What it does not prove |
|---|---|---|
| Smart orchestrator model | Break the goal into bounded tasks, choose workers, inspect evidence, and decide what happens next | That a worker actually ran or that its output is correct |
| Herdr | Create and organize terminals, start agents, submit prompts, observe state, read output, and keep sessions available | Filesystem or credential isolation |
| Isolated worker environment | Give each worker its own checkout, processes, and allowed resources | That the merged result passes review |
We use a high-capability Grok model for the orchestration role and isolated worker environments for execution. Herdr is the connective tissue. You can use another supported agent as the orchestrator; the important part is the contract between layers, not the logo on the model.
Figure 1: Herdr’s official site presents it as the runtime coding agents live on. This is a dated product screenshot, not a mockup of our team workflow.
What you will finish
By the end of this tutorial you will have:
- installed and verified Herdr;
- opened one repository as a persistent workspace;
- started named worker agents in separate panes or environments;
- prompted, waited, and collected results through the agent-aware CLI;
- handled blocked, stalled, timed-out, and unknown states without blindly sending keys; and
- defined a completion gate based on artifacts and tests rather than an agent saying “done.”
Why Herdr works for multi-agent jobs
A normal terminal tells you what is on screen now. A multi-agent runtime must also answer: Which repository owns this process? Which worker is still running? Which one needs approval? Can the orchestrator read the result without stealing my focus? Will the work remain available if I disconnect?
Herdr gives you three primitives:
- a workspace organizes one project, task, or investigation;
- a pane is a real shell for an agent, test process, server, or logs; and
- an agent is a supported coding-agent process that Herdr can identify and address by lifecycle state.
This separation is why the system scales better than a wall of anonymous terminal tabs. The orchestrator can target researcher instead of guessing that “the third pane on the right” still contains the same process.
Herdr’s server owns the panes. The terminal UI is only a client. Detach the client and the processes continue on an awake host; reattach later and the live panes are still there.
Before installing
You need:
- macOS or Linux for the current stable path; native Windows remains preview-only beta;
- at least one supported coding-agent CLI already installed and authenticated;
jqfor the automation examples that parse Herdr’s JSON responses;- a disposable repository for the first run; and
- separate budget controls for every model provider or paid agent subscription.
Check the tools you plan to use:
git --version
jq --version
grok --version # only if Grok CLI is your orchestrator
codex --version # example worker; use the CLI you actually run
Do not paste API keys into Herdr prompts, screenshots, or repository files. Herdr owns real terminals, so processes can see whatever the current OS user and environment can see.
Install Herdr
On macOS or Linux, the official direct installer is:
curl -fsSL https://herdr.dev/install.sh | sh
If you already use Homebrew:
brew install herdr
The direct installer and Homebrew are different update paths. Use one; do not layer both and then guess which binary your shell found.
Open a new shell, then verify the binary:
herdr -V
herdr status
For this tutorial, the expected version line is:
herdr 0.8.0
If your version is newer, use the installed binary as the syntax authority:
herdr --help
herdr agent
herdr pane
herdr workspace
Do not run a mutating nested command without understanding it. herdr workspace create, for example, is valid with defaults and will create state.
Start the first workspace
Move into a disposable repository and launch Herdr:
cd /path/to/disposable-repository
herdr
When the session has no workspaces, Herdr opens one. Use the mouse if you prefer: click panes and workspaces, drag split borders, and right-click to create a pane. Keyboard users can press ctrl+b, then v to split right or - to split down.
Detach without stopping work:
ctrl+b q
Reattach:
herdr
Do not use herdr server stop as a detach command. Stopping the server exits its pane processes.
Teach the orchestrator how to drive Herdr
Herdr ships a release-matched agent skill. Print it from the installed binary:
herdr --skill
For coding agents that support reusable skills, the official installation path is:
npx skills add herdrdev/herdr --skill herdr -g
Omit -g for a project-scoped skill. The skill starts with a useful safety check: the controlling agent should use Herdr only when HERDR_ENV=1 confirms it is running inside a Herdr-managed pane.
If Grok CLI is the orchestrator, install Herdr’s Grok integration after Grok has created its config directory:
herdr integration install grok
herdr integration status
The current integration reports Grok session identity through a SessionStart hook. Lifecycle state still comes from Herdr’s screen detection. The hook lets Herdr resume a stored session with Grok’s native grok --resume <id> path after a server restart; it does not configure the model or grant Grok extra permissions.
Start the orchestrator inside a Herdr pane using the Grok command and model selection already verified for your account. Then give it an operating contract like this:
You are the orchestrator for this repository.
Before delegating:
- restate the acceptance criteria;
- divide work into non-overlapping lanes;
- assign each lane an explicit output file or verifiable result;
- use only the isolated environment path assigned to that worker;
- never approve a permission or destructive action automatically.
For every worker:
- name the agent;
- set a finite timeout;
- if it is blocked, inspect the prompt and ask me for the decision;
- if it becomes idle or done, read the artifact and run the named verification;
- treat unknown state, a completion message, or a plan as unverified.
Finish with changed files, checks run, failures, unresolved decisions, and costs or usage that can be observed.
This prompt is a policy, not an isolation boundary. Enforce access with the worker environment and OS permissions.
Create named workers from the CLI
The simplest first exercise keeps workers in sibling panes of a disposable checkout. From the orchestrator’s Herdr pane:
test "${HERDR_ENV:-}" = 1
research_split=$(herdr pane split --current --direction right --cwd "$PWD" --no-focus)
research_pane=$(printf '%s\n' "$research_split" | jq -r '.result.pane.pane_id')
review_split=$(herdr pane split --current --direction down --cwd "$PWD" --no-focus)
review_pane=$(printf '%s\n' "$review_split" | jq -r '.result.pane.pane_id')
Always parse returned IDs. Do not predict them from sidebar order.
Start the worker CLIs you actually use. These examples use Codex only to show Herdr’s syntax:
herdr agent start researcher --kind codex --pane "$research_pane"
herdr agent start reviewer --kind codex --pane "$review_pane"
Native agent arguments go after --. Model flags belong to that agent CLI, not Herdr:
herdr agent start reviewer --kind codex --pane "$review_pane" -- -m <verified-worker-model>
Name agents for roles and outputs, not model brands. reviewer remains meaningful when you change the provider.
Check the live inventory:
herdr agent list
herdr agent get researcher
herdr agent get reviewer
Give every worker an artifact contract
Avoid prompts such as “research this” or “fix the app.” A worker contract should name its scope, output, and check.
herdr agent prompt researcher "$(cat <<'PROMPT'
Research the current authentication flow in this disposable repository.
Read only; do not edit source files.
Write findings to /tmp/auth-research.md with:
1. entry points,
2. data flow,
3. risks with file-and-line evidence,
4. unresolved questions.
Reply with only the artifact path.
PROMPT
)" --wait --timeout 180000
Then collect the visible result:
herdr agent read researcher --source recent-unwrapped --lines 120
For full-screen agents, terminal history can be incomplete. The artifact file is the durable handoff. Read and verify that file directly before using it.
Use a separate reviewer contract:
herdr agent prompt reviewer "$(cat <<'PROMPT'
Review /tmp/auth-research.md against the current repository.
Do not edit source files.
Report unsupported claims, missing evidence, and the three highest-risk gaps.
Write the review to /tmp/auth-review.md and reply with only that path.
PROMPT
)" --wait --timeout 180000
This structure makes parallel work composable. The orchestrator does not need to reconstruct a long answer from terminal scrollback, and the reviewer checks a named artifact rather than a vague memory of the task.
Put workers in isolated environments
A Herdr workspace is an organizational container, not a security boundary. Two panes in the same checkout can overwrite the same file. A Git worktree reduces file collisions, but every process still runs as the same OS user unless another tool isolates it.
For low-risk local work, Herdr can create separate Git worktree checkouts:
herdr worktree create --cwd "$PWD" --branch agent/research --label research --no-focus
herdr worktree create --cwd "$PWD" --branch agent/build --label build --no-focus
Herdr groups those checkouts as workspaces. Closing a workspace does not delete the checkout or branch. Removing a worktree is a different, potentially destructive command; Git will refuse a dirty checkout unless --force is supplied.
For the team production stack, create the environments with Gas Town first, record their exact paths, and open those paths in Herdr:
research_env=/absolute/path/returned/by/gastown-for-research
build_env=/absolute/path/returned/by/gastown-for-build
test -d "$research_env" && test -d "$build_env"
herdr workspace create --cwd "$research_env" --label research --no-focus
herdr workspace create --cwd "$build_env" --label build --no-focus
Use the exact commands from the linked Gas Town tutorial; do not invent the paths or assume that different directories imply different credentials. The isolation layer should define filesystem access, secrets, network access, and cleanup.
Herdr also supports host-visible VM and sandbox wrappers. When a wrapper hides the real agent process, set the documented process hint on the host-visible command, for example:
HERDR_AGENT=claude <sandbox-wrapper> -- claude
Set that hint only on the wrapper process. Do not export it globally.
Understand the state machine
Herdr reports five useful states:
| State | Operational meaning | What the orchestrator should do |
|---|---|---|
working | The recognized agent appears active | Wait with a finite timeout; do not add duplicate instructions |
blocked | Herdr recognized an approval, permission, or question UI | Read the pane and route the decision to a human |
done | Background work settled and has not yet been viewed | Read the output and verify the artifact |
idle | The agent is ready for input and has been seen | Inspect the result before assigning more work |
unknown | Herdr cannot classify the agent confidently | Inspect manually; never treat this as success |
agent prompt --wait watches lifecycle state, not a unique task ID. If you prompt an already working agent, the end of its current turn may satisfy the wait. For precise production work, assign one bounded task at a time and require a uniquely named artifact.
Handle blocked agents safely
Wait explicitly for a human decision point when that is the event you need:
herdr agent wait reviewer --until blocked --timeout 120000
herdr agent read reviewer --source recent-unwrapped --lines 80
If the UI is a harmless test and you have deliberately decided to dismiss it, use logical keys:
herdr agent send-keys reviewer esc
Do not build an orchestrator that sends enter whenever a worker is blocked. A block can be a deployment approval, a file deletion, a credential prompt, or a question whose answer changes the task.
Verify outcomes outside the agent conversation
An agent’s “done” state means the terminal settled. It does not mean the code is correct.
Use a separate pane for deterministic checks:
test_split=$(herdr pane split --current --direction down --cwd "$PWD" --no-focus)
test_pane=$(printf '%s\n' "$test_split" | jq -r '.result.pane.pane_id')
herdr pane run "$test_pane" "npm test"
herdr pane wait-output "$test_pane" --regex "passed|failed|error" --timeout 180000
herdr pane read "$test_pane" --source recent-unwrapped --lines 160
pane wait-output searches terminal text; it does not understand whether a test suite succeeded. Existing output can also match. Prefer the command’s real exit code or a machine-readable report when your harness exposes it. The terminal read is evidence for a human, not a replacement for CI.
Use this completion language:
- Prepared: the orchestrator created a plan or prompt.
- Running: a named worker is active.
- Reported done: the worker settled and produced a claimed result.
- Verified: a named artifact was inspected and an independent check passed.
- Integrated: accepted changes were merged and the combined checks passed.
Failure handling
The prompt returns agent_blocked
Herdr did not send the prompt because the agent was already at an approval or question UI. Inspect it:
herdr agent get <name>
herdr agent read <name> --source visible
Route the decision to a human. Do not retry the prompt until the block is resolved.
The prompt returns agent_prompt_stalled
Herdr accepted the prompt but observed no lifecycle change within five seconds. The agent may not have received Enter, may be in an unexpected UI state, or detection may be wrong. Read the visible pane and inspect detection:
herdr agent read <name> --source visible
herdr agent explain <name> --verbose
A wait times out
Timeouts and server errors print JSON to stderr and exit with status 1; CLI syntax errors exit with status 2. A timeout means “not settled within this window,” not “failed.” Inspect the current state and output before deciding whether to extend, interrupt, or cancel.
The state looks wrong
Run:
herdr agent explain <name> --verbose
herdr server update-agent-manifests
Herdr’s blocked detection is deliberately strict. A new agent UI may appear idle instead of blocked until its manifest understands that screen shape.
The result scrolled out of view
Try:
herdr agent read <name> --source recent-unwrapped --lines 160
If a full-screen agent still does not expose the complete response, ask it to write a Markdown artifact and return the file path. Do not enable persistent pane history casually; terminal output may contain secrets and tokens.
Herdr updated but the running server is old
Check:
herdr -V
herdr status
For direct-installer versions, herdr update manages the binary. Homebrew, mise, and Nix installations must update through their package manager. Stopping a session to switch servers exits its pane processes, so wait for a safe checkpoint first.
You need diagnostic logs
The default log directory is ~/.config/herdr/:
herdr.log
herdr-client.log
herdr-server.log
For a debug reproduction:
HERDR_LOG=herdr=debug herdr
Review logs for credentials, prompts, repository paths, and customer data before sharing them.
Security boundaries
Treat every agent pane as a process running with your user account’s access unless the environment layer proves otherwise.
- Workspaces are not sandboxes. They organize panes.
- Worktrees are not security isolation. They separate Git checkouts but normally share the same user, network, and credentials.
- Environment variables propagate authority. Pass only the credentials a worker needs; prefer short-lived tokens.
- Blocked input belongs to a human by default. Never auto-approve deployment, deletion, purchase, or credential prompts.
- Terminal output can contain secrets. Pane history is off by default for a reason.
- Remote access uses SSH. Verify ordinary
ssh <host>first, use passphrase-protected keys andssh-agent, and keep the remote host patched. - Stop is destructive to running processes.
herdr server stopandherdr session stop <name>end pane processes. - Plugins and integrations execute code. Install only the official integration or a plugin whose source and permissions you reviewed.
Cost controls
Herdr does not add a per-token model charge. The expensive part is concurrent model work and the compute that hosts it.
Use four controls:
- Give the orchestrator a planning budget and workers smaller execution budgets.
- Set finite timeouts for every wait; an indefinite wait is Herdr’s default when
--timeoutis omitted. - Start with two workers. Add concurrency only when lanes are genuinely independent.
- Require artifacts and verification so failed work is detected before another expensive round begins.
Do not publish static Grok 4.6 pricing until xAI exposes an official model page or your account’s live model record. The xAI API provides a GET /v1/models endpoint that returns the models and prices available to the authenticating account; the account response is the correct current check.
A production orchestration loop
Use this sequence for real work:
- Define: write acceptance criteria, excluded scope, and the verification command.
- Isolate: create one Gas Town environment per write-capable lane; record the returned paths.
- Place: open each path as a Herdr workspace or pane without stealing focus.
- Start: launch one named worker per lane and verify it reached
idle. - Prompt: send a bounded task with an artifact path and finite timeout.
- Observe: wait on lifecycle state; route
blockedto a human; inspectunknownmanually. - Collect: read the artifact from the isolated environment, not only the terminal transcript.
- Verify: run tests, lint, type checks, review, or source validation independently.
- Integrate: merge only accepted work and rerun the combined checks.
- Close: record unresolved work and observed costs, then clean up environments deliberately.
This is why the stack works: the smart model spends its attention on decisions, Herdr makes execution observable, the isolation layer prevents workers from trampling one another, and deterministic checks control the final claim.
Completion gate
Do not mark this tutorial complete because three agents are visible. Mark it complete when all six statements are true:
herdr -Vrecords the installed version.- A disposable workspace survives detach and reattach while the host remains awake.
- Two named workers appear in
herdr agent list. - Each worker produced a separate, inspectable artifact.
- A blocked or timed-out test was handled without blind approval.
- An independent check—not the worker’s prose—decided whether the result passed.
Sources and further reading
- Herdr official site
- Install Herdr
- Quick start
- Herdr concepts
- Agent automation
- Supported agents and state authority
- Integrations, including Grok CLI
- Session state and restore
- Persistence and remote access
- CLI reference
- Troubleshooting and logs
- Official Herdr repository and Apache-2.0 license
- xAI models documentation
- xAI REST models endpoint