④ What happens after
The session ends, or is confirmed running, and three things follow: a comment on the issue,
a notification if I asked for one, and — eventually, and only if I turned it on — the disk
back.
What it writes on the issue
Two machines now run this, and a worktree path is not an address. So when a session is
confirmed, the issue is told where it is:
🤖 robot-army dispatched a session for this issue.
- Host: `phoenix`
- Session: `ra-robot-army-38`
- Session id: `2f1c9c3e-6a54-4a0b-9f0d-4c2a1d8e77b1`
- Branch: `robot-army/issue-38-issue-comments-on-dispatch`
- Worktree: `/home/jantman/worktrees/robot-army/issue-38`
Both session handles are there because they are the two different things I search with: the
name is what appears in the tab title and the /resume picker, the id names the
transcript, the log records and the exit spool. The branch is the link to the pull
request — a PR for this work is opened from it — so an issue, its PR and its session logs
are one chain from either end.
Resume or restart an item and the next comment says so rather than repeating itself:
🤖 robot-army reassigned this issue to a new session (attempt 2).
…
- Continues: `2f1c9c3e-…` (that session's context was restored)
A restart says Supersedes: instead, and says the new session starts without that
session’s context — which is the difference between reading the earlier transcript for
context and reading it for facts that no longer apply. If no earlier session is on record
(a rebuilt database), it says that rather than naming one.
A failed attempt gets its own comment, and it is two lines:
🤖 robot-army could not start a session for this issue.
- Host: `phoenix`
- Work item: `126`
The reason is deliberately not on it, and the comment reads the same whatever went
wrong. It used to carry the reason in a fenced block, on the argument that machine text of
unbounded shape needs a fence — which answered the wrong question. A reason interpolates git
exception text, paths under my home directory, a repository’s settings filenames and the
exact local command that clears the condition, and the issue it lands on is often public and
usually not mine. Item 126 published a tool-permission fingerprint diff onto somebody else’s
repository that way, where I cannot take it back.
The host stays, because trust is granted per machine and “it works on the other one” is a
real case. The item number is what replaces the reason: it means nothing to a stranger, and
it is the number robot-army show takes. The reason is recorded in full in three places,
none of them public — the state.work_item audit record, the item’s own columns, and the
notification, if a channel is configured:
robot-army show 126 # prints `failure : …`
It is also on the item’s page and on its card under needs me, which is the surface that
now has to be legible because the issue no longer is.
Three rules hold throughout:
A comment that fails to post is logged and otherwise ignored: GitHub being down is not a
reason to fail a session that is running. Nothing retries it, so a comment lost to a crash
between confirmation and the POST stays lost — robot-army show <id> still knows
everything the comment would have said.
Being told when something happens
Off by default — nothing is sent until I ask. There are two channels, and either, both, or
neither may be configured.
[health]
webhook_url = "https://ntfy.sh/my-private-topic" # a generic JSON POST
[pushover] # a push notification on my phone
token_file = "~/.config/robot-army/pushover-token"
user_key_file = "~/.config/robot-army/pushover-user"
[notifications]
events = ["failure", "needs_info"] # dispatch | completion | failure | needs_info
max_per_cycle = 5
Both channels get every message. Adding Pushover does not replace the webhook, and one
channel failing never stops the other — each outcome is recorded separately, so the log can
say “the webhook took it and Pushover did not”.
Why Pushover needs its own channel rather than the webhook. The webhook posts JSON.
ntfy accepts that; Pushover does not — it takes form-encoded parameters and rejects a
JSON body. health.post_json’s docstring claimed for two milestones that a generic webhook
covered both, and pointing webhook_url at Pushover produced a rejected request rather
than a notification. That is issue #106.
Getting the two credentials. The API token comes from an application registered at
https://pushover.net/apps/build; the user key is on the account dashboard. Each goes in
its own file, and each file must be mode 0600 — the same rule the GitHub and Trello
credential files follow. Both keys must be set or neither: a half-configured channel cannot
send, so it is a load error rather than a channel that silently never fires. A credential
written into config.toml instead of a file is also a load error, because this repository
is public.
At most max_per_cycle messages per daemon tick, then one summary naming how many were
held back and of which kinds. The bound is per burst rather than per event, because a
backlog produces different items and per-item de-duplication would not bound it at all. It
counts messages, not deliveries, so configuring a second channel does not halve how many
things I am told about. Every send is in the audit log whether or not it left the machine.
Messages carry identifiers and state names only. There is no field a credential could
reach, and a test asserts it across a run that includes an authentication failure.
The failed-health alert goes to every configured channel too — the one message that
matters most is the one saying the daemon itself has stopped, and a channel that could not
carry it would be the wrong half. Since issue #52 it also says which stopping: its body
carries the check’s verdict, so DIED and HUNG reach a phone as the different problems
they are — what each one means. Unlike the notifications above, that alert is not gated
by the effect level and never has been: robot-army health --notify takes no
--effect-level flag, so gating it would silently disable the dead-man’s switch whenever
I am running the daemon at local.
The session’s ending
A worker never ends itself. It does the work, opens the pull request, and then sits at a
prompt waiting for someone to type — which is the whole point of these being interactive
sessions, and is also why the exit record that closes a session row never arrived.
Everything downstream of that was working exactly as designed, and the sum of it was a bug.
Merging the PR closes the issue; the next reconciliation pass moves the item to done; a
live worker under a done item is precisely what orphan_session reports; and the session
row is left open on purpose, because reporting fewer running sessions than exist would
oversubscribe the cap. So the ordinary successful path ended in an anomaly and a capacity
slot held for as long as the machine stayed up. Three successful items at
max_concurrent_sessions = 3 were enough to stop dispatch permanently, reporting only that
the machine was full — and the worktrees could never be reclaimed either, because cleanup’s
session guard kept recording skipped, which means “not yet”. That is issue #138.
So the lifecycle has an ending now. An item that is done and whose worker is idle has
that worker stopped: the process ends, the row closes with a reason naming retirement, and
the slot comes back. Its terminal tab goes too, on the same pass — see the tab
below, which is a separate act and was originally, wrongly, assumed to be free.
What says “now”: the merge, or failing that the clock
Two signals authorise it, and the merged pull request is the one that matters. If I have
merged the PR I have said “yes, this is complete” in as many words, which is a stronger and
earlier statement than any inference from how long a process has been quiet. From that point
the worker has nothing left to do and its tab has nothing left in it I am about to read, so
it goes on the very pass the item reaches done — no floor, no further wait.
The item is done and… |
It is retired |
| it has a merged pull request |
as soon as the worker is observed idle |
| it has none — the issue was closed by hand, or as not-planned |
after the worker has been idle 30 minutes |
The second row is unchanged and is the guard on the first. With no merge there is no
explicit acceptance of anything, the session may be exactly what I am about to attach to,
and idleness is the only evidence there is.
This is a correction, and the shape of the mistake is worth keeping. Retirement shipped
with the 30-minute clock as the whole gate, on the reasoning that erring long was nearly
free. That reasoning was measured against the wrong case. On the real path the worker goes
quiet, I merge within a few minutes, the issue closes, and the item goes done — so done
reliably arrives inside the quiet period and never after it. The gate declined every time,
orphan_session was raised by the sweep a few lines later, and for the next ~29 minutes
there was an anomaly on the list, a capacity slot held, a tab still open and a worktree
reported skipped. Of the four session.retire records in the log when this was found — at
idle 2477s, 11205s, 18049s and 35052s — not one was an item that had finished normally.
They were all backlog from before the feature existed. That is issue #149, and it is the
bounded version of the same bug #138 reported.
No floor on the merged path, and that is arithmetic rather than taste. A 60-second one
was considered. On the completion #149 was filed about the worker had been idle 47 seconds
when its item reached done, so a 60-second floor declines on exactly the pass that matters
and the anomaly is raised anyway. The case a floor would cover — merging from the web
interface while still reading the session — is already covered by the fact that retirement
destroys nothing: the transcript survives and claude --resume brings it back, so the cost
is a keystroke.
What the merge does not remove is the idleness check. A worker whose status is not
idle, or whose idleness cannot be established at all, is left alone however long ago
anything was merged. Merging is a statement about the work; whether the process is between
tool calls is a different question, and only the registry answers it. That is what keeps a
worker from being ended in the middle of a tool call, and it is why being wrong about the
registry can still only ever delay a retirement.
The audit record says which of the two fired: session.retire carries
signal: merged_pull_request or signal: quiet_period. With one gate the idle_s field
implied the reason; with two it does not, and a reader should not have to know that a low
idle_s means a merge.
done is the whole precondition, and it means more than it looks like. The only thing
in the system that writes that state is the pass that observed the issue closed, so done
is “the work was accepted”. A test asserts that stays true, because a second route to
done would quietly widen which workers get stopped.
abandoned and failed keep their workers. Those are the states where the work is not
finished and the session may be exactly what I am about to attach to. `robot-army cancel
` is the route out of those, and it now settles a terminal item's session correctly
rather than reporting an ending it did not observe.
`cancel` asks before it signals anything. Ctrl-C at that question, or running it with no
stdin, stops the command with nothing signalled and the item exactly where it was — and
writes a `session.cancel` record saying which session I reached for and which way I gave up.
Until issue #23 a closed stdin tracebacked here, and a Ctrl-C — which `main` has always caught
— left nothing behind at all. `cancel --force` skips the question, and is a different
`--force` from the ones on this page.
### Keeping a session after the merge
**Sometimes the merge is not the end of the work.** In a configuration-management
repository the merge *is* the deploy, and the useful next step is to check the result from
inside the session that made the change — it already knows what it changed and why. The
merged path above retires that worker on the first idle pass, which is a minute after I
merge. A **keep** is how I say "not this one":
```bash
uv run robot-army keep # this item's session outlives its issue closing
uv run robot-army unkeep # back to the ordinary rules
uv run robot-army keeps # every keep in force, including ones with no session left
```
The web has the same two: a `keep` button on each `/active` row and on the item page, and a
"kept" section on `/active` listing every keep with its `release`.
While a `done` item is kept and its session is open, reconciliation does not retire the
worker, does not close its tab, and does not offer its checkout to automatic cleanup. That is
all a keep does. It starts, stops and signals nothing, so it works with the daemon stopped
and takes effect on the next pass.
- **Place it before merging.** Retirement on the merged path runs on the next pass, so a keep
placed afterwards is usually too late. A keep can go on an item in any state — `ready`,
`active`, `failed` — and simply waits for `done`. Placed after the fact it is accepted
while the worker still runs; once the item is finished and no session is open, `keep`
refuses and prints `claude --resume ` for the transcript instead.
- **Not on an `abandoned` item.** It has no way back to `done`, so a keep there could never
take effect. `keep` refuses and says so; its worker is already left running, and
`robot-army cancel ` is what ends it.
- **It never expires.** No timer, and no clearing when the session ends. A keep whose
session is gone is inert — it holds no slot and blocks no cleanup — and `keeps` shows it as
`no live session` so I can release it.
- **Exiting the session ends it.** When I am done verifying I exit the worker as usual; the
row closes, the tab goes on the next pass, and the keep is left inert. `unkeep` instead
hands the still-running worker back to the rules above: with a merged pull request it is
retired on the next pass that finds it idle, with no further wait.
- **A kept session holds a slot.** It counts against both caps for as long as it runs, and
the capacity reasons say so: `2 ours (1 kept), 0 other`, and
`repository o/n: 1 of 1 sessions (configured), 1 kept`. The fix that names is `unkeep`.
- **A hold and a pause do not keep a session.** They govern what *starts*; a keep governs
what *ends*. A merged, idle `done` item under a paused queue and both holds is retired as
normal.
A kept `done` session is not an orphan, so it raises no `orphan_session`, and one raised
before the keep was placed (the worker was busy when the item finished) is retracted on the
next pass — unless its worker has died by then, in which case it resolves as gone.
**A keep covers the item's current session only.** If the item was resumed while an earlier
attempt's worker was still alive, that earlier worker is not the one I kept: it is retired
when idle, reported as `orphan_session` when busy, and its anomaly is not retracted, exactly
as if there were no keep.
### The tab
**The tab does not close by itself, and I assumed for one whole feature that it did.** The
retirement work above shipped with a sentence here claiming the kitty tab "closes with" the
worker, on the reasoning that the window hosts a chain — `kitty → dtach → wrapper → claude` —
that exists only to run it. The first live retirement disproved that in the most direct way
available: both workers gone, both records closed, both sockets gone, and both tabs still
sitting there. I closed them by hand.
The cause is one flag, passed on every launch since the very first milestone:
```
kitty @ launch --type=tab --hold --cwd … --title …
```
`--hold` keeps a window open **after its command exits**, so that a launch which fails
instantly leaves something readable instead of a window that vanishes before I can see it.
That window is often the only evidence of what went wrong, which is worth more than a tidy
terminal. The chain was reasoned about; the flags were not read.
So closing a tab is its own act, and it has its own rule:
> A tab is closed when its work item is `done` and **none** of its sessions is still open.
Three consequences worth knowing:
- **`failed` and `abandoned` keep their tabs, indefinitely.** That is the whole point of
`--hold` and it survives untouched. The `done` gate is what preserves it — a failed launch
never reaches `done`, so its window is never a candidate.
- **All of an item's tabs go, not just the last one.** A resumed or restarted item left a
window per attempt, all marked with the same item id, and a finished item leaves none of
them behind. The one place this narrows `--hold`: an item that failed, was retried, and
then succeeded loses its failed attempt's tab too. The failure is still in the log and in
the transcript.
- **It is a sweep, not a step of retirement.** Which means a tab left behind by a crash
between the kill and the close is still cleaned up on the next pass, and so is one left by
a version of this that predates the rule.
- **A kept session keeps its tab** — not because the tab reads the keep, but because the
kept session's row stays open, which is the rule above. When I exit the session its row
closes and the tab goes on the next pass, keep or no keep.
- **It goes on the pass the item finishes.** The rule above was right from the start and the
tab still sat there for half an hour, because it waits on the session row and the row was
waiting on a 30-minute clock the ordinary path never reached. Nothing in this rule changed
to fix that: the tab moves earlier because the row closes earlier. That is issue #81,
fixed as a consequence of #149 rather than on its own.
Which tab belongs to which item is decided by a marker the launch writes onto the window —
`ra_item=` — and **never** by the window number recorded on the session row. Kitty
numbers windows per kitty process and starts again at 1 when it restarts, so a stored number
can name something else entirely a week later. A window without the marker is never touched,
whatever it contains.
**Idle is measured, not assumed.** The worker's own session registry entry carries a
`status` and the moment it last changed, and idleness is that pair and nothing else. A
status that is not exactly `idle`, an absent or malformed timestamp, a missing registry
entry, a registry that cannot be read — every one of them means *not retired*, and the
question is asked again next pass. Being wrong about that file can delay a retirement; it
cannot cause one. **A merged pull request does not change any of that**: it removes the
duration requirement, never the idleness one. Transcript file mtime was tried first and measured wrong: it ran 29 and
163 minutes ahead of the last record actually inside the file.
**Nothing is destroyed.** This is the difference between retirement and the cleanup below,
and it is why one is on by default and the other is not. The transcript survives untouched
and `claude --resume ` brings the session back, so a worker ended while I was
reading it costs a keystroke, not work. The worktree, the branch and the item's state are
not touched at all. There is no configuration key, because there is nothing to guard.
If the process survives the attempt, nothing is settled: the row stays open, the slot stays
held, and `orphan_session` is raised — "I tried and could not" is never recorded as "it is
gone".
## Cleaning up
A prepared worktree was measured at up to 499 MB, so disk is a real constraint — but
deleting work is irreversible, so automatic cleanup is **off by default** and stays off
until I turn it on.
```toml
[cleanup]
on_issue_close = false # true: reclaim a finished item's worktree and branch
```
```bash
uv run robot-army worktree list # size, branch, condition, cleanup state
uv run robot-army worktree remove # refuses if a session is live, or if dirty
uv run robot-army worktree remove # a worktree no item claims (see below)
uv run robot-army worktree prune
uv run robot-army cleanup # every eligible item, under the same guards
uv run robot-army cleanup # one item, reconsidering a retained decision
```
**Discarding an unfinished item's checkout is `reset`, not cleanup** (issue #179). Cleanup and
`worktree remove` are about reclaiming *disk*; `reset` is about throwing away *work* that went
the wrong way, and it removes the checkout as one step of putting the item back in the queue
against a freshly read issue. It uses `worktree remove`'s guards unchanged — a live session
refuses it, and git's refusal over uncommitted work refuses it unless `--force` is given from a
terminal — so everything below about what those guards protect applies to a reset too. What it
adds is the two steps either side: the issue is re-read before anything is deleted, and the item
returns to `ready` after. See [operating](/robot-army/guide/operating.html).
**Worktrees no row claims** (issue #59). `worktree remove ` and `cleanup` both start from
a work item, so a worktree whose row is gone is beyond both. `purge-simulated` was how that
happened: it deleted a rehearsal's rows, left their worktrees on disk, and pointed at `worktree
remove ` for rows it had just deleted. Now it names the worktrees those rows own — path,
branch, size — and asks, *separately* and defaulting to no, whether to remove them too, before
anything is deleted. `--yes` still answers only the rows question, so a script already passing
it does not start deleting directories; `--remove-worktrees` answers the other. Removal is the
same two steps under the same guards as `worktree remove`, never forced, with one difference: a
session row carrying the simulated host's signature (`pid 0`, no start time) does not refuse
it, because at `local` every row has one and nothing ever closes it.
Whatever survives is named with `worktree remove `, which removes a worktree no item
claims — worktree and branch — provided it is inside the worktree root, an onboarded clone
lists it as a worktree, and no live worker is in it. With no session rows to ask, that last
guard asks the registry and `/proc`, by pid *and* start time. Git's dirty-tree refusal stands
unless `--force`, which asks me to type the directory's name. A claimed path is refused with
the item id to use instead, so the path form is never a way round the session guard.
Reconciliation reports these directories as `orphan_worktree` (see
[anomalies](/robot-army/guide/operating.html)) and `worktree list` shows them as `unclaimed`, so "no worktrees
recorded" is never printed over disk robot-army made.
**A removal is reported only if the directory is gone.** The simulated version-control
boundary answers every removal with success and touches nothing — right for a worktree it also
only pretended to create, wrong for a real one a `local` round left behind. So a directory still
there after a reported removal is a refusal (`directory_survived`), the branch is left alone,
and an item keeps its recorded path.
**Below `local` the removal verbs say "would"** (issue #70). Where there is no directory to
survive — a simulated item whose worktree was itself only simulated — the removal is the
simulation's and is accepted, but `worktree remove` reports it as `would remove worktree …` and
`would delete branch …`, then a line naming the effect level and the level version control is
real from. It used to print `removed worktree` over a disk it had not touched. `worktree prune`
says each repository was not checked, rather than "nothing to prune". `cleanup` records the reason
as `worktree removal simulated; branch deletion simulated — …`, counts items that *would* have
their worktree removed, and applies the survivor rule too: a real directory still on disk after a
reported removal is `retained`, never `done`, so `cleanup ` at a real level still reaches it.
`--json` and the `worktree.remove` record carry `simulated`.
With `on_issue_close = true`, an item whose issue has closed has its worktree and branch
reclaimed on the next reconciliation pass — provided nothing in either exists only there.
`robot-army cleanup` runs the identical function under the identical guards whether or not
the automatic path is enabled, so the manual route cannot drift from the automatic one.
A **kept** item whose session is still open is not offered to either — automatic cleanup or
bare `robot-army cleanup` — so a keep that stands for days does not write a `skipped`
record every pass. `cleanup ` still considers it and still reports it as live. A keep
whose session has ended does not block cleanup at all.
**The guards are different guards, and that is the whole design.**
- **The session**: is anything still running in there? Asked of the session rows — and of
nothing else. Not of the work item's state, because the reachable case is a *finished*
item: the issue closes, the item goes `done`, the worker types on, and terminal is
exactly the state I reclaim disk from. Not of the process table either, because a row
whose process I cannot see is still a row nothing has closed, and refusing only on a
*confirmed* live process would remove the worktree in every case where liveness could not
be established. `worktree remove` gained this guard in #79, after it deleted a running
worker's directory and its branch in one command and reported success; `cleanup` had it
from the start. That was the wrong way round — `cleanup` runs unattended and is
conservative by design, while `worktree remove` is what I reach for when `/home` is at
93%, and it is the one that can override git. `--force` still overrides it, and says so
in the prompt before I type anything. Giving up at that prompt — Ctrl-C, or no stdin at
all — removes nothing: an absent answer is not the item id it asks me to type, and the
attempt is recorded against the worktree path it named (issue #23).
- **The worktree**: git's own refusal, taken as-is. `git worktree remove` refuses on a dirty
tree — *including merely untracked files* — and `--force` is never passed. A refused
worktree is recorded as `retained` with git's own message, and the branch half is not
attempted, because a dirty worktree means the branch may hold the only copy of something.
- **The branch**: my own containment check, because git's is the wrong one here. `git branch
-d` accepts only a branch merged into the clone's current `HEAD`; the normal case is a PR
merged on GitHub while my clone has a stale `main` checked out, so `-d` would refuse every
time and `robot-army/*` branches would accumulate forever. Instead the branch is deleted
only if every commit on it is provably on the remote — contained in the published base,
which is fetched first, or on the remote under its own name, which the **remote is asked
about during the check**. A remote-tracking ref is not taken as the remote's answer: it is
a cache of what the remote said last time, a fetch scoped to the base branch neither
refreshes nor prunes it, and #105 measured a branch deleted on the remote going on proving
itself "pushed and up to date" from that leftover until a `gc` on the remote made the loss
permanent. If the remote cannot be asked, or git cannot answer, that is "unproven", never
"safe", and the branch is kept.
Four outcomes, all visible in `robot-army show ` and on the item's web page:
| `cleanup_state` | Worktree | Branch | Meaning |
|---|---|---|---|
| `done` | removed | removed | both guards passed |
| `branch_retained` | removed | kept | containment could not be proved |
| `retained` | kept | kept | git refused the worktree |
| `skipped` | kept | kept | a session was still live — reconsidered next pass |
`skipped` is the only one the automatic pass revisits: it means "not yet", where `retained`
means "we looked and decided no". `worktree_path` and `branch` are kept on the record even
after a successful removal, so "what was at this path?" stays answerable.
**`worktree remove ` leaves the same record** (issue #113). On a finished item — `done` or
`abandoned` — a manual removal writes `done` or `branch_retained` exactly as cleanup would,
with a reason that ends by naming `robot-army worktree remove`, and keeps the path and branch. It
used to write nothing and forget the path, so `show` could not say where the worktree went and
reconciliation could not tell the removal from an `rm -rf`. An *unfinished* item still forgets
the path and gets no record: `retry` and `reset` can give it a fresh worktree, and a record
describing the old one would quietly exempt the new one from the missing-worktree report and
from cleanup.
It is also the command that settles a finished item whose directory was deleted by hand, which
reconciliation reports as `prunable_worktree` (see [anomalies](/robot-army/guide/operating.html)). A directory that
is already gone is not refused — once `worktree prune` has cleared git's record, git answers "is
not a working tree", which is about its record, and there are no contents to protect — and a
branch that is already gone is reported as gone rather than as surviving. Run it on an item
whose record is `done` — both halves gone — and it refuses (`already_removed`, exit 3) without
asking git anything. A `branch_retained` item is *not* refused: that is what the first run leaves
when git's `-d` will not delete an unmerged branch, which is the ordinary case for abandoned
work, and re-running with `--force` is how that branch goes. For an abandoned item it is the
only way, since `cleanup` considers `done` items only.
---
Next: [operating it](/robot-army/guide/operating.html).