Three Ways a Nightly Agent Dies Quietly

Context

Kyogre is a blog agent that runs on a schedule. A launchd job wakes up at night, the shell script daily_with_iterations.sh collects the day's commits from the GitHub search API, collects the diffs for those commits, hands both files to Claude, scores the resulting draft, rewrites it until the score clears the bar, and opens a PR against the portfolio repo.

It worked. That is the problem with jobs like this: once they work, the only signal that they are still working is output showing up. And "no output today" is indistinguishable from "no commits worth writing about today," which the agent is explicitly designed to produce.

Problem

At some point the nightly run stopped producing anything, and nothing anywhere said so. No alert, no failed job, no error in a terminal anyone was looking at. The eventual investigation turned up not one bug but three independent failures with overlapping timelines — the deployed clone had stopped updating on Aug 18, and the run on Aug 24 hung until Sep 4 — each invisible for a different reason.

What was tried

The first instinct was to look at the collection step, since that is where the agent talks to the network. That turned out to be right, but for a reason a quick read wouldn't reveal:

resp = requests.get(url, headers=headers, params=params)

No timeout. requests has no default timeout, so this call will wait indefinitely. On Aug 24, with DNS unavailable, it did exactly that: the run hung inside that single call for eleven days. It never crashed, never returned, never logged anything. It just sat there.

And because launchd tracks a job by its process, a hung process is a running job. Every subsequent nightly trigger found the slot occupied and did nothing. One missing keyword argument turned a single failed night into every failed night after it.

The second failure was older, quieter, and had a completely different shape: it did not stop the agent, it just froze the code the agent was running. The deployed clone pulls before each run, and temp/ and logs/ were tracked in git — the same temp/commits.md, temp/diffs.md, and logs/agent.log that the agent overwrites at runtime. So every run rewrote tracked files, and every subsequent git pull refused to proceed with local changes in the working tree. The pull had been failing since Aug 18.

It failed silently because it ran under capture_output. The error was produced, captured into a variable, and never inspected. Text written to a Python string that nobody reads is not an error message; it is a rounding error.

The third failure is visible in the repository itself, in a log file that was committed by accident:

shell-init: error retrieving current directory: getcwd: cannot access parent directories: Operation not permitted
/bin/bash: /Users/manishbisht/Desktop/langchain/kyogre/src/kyogre/scripts/daily.sh: Operation not permitted

The scheduled job was pointing at a path the agent had since moved away from, and macOS denied access to it. The accidentally-tracked log file that broke git pull was, at the same time, the only durable record of the run failing. The bug and its own evidence were the same file.

What failed

The interesting failure here is not any single bug. It is that all three had the same property: the failure path produced no observable output. A hang produces nothing by definition. A captured stderr produces nothing by construction. A log file written into a directory nobody reads produces nothing in practice.

None of these are exotic. Each one is a line of code that a code review would wave through.

Final solution

The fixes are small enough to feel anticlimactic.

Add the timeout, in both collectors:

resp = requests.get(url, headers=headers, params=params, timeout=30)

Stop tracking the scratch directories, so the runtime files and the repository stop fighting over the same paths:

# Runtime scratch
logs/
temp/
drafts/
evaluations/

Then, because seventeen days of commits had piled up while the agent was asleep, the agent needed to be able to look backwards. collect_commits.py now takes an optional GitHub date qualifier instead of hardcoding today:

query = f"author:{username} author-date:{when}"

when defaults to today, so the nightly path is unchanged, but KYOGRE_DATE=2026-08-21..2026-09-04 makes a run cover a range.

That exposed a fourth problem, this one purely structural. A multi-day range is likely to contain more than one story, and the script's pipeline hard-errored on more than one draft. It now loops over every draft — and the loop revealed its own bug: the second PR contained the first blog, because the script branched from wherever the previous iteration had left the portfolio repo. Hence the least glamorous and most necessary line in the diff:

# Branch from master, not from the branch a previous draft left us on,
# or the second PR would also contain the first blog.
git checkout master

Finally, a backfill spanning weeks can rediscover stories already published. The script now scrapes the title: frontmatter out of the live blog directory and injects the list into the prompt as a do-not-retell block. The agent's constraint is the published record, not its own memory.

Technical reasoning

Every one of these bugs comes from the same assumption: that a step which usually works will tell you when it doesn't. That assumption is fine in a terminal, where a hung command is obvious and stderr goes to your face. It is false in a scheduled job, where nobody is looking at either.

Two things specifically deserve blame. First, requests making timeout optional and defaulting to infinite — an unbounded wait is never the behaviour you want in automation, and a hung process is worse than a crashed one because supervisors treat it as healthy. Second, capture_output=True without a corresponding check, which is a deliberate decision to discard error information written as a convenience flag.

The scratch-directory collision is a subtler point about deployment. A repository that is both source and runtime working directory has two writers with different intentions: git wants the working tree to match a commit, and the agent wants to overwrite files whenever it likes. Those goals are incompatible, and .gitignore is where you declare which files belong to which writer.

Lessons learned

Timeouts are not error handling; they are liveness. A call without a timeout can convert one bad night into permanent silence. Set them everywhere you cross a network boundary, even in scripts you think of as throwaway.

A hung job is the worst failure mode, because process-based schedulers count it as running. Prefer crashing loudly to waiting politely.

capture_output without an inspected exit code is a mute button. If you capture output, you have taken responsibility for reading it.

Never track files your runtime overwrites. The collision surfaces as a confusing git failure in a place you weren't debugging.

The absence of output is not a signal. An agent designed to sometimes produce nothing cannot use "produced nothing" to mean "healthy." That ambiguity is the actual root cause — the three bugs just took turns exploiting it. A run that reports "ran, found nothing" is a fundamentally different artifact from a run that never happened, and only the first can be missed on purpose.