Investigate the kitchenowl-backend worker timeout and add supervision #57

Open
opened 2026-11-10 19:47:06 +00:00 by martin · 0 comments
Owner

Summary

A gunicorn worker timed out and was killed on 2026-11-09, causing a service interruption. The cause is unknown and the unit has no timeout, restart, or memory-supervision settings.

Context

Journal excerpt:

[2026-11-09 20:15:19 +0000] [2046695] [CRITICAL] WORKER TIMEOUT (pid:2046721)
[2026-11-09 20:16:39 +0000] [2046695] [ERROR] Worker (pid:2046721) was sent SIGKILL! Perhaps out of memory?
[2026-11-09 20:16:40 +0000] [556800] [INFO] Booting worker with pid: 556800
  • No OOM entries in the kernel log for that window; the SIGKILL came from gunicorn's 30 s default timeout.
  • The unit sets only --worker-class gevent --pythonpath; no --timeout, Restart=, or MemoryMax=.
  • Memory usage at time of investigation was about 483 MB with no limit.
  • The triggering request is unknown; recipe scraping and LiteLLM calls are plausible culprits.

Scope

  • Identify the hanging request or blocking operation, if the evidence still exists.
  • Add supervision: an explicit gunicorn timeout, Restart=on-failure, and a MemoryMax guard appropriate for the host.
  • Capture enough logging to attribute the next occurrence.

Acceptance criteria

  • The timeout cause is identified, or the evidence and next diagnostic step are recorded.
  • kitchenowl-backend.service sets an explicit timeout and restart policy.
  • A worker timeout no longer leaves the service without a replacement worker.
  • The next occurrence can be attributed to a request in the journal.
## Summary A gunicorn worker timed out and was killed on 2026-11-09, causing a service interruption. The cause is unknown and the unit has no timeout, restart, or memory-supervision settings. ## Context Journal excerpt: ```text [2026-11-09 20:15:19 +0000] [2046695] [CRITICAL] WORKER TIMEOUT (pid:2046721) [2026-11-09 20:16:39 +0000] [2046695] [ERROR] Worker (pid:2046721) was sent SIGKILL! Perhaps out of memory? [2026-11-09 20:16:40 +0000] [556800] [INFO] Booting worker with pid: 556800 ``` - No OOM entries in the kernel log for that window; the SIGKILL came from gunicorn's 30 s default timeout. - The unit sets only `--worker-class gevent --pythonpath`; no `--timeout`, `Restart=`, or `MemoryMax=`. - Memory usage at time of investigation was about 483 MB with no limit. - The triggering request is unknown; recipe scraping and LiteLLM calls are plausible culprits. ## Scope - Identify the hanging request or blocking operation, if the evidence still exists. - Add supervision: an explicit gunicorn timeout, `Restart=on-failure`, and a `MemoryMax` guard appropriate for the host. - Capture enough logging to attribute the next occurrence. ## Acceptance criteria - [ ] The timeout cause is identified, or the evidence and next diagnostic step are recorded. - [ ] `kitchenowl-backend.service` sets an explicit timeout and restart policy. - [ ] A worker timeout no longer leaves the service without a replacement worker. - [ ] The next occurrence can be attributed to a request in the journal.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
Personal/nixos-configuration#57
No description provided.