hmo
9509b80d8b
feat(xmpp_monitor): dynamic key switching + coalesce auto-heal
- xmpp_logger.py:
- Add switch_key(key_id): switch Hermes model.provider via sed + systemctl restart
- Add current_provider() reading model: block correctly (not just first ' provider:' line)
- Add KEY_TO_PROVIDER mapping (AgentsMeeting key_id -> Hermes provider name)
- Add RESTART_COOLDOWN_FILE/SEC = 180s to prevent restart loops
- auto_heal(): detect HTTP 429 / Weekly usage limit -> call best_key() -> switch_key()
- auto_heal(): detect timeout/error -> async systemctl restart (Popen, not blocking)
- _verify_llm(): bump timeout 25s -> 90s (cold-start gateway takes 20-40s)
- health(): urlopen timeout 8s -> 90s (match verify window)
- Use sudo NOPASSWD (hmo ALL=(ALL) NOPASSWD: ALL already configured)
- agents_health_check.py:
- Replace inline pkill+Popen restart logic (caused multiple instances) with systemctl
- Add RESTART_COOLDOWN_FILE state to skip restart within 3 min of last
- Call xmpp_logger.auto_heal() at end of every cron cycle
- Both inline restart and auto_heal restarts share cooldown file
- scripts/key_status.py: Reports weekly/monthly/rolling status of all 6 OCG keys
- scripts/test_llm.py: 90s timeout test (was 15s, gateway cold-start >= 30s)
- scripts/test_production.py: smoke test on /home/hmo/MoFin/ (hardlinked to web-dashboard)
Fixes:
- Old assumption 'Cloudflare blocks Python User-Agent' was WRONG.
True cause was HTTP 429 Weekly usage limit on key5 (12hr reset window).
Hermes silently ignored providers.X.headers config keys; only model.default_headers works.
Config already has model.default_headers: User-Agent: curl/8.5.0 (defense in depth).
- Multiple gateway instances were caused by 3 competing systemd units
(hermes-gateway@.service template + hermes-gateway-zhiwei.service named).
Masked the template unit @position-analyst and @zhiwei so only the named one wins.
2026-07-19 20:12:47 +08:00
..
2026-07-08 12:34:39 +08:00
2026-07-06 14:01:54 +08:00
2026-07-08 23:54:01 +08:00
2026-07-09 17:57:17 +08:00
2026-07-08 12:37:36 +08:00
2026-07-10 13:58:23 +08:00
2026-07-04 09:30:50 +08:00
2026-07-08 23:54:01 +08:00
2026-07-09 19:42:35 +08:00
2026-07-10 14:06:20 +08:00
2026-07-19 20:12:47 +08:00
2026-07-19 11:06:50 +08:00
2026-07-19 11:06:50 +08:00
2026-07-19 11:06:50 +08:00
2026-07-19 11:06:50 +08:00
2026-07-06 14:01:54 +08:00
2026-07-08 23:54:01 +08:00
2026-07-08 23:54:01 +08:00
2026-07-08 19:50:44 +08:00
2026-07-14 10:56:38 +08:00
2026-07-14 12:14:08 +08:00
2026-07-06 14:01:54 +08:00
2026-07-08 23:54:01 +08:00
2026-07-08 19:50:44 +08:00
2026-07-10 11:59:08 +08:00
2026-07-14 11:08:07 +08:00
2026-07-19 20:12:47 +08:00
2026-07-14 15:00:56 +08:00
2026-07-19 11:06:50 +08:00
2026-07-19 11:06:50 +08:00
2026-07-10 10:26:14 +08:00
2026-07-06 14:01:54 +08:00
2026-07-01 23:00:28 +08:00
2026-07-06 14:01:54 +08:00
2026-07-03 12:12:05 +08:00
2026-07-09 19:42:35 +08:00
2026-07-17 23:08:11 +08:00
2026-07-14 10:56:38 +08:00
2026-07-19 20:12:47 +08:00
2026-07-19 11:06:50 +08:00
2026-07-02 13:46:53 +08:00
2026-07-01 23:00:28 +08:00
2026-07-08 12:37:36 +08:00
2026-07-19 12:47:15 +08:00
2026-07-08 20:04:32 +08:00
2026-07-08 12:37:36 +08:00
2026-06-27 01:37:18 +08:00
2026-06-24 00:15:33 +08:00
2026-07-08 12:34:39 +08:00
2026-07-06 14:01:54 +08:00
2026-07-08 23:54:01 +08:00
2026-07-09 16:35:27 +08:00
2026-07-06 14:01:54 +08:00
2026-07-06 14:01:54 +08:00
2026-07-08 23:54:01 +08:00
2026-07-08 23:54:01 +08:00
2026-07-14 11:01:59 +08:00
2026-07-14 10:56:38 +08:00
2026-07-08 12:37:36 +08:00
2026-07-06 14:01:54 +08:00
2026-07-09 11:13:45 +08:00
2026-07-08 12:34:39 +08:00
2026-07-08 23:54:01 +08:00
2026-06-30 02:56:35 +08:00
2026-07-13 12:09:25 +08:00
2026-07-19 20:12:47 +08:00
2026-07-08 23:54:01 +08:00
2026-07-09 13:43:30 +08:00
2026-07-10 12:20:18 +08:00
2026-07-08 16:18:26 +08:00
2026-07-14 15:33:35 +08:00
2026-07-06 14:01:54 +08:00
2026-07-13 12:09:25 +08:00
2026-07-04 09:30:50 +08:00
2026-07-08 23:54:01 +08:00
2026-07-06 14:01:54 +08:00
2026-07-03 12:12:05 +08:00
2026-07-04 09:45:27 +08:00
2026-07-14 10:56:38 +08:00
2026-07-08 12:34:39 +08:00
2026-07-06 14:01:54 +08:00
2026-07-13 12:09:25 +08:00
2026-07-13 12:09:25 +08:00
2026-07-08 23:54:01 +08:00
2026-07-08 23:54:01 +08:00
2026-06-29 23:25:54 +08:00
2026-07-08 12:37:36 +08:00
2026-07-08 23:54:01 +08:00
2026-07-08 12:37:36 +08:00
2026-07-17 23:08:11 +08:00
2026-07-08 23:54:01 +08:00
2026-07-08 12:34:39 +08:00
2026-07-08 23:54:01 +08:00
2026-07-19 10:56:42 +08:00
2026-07-09 13:43:30 +08:00
2026-07-06 14:01:54 +08:00
2026-07-07 11:20:04 +08:00
2026-07-17 23:08:11 +08:00
2026-07-08 12:34:39 +08:00
2026-07-10 21:07:59 +08:00
2026-07-19 20:12:47 +08:00
2026-07-19 20:12:47 +08:00
2026-07-19 20:12:47 +08:00
2026-07-19 11:06:50 +08:00
2026-07-19 20:12:47 +08:00
2026-07-08 12:37:36 +08:00
2026-07-08 10:13:49 +08:00
2026-07-07 15:33:14 +08:00
2026-07-19 11:06:50 +08:00
2026-07-10 12:20:18 +08:00
2026-07-08 12:37:36 +08:00
2026-07-01 23:00:28 +08:00
2026-07-06 14:01:54 +08:00
2026-07-09 10:35:34 +08:00
2026-07-02 01:43:33 +08:00