Commit Graph
11 Commits
Author SHA1 Message Date
hmo 2de527b883 fix(health): drop live LLM ping — scan agent.log instead (38s -> 0.1s)
Every /api/xmpp/health fetch ran a real LLM call (22k token system prompt
each). Dashboard refreshes every 10s -> thousands of paid LLM calls/day,
plus 38s latency hanging the health tab on '加载中...'.

LLM health is now derived from the gateway's own agent.log (zero cost,
more accurate than synthetic ping — real traffic results):
- last 'API call #N latency=Xs' -> ok
- last 'API call failed ... HTTP 429...' -> error with summary
- health() runtime 38s -> 0.1s; endpoint 38s -> 0.097s
2026-07-20 00:21:49 +08:00
hmo 058c42ce27 feat(health): recent XMPP conversation log panel + stale error fix
User requirement: health tab should show recent XMPP conversations.
- bot hooks log_xmpp on inbound (on_msg) and outbound (_deliver_loop)
  so real chats land in xmpp_messages.jsonl (was: only cron/scanner)
- index.html health tab: new '最近对话' panel (last 10 msgs, dir arrow,
  preview, status, time); refreshHealth updates it incrementally
- last_error now shows age and resolved state: once a successful
  outbound happens after an error, it's shown gray as '已恢复'
  instead of alarming red forever; unresolved errors still red
- health() status no longer degraded by errors that were later
  resolved by successful outbound
2026-07-20 00:07:28 +08:00
hmo cb334ddd54 fix(bot+auto_heal): stop auto_heal from killing bot during slow LLM calls
Root cause of 'no response': auto_heal restart_zhiwei_bot fired whenever
inbound>0 + outbound=0 + errors>0 — which is exactly the normal state while
the bot waits for a slow LLM call (agent tool-use turns take 1-10 min).
Restart killed the in-flight LLM call -> user never got reply -> next cron
cycle saw same state -> restart again. Death loop (fired 22:30,22:35,22:40,
22:45,22:50,23:35).

Fixes:
- CALL_HERMES_TIMEOUT 180s -> 600s (agent tool calls need minutes)
- health(): parse last_inbound/outbound timestamps from journal
- auto_heal: restart bot ONLY if last inbound >600s old with no outbound
  since (truly stuck), plus 600s bot-restart cooldown
- slow-but-normal state now logs bot_busy_not_stuck instead of killing
2026-07-19 23:43:28 +08:00
hmo e0f47f9349 fix(xmpp_monitor): exclude key7(kimi) from auto-heal switch candidates
zhiwei must stay on ocg-key6 (deepseek-v4-flash) per user decision.
key7 is kimi — different model provider, incompatible model name.
If best_key returns key7, auto_heal logs skip_switch instead of switching.
2026-07-19 22:09:15 +08:00
hmo 9509b80d8b feat(xmpp_monitor): dynamic key switching + coalesce auto-heal
- xmpp_logger.py:
  - Add switch_key(key_id): switch Hermes model.provider via sed + systemctl restart
  - Add current_provider() reading model: block correctly (not just first '  provider:' line)
  - Add KEY_TO_PROVIDER mapping (AgentsMeeting key_id -> Hermes provider name)
  - Add RESTART_COOLDOWN_FILE/SEC = 180s to prevent restart loops
  - auto_heal(): detect HTTP 429 / Weekly usage limit -> call best_key() -> switch_key()
  - auto_heal(): detect timeout/error -> async systemctl restart (Popen, not blocking)
  - _verify_llm(): bump timeout 25s -> 90s (cold-start gateway takes 20-40s)
  - health(): urlopen timeout 8s -> 90s (match verify window)
  - Use sudo NOPASSWD (hmo ALL=(ALL) NOPASSWD: ALL already configured)

- agents_health_check.py:
  - Replace inline pkill+Popen restart logic (caused multiple instances) with systemctl
  - Add RESTART_COOLDOWN_FILE state to skip restart within 3 min of last
  - Call xmpp_logger.auto_heal() at end of every cron cycle
  - Both inline restart and auto_heal restarts share cooldown file

- scripts/key_status.py: Reports weekly/monthly/rolling status of all 6 OCG keys
- scripts/test_llm.py: 90s timeout test (was 15s, gateway cold-start >= 30s)
- scripts/test_production.py: smoke test on /home/hmo/MoFin/ (hardlinked to web-dashboard)

Fixes:
- Old assumption 'Cloudflare blocks Python User-Agent' was WRONG.
  True cause was HTTP 429 Weekly usage limit on key5 (12hr reset window).
  Hermes silently ignored providers.X.headers config keys; only model.default_headers works.
  Config already has model.default_headers: User-Agent: curl/8.5.0 (defense in depth).
- Multiple gateway instances were caused by 3 competing systemd units
  (hermes-gateway@.service template + hermes-gateway-zhiwei.service named).
  Masked the template unit @position-analyst and @zhiwei so only the named one wins.
2026-07-19 20:12:47 +08:00
hmo bff246313b feat: auto-heal checks key availability before restarting gateway 2026-07-19 14:46:46 +08:00
hmo f791d8dee7 feat: API key availability monitoring — auto-select best key from AgentsMeeting 2026-07-19 14:41:18 +08:00
hmo 6398f595b2 feat: auto-heal pipeline — detect+restart gateway/bot/ejabberd on failure 2026-07-19 14:18:45 +08:00
hmo 0802046d5c feat: Hermes Gateway + LLM provider monitoring, root cause visible in health tab 2026-07-19 13:25:37 +08:00
hmo 50ae4e1cc9 feat: XMPP bot journal monitoring — detect inbound/outbound/errors, smooth health refresh 2026-07-19 13:15:37 +08:00
hmo b41bed7959 feat: XMPP observability — logger, monitor endpoints, clean dead xiaoguo refs 2026-07-19 12:47:15 +08:00