feat(xmpp_monitor): dynamic key switching + coalesce auto-heal
- xmpp_logger.py: - Add switch_key(key_id): switch Hermes model.provider via sed + systemctl restart - Add current_provider() reading model: block correctly (not just first ' provider:' line) - Add KEY_TO_PROVIDER mapping (AgentsMeeting key_id -> Hermes provider name) - Add RESTART_COOLDOWN_FILE/SEC = 180s to prevent restart loops - auto_heal(): detect HTTP 429 / Weekly usage limit -> call best_key() -> switch_key() - auto_heal(): detect timeout/error -> async systemctl restart (Popen, not blocking) - _verify_llm(): bump timeout 25s -> 90s (cold-start gateway takes 20-40s) - health(): urlopen timeout 8s -> 90s (match verify window) - Use sudo NOPASSWD (hmo ALL=(ALL) NOPASSWD: ALL already configured) - agents_health_check.py: - Replace inline pkill+Popen restart logic (caused multiple instances) with systemctl - Add RESTART_COOLDOWN_FILE state to skip restart within 3 min of last - Call xmpp_logger.auto_heal() at end of every cron cycle - Both inline restart and auto_heal restarts share cooldown file - scripts/key_status.py: Reports weekly/monthly/rolling status of all 6 OCG keys - scripts/test_llm.py: 90s timeout test (was 15s, gateway cold-start >= 30s) - scripts/test_production.py: smoke test on /home/hmo/MoFin/ (hardlinked to web-dashboard) Fixes: - Old assumption 'Cloudflare blocks Python User-Agent' was WRONG. True cause was HTTP 429 Weekly usage limit on key5 (12hr reset window). Hermes silently ignored providers.X.headers config keys; only model.default_headers works. Config already has model.default_headers: User-Agent: curl/8.5.0 (defense in depth). - Multiple gateway instances were caused by 3 competing systemd units (hermes-gateway@.service template + hermes-gateway-zhiwei.service named). Masked the template unit @position-analyst and @zhiwei so only the named one wins.
This commit is contained in:
@@ -0,0 +1,20 @@
|
||||
import urllib.request, json
|
||||
req = urllib.request.Request('http://127.0.0.1:5803/api/keys')
|
||||
data = json.loads(urllib.request.urlopen(req, timeout=10).read())
|
||||
print(f"{'key':6} {'weekly':15} {'monthly':15} {'rolling':15} {'workspace':40}")
|
||||
print('-'*95)
|
||||
for k in data['keys']:
|
||||
print(f"{k['key_id']:6} "
|
||||
f"{k['weekly']['status']:15} "
|
||||
f"{k['monthly']['status']:15} "
|
||||
f"{k['rolling']['status']:15} "
|
||||
f"{k['workspace_id'][:40]:40}")
|
||||
|
||||
best = None
|
||||
for k in data['keys']:
|
||||
if (k['weekly']['status'] == 'ok' and
|
||||
k['monthly']['status'] == 'ok' and
|
||||
k['rolling']['status'] == 'ok'):
|
||||
if best is None or k['weekly']['usage_percent'] < best['weekly']['usage_percent']:
|
||||
best = k
|
||||
print('\nBEST:', best['key_id'] if best else 'NONE', '- all weekly rate-limited' if not best else '')
|
||||
Reference in New Issue
Block a user