feat(xmpp_monitor): dynamic key switching + coalesce auto-heal

- xmpp_logger.py:
  - Add switch_key(key_id): switch Hermes model.provider via sed + systemctl restart
  - Add current_provider() reading model: block correctly (not just first '  provider:' line)
  - Add KEY_TO_PROVIDER mapping (AgentsMeeting key_id -> Hermes provider name)
  - Add RESTART_COOLDOWN_FILE/SEC = 180s to prevent restart loops
  - auto_heal(): detect HTTP 429 / Weekly usage limit -> call best_key() -> switch_key()
  - auto_heal(): detect timeout/error -> async systemctl restart (Popen, not blocking)
  - _verify_llm(): bump timeout 25s -> 90s (cold-start gateway takes 20-40s)
  - health(): urlopen timeout 8s -> 90s (match verify window)
  - Use sudo NOPASSWD (hmo ALL=(ALL) NOPASSWD: ALL already configured)

- agents_health_check.py:
  - Replace inline pkill+Popen restart logic (caused multiple instances) with systemctl
  - Add RESTART_COOLDOWN_FILE state to skip restart within 3 min of last
  - Call xmpp_logger.auto_heal() at end of every cron cycle
  - Both inline restart and auto_heal restarts share cooldown file

- scripts/key_status.py: Reports weekly/monthly/rolling status of all 6 OCG keys
- scripts/test_llm.py: 90s timeout test (was 15s, gateway cold-start >= 30s)
- scripts/test_production.py: smoke test on /home/hmo/MoFin/ (hardlinked to web-dashboard)

Fixes:
- Old assumption 'Cloudflare blocks Python User-Agent' was WRONG.
  True cause was HTTP 429 Weekly usage limit on key5 (12hr reset window).
  Hermes silently ignored providers.X.headers config keys; only model.default_headers works.
  Config already has model.default_headers: User-Agent: curl/8.5.0 (defense in depth).
- Multiple gateway instances were caused by 3 competing systemd units
  (hermes-gateway@.service template + hermes-gateway-zhiwei.service named).
  Masked the template unit @position-analyst and @zhiwei so only the named one wins.

scripts/check_dashboard.py

@@ -0,0 +1,9 @@
+import urllib.request, json
+req = urllib.request.Request('http://127.0.0.1:8899/api/xmpp/health')
+data = json.loads(urllib.request.urlopen(req, timeout=120).read())
+print('status:', data.get('status'))
+print('gateways:', list(k + '=' + str(v.get('alive')) for k, v in data.get('gateways', {}).items()))
+print('llm:', data.get('llm_provider'))
+print('best_key:', data.get('best_key', {}).get('key_id'),
+      '| weekly:', data.get('best_key', {}).get('weekly', {}).get('usage_percent'), '%',
+      '| rolling:', data.get('best_key', {}).get('rolling', {}).get('usage_percent'), '%')

scripts/dump_health.py

@@ -0,0 +1,5 @@
+import json, sys
+line = sys.stdin.read().strip()
+d = json.loads(line)
+llm = d.get('llm_provider', {})
+print(d['timestamp'], '| status:', d['status'], '| llm:', llm.get('status', '?'), '| best:', d.get('best_key', {}).get('key_id', '?'))

scripts/key_status.py

@@ -0,0 +1,20 @@
+import urllib.request, json
+req = urllib.request.Request('http://127.0.0.1:5803/api/keys')
+data = json.loads(urllib.request.urlopen(req, timeout=10).read())
+print(f"{'key':6} {'weekly':15} {'monthly':15} {'rolling':15} {'workspace':40}")
+print('-'*95)
+for k in data['keys']:
+    print(f"{k['key_id']:6} "
+          f"{k['weekly']['status']:15} "
+          f"{k['monthly']['status']:15} "
+          f"{k['rolling']['status']:15} "
+          f"{k['workspace_id'][:40]:40}")
+
+best = None
+for k in data['keys']:
+    if (k['weekly']['status'] == 'ok' and
+        k['monthly']['status'] == 'ok' and
+        k['rolling']['status'] == 'ok'):
+        if best is None or k['weekly']['usage_percent'] < best['weekly']['usage_percent']:
+            best = k
+print('\nBEST:', best['key_id'] if best else 'NONE', '- all weekly rate-limited' if not best else '')

scripts/pp.py

@@ -0,0 +1,3 @@
+import sys, json
+d = json.loads(sys.stdin.read())
+print(d['timestamp'], d['status'], 'llm:', d['llm_provider']['status'], '| best_key:', d['best_key']['key_id'])

scripts/test_auto_heal.py

@@ -0,0 +1,7 @@
+import sys
+sys.path.insert(0, '/home/hmo/projects/MoFin')
+import xmpp_logger as x
+import json
+print("=== AUTO HEAL RESULT ===")
+result = x.auto_heal()
+print(json.dumps(result, ensure_ascii=False, indent=2))

scripts/test_llm.py

@@ -0,0 +1,9 @@
+import urllib.request, json
+data = json.dumps({'model':'deepseek-v4-flash','messages':[{'role':'user','content':'say hi'}],'max_tokens':10}).encode()
+req = urllib.request.Request('http://127.0.0.1:8643/v1/chat/completions', data=data,
+    headers={'Content-Type':'application/json','Authorization':'Bearer hermes123'})
+try:
+    resp = urllib.request.urlopen(req, timeout=90)
+    print('OK:', resp.read().decode()[:300])
+except Exception as e:
+    print('FAIL:', e)

scripts/test_production.py

@@ -0,0 +1,8 @@
+import sys, json
+sys.path.insert(0, '/home/hmo/MoFin')
+import xmpp_logger as x
+print("current provider:", x.current_provider())
+print("health status:", x.health().get("status"))
+print("verify_llm:", x._verify_llm())
+print("=== auto_heal ===")
+print(json.dumps(x.auto_heal(), ensure_ascii=False, indent=2))

scripts/test_xmpp_logger.py

@@ -0,0 +1,11 @@
+import sys
+sys.path.insert(0, '/home/hmo/projects/MoFin')
+import xmpp_logger as x
+print("current provider:", x.current_provider())
+h = x.health()
+print("health status:", h.get("status"))
+print("llm_provider:", h.get("llm_provider"))
+print("=== best_key ===")
+print(x.best_key())
+print("=== verify_llm ===")
+print(x._verify_llm())