feat(self-check): L0-L4 layered self-check architecture with LLM auto-repair

User directive: daily not weekly; clear responsibilities per layer with no
overlap; functional criteria (does the function WORK) not process liveness;
problems get FIXED via LLM with file-and-report discipline (act first,
report after); plus a meta-layer watching the watchers; deeply integrated
into F健康.

Architecture (responsibility matrix in dev-spec.md):
- L0 agents_health_check (5min): port/HTTP/DB liveness + auto_heal executor
- L1 functional_health_check (15min trading): per-module FUNCTIONAL
  criteria — output freshness/validity per REGISTRY (live_prices/market_
  snapshots/mtf_cache/macro_context/bot/LLM/cron engine), not process alive
- L2 system_hygiene_audit (daily 08:20, was weekly): divergence/hardlink/
  zombie/orphan/dead-cron/db-freshness
- L3 self_repair (30min): reads L1/L2 failures -> LLM diagnoses -> executes
  WHITELISTED repair actions directly (rerun_script/restart_service/
  sync_links/switch_llm_key/none) -> repair_log.jsonl + XMPP report.
  Max 2 repairs/module/day anti-loop. LLM unavailable -> rule fallback.
- L4 meta_watchdog (hourly): checks L0-L3 output freshness + L3 cron
  registration + XMPP bridge; direct XMPP alert as last resort

Retired (overlap): Cron监护-高频 (cron_watchdog -> L3), 全局cron健康监控
(cron_health_monitor -> L1).

Dashboard: mofin_health.py now emits self_check section (functional/meta/
hygiene/recent_repairs); mofin_health.html new '🩺 自检体系' tab rendering
L4 layers, L1 module checks, L2 issues, L3 repair history.

E2E verified: stopped xmpp bot -> L1 flagged fail -> systemd recovered ->
L3 LLM correctly diagnosed 'none needed' and logged; rerun_script whitelist
path executes real scripts successfully; meta_watchdog all-green after fix.
This commit is contained in:
hmo
2026-07-20 19:39:58 +08:00
parent 4f83ee8a01
commit 08eef1e181
11 changed files with 874 additions and 1 deletions
+42
View File
@@ -955,6 +955,47 @@ def build_report():
"profile": j.get("profile", "?"),
})
# ── 自检体系状态(L1功能健康/L3修复记录/L4元监控/L2卫生)──
self_check = {}
LOGS = Path('/home/hmo/MoFin/gateway/logs')
try:
fh = json.loads((LOGS / 'functional_health.json').read_text(encoding='utf-8'))
self_check['functional'] = {
'generated_at': fh.get('generated_at'), 'status': fh.get('status'),
'summary': fh.get('summary'), 'checks': fh.get('checks', []),
}
except Exception:
self_check['functional'] = None
try:
mw = json.loads((LOGS / 'meta_watchdog.json').read_text(encoding='utf-8'))
self_check['meta_watchdog'] = {
'generated_at': mw.get('generated_at'), 'status': mw.get('status'),
'layers': mw.get('layers', []),
}
except Exception:
self_check['meta_watchdog'] = None
try:
hy = json.loads((LOGS / 'hygiene_report.json').read_text(encoding='utf-8'))
self_check['hygiene'] = {
'generated_at': hy.get('generated_at'), 'status': hy.get('status'),
'issue_count': hy.get('issue_count'), 'issues': hy.get('issues', [])[:10],
}
except Exception:
self_check['hygiene'] = None
try:
repairs = []
rp = LOGS / 'repair_log.jsonl'
if rp.exists():
for line in rp.read_text(encoding='utf-8').splitlines()[-10:]:
try:
repairs.append(json.loads(line))
except Exception:
pass
repairs.reverse()
self_check['recent_repairs'] = repairs
except Exception:
self_check['recent_repairs'] = []
# ── 写JSON ──
report = {
"generated_at": now.strftime("%Y-%m-%d %H:%M:%S"),
@@ -963,6 +1004,7 @@ def build_report():
"json_files": json_entities,
"pipelines": pipelines,
"db_freshness": db_freshness,
"self_check": self_check,
}
out_path = WEB_DATA / "mofin_health.json"
with open(out_path, "w") as f: