feat(self-check): L0-L4 layered self-check architecture with LLM auto-repair
User directive: daily not weekly; clear responsibilities per layer with no
overlap; functional criteria (does the function WORK) not process liveness;
problems get FIXED via LLM with file-and-report discipline (act first,
report after); plus a meta-layer watching the watchers; deeply integrated
into F健康.
Architecture (responsibility matrix in dev-spec.md):
- L0 agents_health_check (5min): port/HTTP/DB liveness + auto_heal executor
- L1 functional_health_check (15min trading): per-module FUNCTIONAL
criteria — output freshness/validity per REGISTRY (live_prices/market_
snapshots/mtf_cache/macro_context/bot/LLM/cron engine), not process alive
- L2 system_hygiene_audit (daily 08:20, was weekly): divergence/hardlink/
zombie/orphan/dead-cron/db-freshness
- L3 self_repair (30min): reads L1/L2 failures -> LLM diagnoses -> executes
WHITELISTED repair actions directly (rerun_script/restart_service/
sync_links/switch_llm_key/none) -> repair_log.jsonl + XMPP report.
Max 2 repairs/module/day anti-loop. LLM unavailable -> rule fallback.
- L4 meta_watchdog (hourly): checks L0-L3 output freshness + L3 cron
registration + XMPP bridge; direct XMPP alert as last resort
Retired (overlap): Cron监护-高频 (cron_watchdog -> L3), 全局cron健康监控
(cron_health_monitor -> L1).
Dashboard: mofin_health.py now emits self_check section (functional/meta/
hygiene/recent_repairs); mofin_health.html new '🩺 自检体系' tab rendering
L4 layers, L1 module checks, L2 issues, L3 repair history.
E2E verified: stopped xmpp bot -> L1 flagged fail -> systemd recovered ->
L3 LLM correctly diagnosed 'none needed' and logged; rerun_script whitelist
path executes real scripts successfully; meta_watchdog all-green after fix.
This commit is contained in:
@@ -955,6 +955,47 @@ def build_report():
|
||||
"profile": j.get("profile", "?"),
|
||||
})
|
||||
|
||||
# ── 自检体系状态(L1功能健康/L3修复记录/L4元监控/L2卫生)──
|
||||
self_check = {}
|
||||
LOGS = Path('/home/hmo/MoFin/gateway/logs')
|
||||
try:
|
||||
fh = json.loads((LOGS / 'functional_health.json').read_text(encoding='utf-8'))
|
||||
self_check['functional'] = {
|
||||
'generated_at': fh.get('generated_at'), 'status': fh.get('status'),
|
||||
'summary': fh.get('summary'), 'checks': fh.get('checks', []),
|
||||
}
|
||||
except Exception:
|
||||
self_check['functional'] = None
|
||||
try:
|
||||
mw = json.loads((LOGS / 'meta_watchdog.json').read_text(encoding='utf-8'))
|
||||
self_check['meta_watchdog'] = {
|
||||
'generated_at': mw.get('generated_at'), 'status': mw.get('status'),
|
||||
'layers': mw.get('layers', []),
|
||||
}
|
||||
except Exception:
|
||||
self_check['meta_watchdog'] = None
|
||||
try:
|
||||
hy = json.loads((LOGS / 'hygiene_report.json').read_text(encoding='utf-8'))
|
||||
self_check['hygiene'] = {
|
||||
'generated_at': hy.get('generated_at'), 'status': hy.get('status'),
|
||||
'issue_count': hy.get('issue_count'), 'issues': hy.get('issues', [])[:10],
|
||||
}
|
||||
except Exception:
|
||||
self_check['hygiene'] = None
|
||||
try:
|
||||
repairs = []
|
||||
rp = LOGS / 'repair_log.jsonl'
|
||||
if rp.exists():
|
||||
for line in rp.read_text(encoding='utf-8').splitlines()[-10:]:
|
||||
try:
|
||||
repairs.append(json.loads(line))
|
||||
except Exception:
|
||||
pass
|
||||
repairs.reverse()
|
||||
self_check['recent_repairs'] = repairs
|
||||
except Exception:
|
||||
self_check['recent_repairs'] = []
|
||||
|
||||
# ── 写JSON ──
|
||||
report = {
|
||||
"generated_at": now.strftime("%Y-%m-%d %H:%M:%S"),
|
||||
@@ -963,6 +1004,7 @@ def build_report():
|
||||
"json_files": json_entities,
|
||||
"pipelines": pipelines,
|
||||
"db_freshness": db_freshness,
|
||||
"self_check": self_check,
|
||||
}
|
||||
out_path = WEB_DATA / "mofin_health.json"
|
||||
with open(out_path, "w") as f:
|
||||
|
||||
Reference in New Issue
Block a user