feat(self-check): L0-L4 layered self-check architecture with LLM auto-repair

User directive: daily not weekly; clear responsibilities per layer with no
overlap; functional criteria (does the function WORK) not process liveness;
problems get FIXED via LLM with file-and-report discipline (act first,
report after); plus a meta-layer watching the watchers; deeply integrated
into F健康.

Architecture (responsibility matrix in dev-spec.md):
- L0 agents_health_check (5min): port/HTTP/DB liveness + auto_heal executor
- L1 functional_health_check (15min trading): per-module FUNCTIONAL
  criteria — output freshness/validity per REGISTRY (live_prices/market_
  snapshots/mtf_cache/macro_context/bot/LLM/cron engine), not process alive
- L2 system_hygiene_audit (daily 08:20, was weekly): divergence/hardlink/
  zombie/orphan/dead-cron/db-freshness
- L3 self_repair (30min): reads L1/L2 failures -> LLM diagnoses -> executes
  WHITELISTED repair actions directly (rerun_script/restart_service/
  sync_links/switch_llm_key/none) -> repair_log.jsonl + XMPP report.
  Max 2 repairs/module/day anti-loop. LLM unavailable -> rule fallback.
- L4 meta_watchdog (hourly): checks L0-L3 output freshness + L3 cron
  registration + XMPP bridge; direct XMPP alert as last resort

Retired (overlap): Cron监护-高频 (cron_watchdog -> L3), 全局cron健康监控
(cron_health_monitor -> L1).

Dashboard: mofin_health.py now emits self_check section (functional/meta/
hygiene/recent_repairs); mofin_health.html new '🩺 自检体系' tab rendering
L4 layers, L1 module checks, L2 issues, L3 repair history.

E2E verified: stopped xmpp bot -> L1 flagged fail -> systemd recovered ->
L3 LLM correctly diagnosed 'none needed' and logged; rerun_script whitelist
path executes real scripts successfully; meta_watchdog all-green after fix.
This commit is contained in:
hmo
2026-07-20 19:39:58 +08:00
parent 4f83ee8a01
commit 08eef1e181
11 changed files with 874 additions and 1 deletions
+21 -1
View File
@@ -156,7 +156,27 @@ specs/{module}.json
---
## 三、开发流程
## 三、自检体系责任矩阵(L0-L4
> 2026-07-20 确立。原则:**每层职责单一,不重叠不疏漏;判据是"功能是否达成",不是"进程是否活着"**。
| 层 | 组件 | 频率 | 职责(唯一) | 判据 |
|---|------|------|-------------|------|
| L0 执行心跳 | `agents_health_check.py`(系统 crontab | 5min | 端口/HTTP/DB 存活 + auto_heal 执行(gateway 重启/key 切换/bot 重启) | 端口通 + HTTP 200 + DB 可写 |
| L1 功能健康 | `functional_health_check.py`hermes cron | 交易时段 15min | 核心模块**功能是否达成**:检查输出物新鲜度/有效性(live_prices/market_snapshots/mtf_cache/macro_context/bot/LLM/cron引擎) | 每模块注册表判据(REGISTRY),输出 `functional_health.json` |
| L2 系统卫生 | `system_hygiene_audit.py`hermes cron | 每日 08:20 | 分叉副本/断裂硬链接/僵尸进程/孤儿文件/死cron/DB新鲜度(红线6-10 enforcement | 输出 `hygiene_report.json` |
| L3 修复循环 | `self_repair.py`hermes cron | 30min | 读 L1/L2 失败项 → LLM 诊断 → **白名单动作直接修复(报备制)**`repair_log.jsonl` + XMPP | LLM 只能选白名单动作;每模块每天≤2次防循环 |
| L4 元监控 | `meta_watchdog.py`hermes cron | 每小时 | **自检系统的自检**:L0-L3 输出物新鲜度 + L3 注册状态 + XMPP 桥 | 任何一层死亡直接 XMPP 点名(最后兜底) |
**已退休(职责被合并)**`Cron监护-高频`cron_watchdog,并入 L3)、`全局cron健康监控-每10分`cron_health_monitor,并入 L1)。
### 报备制(L3 的核心纪律)
发现问题 → **直接修复** → 记录日志 → XMPP 报备。不是"发现问题→报告→等指示"。LLM 介入诊断但**只能执行白名单动作**rerun_script/restart_service/sync_links/switch_llm_key/none),不允许任意代码执行。白名单兜不住的,在 XMPP 里明确说"需要人工"。
---
## 四、开发流程
### 新增功能流程