hmo
08eef1e181
feat(self-check): L0-L4 layered self-check architecture with LLM auto-repair
User directive: daily not weekly; clear responsibilities per layer with no
overlap; functional criteria (does the function WORK) not process liveness;
problems get FIXED via LLM with file-and-report discipline (act first,
report after); plus a meta-layer watching the watchers; deeply integrated
into F健康.
Architecture (responsibility matrix in dev-spec.md):
- L0 agents_health_check (5min): port/HTTP/DB liveness + auto_heal executor
- L1 functional_health_check (15min trading): per-module FUNCTIONAL
criteria — output freshness/validity per REGISTRY (live_prices/market_
snapshots/mtf_cache/macro_context/bot/LLM/cron engine), not process alive
- L2 system_hygiene_audit (daily 08:20, was weekly): divergence/hardlink/
zombie/orphan/dead-cron/db-freshness
- L3 self_repair (30min): reads L1/L2 failures -> LLM diagnoses -> executes
WHITELISTED repair actions directly (rerun_script/restart_service/
sync_links/switch_llm_key/none) -> repair_log.jsonl + XMPP report.
Max 2 repairs/module/day anti-loop. LLM unavailable -> rule fallback.
- L4 meta_watchdog (hourly): checks L0-L3 output freshness + L3 cron
registration + XMPP bridge; direct XMPP alert as last resort
Retired (overlap): Cron监护-高频 (cron_watchdog -> L3), 全局cron健康监控
(cron_health_monitor -> L1).
Dashboard: mofin_health.py now emits self_check section (functional/meta/
hygiene/recent_repairs); mofin_health.html new '🩺 自检体系' tab rendering
L4 layers, L1 module checks, L2 issues, L3 repair history.
E2E verified: stopped xmpp bot -> L1 flagged fail -> systemd recovered ->
L3 LLM correctly diagnosed 'none needed' and logged; rerun_script whitelist
path executes real scripts successfully; meta_watchdog all-green after fix.
2026-07-20 19:39:58 +08:00
..
2026-07-08 12:34:39 +08:00
2026-07-06 14:01:54 +08:00
2026-07-08 23:54:01 +08:00
2026-07-09 17:57:17 +08:00
2026-07-20 19:04:05 +08:00
2026-07-08 12:37:36 +08:00
2026-07-20 01:12:38 +08:00
2026-07-20 19:04:05 +08:00
2026-07-20 19:04:05 +08:00
2026-07-20 19:04:05 +08:00
2026-07-20 19:04:05 +08:00
2026-07-20 19:04:05 +08:00
2026-07-20 19:04:05 +08:00
2026-07-20 17:46:57 +08:00
2026-07-10 13:58:23 +08:00
2026-07-04 09:30:50 +08:00
2026-07-08 23:54:01 +08:00
2026-07-09 19:42:35 +08:00
2026-07-10 14:06:20 +08:00
2026-07-20 18:10:05 +08:00
2026-07-20 08:48:08 +08:00
2026-07-20 01:34:37 +08:00
2026-07-19 20:12:47 +08:00
2026-07-19 11:06:50 +08:00
2026-07-20 08:48:08 +08:00
2026-07-20 17:30:15 +08:00
2026-07-20 17:46:57 +08:00
2026-07-20 17:30:15 +08:00
2026-07-20 01:12:38 +08:00
2026-07-19 11:06:50 +08:00
2026-07-20 17:30:15 +08:00
2026-07-19 11:06:50 +08:00
2026-07-20 17:46:57 +08:00
2026-07-20 17:30:15 +08:00
2026-07-19 11:06:50 +08:00
2026-07-06 14:01:54 +08:00
2026-07-08 23:54:01 +08:00
2026-07-20 19:04:05 +08:00
2026-07-08 23:54:01 +08:00
2026-07-20 19:04:05 +08:00
2026-07-08 19:50:44 +08:00
2026-07-14 10:56:38 +08:00
2026-07-20 01:12:38 +08:00
2026-07-14 12:14:08 +08:00
2026-07-06 14:01:54 +08:00
2026-07-08 23:54:01 +08:00
2026-07-08 19:50:44 +08:00
2026-07-10 11:59:08 +08:00
2026-07-14 11:08:07 +08:00
2026-07-19 20:12:47 +08:00
2026-07-20 01:12:38 +08:00
2026-07-20 19:04:05 +08:00
2026-07-20 19:04:05 +08:00
2026-07-14 15:00:56 +08:00
2026-07-19 11:06:50 +08:00
2026-07-20 01:12:38 +08:00
2026-07-19 11:06:50 +08:00
2026-07-20 17:30:15 +08:00
2026-07-10 10:26:14 +08:00
2026-07-20 01:34:37 +08:00
2026-07-06 14:01:54 +08:00
2026-07-20 17:30:15 +08:00
2026-07-01 23:00:28 +08:00
2026-07-06 14:01:54 +08:00
2026-07-03 12:12:05 +08:00
2026-07-09 19:42:35 +08:00
2026-07-17 23:08:11 +08:00
2026-07-20 19:04:05 +08:00
2026-07-14 10:56:38 +08:00
2026-07-20 17:30:15 +08:00
2026-07-20 17:30:15 +08:00
2026-07-20 17:30:15 +08:00
2026-07-20 17:30:15 +08:00
2026-07-20 17:30:15 +08:00
2026-07-19 20:12:47 +08:00
2026-07-19 11:06:50 +08:00
2026-07-02 13:46:53 +08:00
2026-07-01 23:00:28 +08:00
2026-07-08 12:37:36 +08:00
2026-07-19 12:47:15 +08:00
2026-07-08 20:04:32 +08:00
2026-07-08 12:37:36 +08:00
2026-06-27 01:37:18 +08:00
2026-07-20 19:04:05 +08:00
2026-06-24 00:15:33 +08:00
2026-07-08 12:34:39 +08:00
2026-07-06 14:01:54 +08:00
2026-07-20 18:10:05 +08:00
2026-07-09 16:35:27 +08:00
2026-07-06 14:01:54 +08:00
2026-07-06 14:01:54 +08:00
2026-07-08 23:54:01 +08:00
2026-07-08 23:54:01 +08:00
2026-07-20 18:10:05 +08:00
2026-07-14 10:56:38 +08:00
2026-07-08 12:37:36 +08:00
2026-07-06 14:01:54 +08:00
2026-07-09 11:13:45 +08:00
2026-07-08 12:34:39 +08:00
2026-07-08 23:54:01 +08:00
2026-07-20 17:33:39 +08:00
2026-07-20 18:10:05 +08:00
2026-07-20 19:04:05 +08:00
2026-07-20 19:04:05 +08:00
2026-07-20 19:39:58 +08:00
2026-07-20 17:30:15 +08:00
2026-06-30 02:56:35 +08:00
2026-07-13 12:09:25 +08:00
2026-07-19 20:12:47 +08:00
2026-07-08 23:54:01 +08:00
2026-07-09 13:43:30 +08:00
2026-07-10 12:20:18 +08:00
2026-07-08 16:18:26 +08:00
2026-07-20 18:10:05 +08:00
2026-07-06 14:01:54 +08:00
2026-07-13 12:09:25 +08:00
2026-07-04 09:30:50 +08:00
2026-07-08 23:54:01 +08:00
2026-07-06 14:01:54 +08:00
2026-07-20 19:39:58 +08:00
2026-07-20 19:39:58 +08:00
2026-07-20 19:04:05 +08:00
2026-07-20 19:04:05 +08:00
2026-07-03 12:12:05 +08:00
2026-07-04 09:45:27 +08:00
2026-07-20 00:40:32 +08:00
2026-07-20 08:48:08 +08:00
2026-07-20 18:10:05 +08:00
2026-07-14 10:56:38 +08:00
2026-07-08 12:34:39 +08:00
2026-07-06 14:01:54 +08:00
2026-07-13 12:09:25 +08:00
2026-07-13 12:09:25 +08:00
2026-07-08 23:54:01 +08:00
2026-07-08 23:54:01 +08:00
2026-06-29 23:25:54 +08:00
2026-07-08 12:37:36 +08:00
2026-07-08 23:54:01 +08:00
2026-07-08 12:37:36 +08:00
2026-07-17 23:08:11 +08:00
2026-07-08 23:54:01 +08:00
2026-07-08 12:34:39 +08:00
2026-07-08 23:54:01 +08:00
2026-07-19 10:56:42 +08:00
2026-07-09 13:43:30 +08:00
2026-07-06 14:01:54 +08:00
2026-07-07 11:20:04 +08:00
2026-07-19 23:23:46 +08:00
2026-07-17 23:08:11 +08:00
2026-07-08 12:34:39 +08:00
2026-07-10 21:07:59 +08:00
2026-07-19 20:12:47 +08:00
2026-07-20 18:10:05 +08:00
2026-07-20 17:46:57 +08:00
2026-07-20 17:46:57 +08:00
2026-07-20 17:30:15 +08:00
2026-07-19 23:23:46 +08:00
2026-07-19 22:09:15 +08:00
2026-07-19 20:12:47 +08:00
2026-07-20 01:12:38 +08:00
2026-07-20 00:21:49 +08:00
2026-07-19 20:12:47 +08:00
2026-07-20 17:46:57 +08:00
2026-07-20 19:39:58 +08:00
2026-07-19 11:06:50 +08:00
2026-07-19 23:23:46 +08:00
2026-07-19 20:12:47 +08:00
2026-07-08 12:37:36 +08:00
2026-07-20 19:04:05 +08:00
2026-07-20 08:48:08 +08:00
2026-07-08 10:13:49 +08:00
2026-07-20 01:12:38 +08:00
2026-07-20 08:48:08 +08:00
2026-07-20 00:07:28 +08:00
2026-07-20 17:30:15 +08:00
2026-07-07 15:33:14 +08:00
2026-07-20 19:39:58 +08:00
2026-07-19 11:06:50 +08:00
2026-07-10 12:20:18 +08:00
2026-07-08 12:37:36 +08:00
2026-07-01 23:00:28 +08:00
2026-07-06 14:01:54 +08:00
2026-07-09 10:35:34 +08:00
2026-07-02 01:43:33 +08:00