hmo
08eef1e181
User directive: daily not weekly; clear responsibilities per layer with no
overlap; functional criteria (does the function WORK) not process liveness;
problems get FIXED via LLM with file-and-report discipline (act first,
report after); plus a meta-layer watching the watchers; deeply integrated
into F健康.
Architecture (responsibility matrix in dev-spec.md):
- L0 agents_health_check (5min): port/HTTP/DB liveness + auto_heal executor
- L1 functional_health_check (15min trading): per-module FUNCTIONAL
criteria — output freshness/validity per REGISTRY (live_prices/market_
snapshots/mtf_cache/macro_context/bot/LLM/cron engine), not process alive
- L2 system_hygiene_audit (daily 08:20, was weekly): divergence/hardlink/
zombie/orphan/dead-cron/db-freshness
- L3 self_repair (30min): reads L1/L2 failures -> LLM diagnoses -> executes
WHITELISTED repair actions directly (rerun_script/restart_service/
sync_links/switch_llm_key/none) -> repair_log.jsonl + XMPP report.
Max 2 repairs/module/day anti-loop. LLM unavailable -> rule fallback.
- L4 meta_watchdog (hourly): checks L0-L3 output freshness + L3 cron
registration + XMPP bridge; direct XMPP alert as last resort
Retired (overlap): Cron监护-高频 (cron_watchdog -> L3), 全局cron健康监控
(cron_health_monitor -> L1).
Dashboard: mofin_health.py now emits self_check section (functional/meta/
hygiene/recent_repairs); mofin_health.html new '🩺 自检体系' tab rendering
L4 layers, L1 module checks, L2 issues, L3 repair history.
E2E verified: stopped xmpp bot -> L1 flagged fail -> systemd recovered ->
L3 LLM correctly diagnosed 'none needed' and logged; rerun_script whitelist
path executes real scripts successfully; meta_watchdog all-green after fix.
59 lines
2.6 KiB
Python
59 lines
2.6 KiB
Python
import json, shutil, uuid
|
||
from datetime import datetime
|
||
|
||
jf = '/home/hmo/.hermes/profiles/position-analyst/cron/jobs.json'
|
||
shutil.copy(jf, jf + '.bak-20260720-l134')
|
||
d = json.load(open(jf))
|
||
is_list = isinstance(d, list)
|
||
jobs = d if is_list else d.get('jobs', [])
|
||
|
||
def has_script(s):
|
||
return any(j.get('script') == s for j in jobs)
|
||
|
||
def mkjob(name, script, schedule, display, next_run):
|
||
return {
|
||
"id": uuid.uuid4().hex[:12], "name": name, "prompt": "", "skills": [], "skill": None,
|
||
"model": None, "provider": None, "base_url": None, "script": script,
|
||
"no_agent": True, "context_from": None,
|
||
"schedule": {"kind": "cron", "expr": schedule, "display": display},
|
||
"schedule_display": display,
|
||
"repeat": {"times": None, "completed": 0}, "enabled": True, "state": "scheduled",
|
||
"paused_at": None, "paused_reason": None, "created_at": datetime.now().isoformat(),
|
||
"next_run_at": next_run, "last_run_at": None, "last_status": None,
|
||
}
|
||
|
||
# 1. 新增 L1/L3/L4 jobs
|
||
added = []
|
||
if not has_script('functional_health_check.py'):
|
||
jobs.append(mkjob('功能健康检查-L1', 'functional_health_check.py', '*/15 9-16,20-22 * * 1-5', '*/15 9-16,20-22 * * 1-5', '2026-07-20T20:30:00+08:00'))
|
||
added.append('功能健康检查-L1(15min)')
|
||
if not has_script('self_repair.py'):
|
||
jobs.append(mkjob('LLM修复循环-L3', 'self_repair.py', '*/30 9-16,20-22 * * 1-5', '*/30 9-16,20-22 * * 1-5', '2026-07-20T20:30:00+08:00'))
|
||
added.append('LLM修复循环-L3(30min)')
|
||
if not has_script('meta_watchdog.py'):
|
||
jobs.append(mkjob('元监控-自检系统的自检-L4', 'meta_watchdog.py', '5 * * * *', '5 * * * *', '2026-07-20T20:05:00+08:00'))
|
||
added.append('元监控-L4(每小时)')
|
||
|
||
# 2. hygiene 改每日（原每周一 07:30 → 每日 08:20）
|
||
for j in jobs:
|
||
if j.get('script') == 'system_hygiene_audit.py':
|
||
j['schedule'] = {"kind": "cron", "expr": "20 8 * * *", "display": "20 8 * * *"}
|
||
j['schedule_display'] = "20 8 * * *"
|
||
j['name'] = '系统卫生审计-每日'
|
||
j['next_run_at'] = '2026-07-21T08:20:00+08:00'
|
||
added.append('hygiene改每日08:20')
|
||
|
||
# 3. 退休重叠组件：Cron监护-高频(cron_watchdog) + 全局cron健康监控(cron_health_monitor)
|
||
REMOVE = {'Cron监护-高频', '全局cron健康监控-每10分'}
|
||
removed = [j.get('name') for j in jobs if j.get('name') in REMOVE]
|
||
jobs = [j for j in jobs if j.get('name') not in REMOVE]
|
||
|
||
if is_list:
|
||
json.dump(jobs, open(jf, 'w'), ensure_ascii=False, indent=2)
|
||
else:
|
||
d['jobs'] = jobs
|
||
json.dump(d, open(jf, 'w'), ensure_ascii=False, indent=2)
|
||
|
||
print('added:', added)
|
||
print('removed:', removed)
|
||
print('total jobs:', len(jobs)) |