hmo
4f83ee8a01
Root cause analysis of the 2026-07-20 redundancy incident: 1. No single-source-of-truth rule -> same file legitimately lived in 4+ locations, diverging silently 2. Relative path resolution (Path(__file__).parent/'data') -> each hardlinked copy of mofin_db.py pointed to a DIFFERENT database 3. 'Backup habit' left .bak/legacy files in production dirs, which monitoring then scanned and reported as false alarms 4. Half-done migrations: DB tables created but old JSON writers/readers stayed (price_events), old files stayed 5. Dead modules never got buried: xiaoguo 'dead' but bot ran 8 days as root eating 2.5GB 6. Monitoring checked 'does it exist' not 'is it alive' -> stale file mtime reported as 'pipeline stalled 14 days' (false alarm) 7. No 'system hygiene' as a check category at all Prevention implemented: - dev-spec.md v2.0: 五条红线 -> 十条红线 #6 single source of truth (hardlink only, no independent copies) #7 absolute data paths only (no __file__-relative data resolution) #8 no backups/legacy in production data dirs (archive immediately) #9 dead module burial checklist (6 mandatory steps) #10 monitor liveness (DB table freshness) not existence - File Location Constitution: canonical location per content type - NEW system_hygiene_audit.py: weekly Monday 07:30 cron checking diverged copies / broken hardlinks / zombie processes / orphan data files / dead cron scripts / DB freshness -> hygiene_report.json + XMPP - specs/hygiene.json: module spec per red line #1 - Verified: audit found 5 real issues on first run, all fixed, re-run clean
46 lines
2.1 KiB
Python
46 lines
2.1 KiB
Python
#!/usr/bin/env python3
|
||
"""audit_runtime.py — 运行中的进程/服务/cron 审计"""
|
||
import subprocess, json, os
|
||
|
||
def sh(cmd, timeout=10):
|
||
try:
|
||
r = subprocess.run(cmd, shell=True, capture_output=True, text=True, timeout=timeout)
|
||
return r.stdout.strip()
|
||
except Exception as e:
|
||
return f"ERR: {e}"
|
||
|
||
print("=== 1. 长期运行的 Python 进程（>1天）===")
|
||
out = sh("ps -eo pid,etime,user,args --sort=-etime | grep -E 'python|node' | grep -v grep | head -25")
|
||
print(out)
|
||
|
||
print("\n=== 2. systemd 服务状态（全部 mofin/hermes/xmpp/ejabberd 相关）===")
|
||
out = sh("systemctl list-units --all --type=service 2>/dev/null | grep -iE 'mofin|hermes|xmpp|ejabberd|wechat|gitea|kanban|wiki|stock|xiaoguo|zhiwei|mohe' | head -25")
|
||
print(out)
|
||
print("\n--- user services ---")
|
||
out = sh("systemctl --user list-units --all --type=service 2>/dev/null | grep -iE 'hermes|xmpp|mofin|wiki' | head -10")
|
||
print(out)
|
||
|
||
print("\n=== 3. DISABLED cron jobs（hermes 两个 profile）===")
|
||
for jf, prof in [('/home/hmo/.hermes/profiles/position-analyst/cron/jobs.json', 'position-analyst'),
|
||
('/home/hmo/.hermes/cron/jobs.json', 'default')]:
|
||
try:
|
||
d = json.load(open(jf))
|
||
jobs = d if isinstance(d, list) else d.get('jobs', [])
|
||
for j in jobs:
|
||
if not j.get('enabled', True):
|
||
lr = str(j.get('last_run_at') or '?')[:10]
|
||
print(f" [{prof}] {j.get('name')} | script={j.get('script')} | last={lr} | paused_reason={j.get('paused_reason')}")
|
||
except Exception as e:
|
||
print(f" {prof}: {e}")
|
||
|
||
print("\n=== 4. 系统 crontab 全文（非注释行）===")
|
||
out = sh("crontab -l | grep -vE '^\\s*#' | grep -vE '^\\s*$'")
|
||
print(out)
|
||
|
||
print("\n=== 5. 僵尸/异常进程（root跑的python、很老的进程）===")
|
||
out = sh("ps -eo pid,etime,user,args | grep -E '^\\s*\\S+\\s+\\S+\\s+root' | grep python | grep -v grep")
|
||
print(out if out else "(none)")
|
||
|
||
print("\n=== 6. 监听端口清单 ===")
|
||
out = sh("ss -tlnp 2>/dev/null | grep -E '8642|8643|8645|8646|8899|5801|5802|5803|5804|5805|5807|5808|5810|9090|9580|5222|5443|5280|19088|19099|9877|9878' | awk '{print $4, $6}'")
|
||
print(out) |