Root causes of the screenshot 404 incident:
1. RACE: client uploads image AND sends message concurrently; bot received
the message before the upload finished writing, so its GET hit a 404
error page (<100B treated as failure). FIX: _download_image now retries
3x with 2s backoff.
2. Zhiwei mentioned tesseract/小果 because the failure text never told her
the pipeline IS SenseNova. FIX: failure messages now name SenseNova
explicitly and ask for resend.
3. log_xmpp never worked for the bot: sys.path used relative '../..' from
a symlinked __file__ which resolved to '/' instead of MoFin root. This
is why the '最近对话' panel never had bot chat data (only cron script
entries). FIX: absolute path per red line #7. Verified: test message
now lands in xmpp_messages.jsonl.
4. My PowerShell -replace corrupted the file encoding (UnicodeDecodeError
crash loop on restart). Restored from git HEAD and re-applied edits with
the edit tool. Lesson: never use PowerShell string replace on UTF-8
source files with Chinese content.
5. functional_health: new sense_ocr module (OCR config presence +
SenseNova API TCP reachability), no token cost.
Zhiwei flagged: '十条红线' but she counted 14. Real issues found:
- duplicate section numbers: two '三、' (验证闭环 + 自检体系矩阵),
two '四、' (开发流程 + 部署环境) -> renumbered 一~七 sequentially
- doc index said '含五条红线' (stale from v1) -> '含十条红线'
- F 小节 still used old Tier1/Tier2 framing -> aligned to L0/L1-L2
with pointer to the L0-L4 matrix section
- version bump v2.0 -> v2.1
Minimal edit: no content changes beyond numbering/consistency.
Gateway看门狗-知微 was erroring (exit -15): its check_session_health did a
live LLM ping with 25s timeout. Cold-start LLM latency is 20-100s so the
ping always timed out -> false '不健康' verdict -> false gateway restart
-> and each 10-min run burned 22k tokens.
Now uses xmpp_logger._scan_agent_log (zero cost, reads real call results):
- ok if last real call succeeded
- unhealthy only if last call explicitly failed
- idle (no recent calls) counts as healthy
Verified: watchdog job now status=ok.
Also: triggered all 6 weekend 'Blocked' jobs via hermes cron run — all
now status=ok, proving the hardlink fix holds.
User insight: hardlink breakage only happens at deploy time (scp file
replacement / git checkout-merge), so detection must be welded INTO the
deploy pipeline, not left to daily audit.
Three automatic layers, no reliance on discipline:
1. systemd path watcher (profile-scripts-sync.path): watches
deploy/profile-scripts/ directory, auto-fires sync_profile_scripts.sh
on any change. Verified: fires within 4s of file replacement, logs to
gateway/logs/link_sync.log (runs as hmo user)
2. git hooks (.git/hooks/post-merge + post-checkout on 246 repo):
auto re-link after git operations
3. Manual fallback: sync_profile_scripts.sh (now self-logging)
dev-spec red line #6 updated: SSOT rule now documents the three layers
and states breakage only happens at deploy time.
Root cause analysis of the 2026-07-20 redundancy incident:
1. No single-source-of-truth rule -> same file legitimately lived in 4+
locations, diverging silently
2. Relative path resolution (Path(__file__).parent/'data') -> each
hardlinked copy of mofin_db.py pointed to a DIFFERENT database
3. 'Backup habit' left .bak/legacy files in production dirs, which
monitoring then scanned and reported as false alarms
4. Half-done migrations: DB tables created but old JSON writers/readers
stayed (price_events), old files stayed
5. Dead modules never got buried: xiaoguo 'dead' but bot ran 8 days
as root eating 2.5GB
6. Monitoring checked 'does it exist' not 'is it alive' -> stale file
mtime reported as 'pipeline stalled 14 days' (false alarm)
7. No 'system hygiene' as a check category at all
Prevention implemented:
- dev-spec.md v2.0: 五条红线 -> 十条红线
#6 single source of truth (hardlink only, no independent copies)
#7 absolute data paths only (no __file__-relative data resolution)
#8 no backups/legacy in production data dirs (archive immediately)
#9 dead module burial checklist (6 mandatory steps)
#10 monitor liveness (DB table freshness) not existence
- File Location Constitution: canonical location per content type
- NEW system_hygiene_audit.py: weekly Monday 07:30 cron checking
diverged copies / broken hardlinks / zombie processes / orphan data
files / dead cron scripts / DB freshness -> hygiene_report.json + XMPP
- specs/hygiene.json: module spec per red line #1
- Verified: audit found 5 real issues on first run, all fixed, re-run clean
User directive: no JSON, retire it fully, fix all related code.
Changes:
- price_monitor.py: record_event writes DB only; removed EVENTS_PATH/
load_events/save_events entirely
- strategy_feedback.py: price events read from DB only (removed JSON fallback)
- system_health_check.py: removed price_events.json from file-check list,
DB-only event stats (was showing 0/0 due to wrong-DB resolution)
- mo_config.py: removed dead price_events_path property (no callers)
- mofin_health.py: price_events freshness reads DB table (authoritative now)
- mofin_db.py: DATA_DIR/DB_PATH now ABSOLUTE (/home/hmo/MoFin/data) —
was relative __file__.parent, so each hardlinked copy of mofin_db.py
resolved to a DIFFERENT database (canonical vs web-dashboard vs
profile-local third DB with 0 rows of everything except market_snapshots).
This fragmentation was the real cause of health checks reading empty tables.
- Unified all 4 mofin_db copies (root/scripts/deploy/profile) via hardlink
- price_events.json archived to trashbox (fully backfilled: 6353 rows in DB)
Verified:
- record_event lands in DB only, JSON not recreated
- system_health_check: 历史事件 6353 / 今日事件 2965 (was 0/0)
- strategy_feedback + price_monitor full runs clean
User caught the inconsistency: system claims DB-first but price events
only went to price_events.json, leaving DB table stale since Jul 6.
Root cause chain found:
- record_event() only wrote JSON, never called mofin_db.write_price_event
- price_events.code has FK -> stocks(code); events for unregistered stocks
(new candidates, HK) silently failed INSERT and were lost to DB
- mofin_db.write_price_event swallows errors (returns False silently)
Fixes:
- record_event now dual-writes: DB (authoritative) + JSON (compat for
legacy readers mo_config/strategy_feedback/system_health_check)
- auto-registers unknown codes into stocks table before event insert
- one-time backfill: 4064 JSON events -> DB (total 6353 rows, last=today)
- verified: record_event TEST99 lands in both DB and JSON
Real errors fixed (all verified by manual run):
- price_monitor.py: shares None -> TypeError at L584 (now completes 3m7s,
full 39-stock reassess + zone triggers + Dad push)
- market_insight.py: net_inflow None -> TypeError at L142 (now 0.3s, 5 insights)
- promote_candidates.py: add busy_timeout=30s (DB lock under concurrent writes)
- premarket_full_review.py: 12-dim analysis now detached background launch
(was doomed by cron 120s script timeout no matter what)
Systemic:
- HERMES_CRON_SCRIPT_TIMEOUT=600 drop-in for both gateway services
(fixes mofin_health SIGTERM, market_watch timeout, memory_guardian timeout)
- sync_profile_scripts.sh: re-hardlink deploy->profile scripts after every
deploy (scp replaces files = new inode = broken hardlink = cron silently
runs stale code; this caused promote to keep failing after my first fix)
Monitoring false-alarm fixes (the '花瓶' problem):
- mofin_health.py: legacy JSONs that migrated to DB (multi_tf_cache/
macro_context/market/live_prices/price_history/macro_risk_state) no longer
warn 'no readers'; marked as migrated
- NEW db_freshness section: real pipeline health from DB tables
(mtf_cache 0.4h / macro_context_log 2h / market_snapshots 2h /
live_prices 0.4h / price_events.json 0.4h — ALL HEALTHY)
- price_events freshness reads live JSON store (DB table is legacy)
- market.json placeholder created (13+ scripts have fallback paths)
Investigation notes: wiki-self-growth 03:04 key1 429 predates full key6
activation on default gateway; current 8642 verified on key6 and working.
Weekend 'Blocked' jobs verified fixed (vacuum_state_db passes).
Gap (reported by user via zhiwei): premarket full review updated technical
params but full_analysis (12-dim LLM matrix) was empty for new holdings
and stale for old ones — batch_reassess existed but was never wired into
the daily pipeline and only covered watchlist.
System fix:
- premarket_full_review.py: new Step 1.5 runs batch_reassess --type
holding --today every trading day 08:10 (force-refresh today's analysis,
timeout 3600s, result in summary.json)
- batch_reassess.py:
- coverage: --type holding|watchlist|all (was watchlist-only)
- staleness: analysis >20h stale gets refreshed (was: skip if any
analysis exists = forever stale)
- --today flag: force re-analyze if not reassessed since 04:00 today
- cash/total read live from portfolio_summary (was hardcoded 321271/
952879 from weeks ago)
- HK stock prefix fix (5-digit codes -> hk, was sending sz00700)
- watchlist_12d_backfill.py: wrapper for hermes cron (no args support)
- cron job '批量补全九维分析-一次性' -> '自选12维分析补全-每日午间'
(daily 12:30 weekdays, covers 109 watchlist stocks missing analysis)
Verified: 300308 got 1920-char 12-dim analysis written to DB at 08:41,
signal=观望, stop/take-profit updated.
Pre-open error sweep (all verified by manual run):
- promote_candidates.py: INSERT OR IGNORE + only newly-added stocks count
toward promotion/XMPP (was crashing on first duplicate, never finishing;
now completes 40s, promoted 74 with correct skip marking)
- candidate_filter.py: PRAGMA busy_timeout=30s (was dying on transient
'database is locked' under concurrent cron writes; now completes 9.6s)
- price_monitor.py: verified completes 1m55s (< 120s cron timeout) with
working LLM reassess via key6
- macro_context_collector.py / divergence_detector.py: previously
'Blocked' by symlink check, now run fine (8.7s / 1m55s)
- memory_guardian.py: completes 1m55s with key6
- preflight sync diff: historical, files now identical
Production proof: 大脑任务执行 (was 429 every 10min) now status=ok at 01:21
Two regressions fixed:
1. RESTORED: original health monitoring (功能树/全部Cron/数据实体/数据流)
was an iframe to /mofin_health.html — refactor replaced it with a
minimal services-only panel and lost all of it. mofin_health.json
(86KB, fresh) was still being generated the whole time.
- 健康 tab: XMPP panel (native) + restored iframe below
2. ADDED: 开发原则 parent tab replicating AgentsMeeting structure:
- G 规范: /api/spec renders docs/dev-spec.md + git history
- K 测试: /api/tests renders agents_health_check report as pass/fail
- F 健康: iframe to /mofin_health.html + link to 健康 tab
- H 需求: /api/prd (placeholder — prd.md not yet created)
- mdRender() ported from AgentsMeeting dashboard
Every /api/xmpp/health fetch ran a real LLM call (22k token system prompt
each). Dashboard refreshes every 10s -> thousands of paid LLM calls/day,
plus 38s latency hanging the health tab on '加载中...'.
LLM health is now derived from the gateway's own agent.log (zero cost,
more accurate than synthetic ping — real traffic results):
- last 'API call #N latency=Xs' -> ok
- last 'API call failed ... HTTP 429...' -> error with summary
- health() runtime 38s -> 0.1s; endpoint 38s -> 0.097s
User requirement: health tab should show recent XMPP conversations.
- bot hooks log_xmpp on inbound (on_msg) and outbound (_deliver_loop)
so real chats land in xmpp_messages.jsonl (was: only cron/scanner)
- index.html health tab: new '最近对话' panel (last 10 msgs, dir arrow,
preview, status, time); refreshHealth updates it incrementally
- last_error now shows age and resolved state: once a successful
outbound happens after an error, it's shown gray as '已恢复'
instead of alarming red forever; unresolved errors still red
- health() status no longer degraded by errors that were later
resolved by successful outbound
Root cause of 'no response': auto_heal restart_zhiwei_bot fired whenever
inbound>0 + outbound=0 + errors>0 — which is exactly the normal state while
the bot waits for a slow LLM call (agent tool-use turns take 1-10 min).
Restart killed the in-flight LLM call -> user never got reply -> next cron
cycle saw same state -> restart again. Death loop (fired 22:30,22:35,22:40,
22:45,22:50,23:35).
Fixes:
- CALL_HERMES_TIMEOUT 180s -> 600s (agent tool calls need minutes)
- health(): parse last_inbound/outbound timestamps from journal
- auto_heal: restart bot ONLY if last inbound >600s old with no outbound
since (truly stuck), plus 600s bot-restart cooldown
- slow-but-normal state now logs bot_busy_not_stuck instead of killing
D fix: screenshots were silently dropped (empty body + OOB url).
- register xep_0066, capture msg['oob']['url'] when body empty
- also handle body-as-URL messages (some clients put URL in body)
- download from upload.yoin.fun, OCR via SenseNova (sensenova-6.7-flash-lite)
- inject OCR text as context into LLM call
- config at /home/hmo/.config/mofin/ocr_config.json (outside repo)
- replaces dead node122 GLM-OCR path (host unreachable)
A+B fix: ACK_DELAY 15s -> 120s. 15s fired on every normal LLM
cold-start (20-100s), now only signals genuine hangs.
zhiwei must stay on ocg-key6 (deepseek-v4-flash) per user decision.
key7 is kimi — different model provider, incompatible model name.
If best_key returns key7, auto_heal logs skip_switch instead of switching.
hmo