Root cause: holding_strategies.shares field got set to literal string
'write_holding_strategy' for 6 records, causing TypeError at
price_monitor.py line 611 ('>' not supported between str and int).
Fixes:
1. price_monitor.py: Replace set comprehension with safe loop that
checks isinstance before comparison. Non-numeric shares treated
as holdings for safety.
2. mofin_db.py write_holding_strategy: Add type guard that resets
non-numeric shares to 0 with warning.
3. Data fix: Updated 5 corrupted holding_strategies records from
holdings table (300035/300308/300750/518880/00700).
Set 002594 (watchlist) shares=0.
Root causes of the screenshot 404 incident:
1. RACE: client uploads image AND sends message concurrently; bot received
the message before the upload finished writing, so its GET hit a 404
error page (<100B treated as failure). FIX: _download_image now retries
3x with 2s backoff.
2. Zhiwei mentioned tesseract/小果 because the failure text never told her
the pipeline IS SenseNova. FIX: failure messages now name SenseNova
explicitly and ask for resend.
3. log_xmpp never worked for the bot: sys.path used relative '../..' from
a symlinked __file__ which resolved to '/' instead of MoFin root. This
is why the '最近对话' panel never had bot chat data (only cron script
entries). FIX: absolute path per red line #7. Verified: test message
now lands in xmpp_messages.jsonl.
4. My PowerShell -replace corrupted the file encoding (UnicodeDecodeError
crash loop on restart). Restored from git HEAD and re-applied edits with
the edit tool. Lesson: never use PowerShell string replace on UTF-8
source files with Chinese content.
5. functional_health: new sense_ocr module (OCR config presence +
SenseNova API TCP reachability), no token cost.
Gateway看门狗-知微 was erroring (exit -15): its check_session_health did a
live LLM ping with 25s timeout. Cold-start LLM latency is 20-100s so the
ping always timed out -> false '不健康' verdict -> false gateway restart
-> and each 10-min run burned 22k tokens.
Now uses xmpp_logger._scan_agent_log (zero cost, reads real call results):
- ok if last real call succeeded
- unhealthy only if last call explicitly failed
- idle (no recent calls) counts as healthy
Verified: watchdog job now status=ok.
Also: triggered all 6 weekend 'Blocked' jobs via hermes cron run — all
now status=ok, proving the hardlink fix holds.
User insight: hardlink breakage only happens at deploy time (scp file
replacement / git checkout-merge), so detection must be welded INTO the
deploy pipeline, not left to daily audit.
Three automatic layers, no reliance on discipline:
1. systemd path watcher (profile-scripts-sync.path): watches
deploy/profile-scripts/ directory, auto-fires sync_profile_scripts.sh
on any change. Verified: fires within 4s of file replacement, logs to
gateway/logs/link_sync.log (runs as hmo user)
2. git hooks (.git/hooks/post-merge + post-checkout on 246 repo):
auto re-link after git operations
3. Manual fallback: sync_profile_scripts.sh (now self-logging)
dev-spec red line #6 updated: SSOT rule now documents the three layers
and states breakage only happens at deploy time.
Root cause analysis of the 2026-07-20 redundancy incident:
1. No single-source-of-truth rule -> same file legitimately lived in 4+
locations, diverging silently
2. Relative path resolution (Path(__file__).parent/'data') -> each
hardlinked copy of mofin_db.py pointed to a DIFFERENT database
3. 'Backup habit' left .bak/legacy files in production dirs, which
monitoring then scanned and reported as false alarms
4. Half-done migrations: DB tables created but old JSON writers/readers
stayed (price_events), old files stayed
5. Dead modules never got buried: xiaoguo 'dead' but bot ran 8 days
as root eating 2.5GB
6. Monitoring checked 'does it exist' not 'is it alive' -> stale file
mtime reported as 'pipeline stalled 14 days' (false alarm)
7. No 'system hygiene' as a check category at all
Prevention implemented:
- dev-spec.md v2.0: 五条红线 -> 十条红线
#6 single source of truth (hardlink only, no independent copies)
#7 absolute data paths only (no __file__-relative data resolution)
#8 no backups/legacy in production data dirs (archive immediately)
#9 dead module burial checklist (6 mandatory steps)
#10 monitor liveness (DB table freshness) not existence
- File Location Constitution: canonical location per content type
- NEW system_hygiene_audit.py: weekly Monday 07:30 cron checking
diverged copies / broken hardlinks / zombie processes / orphan data
files / dead cron scripts / DB freshness -> hygiene_report.json + XMPP
- specs/hygiene.json: module spec per red line #1
- Verified: audit found 5 real issues on first run, all fixed, re-run clean
User directive: no JSON, retire it fully, fix all related code.
Changes:
- price_monitor.py: record_event writes DB only; removed EVENTS_PATH/
load_events/save_events entirely
- strategy_feedback.py: price events read from DB only (removed JSON fallback)
- system_health_check.py: removed price_events.json from file-check list,
DB-only event stats (was showing 0/0 due to wrong-DB resolution)
- mo_config.py: removed dead price_events_path property (no callers)
- mofin_health.py: price_events freshness reads DB table (authoritative now)
- mofin_db.py: DATA_DIR/DB_PATH now ABSOLUTE (/home/hmo/MoFin/data) —
was relative __file__.parent, so each hardlinked copy of mofin_db.py
resolved to a DIFFERENT database (canonical vs web-dashboard vs
profile-local third DB with 0 rows of everything except market_snapshots).
This fragmentation was the real cause of health checks reading empty tables.
- Unified all 4 mofin_db copies (root/scripts/deploy/profile) via hardlink
- price_events.json archived to trashbox (fully backfilled: 6353 rows in DB)
Verified:
- record_event lands in DB only, JSON not recreated
- system_health_check: 历史事件 6353 / 今日事件 2965 (was 0/0)
- strategy_feedback + price_monitor full runs clean
hmo
知微