hmo/MoFin
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
417 Commits 4 Branches 0 Tags
57377e9dd39b84e3a17ee922a0f8f7f6b71c6399
Commit Graph
50 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
hmo 57377e9dd3 feat(auto_heal): multi-profile key switching + fix two error classes 
Error investigation (from restored monitoring) found 2 root causes:

1. 'Blocked: script path resolves outside scripts dir' (5+ cron jobs):
   Jul 17 symlink refactor replaced real scripts with symlinks; the hermes
   cron scheduler's security check (Path.resolve + relative_to) rejects
   symlink escape. ALL no_agent script jobs blocked since Jul 17 23:12.
   FIX: converted 102 symlinks to hardlinks (same inode, resolve() stays
   inside scripts_dir, single-source still works). Permanent structural fix.

2. HTTP 429 on default profile (知识研究/梦境循环/wiki-self-growth/
   evolution-pulse/大脑任务执行): default gateway used ocg-key1 (weekly
   100%). FIX: switched default profile to ocg-key6 + added missing
   provider block. LLM verified working (3.1s).

auto_heal extended to actually cover these automatically next time:
- PROFILES registry: zhiwei (8643, system svc) + default (8642, user svc)
- current_provider/switch_key/_scan_agent_log parameterized by profile
- health() now reports llm_provider_default (agent.log scan)
- auto_heal: per-profile 429 detection -> best_key -> switch_key(profile)
- _ensure_provider_block: injects missing provider credentials from
  zhiwei config (single source of truth) into target config
 2026-07-20 01:12:38 +08:00
hmo a05c118cbc feat(dashboard): restore lost monitoring + add 开发原则 tab (AgentsMeeting parity) 
Two regressions fixed:

1. RESTORED: original health monitoring (功能树/全部Cron/数据实体/数据流)
   was an iframe to /mofin_health.html — refactor replaced it with a
   minimal services-only panel and lost all of it. mofin_health.json
   (86KB, fresh) was still being generated the whole time.
   - 健康 tab: XMPP panel (native) + restored iframe below

2. ADDED: 开发原则 parent tab replicating AgentsMeeting structure:
   - G 规范: /api/spec renders docs/dev-spec.md + git history
   - K 测试: /api/tests renders agents_health_check report as pass/fail
   - F 健康: iframe to /mofin_health.html + link to 健康 tab
   - H 需求: /api/prd (placeholder — prd.md not yet created)
   - mdRender() ported from AgentsMeeting dashboard
 2026-07-20 00:40:32 +08:00
hmo 2de527b883 fix(health): drop live LLM ping — scan agent.log instead (38s -> 0.1s) 
Every /api/xmpp/health fetch ran a real LLM call (22k token system prompt
each). Dashboard refreshes every 10s -> thousands of paid LLM calls/day,
plus 38s latency hanging the health tab on '加载中...'.

LLM health is now derived from the gateway's own agent.log (zero cost,
more accurate than synthetic ping — real traffic results):
- last 'API call #N latency=Xs' -> ok
- last 'API call failed ... HTTP 429...' -> error with summary
- health() runtime 38s -> 0.1s; endpoint 38s -> 0.097s
 2026-07-20 00:21:49 +08:00
hmo 058c42ce27 feat(health): recent XMPP conversation log panel + stale error fix 
User requirement: health tab should show recent XMPP conversations.
- bot hooks log_xmpp on inbound (on_msg) and outbound (_deliver_loop)
  so real chats land in xmpp_messages.jsonl (was: only cron/scanner)
- index.html health tab: new '最近对话' panel (last 10 msgs, dir arrow,
  preview, status, time); refreshHealth updates it incrementally
- last_error now shows age and resolved state: once a successful
  outbound happens after an error, it's shown gray as '已恢复'
  instead of alarming red forever; unresolved errors still red
- health() status no longer degraded by errors that were later
  resolved by successful outbound
 2026-07-20 00:07:28 +08:00
hmo cb334ddd54 fix(bot+auto_heal): stop auto_heal from killing bot during slow LLM calls 
Root cause of 'no response': auto_heal restart_zhiwei_bot fired whenever
inbound>0 + outbound=0 + errors>0 — which is exactly the normal state while
the bot waits for a slow LLM call (agent tool-use turns take 1-10 min).
Restart killed the in-flight LLM call -> user never got reply -> next cron
cycle saw same state -> restart again. Death loop (fired 22:30,22:35,22:40,
22:45,22:50,23:35).

Fixes:
- CALL_HERMES_TIMEOUT 180s -> 600s (agent tool calls need minutes)
- health(): parse last_inbound/outbound timestamps from journal
- auto_heal: restart bot ONLY if last inbound >600s old with no outbound
  since (truly stuck), plus 600s bot-restart cooldown
- slow-but-normal state now logs bot_busy_not_stuck instead of killing
 2026-07-19 23:43:28 +08:00
hmo bb1529909b feat(bot): screenshot OCR pipeline + ack delay 120s 
D fix: screenshots were silently dropped (empty body + OOB url).
- register xep_0066, capture msg['oob']['url'] when body empty
- also handle body-as-URL messages (some clients put URL in body)
- download from upload.yoin.fun, OCR via SenseNova (sensenova-6.7-flash-lite)
- inject OCR text as context into LLM call
- config at /home/hmo/.config/mofin/ocr_config.json (outside repo)
- replaces dead node122 GLM-OCR path (host unreachable)

A+B fix: ACK_DELAY 15s -> 120s. 15s fired on every normal LLM
cold-start (20-100s), now only signals genuine hangs.
 2026-07-19 23:23:46 +08:00
hmo e0f47f9349 fix(xmpp_monitor): exclude key7(kimi) from auto-heal switch candidates 
zhiwei must stay on ocg-key6 (deepseek-v4-flash) per user decision.
key7 is kimi — different model provider, incompatible model name.
If best_key returns key7, auto_heal logs skip_switch instead of switching.
 2026-07-19 22:09:15 +08:00
hmo 9509b80d8b feat(xmpp_monitor): dynamic key switching + coalesce auto-heal 
- xmpp_logger.py:
  - Add switch_key(key_id): switch Hermes model.provider via sed + systemctl restart
  - Add current_provider() reading model: block correctly (not just first '  provider:' line)
  - Add KEY_TO_PROVIDER mapping (AgentsMeeting key_id -> Hermes provider name)
  - Add RESTART_COOLDOWN_FILE/SEC = 180s to prevent restart loops
  - auto_heal(): detect HTTP 429 / Weekly usage limit -> call best_key() -> switch_key()
  - auto_heal(): detect timeout/error -> async systemctl restart (Popen, not blocking)
  - _verify_llm(): bump timeout 25s -> 90s (cold-start gateway takes 20-40s)
  - health(): urlopen timeout 8s -> 90s (match verify window)
  - Use sudo NOPASSWD (hmo ALL=(ALL) NOPASSWD: ALL already configured)

- agents_health_check.py:
  - Replace inline pkill+Popen restart logic (caused multiple instances) with systemctl
  - Add RESTART_COOLDOWN_FILE state to skip restart within 3 min of last
  - Call xmpp_logger.auto_heal() at end of every cron cycle
  - Both inline restart and auto_heal restarts share cooldown file

- scripts/key_status.py: Reports weekly/monthly/rolling status of all 6 OCG keys
- scripts/test_llm.py: 90s timeout test (was 15s, gateway cold-start >= 30s)
- scripts/test_production.py: smoke test on /home/hmo/MoFin/ (hardlinked to web-dashboard)

Fixes:
- Old assumption 'Cloudflare blocks Python User-Agent' was WRONG.
  True cause was HTTP 429 Weekly usage limit on key5 (12hr reset window).
  Hermes silently ignored providers.X.headers config keys; only model.default_headers works.
  Config already has model.default_headers: User-Agent: curl/8.5.0 (defense in depth).
- Multiple gateway instances were caused by 3 competing systemd units
  (hermes-gateway@.service template + hermes-gateway-zhiwei.service named).
  Masked the template unit @position-analyst and @zhiwei so only the named one wins.
 2026-07-19 20:12:47 +08:00
hmo 6d66cc6cf2 feat: auto-heal — gateway down auto-restart via cron every 5min 2026-07-19 18:22:45 +08:00
hmo 04308facde feat: XMPP health auto-collection every 5min via cron pipeline 2026-07-19 15:17:08 +08:00
hmo bff246313b feat: auto-heal checks key availability before restarting gateway 2026-07-19 14:46:46 +08:00
hmo f791d8dee7 feat: API key availability monitoring — auto-select best key from AgentsMeeting 2026-07-19 14:41:18 +08:00
hmo 6398f595b2 feat: auto-heal pipeline — detect+restart gateway/bot/ejabberd on failure 2026-07-19 14:18:45 +08:00
hmo 0802046d5c feat: Hermes Gateway + LLM provider monitoring, root cause visible in health tab 2026-07-19 13:25:37 +08:00
hmo 50ae4e1cc9 feat: XMPP bot journal monitoring — detect inbound/outbound/errors, smooth health refresh 2026-07-19 13:15:37 +08:00
hmo 9f3d1d8106 refactor: remove redundant dashboard tab, health tab now unified 2026-07-19 12:57:08 +08:00
hmo b41bed7959 feat: XMPP observability — logger, monitor endpoints, clean dead xiaoguo refs 2026-07-19 12:47:15 +08:00
hmo fec2bc9106 refactor: integrate spec system into all 9 tabs — ?§ buttons, native health monitoring 2026-07-19 11:22:56 +08:00
hmo 49ccd7467d docs: update module list — 10 specs completed 2026-07-19 11:18:22 +08:00
hmo 06e59f3d9c feat: add prompts and reports module specs (10 specs total) 2026-07-19 11:18:00 +08:00
hmo 8356946de3 refactor: health tab now uses new dashboard, added dashboard tab 2026-07-19 11:11:59 +08:00
hmo e45c3bb01f feat: integrate dashboard tab into main index.html navbar 2026-07-19 11:11:18 +08:00
hmo b08bfa5d03 refactor: integrate dashboard into server.py :8899, remove standalone dashboard 2026-07-19 11:06:50 +08:00
hmo 782a914a3c fix: dashboard port 5804→5807 (avoid conflicts with wechat_webhook and zhiwei bot) 2026-07-19 10:56:42 +08:00
hmo 747fdfe467 feat: introduce spec system + dashboard + health pipeline (AgentsMeeting template) 2026-07-19 10:50:34 +08:00
hmo bfeb3cdfb1 docs: DSA Web + AlphaSift + 选股链路修复 + 小果EasyTier 2026-06-30 02:35:40 +08:00
hmo d4cfc5c931 fix: market_screener call_xiaoguo direct to node122 instead of Hermes gateway 2026-06-30 02:30:32 +08:00
hmo f4b2467ae9 fix: replace hardcoded 192.168.1.122 with node122 (hostname resolves via /etc/hosts to LAN or EasyTier) 
- mo_config.py: xiaoguo_host=node122, xiaoguo_api_url property
- market_screener.py, xiaoguo_scanner.py, xiaoguo_news_processor.py: use mo_config or node122 fallback
- scripts/intraday_health_check.py, scripts/ocr_client.py: node122
- EasyTier connects at 10.144.144.2 when off-LAN
 2026-06-30 02:27:52 +08:00
hmo efaa20d775 feat: AlphaSift disabled by default (ALPHASIFT_ENABLED=false), add --enable flag 2026-06-30 02:21:23 +08:00
hmo c38787aea3 fix: sort candidates by score across strategies before limiting MAX_ADD 2026-06-30 02:15:44 +08:00
hmo 992b283d3f feat: multi-strategy parallel screening (balanced_alpha+dual_low+quality_value) 2026-06-30 02:08:48 +08:00
hmo 790b0f9acc feat: mo_alphasift_bridge v2 — async mode + source tracking (date/strategy/notes) 2026-06-30 01:57:01 +08:00
hmo 897bc54bab feat: mo_alphasift_bridge.py — AlphaSift screening → MoFin watchlist auto-bridge 2026-06-30 01:51:48 +08:00
hmo 8fd134a063 fix: mo_bridge cache path (reports/ + data/market_review/) + mo_dsa_opinion.py standalone DSA strategy reference script 2026-06-30 01:31:42 +08:00
hmo 18081c05a4 fix: mo_bridge news fallback to akshare + market review cache-only in cron 2026-06-30 01:24:09 +08:00
hmo ab23dfd234 feat: DSA full integration — mo_bridge v2 + strategy_lifecycle injection 
mo_bridge.py (rewrite):
- get_stock_news(): DSA SearchService 7 engines → MoFin analysis context
- get_market_review(): DSA run_market_review() with 24h cache
- get_stock_analysis(): DSA AgentExecutor.run() with 15 strategies
- enrich_analysis_context(): one-call context injection

strategy_lifecycle.py:
- reassess_with_context() now injects DSA market + news context
- Auto-detects HK vs A-share region for market review
- Graceful fallback if DSA unavailable
 2026-06-30 01:17:23 +08:00
hmo 8bcba05e7d docs: add DSA web server info to CHANGELOG 2026-06-30 00:50:56 +08:00
hmo 5264fd5241 docs: MoFin architecture reform CHANGELOG 2026-06-30 00:36:08 +08:00
hmo d9b48ea5f0 fix: Eastmoney individual query + remove /100 price bug 2026-06-30 00:29:37 +08:00
hmo 8710dfe366 feat: HK realtime via Eastmoney push2 API — no more 15-min delay 
price_monitor.py now splits A-share (Tencent realtime) and HK (Eastmoney realtime):
- fetch_all_prices() → A-shares only via qt.gtimg.cn
- fetch_hk_eastmoney() → HK stocks via push2.eastmoney.com (real-time, free)
- _fetch_hk_tencent_fallback() → fallback if Eastmoney fails (15-min delay)
- Eastmoney API: market code 116, fields f43(price)/f170(chg%)/f60(prev_close)
 2026-06-30 00:25:56 +08:00
hmo 89153832b4 fix: DSA path auto-detect — check server path /home/hmo/daily-stock-analysis first 2026-06-29 23:47:59 +08:00
hmo 6abc2e45b0 refactor: phase 0-2 MoFin architecture reform — single source of truth 
Phase 0 (止血):
- mo_models.py: unified calc_total_assets(), is_hk_stock(), get_hk_rate() — single source of truth
- Fixed 3 files missing frozen_cash: holdings_reconciliation, server, import_holding_xls
- Fixed stale_push_wlin: unified is_hk_stock detection, removed hardcoded 0.866
- Fixed price_monitor: consolidated 2 duplicate total_assets blocks into mo_models calls
- Fixed stock_scorer: replaced broken len()<=5 is_hk_stock heuristic
- Fixed strategy_lifecycle: replaced non-existent currency_utils import with mo_models

Phase 1 (DSA adapter):
- mo_provider.py: wraps DSA DataFetcherManager (16 fetchers, auto-fallback)
  - TDX relay as primary, DSA as backup for realtime/kline/news/fundamentals

Phase 2 (Integration):
- mo_bridge.py: injects DSA market review + news context into MoFin analysis prompts
- Graceful degradation if DSA not installed

Infrastructure:
- mo_config.py: centralized Config singleton replacing scattered hardcoded paths
- All 11 changed files pass python compile check

Impact: total_assets now computed in ONE place (mo_models).
        is_hk_stock now ONE implementation (no more false negatives).
        HK rate now ONE source (hk_rate API → cache → 0.87 fallback).
        No more hardcoded 0.866/0.8664/0.8700 divergence.
 2026-06-29 23:25:54 +08:00
hmo c80f814632 fix: K线迁移 + 清理重复文件 
migrate_all.py:
- 新增 migrate_klines(): multi_tf_cache.json → stock_daily/weekly/monthly + stock_fundamentals
- 迁移量: daily=5520, weekly=1104, monthly=552 (46只股票)
- 验证表新增 stock_daily/weekly/monthly/fundamentals

清理:
- 删除 web/static/ (与根目录 static/ 重复，server.py 使用 static/)
 2026-06-20 20:46:25 +08:00
hmo d820ff2ad8 docs: 更新协作文档 — 记录数据层重构完成状态 2026-06-20 20:08:09 +08:00
hmo 6182ff081d docs: 统一数据库架构实施文档 
覆盖: 表结构设计(13张表)、数据流架构、核心模块说明、
部署指南、架构决策记录(ADR)、已知限制
 2026-06-20 17:54:10 +08:00
hmo 25f8c6ec67 refactor: 消费者切 SQLite 优先读取 
切换策略: SQLite 优先 → 失败回退 JSON

price_events (100%覆盖):
- strategy_feedback.py: run() 优先 query_price_events()
- system_health_check.py: 优先 query_price_events() + query_price_events_by_date()

stock_sector_map (100%覆盖):
- strategy_lifecycle.py: load_stock_sector_map() 优先 stock_sectors 表

market.json (85%覆盖):
- strategy_lifecycle.py: load_market_context() 优先 query_latest_market()
- market_insight.py: generate() 优先 query_latest_market()

portfolio.json + watchlist.json (70%覆盖):
- strategy_lifecycle.py: regenerate_all() 优先 query_holdings() + query_watchlist()
- server.py: /api/portfolio, /api/watchlist, /api/overview, /api/market 优先 SQLite

所有改动保留 JSON 回退路径，SQLite 不可用时自动降级
 2026-06-20 17:50:15 +08:00
hmo 1610f184a0 feat: 补全 SQLite 表结构 + 查询函数 + 迁移覆盖 
mofin_db.py 新增:
- 4 张表: portfolio_summary, advice_timeline, accuracy_stats, strategy_feedback
- 18 个查询函数: query_holdings, query_watchlist, query_strategies,
  query_advice_timeline, query_candidates, query_candidate_scores,
  query_price_events, query_price_events_by_date, query_stock_sectors,
  query_sector_stocks, query_accuracy_stats, query_strategy_feedback,
  query_strategy_evaluations, query_latest_market, query_holding_by_code,
  query_portfolio_summary

migrate_all.py 新增:
- 4 个迁移函数: migrate_portfolio_summary, migrate_advice_timeline,
  migrate_accuracy_stats, migrate_strategy_feedback
- 迁移量: portfolio_summary(1), advice_timeline(2547),
  accuracy_stats(1), strategy_feedback(37)

现在 13 张表全部覆盖，JSON→SQLite 数据完整迁移
 2026-06-20 16:59:24 +08:00
hmo 0650673038 feat: migrate_all.py — 完整数据迁移脚本 (JSON → SQLite) 
一次性迁移全部生产数据到 mofin.db:
- stock_profiles.json → stocks (55只)
- portfolio.json → holdings (21只) + holding_strategies (21条)
- watchlist.json → watchlist_stocks (1只) + holding_strategies (1条)
- decisions.json → holding_strategies (316条, 含changelog历史)
- candidate_pool.json → candidates (10只) + candidate_score_history (21条)
- price_events.json → price_events (193条)
- evaluation.json → strategy_evaluations (36条)
- stock_sector_map.json → stock_sectors (62条)

特性:
- 自动从所有JSON源收集股票代码 (collect_all_stocks)
- 代码格式归一化 (_normalize_code: 整数→补零字符串)
- 迁移期间关闭外键约束 (兼容旧数据格式不一致)
- 幂等可重跑 (INSERT OR REPLACE/IGNORE)
- JSON文件不修改，可安全重复执行

替换旧的 migrate_sectors.py (功能已合并)
 2026-06-20 16:40:39 +08:00
hmo 0924cf3124 refactor: 数据层重构 — 统一 SQLite 访问层 + 多脚本双写 
新建 mofin_db.py 共享数据库模块:
- get_conn() 统一连接管理 (WAL + Row factory + 外键)
- init_all_tables() 幂等建表 (12张表: market/sector/stock/kline/fundamentals/sectors/holdings/strategies/watchlist/candidates/score_history/events/evaluations)
- write_market_snapshot() 市场快照双写
- write_klines() K线数据双写 (stocks + daily/weekly/monthly + fundamentals)
- write_price_event() 价格事件双写
- migrate_stock_sectors() 一次性迁移 stock_sector_map.json
- query_*() 通用查询函数 (sector_trend/top_inflow/consecutive_inflow/market_mood/db_stats)

重构现有脚本:
- market_watch.py: 删除内联 DB 代码，改用 mofin_db
- multi_timeframe.py: _save_local_history() 加 SQLite 双写
- price_monitor.py: record_event() 加 SQLite 双写
- mofin_query.py: 改用 mofin_db 查询函数

新增:
- migrate_sectors.py: 一次性迁移脚本

清理:
- get_realtime_prices.py: 死代码 (只读 portfolio.json，不调API)
 2026-06-20 16:26:17 +08:00
hmo a293119a31 feat: 阶段1 — market_watch 双写 SQLite + 查询工具 
- market_watch.py: 新增 init_db() 建表 + write_snapshot() 双写 SQLite
  - market_snapshots: 每次采集的元信息（时间、来源、涨跌比、情绪）
  - sector_snapshots: 每个板块的涨跌幅、资金流向、领涨股等
  - JSON 写入保留不变，SQLite 写入失败不影响 JSON 管道
- mofin_query.py: 通用查询工具
  - 板块趋势查询：「半导体最近5次采集的涨跌幅」
  - 资金流向排行：「净流入最多的5个板块」
  - 连续净流入检测：「最近3天连续净流入的板块」
  - 市场情绪趋势 + 数据库概览
  - 支持直接 SQL 查询
 2026-06-20 12:51:02 +08:00