知微 (MoFin)
aa0f740381
完整数据采集+分析管道： - market_watch.py：90行业板块采集（同花顺/东方财富） - 市场精选推荐 cron：全市场分析+候选池+星级推荐 - price_monitor.py：持仓/自选高频价格监控 - refresh_mtf_cache.py：多周期K线缓存 - 策略评估/知识萃取管道 文档：docs/ 含完整需求+架构设计 注意：尚未配置 git remote，笑笑接手后自行配置
396 lines
14 KiB
Markdown
396 lines
14 KiB
Markdown
# MoFin 统一数据库架构设计
|
||
|
||
## 目标
|
||
|
||
将 MoFin 系统目前散落在各 JSON 文件中的数据统一纳入 SQLite 数据库，实现：
|
||
- 数据关系化（板块 ↔ 个股 ↔ 持仓 ↔ 策略，SQL join 直连）
|
||
- 历史可追溯（板块快照、价格序列、评分变更全部时序化）
|
||
- 操作 no_agent 化（纯 SQL 即可完成大部分查询和分析）
|
||
- 增量迁移（现有 JSON 不动，新数据双写，逐步切换）
|
||
|
||
---
|
||
|
||
## 一、数据库概览
|
||
|
||
```
|
||
数据库文件: /home/hmo/web-dashboard/data/mofin.db
|
||
引擎: SQLite 3
|
||
```
|
||
|
||
### 逻辑分组
|
||
|
||
| 组 | 表 | 说明 |
|
||
|----|-----|------|
|
||
| 市场 | market_snapshots, sector_snapshots | 全市场板块快照（每30分） |
|
||
| 个股 | stocks, stock_daily, stock_weekly, stock_monthly, stock_fundamentals | 个股K线+基本面 |
|
||
| 板块成分 | stock_sectors | 个股所属板块映射 |
|
||
| 持仓 | holdings, holding_strategies | 持仓股+策略参数 |
|
||
| 自选 | watchlist_stocks | 自选股+买入区 |
|
||
| 候选池 | candidates, candidate_score_history | 小果+知微潜力股推荐 |
|
||
| 事件 | price_events, zone_breaches | 价格预警、区间触发 |
|
||
| 策略评估 | strategy_evaluations | 策略重评记录 |
|
||
|
||
---
|
||
|
||
## 二、表结构详述
|
||
|
||
### 2.1 市场/板块快照
|
||
|
||
核心时间序列表。每次 market_watch 执行（每30分）写一条 snapshot，再把每个板块的数据写入 sector_snapshots。
|
||
|
||
```sql
|
||
-- 每次采集的元信息
|
||
CREATE TABLE market_snapshots (
|
||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||
timestamp TEXT NOT NULL, -- '2026-06-19 15:30'
|
||
source TEXT NOT NULL DEFAULT 'ths', -- ths / eastmoney
|
||
up_ratio REAL, -- 上涨板块占比(%)
|
||
mood TEXT, -- bullish / neutral / bearish
|
||
created_at TEXT DEFAULT (datetime('now','localtime'))
|
||
);
|
||
|
||
CREATE INDEX idx_snapshots_time ON market_snapshots(timestamp);
|
||
|
||
-- 每个板块在每次采集中的快照数据
|
||
CREATE TABLE sector_snapshots (
|
||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||
snapshot_id INTEGER NOT NULL REFERENCES market_snapshots(id),
|
||
name TEXT NOT NULL, -- 板块名称（同花顺行业名）
|
||
change_pct REAL, -- 涨跌幅(%)
|
||
up_count INTEGER, -- 上涨家数
|
||
down_count INTEGER, -- 下跌家数
|
||
net_inflow REAL, -- 资金净流入(亿)
|
||
lead_stock TEXT, -- 领涨股名
|
||
lead_stock_change REAL, -- 领涨股涨跌幅(%)
|
||
volume REAL, -- 成交量(万手)
|
||
turnover REAL -- 成交额(亿)
|
||
);
|
||
|
||
CREATE INDEX idx_sector_name ON sector_snapshots(name);
|
||
CREATE INDEX idx_sector_snapshot ON sector_snapshots(snapshot_id);
|
||
CREATE INDEX idx_sector_name_time ON sector_snapshots(name, snapshot_id);
|
||
```
|
||
|
||
**典型查询：**
|
||
```sql
|
||
-- 半导体板块最近5天的涨跌幅趋势
|
||
SELECT s.timestamp, ss.change_pct, ss.net_inflow
|
||
FROM sector_snapshots ss
|
||
JOIN market_snapshots s ON ss.snapshot_id = s.id
|
||
WHERE ss.name = '半导体'
|
||
ORDER BY s.timestamp DESC LIMIT 10;
|
||
|
||
-- 资金连续3天净流入的板块（最近3次采集）
|
||
SELECT name, COUNT(*) as times, AVG(net_inflow) as avg_inflow
|
||
FROM sector_snapshots ss
|
||
JOIN market_snapshots s ON ss.snapshot_id = s.id
|
||
WHERE s.timestamp >= date('now', '-3 days')
|
||
AND net_inflow > 0
|
||
GROUP BY name
|
||
HAVING COUNT(*) >= 3
|
||
ORDER BY avg_inflow DESC;
|
||
```
|
||
|
||
---
|
||
|
||
### 2.2 个股K线 + 基本面
|
||
|
||
替代现有的 multi_tf_cache.json 和 data/stocks/*.json。
|
||
|
||
```sql
|
||
CREATE TABLE stocks (
|
||
code TEXT PRIMARY KEY, -- '688981'
|
||
name TEXT NOT NULL, -- '中芯国际'
|
||
exchange TEXT DEFAULT 'SH', -- SH / SZ / HK
|
||
type TEXT DEFAULT 'A', -- A / H
|
||
updated_at TEXT
|
||
);
|
||
|
||
-- 日线数据（时间序列）
|
||
CREATE TABLE stock_daily (
|
||
code TEXT NOT NULL REFERENCES stocks(code),
|
||
date TEXT NOT NULL, -- '2026-06-19'
|
||
open REAL,
|
||
close REAL,
|
||
high REAL,
|
||
low REAL,
|
||
volume REAL, -- 成交量
|
||
amount REAL, -- 成交额
|
||
PRIMARY KEY (code, date)
|
||
);
|
||
|
||
-- 周线数据
|
||
CREATE TABLE stock_weekly (
|
||
code TEXT NOT NULL REFERENCES stocks(code),
|
||
date TEXT NOT NULL, -- 周结束日期
|
||
open REAL, close REAL, high REAL, low REAL,
|
||
volume REAL,
|
||
PRIMARY KEY (code, date)
|
||
);
|
||
|
||
-- 月线数据
|
||
CREATE TABLE stock_monthly (
|
||
code TEXT NOT NULL REFERENCES stocks(code),
|
||
date TEXT NOT NULL, -- 月结束日期
|
||
open REAL, close REAL, high REAL, low REAL,
|
||
volume REAL,
|
||
PRIMARY KEY (code, date)
|
||
);
|
||
|
||
-- 基本面（最新，非时序）
|
||
CREATE TABLE stock_fundamentals (
|
||
code TEXT PRIMARY KEY REFERENCES stocks(code),
|
||
pe REAL, -- 市盈率
|
||
pb REAL, -- 市净率
|
||
eps REAL, -- 每股收益
|
||
mcap_total REAL, -- 总市值
|
||
mcap_flow REAL, -- 流通市值
|
||
updated_at TEXT
|
||
);
|
||
```
|
||
|
||
---
|
||
|
||
### 2.3 板块成分股映射
|
||
|
||
个股和板块的关系表。目前股票不知道自己在哪个板块，板块不知道有哪些成分股。这使 join 分析无法进行。
|
||
|
||
```sql
|
||
CREATE TABLE stock_sectors (
|
||
code TEXT NOT NULL REFERENCES stocks(code),
|
||
sector_name TEXT NOT NULL, -- 板块名称（与 sector_snapshots.name 一致）
|
||
source TEXT DEFAULT 'ths', -- 数据来源
|
||
updated_at TEXT DEFAULT (datetime('now','localtime')),
|
||
PRIMARY KEY (code, sector_name)
|
||
);
|
||
|
||
CREATE INDEX idx_stock_sector ON stock_sectors(sector_name);
|
||
```
|
||
|
||
**典型查询：**
|
||
```sql
|
||
-- 我的持仓股所在的板块今天表现如何
|
||
SELECT h.code, h.name, ss.change_pct, ss.net_inflow
|
||
FROM holdings h
|
||
JOIN stock_sectors ss_map ON h.code = ss_map.code
|
||
JOIN sector_snapshots ss ON ss_map.sector_name = ss.name
|
||
JOIN market_snapshots ms ON ss.snapshot_id = ms.id
|
||
WHERE ms.timestamp = (SELECT MAX(timestamp) FROM market_snapshots)
|
||
ORDER BY ss.change_pct DESC;
|
||
```
|
||
|
||
---
|
||
|
||
### 2.4 持仓
|
||
|
||
```sql
|
||
CREATE TABLE holdings (
|
||
code TEXT PRIMARY KEY REFERENCES stocks(code),
|
||
name TEXT NOT NULL,
|
||
shares INTEGER NOT NULL, -- 持股数
|
||
cost REAL, -- 成本价
|
||
position_pct REAL, -- 仓位占比(%)
|
||
added_at TEXT, -- 买入时间
|
||
is_active INTEGER DEFAULT 1, -- 1=持仓中, 0=已卖出
|
||
closed_at TEXT, -- 卖出时间
|
||
close_pnl REAL -- 最终盈亏(%)
|
||
);
|
||
|
||
-- 持仓策略参数（可重评，保留历史）
|
||
CREATE TABLE holding_strategies (
|
||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||
code TEXT NOT NULL REFERENCES holdings(code),
|
||
version INTEGER DEFAULT 1, -- 策略版本号
|
||
stop_loss REAL, -- 止损价
|
||
take_profit REAL, -- 止盈价
|
||
entry_low REAL, -- 买入区下沿
|
||
entry_high REAL, -- 买入区上沿
|
||
strategy_type TEXT DEFAULT 'holding', -- holding / watch
|
||
source TEXT, -- reassess / manual / initial
|
||
reason TEXT, -- 策略理由
|
||
created_at TEXT DEFAULT (datetime('now','localtime')),
|
||
superseded_at TEXT -- 被新版本替代的时间
|
||
);
|
||
|
||
CREATE INDEX idx_strategy_code ON holding_strategies(code);
|
||
```
|
||
|
||
---
|
||
|
||
### 2.5 自选股
|
||
|
||
```sql
|
||
CREATE TABLE watchlist_stocks (
|
||
code TEXT PRIMARY KEY REFERENCES stocks(code),
|
||
name TEXT NOT NULL,
|
||
added_at TEXT DEFAULT (datetime('now','localtime')),
|
||
is_active INTEGER DEFAULT 1
|
||
);
|
||
|
||
-- 自选股买入区策略（关联到 holding_strategies，共用策略表）
|
||
```
|
||
|
||
说明：自选股的买入区策略直接使用 `holding_strategies` 表（`strategy_type='watch'`），不需要单独建表。
|
||
|
||
---
|
||
|
||
### 2.6 候选池
|
||
|
||
```sql
|
||
CREATE TABLE candidates (
|
||
code TEXT PRIMARY KEY REFERENCES stocks(code),
|
||
name TEXT NOT NULL,
|
||
sector TEXT, -- 所属板块
|
||
reason TEXT, -- 推荐理由
|
||
entry_range TEXT, -- '49.00-52.00'
|
||
stop_loss REAL,
|
||
target REAL,
|
||
zhiwei_star REAL, -- 知微星级 5.0/4.5/4.0/3.5/3.0
|
||
zhiwei_reviewed INTEGER DEFAULT 0,
|
||
zhiwei_reviewed_at TEXT,
|
||
promoted INTEGER DEFAULT 0, -- 是否已向老爸推荐过
|
||
promoted_at TEXT,
|
||
dropped INTEGER DEFAULT 0,
|
||
drop_reason TEXT,
|
||
created_at TEXT DEFAULT (datetime('now','localtime'))
|
||
);
|
||
|
||
-- 评分历史（每次小果/知微评分的记录）
|
||
CREATE TABLE candidate_score_history (
|
||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||
code TEXT NOT NULL REFERENCES candidates(code),
|
||
score REAL NOT NULL, -- 评分 1-10
|
||
source TEXT NOT NULL, -- xiaoguo / zhiwei
|
||
reason TEXT,
|
||
created_at TEXT DEFAULT (datetime('now','localtime'))
|
||
);
|
||
|
||
CREATE INDEX idx_candidate_history ON candidate_score_history(code, created_at);
|
||
```
|
||
|
||
**典型查询：**
|
||
```sql
|
||
-- 候选池状态（活跃候选及其最新评分）
|
||
SELECT c.code, c.name, c.sector, c.zhiwei_star, c.promoted,
|
||
(SELECT score FROM candidate_score_history
|
||
WHERE code = c.code ORDER BY created_at DESC LIMIT 1) as latest_score
|
||
FROM candidates c
|
||
WHERE c.dropped = 0
|
||
ORDER BY c.zhiwei_star DESC NULLS LAST;
|
||
|
||
-- 某候选的评分趋势
|
||
SELECT created_at, score, source
|
||
FROM candidate_score_history
|
||
WHERE code = '688981'
|
||
ORDER BY created_at;
|
||
```
|
||
|
||
---
|
||
|
||
### 2.7 价格事件
|
||
|
||
```sql
|
||
CREATE TABLE price_events (
|
||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||
code TEXT NOT NULL REFERENCES stocks(code),
|
||
name TEXT,
|
||
event_type TEXT NOT NULL, -- entry_zone / stop_breach / take_profit / zone_warning
|
||
price REAL, -- 触发时价格
|
||
trigger_value TEXT, -- '15.36-16.49' 区间描述
|
||
event_label TEXT, -- '买入区' / '止损预警'
|
||
created_at TEXT DEFAULT (datetime('now','localtime')),
|
||
date TEXT -- 触发日期
|
||
);
|
||
|
||
CREATE INDEX idx_events_code ON price_events(code);
|
||
CREATE INDEX idx_events_date ON price_events(date);
|
||
```
|
||
|
||
---
|
||
|
||
### 2.8 策略评估记录
|
||
|
||
```sql
|
||
CREATE TABLE strategy_evaluations (
|
||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||
code TEXT NOT NULL REFERENCES stocks(code),
|
||
eval_type TEXT NOT NULL, -- daily / weekly / adhoc
|
||
status TEXT DEFAULT 'pending', -- pending / completed / error
|
||
old_stop_loss REAL,
|
||
new_stop_loss REAL,
|
||
old_tp REAL,
|
||
new_tp REAL,
|
||
reason TEXT,
|
||
created_at TEXT DEFAULT (datetime('now','localtime'))
|
||
);
|
||
```
|
||
|
||
---
|
||
|
||
## 三、与现有系统的关系
|
||
|
||
### 3.1 分步迁移计划
|
||
|
||
```
|
||
阶段1：新库就位
|
||
→ 创建 mofin.db + 所有表
|
||
→ market_watch.py 改：写 market.json（保留兼容）+ 同时写 market_snapshots / sector_snapshots
|
||
→ 验证：数据正常双写，旧管道不受影响
|
||
|
||
阶段2：个股数据迁移
|
||
→ refresh_mtf_cache.py 改：写 multi_tf_cache.json + 同时写 stock_daily/weekly/monthly
|
||
→ 新增 stock_sectors 填充（从 market.json 的板块数据提取成分股）
|
||
→ 验证：K线数据在SQLite可查
|
||
|
||
阶段3：业务表迁移
|
||
→ portfolio / watchlist / candidate_pool 逐步迁移
|
||
→ 每个模块：先写JSON+SQLite双写，确认无问题后切到SQLite
|
||
→ 淘汰对应JSON文件
|
||
|
||
阶段4：no_agent 化
|
||
→ 通用查询脚本：mofin_query.py "SELECT ..." → 输出JSON/TEXT
|
||
→ 定时报表：纯SQL生成，无需LLM
|
||
→ 预警规则：SQL条件 → 推送通知
|
||
```
|
||
|
||
### 3.2 JSON 文件保留策略
|
||
|
||
| 文件 | 阶段1 | 阶段2 | 阶段3 | 阶段4 |
|
||
|------|-------|-------|-------|-------|
|
||
| market.json | ✅ 双写 | ✅ 双写 | ❌ 淘汰 | — |
|
||
| multi_tf_cache.json | — | ✅ 双写 | ❌ 淘汰 | — |
|
||
| portfolio.json | — | — | ✅ 双写 | ❌ 淘汰 |
|
||
| watchlist.json | — | — | ✅ 双写 | ❌ 淘汰 |
|
||
| candidate_pool.json | — | — | ✅ 双写 | ❌ 淘汰 |
|
||
| price_events.json | — | — | ✅ 双写 | ❌ 淘汰 |
|
||
| decisions.json | — | — | ⚠️ 按需迁移 | — |
|
||
|
||
---
|
||
|
||
## 四、关键关联路径
|
||
|
||
以下是最有价值的跨表查询链：
|
||
|
||
```
|
||
持仓股 → stock_sectors（查所属板块）→ sector_snapshots（板块最近表现）
|
||
↓
|
||
sector_snapshots 历史（板块趋势）
|
||
↓
|
||
stock_daily（个股K线与板块对比）
|
||
↓
|
||
candidate_score_history（候选评分变化）
|
||
|
||
事件触发 → stock_daily（看当日量价）→ holding_strategies（检查策略偏离）
|
||
↓
|
||
strategy_evaluations（策略重评记录）
|
||
```
|
||
|
||
---
|
||
|
||
## 五、迁移风险控制
|
||
|
||
- 每条写操作先写 JSON（现有流程不受影响），再写 SQLite
|
||
- SQLite 写失败时降级报警，不阻塞 JSON 写
|
||
- 全部验证通过后才切掉 JSON
|
||
- 所有变更通过 market_watch.py / refresh_mtf_cache.py 的 patch 实现，不重写
|