fix(pipelines): clear today's real cron errors + kill monitoring false alarms

Real errors fixed (all verified by manual run):
- price_monitor.py: shares None -> TypeError at L584 (now completes 3m7s,
  full 39-stock reassess + zone triggers + Dad push)
- market_insight.py: net_inflow None -> TypeError at L142 (now 0.3s, 5 insights)
- promote_candidates.py: add busy_timeout=30s (DB lock under concurrent writes)
- premarket_full_review.py: 12-dim analysis now detached background launch
  (was doomed by cron 120s script timeout no matter what)

Systemic:
- HERMES_CRON_SCRIPT_TIMEOUT=600 drop-in for both gateway services
  (fixes mofin_health SIGTERM, market_watch timeout, memory_guardian timeout)
- sync_profile_scripts.sh: re-hardlink deploy->profile scripts after every
  deploy (scp replaces files = new inode = broken hardlink = cron silently
  runs stale code; this caused promote to keep failing after my first fix)

Monitoring false-alarm fixes (the '花瓶' problem):
- mofin_health.py: legacy JSONs that migrated to DB (multi_tf_cache/
  macro_context/market/live_prices/price_history/macro_risk_state) no longer
  warn 'no readers'; marked as migrated
- NEW db_freshness section: real pipeline health from DB tables
  (mtf_cache 0.4h / macro_context_log 2h / market_snapshots 2h /
  live_prices 0.4h / price_events.json 0.4h — ALL HEALTHY)
- price_events freshness reads live JSON store (DB table is legacy)
- market.json placeholder created (13+ scripts have fallback paths)

Investigation notes: wiki-self-growth 03:04 key1 429 predates full key6
activation on default gateway; current 8642 verified on key6 and working.
Weekend 'Blocked' jobs verified fixed (vacuum_state_db passes).
This commit is contained in:
hmo
2026-07-20 17:30:15 +08:00
parent a40b97f5ca
commit 17305bed0b
20 changed files with 417 additions and 20 deletions
+60
View File
@@ -0,0 +1,60 @@
import json, os, subprocess
from datetime import datetime
now = datetime.now()
print("=== 1. DATA FILES ===")
for f in ['multi_tf_cache.json', 'macro_context.json']:
p = f'/home/hmo/MoFin/data/{f}'
if os.path.exists(p):
mtime = datetime.fromtimestamp(os.path.getmtime(p))
h = (now - mtime).total_seconds() / 3600
print(f'{f}: {mtime} ({h:.0f}h ago)')
else:
print(f'{f}: MISSING')
pm = '/home/hmo/web-dashboard/data/market.json'
print(f'market.json: {"EXISTS" if os.path.exists(pm) else "MISSING"}')
print()
print("=== 2. CRON MISSING SCRIPTS in profile scripts dir ===")
d = json.load(open('/home/hmo/.hermes/profiles/position-analyst/cron/jobs.json'))
jobs = d if isinstance(d, list) else d.get('jobs', [])
sdir = '/home/hmo/.hermes/profiles/position-analyst/scripts'
exist = set(os.listdir(sdir))
for j in jobs:
s = j.get('script', '')
if s and s not in exist and j.get('no_agent'):
f = subprocess.run(['find', '/home/hmo/MoFin', '-name', s, '-not', '-path', '*/venv/*'],
capture_output=True, text=True, timeout=15)
locs = [l for l in f.stdout.splitlines() if l.strip()]
print(f"MISSING: {s} (job: {j.get('name')}) -> {locs[:2] if locs else 'NOT FOUND'}")
print()
print("=== 3. suggestions table ===")
r = subprocess.run(['grep', '-rn', 'suggestions', '/home/hmo/MoFin/deploy/profile-scripts/', '--include=*.py'],
capture_output=True, text=True, timeout=10)
for l in r.stdout.splitlines():
if 'suggestions' in l.lower():
print(l.strip())
# Check if any cron references it
r2 = subprocess.run(['python3', '-c', '''
import json
d = json.load(open("/home/hmo/.hermes/profiles/position-analyst/cron/jobs.json"))
jobs = d if isinstance(d, list) else d.get("jobs", [])
for j in jobs:
if "suggest" in str(j):
print(j.get("name",""), j.get("script",""), j.get("prompt","")[:100])
'''], capture_output=True, text=True, timeout=5)
print(r2.stdout)
print()
print("=== 4. price_monitor newspaper ===")
r = subprocess.run(['grep', '-n', 'newspaper', '/home/hmo/MoFin/deploy/profile-scripts/price_monitor.py'], capture_output=True, text=True, timeout=5)
print(r.stdout[-500:] if len(r.stdout) > 500 else r.stdout)
print()
print("=== 5. Which crons generate multi_tf_cache / macro_context ===")
for j in jobs:
if j.get('name') and ('多周期' in j.get('name','') or '宏观' in j.get('name','') or 'multi' in j.get('name','').lower() or 'market' in j.get('name','').lower()):
print(f"{j.get('name')}: script={j.get('script','?')} status={j.get('last_status','?')} last_run={j.get('last_run_at','?')} err={str(j.get('last_error',''))[:80]}")