fix(pipelines): clear today's real cron errors + kill monitoring false alarms
Real errors fixed (all verified by manual run): - price_monitor.py: shares None -> TypeError at L584 (now completes 3m7s, full 39-stock reassess + zone triggers + Dad push) - market_insight.py: net_inflow None -> TypeError at L142 (now 0.3s, 5 insights) - promote_candidates.py: add busy_timeout=30s (DB lock under concurrent writes) - premarket_full_review.py: 12-dim analysis now detached background launch (was doomed by cron 120s script timeout no matter what) Systemic: - HERMES_CRON_SCRIPT_TIMEOUT=600 drop-in for both gateway services (fixes mofin_health SIGTERM, market_watch timeout, memory_guardian timeout) - sync_profile_scripts.sh: re-hardlink deploy->profile scripts after every deploy (scp replaces files = new inode = broken hardlink = cron silently runs stale code; this caused promote to keep failing after my first fix) Monitoring false-alarm fixes (the '花瓶' problem): - mofin_health.py: legacy JSONs that migrated to DB (multi_tf_cache/ macro_context/market/live_prices/price_history/macro_risk_state) no longer warn 'no readers'; marked as migrated - NEW db_freshness section: real pipeline health from DB tables (mtf_cache 0.4h / macro_context_log 2h / market_snapshots 2h / live_prices 0.4h / price_events.json 0.4h — ALL HEALTHY) - price_events freshness reads live JSON store (DB table is legacy) - market.json placeholder created (13+ scripts have fallback paths) Investigation notes: wiki-self-growth 03:04 key1 429 predates full key6 activation on default gateway; current 8642 verified on key6 and working. Weekend 'Blocked' jobs verified fixed (vacuum_state_db passes).
This commit is contained in:
@@ -0,0 +1,60 @@
|
||||
import json, os, subprocess
|
||||
from datetime import datetime
|
||||
|
||||
now = datetime.now()
|
||||
|
||||
print("=== 1. DATA FILES ===")
|
||||
for f in ['multi_tf_cache.json', 'macro_context.json']:
|
||||
p = f'/home/hmo/MoFin/data/{f}'
|
||||
if os.path.exists(p):
|
||||
mtime = datetime.fromtimestamp(os.path.getmtime(p))
|
||||
h = (now - mtime).total_seconds() / 3600
|
||||
print(f'{f}: {mtime} ({h:.0f}h ago)')
|
||||
else:
|
||||
print(f'{f}: MISSING')
|
||||
|
||||
pm = '/home/hmo/web-dashboard/data/market.json'
|
||||
print(f'market.json: {"EXISTS" if os.path.exists(pm) else "MISSING"}')
|
||||
|
||||
print()
|
||||
print("=== 2. CRON MISSING SCRIPTS in profile scripts dir ===")
|
||||
d = json.load(open('/home/hmo/.hermes/profiles/position-analyst/cron/jobs.json'))
|
||||
jobs = d if isinstance(d, list) else d.get('jobs', [])
|
||||
sdir = '/home/hmo/.hermes/profiles/position-analyst/scripts'
|
||||
exist = set(os.listdir(sdir))
|
||||
for j in jobs:
|
||||
s = j.get('script', '')
|
||||
if s and s not in exist and j.get('no_agent'):
|
||||
f = subprocess.run(['find', '/home/hmo/MoFin', '-name', s, '-not', '-path', '*/venv/*'],
|
||||
capture_output=True, text=True, timeout=15)
|
||||
locs = [l for l in f.stdout.splitlines() if l.strip()]
|
||||
print(f"MISSING: {s} (job: {j.get('name')}) -> {locs[:2] if locs else 'NOT FOUND'}")
|
||||
|
||||
print()
|
||||
print("=== 3. suggestions table ===")
|
||||
r = subprocess.run(['grep', '-rn', 'suggestions', '/home/hmo/MoFin/deploy/profile-scripts/', '--include=*.py'],
|
||||
capture_output=True, text=True, timeout=10)
|
||||
for l in r.stdout.splitlines():
|
||||
if 'suggestions' in l.lower():
|
||||
print(l.strip())
|
||||
# Check if any cron references it
|
||||
r2 = subprocess.run(['python3', '-c', '''
|
||||
import json
|
||||
d = json.load(open("/home/hmo/.hermes/profiles/position-analyst/cron/jobs.json"))
|
||||
jobs = d if isinstance(d, list) else d.get("jobs", [])
|
||||
for j in jobs:
|
||||
if "suggest" in str(j):
|
||||
print(j.get("name",""), j.get("script",""), j.get("prompt","")[:100])
|
||||
'''], capture_output=True, text=True, timeout=5)
|
||||
print(r2.stdout)
|
||||
|
||||
print()
|
||||
print("=== 4. price_monitor newspaper ===")
|
||||
r = subprocess.run(['grep', '-n', 'newspaper', '/home/hmo/MoFin/deploy/profile-scripts/price_monitor.py'], capture_output=True, text=True, timeout=5)
|
||||
print(r.stdout[-500:] if len(r.stdout) > 500 else r.stdout)
|
||||
|
||||
print()
|
||||
print("=== 5. Which crons generate multi_tf_cache / macro_context ===")
|
||||
for j in jobs:
|
||||
if j.get('name') and ('多周期' in j.get('name','') or '宏观' in j.get('name','') or 'multi' in j.get('name','').lower() or 'market' in j.get('name','').lower()):
|
||||
print(f"{j.get('name')}: script={j.get('script','?')} status={j.get('last_status','?')} last_run={j.get('last_run_at','?')} err={str(j.get('last_error',''))[:80]}")
|
||||
Reference in New Issue
Block a user