Author SHA1 Message Date
shangfangjian a0ce2e2a6a feat: 数据管线 P2 优化 — 血缘写入 + 质量监控 + 缓存清理 + 重试 worker
P2-1: DataLineage 血缘写入(bzzoiro/understat Bronze→Silver 全链路)
P2-2: DataQualityCheck 质量检查写入(行计数异常检测)
P2-3: RateLimitedClient 集成到 bzzoiro 采集
P2-4: injuries 缓存过期清理(启动时扫描删除 >1h 文件)
P2-5: IngestFailure 重试 worker(FastAPI startup 注册)

新增文件:
  - src/data/retry_worker.py (死信表重试 sweep)
2026-09-16 15:55:08 +08:00
shangfangjian 5de8ffb09d feat: 数据管线架构优化 — Bronze 层 + 死信队列 + 令牌桶 + 血缘追踪
P0: injuries 缓存原子写(tempfile+os.replace) + TTL 分级 + asyncio.Lock 并发安全
P1: 引入 Bronze 层 RawEvent 表(原始事件存档,不可变)
P1: IngestFailure 死信表(失败持久化,支持重试恢复)
P1: understat 改为 xG 覆盖更新模式 + xG 来源追踪字段
P1: 令牌桶限流器(TokenBucket)替换固定 sleep
P1: DataQualityCheck 数据质量监控表
P1: DataLineage 血缘追踪表
P2: 回测并发控制 Semaphore(3)

新增文件:
  - src/data/rate_limiter.py (令牌桶限流器)
  - alembic/versions/0008 (4 张新表迁移)
  - alembic/versions/0009 (MatchStats xG 字段迁移)
2026-09-16 14:12:42 +08:00
shangfangjian 2fd8a80ad9 revert: 移除手动创建的 experts 目录
Octop 为每个专家 Agent 自带存储后端,无需在项目代码中创建
2026-09-16 13:37:00 +08:00
shangfangjian 5676198392 chore: 添加专家工作区目录结构
为代码审查、数据工程、后端架构、安全工程四个角色
建立专属工作区,用于持久化审查报告、设计文档和知识积累
2026-09-16 13:34:42 +08:00
shangfangjian ff0045ad93 fix: 数据库与数据管线 6 个 P1 + 5 个 P2 审查问题修复
P1-1 [context_builder] build_context 共享 session,切片函数传 db 参数,
        回测 20 场并发连接需求从 100+ 降至每场 1 个
P1-2 [bzzoiro] 预加载改为按 raw_events 日期范围 ±30 天按需加载
P1-3 [understat] 批量查询球队 + 比赛,从 1140 次往返降到 3 次
P1-4 [injuries] 批量幂等检查 + 分批 flush,IntegrityError 逐条回退
P1-5 [predict] 删除 threading.Lock,dict 操作原子无需同步锁
P1-6 [models] 添加 (match_id, provider, model) 唯一约束 + 迁移

P2-1 [unit_of_work] get_uow 返回类型改为 AsyncIterator[AsyncSession]
P2-2 [normalize] _parse_date 失败时记录 warning 避免静默丢数据
P2-3 [repositories] find_by_teams_and_date 改用 match_date_date 等值匹配
P2-4 [migration] 幽灵列 cutoff_at 已在 0006 迁移删除(已有)
P2-5 [migration] injuries 约束命名对齐 ORM,UniqueConstraint → 唯一索引
2026-09-16 03:09:39 +08:00
shangfangjian 983b620659 refactor: fix P0/P1/P2 security and performance issues
P0 fixes:
- CORS: replace wildcard methods/headers with configurable lists
- deps.py: remove unsafe global _warned_unset variable

P1 fixes:
- http_client: read default timeout from Settings
- bzzoiro: replace sync urllib with async httpx
- bzzoiro: normalize validation failures use warning level only
- db pool: read pool config from Settings (default 5+10)
- backtest: add asyncio.Semaphore(8) for concurrent execution
- predict/context_builder: add backtest parameter for cutoff buffer

P2 improvements:
- injuries: enforce int conversion for player_id/fixture_id
- injuries: use system temp dir for cache
- utils.py: extract shared actual_1x2/is_correct_1x2
- validation: downgrade 1x2 mismatch log to debug
- docker-compose: use env vars for all credentials
- .env.example: add POSTGRES_USER/PASSWORD/PORT, API_PORT
2026-09-16 02:38:25 +08:00
shangfangjian fa69795d69 fix: 应用代码审查三项发现
- frontend: .tab 补 position:relative,修复激活 tab 下划线定位缺失定位上下文
- core/retry.py: 删除全项目零调用的重试模块(避免"看似生效实际未接入"的误判)
- llm/validation.py: KNOWN_AGENT_NAMES 改为从 orchestrator.SPECIALIST_SPECS 派生,单一权威源
2026-09-15 20:12:47 +08:00
shangfangjian c5f92c9a54 Merge pull request 'refactor(ui): 前端重做为报刊赛程版视觉' (#2) from frontend-beautify into main
Reviewed-on: #2
2026-09-15 20:00:05 +08:00
shangfangjian 7df46544b8 Merge branch 'main' into frontend-beautify 2026-09-15 19:59:13 +08:00
WorkBuddy d847f4f3f4 refactor(ui): 去 AI 味,重做为报刊赛程版视觉
- 色板收敛为纸白/暖墨/印报红三色,移除默认 Tailwind 蓝
- 队名彩色哈希块、shimmer/ping/fade-up 动效、圆角卡片阵全部移除
- 报头改为双线+居中宋体刊名+报眉日期行
- 赛程列表表格化:细线分行,联赛改为文字版面切换
- 预测版:方正墨线框,大号宋体比分,胜平负改文字行,专家意见改汉字编号折叠行,终裁改红竖线引文
- 修复:桌面状态列 sm:contents 错位;移动端对阵未同行
2026-09-15 19:56:23 +08:00
shangfangjian 29e718962f Merge pull request 'feat(ui): 清爽专业风设计系统重构,并修复 Tailwind 从未生效的配置缺陷' (#1) from frontend-beautify into main
Reviewed-on: #1
2026-09-15 19:01:22 +08:00
WorkBuddy e3cacc35e4 chore: 忽略本地审查/预览脚手架
.tools/ 与 .preview/ 为本机校验脚本与截图产物,不参与构建,不入库。
2026-09-15 17:45:21 +08:00
WorkBuddy 77d01450b1 feat(ui): 清爽专业风设计系统与界面重构
设计令牌(tailwind.config.js):
- brand 11 档蓝色阶、ink 11 档冷灰色阶、win/draw/loss 语义色
- 中文字体栈、tabular-nums 数字等宽、2xs 字号
- borderRadius card/pill、boxShadow card/raise/focus、fade-up/shimmer 动画

组件层(index.css):
- @layer base: 全局背景/字色/抗锯齿、统一焦点环、自定义滚动条,
  尊重 prefers-reduced-motion
- @layer components: .card .btn .field .pill .skeleton .segment

界面:
- App.tsx: SVG 队徽替代 emoji、sticky 毛玻璃头部、服务状态指示、页脚免责声明
- Matches.tsx: 比赛列表卡片化;新增 TeamMark(队名哈希色相队标)、
  EdgeBar(home_edge 双向可视化)、OutcomeCard、AgentCard(状态徽章+证据
  列表+无数据说明)、骨架屏;SVG Spinner 替代 emoji 转圈

可用性修复:
- 预测面板拆为独立的胜平负/置信度/模式/耗时卡片,胜平负与比分各有
  独立可视化条
- 移动端横向溢出:比赛行改为 flex-col -> sm:flex-row,元信息与操作按钮重排
- 无数据专家卡片补充说明文案,不再空白

保留原有契约:loadSeq/predictSeq 竞态防护、游标分页、全部类型定义。

验证:tsc --noEmit(strict)0 错误;vite build 成功;
360/390/414/768/1280px 五档实测 scrollWidth === viewport,零横向溢出。
2026-09-15 17:45:16 +08:00
WorkBuddy acea7e699d fix(frontend): 修复 postcss 配置误写导致 Tailwind 从未生效
postcss.config.js 中写入的是 Tailwind 的配置对象(content/theme/plugins),
缺少 PostCSS 插件注册所需的 plugins 字段,导致 PostCSS 从未加载 tailwindcss。
所有 className 上的 Tailwind 工具类一直是无效字符串,页面仅使用浏览器默认样式。

修复为标准写法 plugins: { tailwindcss: {}, autoprefixer: {} }。

验证:构建产物 CSS 由 1.81 kB 增至 24.66 kB,且不再残留未编译的
@tailwind / @apply / @layer 指令。
2026-09-15 17:45:10 +08:00
WorkBuddy f965650c10 test+docs: P0 回归测试与文档同步
回归测试(P0 三项,不依赖 DB 的静态断言):
- tests/test_regressions.py:
  · context_builder 三个查询函数 + backtest 必须 eager-load 关系
  · Match.{stats,home_team,away_team,league} 必须 lazy="selectin"
  · bzzoiro normalized_matches 必须携带并解包 (nm, raw) 配对
  这些是「集成方式」缺陷,原先的单元测试(全 mock slice_fn/provider)
  照不出来,故此专项静态锁定。

文档同步(修正与实际不符的表述):
- docs/01-architecture.md: app.py 标注改为「lifespan 仅验证连接,不建表」;
  补 deps.py / repositories.py / unit_of_work.py / validation.py /
  backtest.py / sources.py / http_client.py / retry.py;迁移范围改 0001~0006
- docs/07-development.md: 目录树修正 leagues.py→matches.py,补齐
  backtest.py / deps.py 及 0003~0006 迁移文件

附带: src/core/retry.py 加 NOTE 说明其当前无调用点(P3),
避免「看似有重试实则未生效」的误判。
2026-09-15 16:55:54 +08:00
WorkBuddy c89bfe2af7 fix(P2): no_data 结构化、权重校验、鉴权与前端竞态
P2-1 no_data 门控依赖文案子串(脆弱):
- context_builder 新增 SliceResult(text/has_data/n_records),
  5 个切片函数改为显式声明 has_data
- base._slice_has_data() 优先取结构化结果,str 返回仍走文案回退
  (兼容既有测试 mock 与自定义切片)
- build_context 的 has_stats/has_injuries 直接取切片声明

P2-2 agent_weights 无校验即落库:
- validation 新增 AgentWeightsSchema / validate_agent_weights:
  未知专家名丢弃、越界值钳制、总和非 1 时归一化
- orchestrator 落库前对 agent_weights 做校验

P2-3 1x2 与比分不一致被静默修正:
- 仍以比分修正,但补 logger.warning 暴露 LLM 自相矛盾

P2-5/P2-6 prompt 缓存不可刷新 + 缓存键不含模板内容:
- 新增 clear_prompt_cache() 供改模板后显式失效
- 缓存键纳入模板内容 hash,模板一改缓存自动失效

P2-7 ingest/backtest/settle 接口无鉴权:
- 新增 require_admin_key 依赖(X-API-Key),
  ADMIN_API_KEY 未设置时放行并告警(不破坏本地开发)
- 挂到 3 个 ingest 接口 + backtest + eval/settle

P2-8 前端请求竞态 + 未使用游标分页:
- Matches.tsx 用递增 seq 丢弃过期响应,避免旧筛选结果覆盖新筛选
- 接入后端已有的 cursor 分页 + 「加载更多」按钮

附带: .env.example 补齐 LLM_TIMEOUT / 分档模型 / ADMIN_API_KEY;
tests 新增 10 个用例覆盖 P2-1/2/3。
2026-09-15 16:54:05 +08:00
WorkBuddy 71bf723a10 fix(P1): 消除迁移漂移、回测缓存污染与历史数据越界风险
P1-1 schema/ORM 漂移:
- 新增 0006 迁移: 回填后 drop 幽灵列 predictions.cutoff_at,
  并幂等重建 ix_match_stats_available_at / ix_predictions_cutoff_at
- models.py 为 MatchStats / Prediction 声明上述两个索引,
  使 autogenerate 不再误建议删除

P1-2 迁移对历史越界数据不安全:
- 0004 在建 CHECK 约束前先 op.execute 清洗越界行
  (负数进球/置信度越界/非法 1x2/mode),避免 ALTER 中途失败
  导致迁移卡在半完成状态

P1-3 回测缓存污染:
- predict_match / _predict_single 新增 use_cache 参数,
  use_cache=False 时既不读也不写进程内缓存
- run_backtest 显式传 use_cache=False,避免 settle 到旧记录

P1-5 understat 关系属性访问:
- MatchRepository.find_by_teams_and_date 加 selectinload(Match.stats)

P1-6 日期键构造不统一:
- bzzoiro 抽取 _to_date()/_match_key() 统一去重键构造,
  消除 str(date) 与 isoformat() 的隐式格式依赖
2026-09-15 16:48:03 +08:00
WorkBuddy 235fb0de97 fix(P0): 修复三处静默失效的阻断缺陷
P0-1 backtest 会话生命周期:
- _get_historical_matches 加 selectinload(league/home_team/away_team),
  并在 session 内物化为 BacktestCandidate 纯数据快照,避免 session
  关闭后访问惰性关系抛 MissingGreenlet
- except 收窄并改用 logger.exception 保留堆栈
- 移除未使用的 and_ / League 导入

P0-2 切片函数关系属性 MissingGreenlet:
- models.py 为 Match.league/home_team/away_team/stats 声明 lazy=selectin
- context_builder 的 _get_form/_get_h2h/_get_home_away 显式 selectinload
  (修复 form_slice/h2h_slice 恒定失败,被 fail-open 掩盖的问题)
- repositories.find_by_teams_and_date 加 selectinload(Match.stats)

P0-3 bzzoiro raw 变量泄漏导致血缘错乱:
- normalized_matches 改为携带 (nm, raw) 元组,内层循环解包
- source_event_id / source_record_id 现在取到正确 event id
- 已有比赛补建 stats 时补齐 source/source_event_id/retrieved_at/available_at
- 抽取 _match_key()/_to_date() 统一日期键构造 (P1-6)
2026-09-15 16:45:09 +08:00
shangfangjian 60e4b89822 docs: 统一文档与代码一致性
- 修复所有 confidence → subjective_confidence 残留(03-api, 04-agents, 05-data, 07-development)
- 修复 5 张表 → 6 张表残留(06-deployment)
- 同步 API schema 示例与实际模型一致
- 更新 predictions 表结构文档(新增 status/cutoff/input_hash 字段)

code: 修复 API 异常处理(eval/predict)和 context_builder stats 时间过滤
2026-09-15 02:27:48 +08:00
shangfangjian 0980c2242a feat: P0-02/P0-03/P0-05 数据库时间语义与约束
P0-02: MatchStats 增加 source/source_record_id/retrieved_at/available_at
        context_builder 过滤统计数据时检查 available_at <= cutoff
P0-03: Prediction 增加 match_kickoff_at/prediction_created_at/prediction_cutoff_at
        明确区分比赛时间/预测创建时间/数据截止时间
P0-05: Prediction 增加 status 字段(success/failed/degraded)
        数据库 CHECK 约束

Alembic: 0005_prediction_status_and_stats_provenance

P1-02: injuries as_of 不再截断为 date,保持 datetime 精度
2026-09-15 02:04:25 +08:00
shangfangjian c657e04679 docs: 同步文档与实现一致性
- 5 张表 → 6 张表(添加 injuries)
- 同步所有 docs 和 models.py docstring
2026-09-15 01:44:49 +08:00
shangfangjian 98e3d07cf5 fix: 补充 Alembic 迁移 + bzzoiro 批量优化 + validation 兼容层
- 新增 0004_snapshot_and_constraints.py: 重命名 confidence → subjective_confidence,
  新增 cutoff_at/input_hash, 添加 CHECK 约束
- bzzoiro.py: 预加载完整 Match 对象到内存,更新路径不再重复查询
- validation.py: 旧字段 confidence 兼容并打 deprecation 日志
2026-09-15 01:42:52 +08:00
shangfangjian 2c205c68b1 fix: 修复剩余审查问题
- ingest.py: 异常不再暴露客户端,改为通用错误消息+日志记录
- bzzoiro.py: 修复 normalize 重复调用,改为一次遍历缓存结果
- understat.py: 使用 Repository 替代原始 SQL
2026-09-15 01:26:15 +08:00
shangfangjian 53e602b4f6 fix: 代码审查问题修复
- unit_of_work.py: 简化为纯 session 上下文管理器,消除 double-close 风险
- validation.py: 移除未使用的 import,简化 _score_to_1x2 逻辑
- repositories.py: 将 func 导入移到模块顶层
- 更新所有 get_uow() 调用点使用新接口(yield session 而非 uow 对象)
2026-09-15 01:18:35 +08:00
shangfangjian 74586aa5b7 docs: 更新 README 反映当前架构
- 更新架构图(添加分层架构、backtest 端点)
- 更新项目结构(添加 unit_of_work.py、repositories.py、backtest.py)
- 更新核心模块表
- 添加数据正确性保障章节
- 修正表数量(6 张表)
- confidence → subjective_confidence
2026-09-15 00:48:43 +08:00
shangfangjian 483cb956ba refactor: Sprint 3 - 引入 UnitOfWork + Repository 架构
新增:
- src/db/unit_of_work.py: UnitOfWork 事务封装
- src/db/repositories.py: Match/Team/League/Prediction Repository

重构:
- 删除 src/data/match_lookup.py(由 Repository 替代)
- 数据源(bzzoiro/understat/injuries)不再自行 commit
- API 路由(ingest)改用 UnitOfWork
- LLM 服务(predict/orchestrator/eval/backtest)改用 UnitOfWork

事务边界统一由调用方控制,数据层不再自行决定 commit。
2026-09-15 00:42:05 +08:00
shangfangjian cb36dc3ef9 feat: Sprint 2 - 概率语义 + 轻量快照 + 数据库约束
P0-05: confidence → subjective_confidence 改名(15 文件)
       LLM 主观置信度与概率分离
P0-02: predictions 增加 cutoff_at + input_hash(轻量快照)
       MatchContext 暴露 match_dt
       单/多 Agent 路径均记录快照元数据
P2-05: 数据库 CHECK 约束
       - pred_home_goals >= 0
       - pred_away_goals >= 0
       - subjective_confidence 0~1
       - pred_1x2 IN (1,X,2)
       - mode IN (single,multi)
P2-06: limit 分页约束(ge=1, le=200)
P2-01: 新增 /health/ready 就绪检查
2026-09-15 00:14:24 +08:00
shangfangjian f3160e3062 feat: Sprint 1 - 数据正确性整改
P2-01: 移除 lifespan create_all,改为仅验证连接
       新增 /health/ready 就绪检查
P0-04: LLM 输出严格 Pydantic 校验
       - Agent 输出越界/非法 → parse_error
       - 预测输出自动修正 1X2 与比分一致性
P0-01: injuries cutoff 修复
       - get_injuries_for_match 增加 as_of 参数
       - injuries_slice 使用 as_of 过滤 retrieved_at
       - 防止回测时未来采集数据泄漏
P1-12: 批量入库优化
       - 预加载 teams 到内存 dict
       - 预加载 existing matches 到内存 set
       - 消灭 N+1 查询
2026-09-14 23:36:35 +08:00
55 changed files with 4067 additions and 840 deletions
+14
View File
@@ -3,6 +3,10 @@ APP_ENV=development
LOG_LEVEL=INFO
# ---- 数据库 ----
POSTGRES_USER=football
POSTGRES_PASSWORD=football
POSTGRES_DB=football
POSTGRES_PORT=5432
DATABASE_URL=postgresql+asyncpg://football:football@localhost:5432/football
# ---- LLM (OpenAI-compatible,必填一个) ----
@@ -10,6 +14,11 @@ LLM_PROVIDER=openai
LLM_API_KEY=sk-xxxx
LLM_BASE_URL=https://api.openai.com/v1
LLM_MODEL=gpt-4o
# 多 Agent 分档模型(留空则回落 LLM_MODEL)
LLM_SPECIALIST_MODEL=
LLM_AGGREGATOR_MODEL=
# 单次 LLM 调用超时(秒)
LLM_TIMEOUT=60
# ---- 数据源 ----
BZZOIRO_KEY=
@@ -17,3 +26,8 @@ API_FOOTBALL_KEY=
# ---- CORS ----
CORS_ORIGINS=http://localhost:5173,http://localhost:3000
# ---- 管理接口鉴权 ----
# 采集/回测/回填接口的访问密钥(请求头 X-API-Key)。
# 留空 = 不启用鉴权(本地开发默认);生产环境必须设置强随机值。
ADMIN_API_KEY=
+4
View File
@@ -10,3 +10,7 @@ frontend/dist/
CLAUDE.md
docs/AGENTS.md
docs/agents/
# 本地审查/预览脚手架(不入库)
.tools/
.preview/
+51 -67
View File
@@ -15,12 +15,13 @@
│ ├── /api/v1/matches 比赛查询 │
│ ├── /api/v1/predict LLM 预测 (单/多 Agent) │
│ ├── /api/v1/ingest/* 数据采集 │
── /api/v1/eval/* 评估回填 │
── /api/v1/eval/* 评估回填 │
│ └── /api/v1/backtest 回测 │
└──────────┬─────────────────────────────┬────────────┘
│ │
┌──────────▼──────────┐ ┌─────────────▼────────────┐
│ PostgreSQL │ │ LLM (OpenAI-compatible) │
5 张表 │ │ OpenAI / Deepseek / │
6 张表 │ │ OpenAI / Deepseek / │
│ leagues/teams/ │ │ Ollama / 任意网关 │
│ matches/match_ │ └──────────────────────────┘
│ stats/predictions/ │
@@ -36,6 +37,16 @@
└────────────────────────────────────────────────────┘
```
### 分层架构
```
API Route → Application Service → Repository → UnitOfWork → DB
```
- **UnitOfWork**: 统一事务边界,业务层不再自行 commit
- **Repository**: 封装数据访问,提供类型化查询接口
- **DataSource**: 采集外部数据,通过注册表动态分发
### 多 Agent 预测
默认模式 (`mode=multi`) 采用 **5 专家 + 终裁** 架构:
@@ -51,7 +62,14 @@
- 各专家**只看到自己维度的数据切片**,避免信息过载
- **fail-open**: 单个专家失败不影响整体
- **no_data 门控**: 无数据维度跳过 LLM 调用,省 token 防幻觉
- 终裁根据各报告的 `confidence` / `data_sufficiency` 加权输出 `agent_weights`
- 终裁根据各报告的 `subjective_confidence` / `data_sufficiency` 输出 `agent_weights`
### 数据正确性保障
- **Cutoff 机制**: 回测时只使用 `cutoff_at` 之前已采集的数据
- **Injury 防泄漏**: 伤停查询强制 `retrieved_at <= cutoff`
- **LLM 输出校验**: Pydantic 严格校验 + 语义一致性检查
- **数据库约束**: CHECK 约束作为最后一道防线
## 快速开始
@@ -64,7 +82,6 @@
### 1. 安装
```bash
# 克隆
git clone https://git.bilidili.cn/shangfangjian/Profeto.git
cd Profeto
@@ -80,6 +97,7 @@ cp .env.example .env
```bash
docker compose up -d postgres
alembic upgrade head # 首次运行需要执行迁移
```
### 3. 启动服务
@@ -94,58 +112,22 @@ cd frontend && npm install && npm run dev
后端运行在 `http://localhost:8000`,前端在 `http://localhost:5173`
## 使用流程
## API 概览
### 1. 采集数据
```bash
# 采集 bzzoiro 比分与统计
curl -X POST http://localhost:8000/api/v1/ingest/bzzoiro \
-H "Content-Type: application/json" \
-d '{"leagues":["E0","SP1"],"date_from":"2026-08-01","date_to":"2026-09-06"}'
# 回填 understat xG
curl -X POST http://localhost:8000/api/v1/ingest/understat \
-H "Content-Type: application/json" \
-d '{"league":"E0","season":2026}'
# 采集伤停
curl -X POST http://localhost:8000/api/v1/ingest/injuries \
-H "Content-Type: application/json" \
-d '{"date":"2026-09-09"}'
```
### 2. 查询比赛
```bash
curl "http://localhost:8000/api/v1/matches?league=E0&status=scheduled"
```
### 3. LLM 预测
```bash
# 多 Agent 预测 (默认)
curl -X POST http://localhost:8000/api/v1/predict \
-H "Content-Type: application/json" \
-d '{"match_id": 1}'
# 单次调用模式
curl -X POST http://localhost:8000/api/v1/predict \
-H "Content-Type: application/json" \
-d '{"match_id": 1, "mode": "single"}'
```
### 4. 评估
```bash
# 赛后回填实际比分
curl -X POST http://localhost:8000/api/v1/eval/settle \
-H "Content-Type: application/json" \
-d '{"prediction_id": 1, "home_goals": 2, "away_goals": 1}'
# 查看准确率汇总
curl http://localhost:8000/api/v1/eval/summary
```
| 方法 | 路径 | 说明 |
|---|---|---|
| GET | `/api/v1/matches` | 比赛查询(筛选/分页) |
| GET | `/api/v1/leagues` | 联赛列表 |
| POST | `/api/v1/predict` | LLM 预测 (`mode=single`/`multi`) |
| GET | `/api/v1/predictions` | 预测历史 |
| POST | `/api/v1/ingest/bzzoiro` | 采集比分/统计 |
| POST | `/api/v1/ingest/understat` | 回填 xG |
| POST | `/api/v1/ingest/injuries` | 采集伤停 |
| POST | `/api/v1/eval/settle` | 回填实际结果 |
| GET | `/api/v1/eval/summary` | 准确率汇总 |
| POST | `/api/v1/backtest` | 历史回测 |
| GET | `/health` | 存活检查 |
| GET | `/health/ready` | 就绪检查(含 DB) |
## 项目结构
@@ -159,34 +141,36 @@ Profeto/
│ │ ├── matches.py # 比赛查询
│ │ ├── predict.py # 预测入口
│ │ ├── ingest.py # 数据采集
│ │ ── eval.py # 评估回填
│ │ ── eval.py # 评估回填
│ │ └── backtest.py # 回测
│ ├── core/ # 基础设施
│ │ ├── config.py # pydantic-settings 配置
│ │ ── http_client.py # 共享 httpx 客户端
│ │ ── http_client.py # 共享 httpx 客户端
│ │ └── retry.py # 重试工具(指数退避)
│ ├── data/ # 数据层
│ │ ├── sources.py # DataSource 协议 + 注册表
│ │ ├── match_lookup.py # 比赛匹配辅助函数
│ │ ├── normalize.py # 数据规范化契约
│ │ ├── bzzoiro.py # bzzoiro 数据源
│ │ ├── understat.py # understat xG 数据源
│ │ ├── injuries.py # 伤停数据 (独立领域)
│ │ ├── injuries.py # 伤停数据
│ │ ├── config.py # 联赛映射常量
│ │ └── team_names.py # 队名归一化
│ ├── db/ # 数据库
│ │ ├── base.py # SQLAlchemy async engine
│ │ ── models.py # ORM 模型 (5 表)
│ │ ── models.py # ORM 模型 (6 表)
│ │ ├── unit_of_work.py # UnitOfWork 事务封装
│ │ └── repositories.py # Repository 数据访问
│ └── llm/ # LLM 预测核心
│ ├── predict.py # 预测服务 (缓存 + 单/多模式)
│ ├── context_builder.py # 数据切片 + 上下文拼接
│ ├── eval.py # 评估统计
│ ├── backtest.py # 回测框架
│ ├── provider.py # 多提供商 LLM 抽象
│ ├── validation.py # LLM 输出校验
│ ├── agents/
│ │ ├── base.py # Agent 基础设施 + 解析
│ │ └── orchestrator.py # 多 Agent 编排
│ └── prompts/ # Prompt 模板
│ ├── match_prediction_v1.md
│ ├── match_prediction_v2.md
│ └── agents/ # 各专家 prompt
├── alembic/ # 数据库迁移
├── frontend/ # React 前端
├── docs/ # 详细文档
@@ -204,8 +188,10 @@ Profeto/
| `prompts/` | Prompt 模板 (迭代最频繁) |
| `provider.py` | OpenAI-compatible 多提供商抽象 |
| `sources.py` | 数据源协议 + 注册表 |
| `normalize.py` | 数据清洗契约 (校验/范围/归一) |
| `orchestrator.py` | 多 Agent 编排 (并行专家 + 终裁) |
| `unit_of_work.py` | 统一事务边界 |
| `repositories.py` | 数据访问封装 |
| `validation.py` | LLM 输出严格校验 |
| `backtest.py` | 回测框架(防未来数据泄漏) |
## 配置
@@ -232,8 +218,6 @@ pytest
## 数据库迁移
生产环境建议使用 Alembic:
```bash
alembic upgrade head
```
@@ -0,0 +1,64 @@
"""add snapshot fields and check constraints
Revision ID: 0004_snapshot_and_constraints
Revises: 0003_injuries
Create Date: 2026-09-15
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision: str = '0004_snapshot_and_constraints'
down_revision: Union[str, None] = '0003_injuries'
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# 1. 重命名 confidence → subjective_confidence
op.alter_column('predictions', 'confidence', new_column_name='subjective_confidence')
# 2. 新增快照字段
op.add_column('predictions', sa.Column('cutoff_at', sa.DateTime(timezone=True), nullable=True))
op.add_column('predictions', sa.Column('input_hash', sa.String(length=64), nullable=True))
# 3. 新增 CHECK 约束
# 注意: 对已有数据行加 CHECK 约束时,若存在越界数据 ALTER TABLE 会中途失败,
# 导致迁移卡在半完成状态。这里先做一次性清洗(把越界值收敛到合法域),
# 再建约束,保证在非空库上也能成功。
op.execute("UPDATE predictions SET pred_home_goals = 0 WHERE pred_home_goals IS NOT NULL AND pred_home_goals < 0")
op.execute("UPDATE predictions SET pred_away_goals = 0 WHERE pred_away_goals IS NOT NULL AND pred_away_goals < 0")
op.execute(
"UPDATE predictions SET subjective_confidence = "
"CASE WHEN subjective_confidence < 0 THEN 0 "
" WHEN subjective_confidence > 1 THEN 1 ELSE subjective_confidence END "
"WHERE subjective_confidence IS NOT NULL "
" AND (subjective_confidence < 0 OR subjective_confidence > 1)"
)
op.execute("UPDATE predictions SET pred_1x2 = NULL WHERE pred_1x2 IS NOT NULL AND pred_1x2 NOT IN ('1', 'X', '2')")
op.execute("UPDATE predictions SET mode = 'single' WHERE mode IS NOT NULL AND mode NOT IN ('single', 'multi')")
op.create_check_constraint('ck_pred_home_goals_nonneg', 'predictions', 'pred_home_goals >= 0')
op.create_check_constraint('ck_pred_away_goals_nonneg', 'predictions', 'pred_away_goals >= 0')
op.create_check_constraint('ck_confidence_range', 'predictions', 'subjective_confidence >= 0 AND subjective_confidence <= 1')
op.create_check_constraint('ck_pred_1x2_enum', 'predictions', "pred_1x2 IN ('1', 'X', '2')")
op.create_check_constraint('ck_mode_enum', 'predictions', "mode IN ('single', 'multi')")
def downgrade() -> None:
# 1. 删除 CHECK 约束
op.drop_constraint('ck_mode_enum', 'predictions', type_='check')
op.drop_constraint('ck_pred_1x2_enum', 'predictions', type_='check')
op.drop_constraint('ck_confidence_range', 'predictions', type_='check')
op.drop_constraint('ck_pred_away_goals_nonneg', 'predictions', type_='check')
op.drop_constraint('ck_pred_home_goals_nonneg', 'predictions', type_='check')
# 2. 删除快照字段
op.drop_column('predictions', 'input_hash')
op.drop_column('predictions', 'cutoff_at')
# 3. 恢复列名
op.alter_column('predictions', 'subjective_confidence', new_column_name='confidence')
@@ -0,0 +1,58 @@
"""add prediction status/time semantics and MatchStats provenance
Revision ID: 0005_prediction_status_and_stats_provenance
Revises: 0004_snapshot_and_constraints
Create Date: 2026-09-15
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision: str = '0005_prediction_status_and_stats_provenance'
down_revision: Union[str, None] = '0004_snapshot_and_constraints'
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# 1. Prediction 新增字段
op.add_column('predictions', sa.Column('status', sa.String(length=20), nullable=False, server_default='success'))
op.add_column('predictions', sa.Column('match_kickoff_at', sa.DateTime(timezone=True), nullable=True))
op.add_column('predictions', sa.Column('prediction_created_at', sa.DateTime(timezone=True), nullable=False, server_default=sa.func.now()))
op.add_column('predictions', sa.Column('prediction_cutoff_at', sa.DateTime(timezone=True), nullable=True))
# 重命名 cutoff_at → 保留作为兼容,prediction_cutoff_at 为主字段
# op.drop_column('predictions', 'cutoff_at') # 暂不删除,避免破坏现有数据
# 2. 新增 status CHECK 约束
op.create_check_constraint('ck_status_enum', 'predictions', "status IN ('success', 'failed', 'degraded')")
# 3. MatchStats 新增数据血缘字段
op.add_column('match_stats', sa.Column('source', sa.String(length=30), nullable=True))
op.add_column('match_stats', sa.Column('source_record_id', sa.String(length=100), nullable=True))
op.add_column('match_stats', sa.Column('retrieved_at', sa.DateTime(timezone=True), nullable=True))
op.add_column('match_stats', sa.Column('available_at', sa.DateTime(timezone=True), nullable=True))
# 4. 索引
op.create_index('ix_match_stats_available_at', 'match_stats', ['available_at'])
op.create_index('ix_predictions_cutoff_at', 'predictions', ['prediction_cutoff_at'])
def downgrade() -> None:
op.drop_index('ix_predictions_cutoff_at', table_name='predictions')
op.drop_index('ix_match_stats_available_at', table_name='match_stats')
op.drop_column('match_stats', 'available_at')
op.drop_column('match_stats', 'retrieved_at')
op.drop_column('match_stats', 'source_record_id')
op.drop_column('match_stats', 'source')
op.drop_constraint('ck_status_enum', 'predictions', type_='check')
op.drop_column('predictions', 'prediction_cutoff_at')
op.drop_column('predictions', 'prediction_created_at')
op.drop_column('predictions', 'match_kickoff_at')
op.drop_column('predictions', 'status')
@@ -0,0 +1,80 @@
"""清理 schema 与 ORM 模型的漂移
Revision ID: 0006_schema_model_drift_cleanup
Revises: 0005_prediction_status_and_stats_provenance
Create Date: 2026-09-15
背景(见代码审查报告 P1-1):
0005 迁移在 predictions 表留下了 `cutoff_at` 列(删除语句被注释掉),
但 ORM 模型 `Prediction` 中并无该字段 —— 这是一处会长期存在的 schema 漂移,
而 alembic autogenerate 会持续建议 drop 它,造成噪声。
同时 0005 创建的两个索引 `ix_match_stats_available_at` 与
`ix_predictions_cutoff_at` 未在 ORM 声明,autogenerate 会建议删除它们 ——
一旦被误删,数据血缘相关的时间过滤查询会退化为全表扫描。
本迁移做两件事:
1. drop 掉幽灵列 predictions.cutoff_at(数据已由 prediction_cutoff_at 承载,
迁移前先把非空值回填过去,避免丢数据)
2. 显式重建这两个索引(幂等:先 drop if exists 再 create),使 DB 状态与
修正后的 ORM 声明一致
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision: str = '0006_schema_model_drift_cleanup'
down_revision: Union[str, None] = '0005_prediction_status_and_stats_provenance'
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
bind = op.get_bind()
inspector = sa.inspect(bind)
# --- 1. 幽灵列 cutoff_at: 先把数据回填到 prediction_cutoff_at 再删除 ---
pred_cols = {c["name"] for c in inspector.get_columns("predictions")}
if "cutoff_at" in pred_cols:
if "prediction_cutoff_at" in pred_cols:
# 仅回填尚未有值的行,避免覆盖更权威的数据
op.execute(
"UPDATE predictions "
"SET prediction_cutoff_at = cutoff_at "
"WHERE prediction_cutoff_at IS NULL AND cutoff_at IS NOT NULL"
)
op.drop_column("predictions", "cutoff_at")
# --- 2. 与 ORM 声明对齐的索引(幂等重建) ---
stats_idx = {i["name"] for i in inspector.get_indexes("match_stats")}
if "ix_match_stats_available_at" not in stats_idx:
op.create_index("ix_match_stats_available_at", "match_stats", ["available_at"])
pred_idx = {i["name"] for i in inspector.get_indexes("predictions")}
if "ix_predictions_cutoff_at" not in pred_idx:
op.create_index(
"ix_predictions_cutoff_at", "predictions", ["prediction_cutoff_at"]
)
def downgrade() -> None:
bind = op.get_bind()
inspector = sa.inspect(bind)
# 恢复幽灵列(与 0005 的最终状态一致:列存在但无值)
pred_cols = {c["name"] for c in inspector.get_columns("predictions")}
if "cutoff_at" not in pred_cols:
op.add_column(
"predictions",
sa.Column("cutoff_at", sa.DateTime(timezone=True), nullable=True),
)
stats_idx = {i["name"] for i in inspector.get_indexes("match_stats")}
if "ix_match_stats_available_at" in stats_idx:
op.drop_index("ix_match_stats_available_at", table_name="match_stats")
pred_idx = {i["name"] for i in inspector.get_indexes("predictions")}
if "ix_predictions_cutoff_at" in pred_idx:
op.drop_index("ix_predictions_cutoff_at", table_name="predictions")
@@ -0,0 +1,81 @@
"""修复 injuries 表约束命名与 ORM 声明不一致
Revision ID: 0007_injuries_constraint_naming_align
Revises: 0006_schema_model_drift_cleanup
Create Date: 2026-09-16
背景(见代码审查报告 P2-5):
0003 迁移使用 sa.UniqueConstraint 创建唯一约束,
而 ORM models.py 中声明为 Index(..., unique=True)。
虽然 PostgreSQL 中两者效果相同(都保证唯一性),
但 pg_catalog 中表示不同,会导致:
- alembic autogenerate 持续报告漂移
- 约束命名约定不一致(uc_ 前缀 vs ix_ 前缀)
本迁移将 UniqueConstraint 替换为唯一索引,与 ORM 声明对齐。
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision: str = '0007_injuries_constraint_naming_align'
down_revision: Union[str, None] = '0006_schema_model_drift_cleanup'
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
bind = op.get_bind()
inspector = sa.inspect(bind)
# 检查当前约束类型
constraints = {
c["name"]: c
for c in inspector.get_unique_constraints("injuries")
}
indexes = {
i["name"]: i
for i in inspector.get_indexes("injuries")
}
# 如果存在 UniqueConstraint 形式的 ix_injuries_player_fixture,替换为唯一索引
if "ix_injuries_player_fixture" in constraints:
# 删除唯一约束
op.drop_constraint("ix_injuries_player_fixture", "injuries", type_="unique")
# 如果不存在同名唯一索引,创建它(与 ORM 声明一致)
if "ix_injuries_player_fixture" not in indexes:
op.create_index(
"ix_injuries_player_fixture",
"injuries",
["player_id", "fixture_id", "injury_type"],
unique=True,
)
def downgrade() -> None:
bind = op.get_bind()
inspector = sa.inspect(bind)
indexes = {
i["name"]: i
for i in inspector.get_indexes("injuries")
}
constraints = {
c["name"]: c
for c in inspector.get_unique_constraints("injuries")
}
# 恢复为 UniqueConstraint 形式
if "ix_injuries_player_fixture" in indexes:
op.drop_index("ix_injuries_player_fixture", table_name="injuries")
if "ix_injuries_player_fixture" not in constraints:
op.create_unique_constraint(
"ix_injuries_player_fixture",
"injuries",
["player_id", "fixture_id", "injury_type"],
)
@@ -0,0 +1,59 @@
"""为 predictions 表添加 match_id+provider+model 唯一约束
Revision ID: 0007_predictions_unique_constraint
Revises: 0007_injuries_constraint_naming_align
Create Date: 2026-09-16
背景(见代码审查报告 P1-6):
同一 match_id + provider + model 组合不应产生重复预测。
当前缺少数据库级唯一约束,回测多次运行或并发采集可能产生重复记录,
导致统计偏差。
先清理已存在的重复记录(保留最早创建的那条),再添加唯一约束。
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision: str = '0007_predictions_unique_constraint'
down_revision: Union[str, None] = '0007_injuries_constraint_naming_align'
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# 1. 清理已存在的重复记录(保留 id 最小的)
op.execute(
"""
DELETE FROM predictions
WHERE id NOT IN (
SELECT MIN(id)
FROM predictions
GROUP BY match_id, provider, model
)
AND match_id IN (
SELECT match_id
FROM predictions
GROUP BY match_id, provider, model
HAVING COUNT(*) > 1
)
"""
)
# 2. 添加唯一约束
op.create_unique_constraint(
"uq_predictions_match_provider_model",
"predictions",
["match_id", "provider", "model"],
)
def downgrade() -> None:
op.drop_constraint(
"uq_predictions_match_provider_model",
"predictions",
type_="unique",
)
@@ -0,0 +1,121 @@
"""新增 Bronze 层 + 死信表 + 数据质量表 + 血缘表
Revision ID: 0008_raw_event_and_ingest_failure
Revises: 0007_predictions_unique_constraint
Create Date: 2026-09-17
架构审查报告 P1 实施:
- raw_events: Bronze 层,不可变原始采集记录
- ingest_failures: 采集失败死信表
- data_quality_checks: 数据质量检查结果记录
- data_lineage: ETL 全过程元数据血缘追踪
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
from sqlalchemy.dialects.postgresql import JSONB
# revision identifiers, used by Alembic.
revision: str = '0008_raw_event_and_ingest_failure'
down_revision: Union[str, None] = '0007_predictions_unique_constraint'
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# ── raw_events: Bronze 层原始记录 ──
op.create_table(
"raw_events",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("source_system", sa.String(30), nullable=False),
sa.Column("source_record_id", sa.String(100), nullable=False),
sa.Column("raw_payload", JSONB(), nullable=False),
sa.Column("ingested_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("ingest_batch_id", sa.String(64), nullable=True),
)
op.create_index("ix_raw_events_batch", "raw_events", ["ingest_batch_id"])
op.create_index("ix_raw_events_source_ingested", "raw_events", ["source_system", "ingested_at"])
op.create_unique_constraint("uq_raw_events_source_record", "raw_events", ["source_system", "source_record_id"])
# ── ingest_failures: 采集失败死信表 ──
op.create_table(
"ingest_failures",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("source_system", sa.String(30), nullable=False),
sa.Column("entity_type", sa.String(30), nullable=False),
sa.Column("source_record_id", sa.String(100), nullable=True),
sa.Column("error_type", sa.String(50), nullable=False),
sa.Column("error_detail", sa.Text(), nullable=True),
sa.Column("raw_payload", JSONB(), nullable=True),
sa.Column("retry_count", sa.Integer(), nullable=False, server_default="0"),
sa.Column("next_retry_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("status", sa.String(20), nullable=False, server_default="pending"),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("resolved_at", sa.DateTime(timezone=True), nullable=True),
)
op.create_index("ix_ingest_failures_status_next_retry", "ingest_failures", ["status", "next_retry_at"])
op.create_index("ix_ingest_failures_source", "ingest_failures", ["source_system", "entity_type"])
op.create_check_constraint("ck_ingest_failures_status", "ingest_failures",
"status IN ('pending', 'retrying', 'resolved', 'abandoned')")
op.create_check_constraint("ck_ingest_failures_retry_nonneg", "ingest_failures", "retry_count >= 0")
# ── data_quality_checks: 数据质量检查记录 ──
op.create_table(
"data_quality_checks",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("check_name", sa.String(100), nullable=False),
sa.Column("entity_type", sa.String(30), nullable=False),
sa.Column("entity_id", sa.String(50), nullable=True),
sa.Column("expected_value", sa.Text(), nullable=True),
sa.Column("actual_value", sa.Text(), nullable=True),
sa.Column("passed", sa.Boolean(), nullable=False),
sa.Column("severity", sa.String(10), nullable=False, server_default="warning"),
sa.Column("detail", sa.Text(), nullable=True),
sa.Column("checked_at", sa.DateTime(timezone=True), nullable=False),
)
op.create_index("ix_dqc_check_time", "data_quality_checks", ["check_name", "checked_at"])
op.create_index("ix_dqc_entity", "data_quality_checks", ["entity_type", "entity_id"])
op.create_index("ix_dqc_severity_passed", "data_quality_checks", ["severity", "passed"])
op.create_check_constraint("ck_dqc_severity", "data_quality_checks",
"severity IN ('info', 'warning', 'critical')")
# ── data_lineage: ETL 血缘追踪 ──
op.create_table(
"data_lineage",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("source_system", sa.String(30), nullable=False),
sa.Column("source_record_id", sa.String(100), nullable=False),
sa.Column("target_table", sa.String(50), nullable=False),
sa.Column("target_id", sa.Integer(), nullable=True),
sa.Column("transform_name", sa.String(100), nullable=False),
sa.Column("transform_detail", sa.Text(), nullable=True),
sa.Column("batch_id", sa.String(64), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
)
op.create_index("ix_lineage_source", "data_lineage", ["source_system", "source_record_id"])
op.create_index("ix_lineage_target", "data_lineage", ["target_table", "target_id"])
op.create_index("ix_lineage_batch", "data_lineage", ["batch_id"])
def downgrade() -> None:
# 逆序删除
op.drop_index("ix_lineage_batch", table_name="data_lineage")
op.drop_index("ix_lineage_target", table_name="data_lineage")
op.drop_index("ix_lineage_source", table_name="data_lineage")
op.drop_table("data_lineage")
op.drop_index("ix_dqc_severity_passed", table_name="data_quality_checks")
op.drop_index("ix_dqc_entity", table_name="data_quality_checks")
op.drop_index("ix_dqc_check_time", table_name="data_quality_checks")
op.drop_table("data_quality_checks")
op.drop_index("ix_ingest_failures_source", table_name="ingest_failures")
op.drop_index("ix_ingest_failures_status_next_retry", table_name="ingest_failures")
op.drop_table("ingest_failures")
op.drop_constraint("uq_raw_events_source_record", table_name="raw_events", type_="unique")
op.drop_index("ix_raw_events_source_ingested", table_name="raw_events")
op.drop_index("ix_raw_events_batch", table_name="raw_events")
op.drop_table("raw_events")
@@ -0,0 +1,35 @@
"""MatchStats 新增 xG 追踪字段
Revision ID: 0009_match_stats_xg_fields
Revises: 0008_raw_event_and_ingest_failure
Create Date: 2026-09-17
架构审查报告 P1-4 实施:
understat 允许纠正旧 xG 值。新增字段追踪 xG 具体来源和更新时间,
实现全量覆盖模式:当 understat 数据更新时覆盖旧值而非跳过。
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision: str = '0009_match_stats_xg_fields'
down_revision: Union[str, None] = '0008_raw_event_and_ingest_failure'
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column("match_stats", sa.Column("xg_source", sa.String(30), nullable=True))
op.add_column("match_stats", sa.Column("xg_updated_at", sa.DateTime(timezone=True), nullable=True))
op.add_column("match_stats", sa.Column("xg_source_record_id", sa.String(100), nullable=True))
op.create_index("ix_match_stats_xg_source", "match_stats", ["xg_source", "xg_updated_at"])
def downgrade() -> None:
op.drop_index("ix_match_stats_xg_source", table_name="match_stats")
op.drop_column("match_stats", "xg_source_record_id")
op.drop_column("match_stats", "xg_updated_at")
op.drop_column("match_stats", "xg_source")
+6 -6
View File
@@ -2,15 +2,15 @@ services:
postgres:
image: postgres:16-alpine
environment:
POSTGRES_USER: football
POSTGRES_PASSWORD: football
POSTGRES_DB: football
POSTGRES_USER: ${POSTGRES_USER:?POSTGRES_USER 未设置}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?POSTGRES_PASSWORD 未设置}
POSTGRES_DB: ${POSTGRES_DB:-football}
ports:
- "5432:5432"
- "${POSTGRES_PORT:-5432}:5432"
volumes:
- pgdata:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U football"]
test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:?POSTGRES_USER 未设置}"]
interval: 5s
timeout: 5s
retries: 5
@@ -19,7 +19,7 @@ services:
build: .
command: uvicorn src.api.app:app --host 0.0.0.0 --port 8000 --reload
ports:
- "8000:8000"
- "${API_PORT:-8000}:8000"
env_file: .env
depends_on:
postgres:
+19 -9
View File
@@ -21,7 +21,7 @@
│ │ └────────────┘ └───────────────────┘ │
│ │ │ └ understat (xG) │
│ ┌──┴──────────────┴──┐ └ injuries (伤停) │
│ │ PostgreSQL (5 张表) │ httpx → 外部 API │
│ │ PostgreSQL (6 张表) │ httpx → 外部 API │
│ └────────────────────┘ │
└─────────────────────────────────────────────────────┘
```
@@ -57,38 +57,48 @@
Profeto/
├── src/
│ ├── api/
│ │ ├── app.py # FastAPI 工厂(lifespan 建表)
│ │ ├── app.py # FastAPI 工厂(lifespan 仅验证 DB 连接,不建表)
│ │ ├── deps.py # 依赖:管理接口鉴权(X-API-Key)
│ │ ├── schemas.py # Pydantic v2 请求/响应
│ │ └── routes/
│ │ ├── matches.py # 联赛/比赛查询(游标分页)
│ │ ├── predict.py # 预测 + 预测历史
│ │ ├── ingest.py # 采集触发(自管 session)
│ │ ── eval.py # 赛后回填 + 准确率汇总
│ │ ├── ingest.py # 采集触发(自管 session,需鉴权)
│ │ ── eval.py # 赛后回填 + 准确率汇总
│ │ └── backtest.py # 历史回测(需鉴权)
│ ├── db/
│ │ ├── base.py # async engine + get_db/get_db_read
│ │ ── models.py # 5 张表 ORM
│ │ ── models.py # 6 张表 ORM
│ │ ├── repositories.py # 仓储层
│ │ └── unit_of_work.py # 事务边界
│ ├── data/
│ │ ├── bzzoiro.py # 赛果采集 + 幂等入库
│ │ ├── understat.py # xG 回填
│ │ ├── injuries.py # 伤停采集(带文件缓存)
│ │ ├── normalize.py # NormalizedMatch 清洗契约
│ │ ├── team_names.py # 队名归一映射
│ │ ├── sources.py # 数据源注册表
│ │ └── config.py # 联赛代码映射
│ ├── llm/
│ │ ├── provider.py # OpenAI-compatible 抽象(共享连接池/JSON 兜底解析)
│ │ ├── context_builder.py # 数据切片(h2h/form/stats/home_away/injuries)+ 单 agent 拼接
│ │ ├── predict.py # 预测入口(mode 分派 + 缓存)
│ │ ├── eval.py # 准确率统计
│ │ ├── validation.py # LLM 输出严格校验(Pydantic)
│ │ ├── backtest.py # 回测执行
│ │ ├── agents/
│ │ │ ├── base.py # AgentSpec / AgentReport / run_agent
│ │ │ └── orchestrator.py # 并行专家 → 终裁 → 存库
│ │ └── prompts/
│ │ ├── match_prediction_v1/v2.md # 单 agent 模板
│ │ └── agents/{h2h,form,stats,home_away,injuries,aggregator}_v1.md
│ └── core/config.py # pydantic-settings
├── alembic/versions/ # 0001 建表 + 0002 agent 字段
├── frontend/src/pages/Matches.tsx # 单页(预测面板 + 专家报告折叠区)
├── tests/ # 33 项(核心 13 + agent 20)
│ └── core/
│ ├── config.py # pydantic-settings
│ ├── http_client.py # 共享 httpx 客户端
│ └── retry.py # 重试工具
├── alembic/versions/ # 0001~0006(0001 建表 → 0006 漂移清理)
├── frontend/src/pages/Matches.tsx # 单页(预测面板 + 专家报告折叠区 + 游标分页)
├── tests/ # 核心 + agent 测试
├── docker-compose.yml # api + postgres 两容器
└── docs/ # 本文档
```
+4 -4
View File
@@ -87,17 +87,17 @@ Base URL: `http://localhost:8000` · 交互式文档: `/docs`(Swagger)与 `/redo
"pred_home_goals": 2.1,
"pred_away_goals": 1.0,
"pred_1x2": "1",
"confidence": 0.68,
"subjective_confidence": 0.68,
"reasoning": "综合 xg 报告的进球期望 2.1-1.0 与 form 报告的三连胜势头……",
"agent_outputs": [
{"agent": "h2h", "status": "ok", "data_sufficiency": "medium",
"analysis": "近 5 次交锋主队 3 胜……", "home_edge": 0.4,
"confidence": 0.7, "key_evidence": ["近5次交锋主队3胜", "主场交锋3连胜"],
"subjective_confidence": 0.7, "key_evidence": ["近5次交锋主队3胜", "主场交锋3连胜"],
"exp_home_goals": null, "exp_away_goals": null, "probable_score": null,
"model": "gpt-4o-mini", "latency_ms": 2100,
"prompt_tokens": 380, "completion_tokens": 120},
{"agent": "injuries", "status": "no_data", "data_sufficiency": "none",
"analysis": "该维度无数据,跳过分析。", "home_edge": null, "confidence": null,
"analysis": "该维度无数据,跳过分析。", "home_edge": null, "subjective_confidence": null,
"key_evidence": [], "exp_home_goals": null, "exp_away_goals": null,
"probable_score": null, "model": "", "latency_ms": null,
"prompt_tokens": null, "completion_tokens": null}
@@ -172,7 +172,7 @@ Base URL: `http://localhost:8000` · 交互式文档: `/docs`(Swagger)与 `/redo
```json
{"summary": [
{"provider": "openai", "model": "gpt-4o", "total": 12,
"accuracy_1x2": 58.3, "avg_score_rmse": 1.21, "avg_confidence": 0.65}
"accuracy_1x2": 58.3, "avg_score_rmse": 1.21, "avg_subjective_confidence": 0.65}
]}
```
+4 -4
View File
@@ -65,7 +65,7 @@ POST /predict {match_id, mode: "multi"}
"data_sufficiency": "high",
"analysis": "近 5 次交锋主队 3 胜 1 平 1 负,主场交锋 3 连胜……",
"home_edge": 0.4,
"confidence": 0.7,
"subjective_confidence": 0.7,
"key_evidence": ["近5次交锋主队3胜", "主场交锋3连胜"]
}
```
@@ -75,7 +75,7 @@ POST /predict {match_id, mode: "multi"}
| `status` | `ok` / `no_data` / `error` / `parse_error` |
| `data_sufficiency` | `high` / `medium` / `low` / `none` |
| `home_edge` | -1.0 ~ 1.0,正数=利主队,负数=利客队 |
| `confidence` | 0.0 ~ 1.0,该专家对自己分析的信心 |
| `subjective_confidence` | 0.0 ~ 1.0,该专家对自己分析的主观信心(非概率) |
| `key_evidence` | 关键证据列表(最多 5 条) |
### 终裁 Agent 输出
@@ -87,13 +87,13 @@ POST /predict {match_id, mode: "multi"}
"pred_home_goals": 2.1,
"pred_away_goals": 1.0,
"1x2": "1",
"confidence": 0.68,
"subjective_confidence": 0.68,
"reasoning": "综合 stats 报告的攻防强度与 form 报告的三连胜势头……",
"agent_weights": {"form": 0.9, "stats": 0.8, "home_away": 0.7, "injuries": 0.0, "h2h": 0.8}
}
```
`agent_weights` 体现终裁对各专家报告的采信度(0–1),可用于后续分析"哪个维度对预测贡献大"。
`agent_weights` 体现终裁对各专家报告的采信度(0–1),可用于后续分析"哪个维度对预测贡献大"。注意:这是 LLM 主观权重,非统计权重。
## 模型分档配置
+7 -2
View File
@@ -59,7 +59,7 @@
## 数据库 Schema
5 张表:
6 张表:
```sql
-- 联赛
@@ -124,7 +124,12 @@ CREATE TABLE predictions (
latency_ms INT,
pred_home_goals FLOAT, pred_away_goals FLOAT,
pred_1x2 VARCHAR(3), -- '1' / 'X' / '2'
confidence FLOAT,
subjective_confidence FLOAT, -- LLM 主观置信度(非概率)
status VARCHAR(20) NOT NULL DEFAULT 'success', -- 'success' / 'failed' / 'degraded'
match_kickoff_at TIMESTAMPTZ, -- 比赛时间
prediction_created_at TIMESTAMPTZ, -- 预测创建时间
prediction_cutoff_at TIMESTAMPTZ, -- 数据截止时间
input_hash VARCHAR(64), -- 输入快照 hash
reasoning TEXT,
raw_response JSONB, -- LLM 完整原始响应
agent_outputs JSONB, -- multi 模式: 5 份专家报告
+1 -1
View File
@@ -128,7 +128,7 @@ alembic revision -m "描述"
```
已有迁移:
- `0001_initial`: 初始 5 张表
- `0001_initial`: 初始 5 张表,0003 增加 injuries,0004 增加约束,0005 增加时间语义
- `0002_agent_outputs`: predictions 加 `mode` + `agent_outputs`
## 备份与恢复
+19 -6
View File
@@ -35,27 +35,34 @@ Profeto/
├── src/ # 后端源码
│ ├── api/
│ │ ├── routes/ # FastAPI 路由
│ │ │ ├── leagues.py # 联赛/比赛查询
│ │ │ ├── matches.py # 联赛/比赛查询
│ │ │ ├── predict.py # 预测入口
│ │ │ ├── ingest.py # 数据采集
│ │ │ ── eval.py # 评估回填
│ │ │ ── eval.py # 评估回填
│ │ │ └── backtest.py # 历史回测
│ │ ├── deps.py # 依赖:管理接口鉴权(X-API-Key)
│ │ ├── schemas.py # Pydantic 模型
│ │ └── app.py # FastAPI 工厂
│ ├── db/
│ │ ├── base.py # SQLAlchemy async engine + session
│ │ ── models.py # 5 张表 ORM
│ │ ── models.py # 6 张表 ORM
│ │ ├── repositories.py # 仓储层(查询封装)
│ │ └── unit_of_work.py # 事务边界
│ ├── data/
│ │ ├── bzzoiro.py # bzzoiro 采集 + 入库
│ │ ├── understat.py # understat xG 回填
│ │ ├── injuries.py # 伤停采集
│ │ ├── normalize.py # 数据清洗契约
│ │ ├── team_names.py # 队名归一化映射
│ │ ├── sources.py # 数据源注册表
│ │ └── config.py # 联赛映射常量
│ ├── llm/
│ │ ├── provider.py # LLM 提供商抽象(OpenAI-compatible)
│ │ ├── context_builder.py # 数据切片 + 拼接
│ │ ├── predict.py # 预测入口(单/多模式分派)
│ │ ├── eval.py # 评估统计
│ │ ├── validation.py # LLM 输出严格校验
│ │ ├── backtest.py # 回测执行
│ │ ├── agents/
│ │ │ ├── base.py # AgentSpec + run_agent
│ │ │ └── orchestrator.py # 多 agent 编排
@@ -69,12 +76,18 @@ Profeto/
│ │ ├── h2h_v1.md
│ │ └── aggregator_v1.md
│ └── core/
── config.py # pydantic-settings 配置
── config.py # pydantic-settings 配置
│ ├── http_client.py # 共享 httpx 客户端
│ └── retry.py # 重试工具
├── frontend/ # React 单页前端
├── alembic/ # 数据库迁移
│ └── versions/
│ ├── 0001_initial.py
── 0002_agent_outputs.py
── 0002_agent_outputs.py
│ ├── 0003_injuries.py
│ ├── 0004_snapshot_and_constraints.py
│ ├── 0005_prediction_status_and_stats_provenance.py
│ └── 0006_schema_model_drift_cleanup.py
├── tests/ # 测试
│ ├── test_core.py # 核心逻辑测试
│ └── test_agents.py # 多 agent 测试
@@ -118,7 +131,7 @@ class MockProvider:
return LLMResponse(
content="{}",
parsed={"data_sufficiency": "high", "analysis": "ok",
"home_edge": 0.5, "confidence": 0.8,
"home_edge": 0.5, "subjective_confidence": 0.8,
"key_evidence": ["证据"]},
prompt_tokens=10, completion_tokens=5, latency_ms=100,
)
+1 -1
View File
@@ -6,7 +6,7 @@
| [02-快速开始](02-quickstart.md) | 安装、启动、首次跑通全流程 |
| [03-API 参考](03-api.md) | 全部 12 个端点、请求/响应示例、curl 全流程 |
| [04-多 Agent 预测](04-agents.md) | 5 专家 + 终裁架构、执行语义、输出契约、prompt 版本化、如何新增 agent |
| [05-数据层与数据库](05-data.md) | 三数据源、清洗契约、5 张表 schema、入库语义、采集建议 |
| [05-数据层与数据库](05-data.md) | 三数据源、清洗契约、6 张表 schema、入库语义、采集建议 |
| [06-部署](06-deployment.md) | Docker Compose、本地部署、环境变量、LLM 提供商配置、迁移、备份 |
| [07-开发指南](07-development.md) | 项目结构、测试、常见开发任务(prompt/agent/数据源/联赛)、前端开发 |
+4 -3
View File
@@ -1,5 +1,6 @@
export default {
content: ['./index.html', './src/**/*.{ts,tsx}'],
theme: { extend: {} },
plugins: [],
plugins: {
tailwindcss: {},
autoprefixer: {},
},
}
+42 -5
View File
@@ -1,17 +1,54 @@
import { ErrorBoundary } from './components/ErrorBoundary'
import Matches from './pages/Matches'
/** 报眉日期行:2026年9月15日 星期二 */
function dateLine(): string {
return new Date().toLocaleDateString('zh-CN', {
year: 'numeric',
month: 'long',
day: 'numeric',
weekday: 'long',
})
}
export default function App() {
return (
<ErrorBoundary>
<div className="min-h-screen bg-gray-50">
<header className="bg-white border-b px-6 py-3 flex items-center justify-between">
<h1 className="text-xl font-bold text-blue-700"> Profeto</h1>
<span className="text-sm text-gray-500"> LLM </span>
<div className="min-h-screen bg-paper-50">
{/* ── 报头:粗线 + 居中刊名 + 报眉 ── */}
<header className="masthead-rule">
<div className="mx-auto max-w-5xl px-5 sm:px-8">
<div className="border-b border-ink-900 py-5 text-center sm:py-6">
<h1 className="font-serif text-4xl font-bold tracking-widest text-ink-900">
<span className="ml-3 align-baseline font-serif text-base font-normal italic tracking-normal text-ink-500">
Profeto
</span>
</h1>
<p className="mt-2 text-2xs tracking-[0.4em] text-ink-500">
· ·
</p>
</div>
<div className="flex items-center justify-between border-b border-ink-200 py-2 text-2xs text-ink-500">
<span>{dateLine()}</span>
<span className="flex items-center gap-1.5">
<span className="inline-block h-1.5 w-1.5 bg-emerald-600" aria-hidden="true" />
</span>
</div>
</div>
</header>
<main className="max-w-5xl mx-auto p-6">
<main className="mx-auto max-w-5xl px-5 py-6 sm:px-8 sm:py-8">
<Matches />
</main>
{/* ── 版底 ── */}
<footer className="mx-auto max-w-5xl px-5 pb-10 sm:px-8">
<div className="border-t border-ink-200 pt-3 text-center text-2xs leading-relaxed text-ink-400">
· ·
</div>
</footer>
</div>
</ErrorBoundary>
)
+99
View File
@@ -1,3 +1,102 @@
@tailwind base;
@tailwind components;
@tailwind utilities;
@layer base {
html {
-webkit-text-size-adjust: 100%;
text-rendering: optimizeLegibility;
}
body {
@apply bg-paper-50 text-ink-800 font-sans antialiased;
font-feature-settings: 'tnum' 1; /* 数字等宽:比分/百分比不跳动 */
}
::selection {
@apply bg-press-wash text-press;
}
/* 统一焦点环:印报红细线,键盘可达 */
:focus-visible {
outline: 2px solid #9e1b1b;
outline-offset: 2px;
}
/* 细滚动条 */
::-webkit-scrollbar {
width: 10px;
height: 10px;
}
::-webkit-scrollbar-track {
background: transparent;
}
::-webkit-scrollbar-thumb {
@apply rounded-full bg-ink-300;
}
::-webkit-scrollbar-thumb:hover {
@apply bg-ink-400;
}
/* 尊重「减少动态效果」偏好 */
@media (prefers-reduced-motion: reduce) {
*,
*::before,
*::after {
animation-duration: 0.01ms !important;
animation-iteration-count: 1 !important;
transition-duration: 0.01ms !important;
}
}
}
@layer components {
/* ── 报头双线:粗线在上、细线在下 ── */
.masthead-rule {
border-top: 3px solid #17140f;
}
/* ── 按钮:方正边框式,悬停反白 ── */
.btn {
@apply inline-flex items-center justify-center gap-1.5 border border-ink-300 bg-transparent px-3 py-1.5
text-sm text-ink-700 transition-colors duration-150
hover:border-ink-900 hover:bg-ink-900 hover:text-paper-50
disabled:cursor-not-allowed disabled:opacity-40
disabled:hover:border-ink-300 disabled:hover:bg-transparent disabled:hover:text-ink-700;
}
.btn-sm {
@apply px-2.5 py-1 text-xs;
}
.btn-solid {
@apply border-ink-900 bg-ink-900 text-paper-50 hover:border-press hover:bg-press;
}
/* ── 表单控件:方正、无圆角 ── */
.field {
@apply border border-ink-300 bg-transparent px-2.5 py-1.5 text-sm text-ink-800
transition-colors hover:border-ink-400
focus:border-press focus:outline-none;
}
/* ── 版面切换文字标签(联赛/状态/模式) ── */
.tab {
@apply relative whitespace-nowrap px-0.5 py-1 text-sm text-ink-500 transition-colors hover:text-ink-900;
}
.tab-on {
@apply font-medium text-press;
}
.tab-on::after {
content: '';
@apply absolute inset-x-0 bottom-0 h-0.5 bg-press;
}
/* ── 小节标题:宋体加粗 + 墨色底线 ── */
.section-head {
@apply border-b border-ink-900 pb-1.5 font-serif text-sm font-bold text-ink-900;
}
/* ── 骨架占位:低调脉动,不用渐变扫光 ── */
.skeleton {
@apply animate-pulse rounded-none bg-ink-200;
}
}
+549 -181
View File
@@ -1,4 +1,4 @@
import { useCallback, useEffect, useState } from 'react'
import { useCallback, useEffect, useRef, useState } from 'react'
interface Match {
id: number
@@ -26,7 +26,7 @@ interface Prediction {
pred_home_goals: number | null
pred_away_goals: number | null
pred_1x2: string | null
confidence: number | null
subjective_confidence: number | null
reasoning: string | null
agent_outputs: AgentReport[] | null
agent_weights: Record<string, number> | null
@@ -40,7 +40,7 @@ interface AgentReport {
data_sufficiency: string
analysis: string
home_edge: number | null
confidence: number | null
subjective_confidence: number | null
key_evidence: string[]
exp_home_goals: number | null
exp_away_goals: number | null
@@ -65,54 +65,207 @@ const LEAGUES = [
{ code: 'F1', name: '法甲' },
]
/** 汉字编号,给专家意见排版用 */
const CN_NUM = ['一', '二', '三', '四', '五', '六', '七', '八']
const STATUS_META: Record<string, { label: string; cls: string }> = {
finished: { label: '已完赛', cls: 'text-ink-400' },
scheduled: { label: '未开赛', cls: 'text-ink-600' },
live: { label: '进行中', cls: 'text-press font-medium' },
}
/** 1x2 → 中文标签 */
const OUTCOME_LABEL: Record<string, string> = { '1': '主胜', X: '平局', '2': '客胜' }
/** 置信度细线:0~1 数值的低调可视化 */
function Meter({ value }: { value: number }) {
const pct = Math.max(0, Math.min(100, Math.round(value * 100)))
return (
<div className="h-px w-full bg-ink-200" role="presentation">
<div className="h-px bg-press transition-[width] duration-500" style={{ width: `${pct}%` }} />
</div>
)
}
/** home_edge(-1~1,正=利主队)的可视化:以中线为原点的双向细条 */
function EdgeBar({ value }: { value: number }) {
const v = Math.max(-1, Math.min(1, value))
const half = Math.abs(v) * 50
return (
<div className="relative h-px w-full bg-ink-200" role="presentation">
<span className="absolute left-1/2 top-1/2 h-2 w-px -translate-x-1/2 -translate-y-1/2 bg-ink-400" />
<span
className={`absolute top-0 h-px transition-all duration-500 ${v >= 0 ? 'bg-press' : 'bg-ink-600'}`}
style={
v >= 0
? { left: '50%', width: `${half}%` }
: { right: '50%', width: `${half}%` }
}
/>
</div>
)
}
/** 骨架占位行:低调脉动灰块 */
function SkeletonRows({ n = 4 }: { n?: number }) {
return (
<>
{Array.from({ length: n }).map((_, i) => (
<div key={i} className="flex items-center gap-4 border-b border-ink-200 px-1 py-3.5">
<div className="skeleton h-3 w-16" />
<div className="skeleton h-3 flex-1" />
<div className="skeleton h-3 w-10" />
<div className="skeleton h-3 flex-1" />
<div className="skeleton h-3 w-16" />
</div>
))}
</>
)
}
function Spinner({ className = '' }: { className?: string }) {
return (
<svg
viewBox="0 0 20 20"
className={`h-3.5 w-3.5 animate-spin ${className}`}
fill="none"
aria-hidden="true"
>
<circle cx="10" cy="10" r="7.5" stroke="currentColor" strokeWidth="1.5" strokeOpacity="0.25" />
<path d="M17.5 10A7.5 7.5 0 0010 2.5" stroke="currentColor" strokeWidth="1.5" strokeLinecap="round" />
</svg>
)
}
/** 胜平负一行文字:选中的红字加方块标记,未选中的退灰 */
function OutcomeLine({
pick,
confidence,
}: {
pick: string | null
confidence: number | null
}) {
const options = ['1', 'X', '2'] as const
return (
<div>
<div className="flex items-baseline justify-center gap-6 sm:gap-10">
{options.map(o => {
const on = pick === o
return (
<div key={o} className="flex flex-col items-center gap-1">
<span className={`flex items-center gap-1.5 text-sm ${on ? 'font-semibold text-press' : 'text-ink-400'}`}>
{on && <span className="inline-block h-2 w-2 bg-press" aria-hidden="true" />}
{OUTCOME_LABEL[o]}
</span>
{on && confidence !== null && (
<span className="text-2xs tabular-nums text-ink-500">
{Math.round(confidence * 100)}%
</span>
)}
</div>
)
})}
</div>
{pick && confidence !== null && (
<div className="mx-auto mt-3 max-w-xs">
<Meter value={confidence} />
<p className="mt-1 text-center text-2xs text-ink-400">,</p>
</div>
)}
</div>
)
}
export default function Matches() {
const [league, setLeague] = useState('E0')
const [status, setStatus] = useState('scheduled')
const [matches, setMatches] = useState<Match[]>([])
const [nextCursor, setNextCursor] = useState<string | null>(null)
const [loadingMore, setLoadingMore] = useState(false)
const [loading, setLoading] = useState(false)
const [predictingId, setPredictingId] = useState<number | null>(null)
const [prediction, setPrediction] = useState<Prediction | null>(null)
const [predictionFor, setPredictionFor] = useState<Match | null>(null)
const [error, setError] = useState<string | null>(null)
const [mode, setMode] = useState<'single' | 'multi'>('multi')
// 请求竞态防护:切换联赛/状态很快时,先发的慢请求可能后返回,
// 把旧结果覆盖到新筛选上。用递增序号只认最后一次请求的响应。
const loadSeq = useRef(0)
const predictSeq = useRef(0)
const load = useCallback(async () => {
const seq = ++loadSeq.current
setLoading(true)
// 切换筛选时作废进行中的「加载更多」,避免其标志位卡住
setLoadingMore(false)
setError(null)
try {
const params = new URLSearchParams({ league, status, limit: '50' })
const res = await fetch(`/api/v1/matches?${params}`)
if (seq !== loadSeq.current) return // 已有更新的请求,丢弃本次结果
if (!res.ok) throw new Error(`HTTP ${res.status}`)
const data = await res.json()
if (seq !== loadSeq.current) return
setMatches(data.items)
setNextCursor(data.next_cursor ?? null)
} catch (e) {
if (seq !== loadSeq.current) return
setError(e instanceof Error ? e.message : String(e))
} finally {
setLoading(false)
if (seq === loadSeq.current) setLoading(false)
}
}, [league, status])
// 加载下一页(游标分页)
const loadMore = async () => {
if (!nextCursor || loadingMore) return
const seq = loadSeq.current // 不做自增:切换筛选会自增,这里只跟随当前序列
setLoadingMore(true)
try {
const params = new URLSearchParams({ league, status, limit: '50', cursor: nextCursor })
const res = await fetch(`/api/v1/matches?${params}`)
if (seq !== loadSeq.current) return
if (!res.ok) throw new Error(`HTTP ${res.status}`)
const data = await res.json()
if (seq !== loadSeq.current) return
setMatches(prev => [...prev, ...data.items])
setNextCursor(data.next_cursor ?? null)
} catch (e) {
if (seq !== loadSeq.current) return
setError(e instanceof Error ? e.message : String(e))
} finally {
if (seq === loadSeq.current) setLoadingMore(false)
}
}
useEffect(() => { load() }, [load])
const predict = async (matchId: number) => {
setPredictingId(matchId)
const predict = async (m: Match) => {
const seq = ++predictSeq.current
setPredictingId(m.id)
setError(null)
setPrediction(null)
setPredictionFor(m)
try {
const res = await fetch('/api/v1/predict', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ match_id: matchId, mode }),
body: JSON.stringify({ match_id: m.id, mode }),
})
if (seq !== predictSeq.current) return
if (!res.ok) {
const t = await res.text()
throw new Error(`HTTP ${res.status}: ${t}`)
}
const data = await res.json()
if (seq !== predictSeq.current) return
setPrediction(data)
} catch (e) {
if (seq !== predictSeq.current) return
setError(e instanceof Error ? e.message : String(e))
} finally {
setPredictingId(null)
if (seq === predictSeq.current) setPredictingId(null)
}
}
@@ -121,195 +274,410 @@ export default function Matches() {
return d.toLocaleString('zh-CN', { month: '2-digit', day: '2-digit', hour: '2-digit', minute: '2-digit' })
}
const leagueName = LEAGUES.find(l => l.code === league)?.name ?? league
/** 状态/模式一组的文字切换 */
const Switch = ({ value, onChange, items }: {
value: string
onChange: (v: string) => void
items: { v: string; label: string; title?: string }[]
}) => (
<span className="inline-flex items-center gap-2.5">
{items.map((it, i) => (
<span key={it.v} className="inline-flex items-center gap-2.5">
{i > 0 && <span className="text-ink-300" aria-hidden="true">/</span>}
<button
onClick={() => onChange(it.v)}
title={it.title}
className={`relative tab ${value === it.v ? 'tab-on' : ''} text-xs`}
>
{it.label}
</button>
</span>
))}
</span>
)
return (
<div className="space-y-4">
{/* 筛选 */}
<div className="flex gap-3 items-center flex-wrap">
<select value={league} onChange={e => setLeague(e.target.value)}
className="border rounded px-3 py-1.5 text-sm">
{LEAGUES.map(l => <option key={l.code} value={l.code}>{l.name}</option>)}
</select>
<select value={status} onChange={e => setStatus(e.target.value)}
className="border rounded px-3 py-1.5 text-sm">
<option value="scheduled"></option>
<option value="finished"></option>
<option value=""></option>
</select>
<div className="flex items-center gap-2 text-sm">
<span className="text-gray-500">:</span>
<div className="space-y-5">
{/* ── 联赛版面切换 ── */}
<nav className="flex items-center gap-6 overflow-x-auto border-b border-ink-900" aria-label="联赛">
{LEAGUES.map(l => (
<button
onClick={() => setMode('single')}
className={`px-2 py-1 rounded text-xs ${mode === 'single' ? 'bg-blue-600 text-white' : 'bg-gray-100 text-gray-600'}`}
></button>
<button
onClick={() => setMode('multi')}
className={`px-2 py-1 rounded text-xs ${mode === 'multi' ? 'bg-blue-600 text-white' : 'bg-gray-100 text-gray-600'}`}
> Agent</button>
</div>
<button onClick={load} disabled={loading}
className="bg-blue-600 text-white text-sm px-4 py-1.5 rounded hover:bg-blue-700 disabled:opacity-50">
{loading ? '加载中...' : '刷新'}
</button>
<span className="text-sm text-gray-500"> {matches.length} </span>
key={l.code}
onClick={() => setLeague(l.code)}
className={`relative tab ${league === l.code ? 'tab-on' : ''} font-serif`}
>
{l.name}
</button>
))}
</nav>
{/* ── 第二行:状态 / 模式 / 计数 / 刷新 ── */}
<div className="flex flex-wrap items-center gap-x-5 gap-y-2 text-xs text-ink-500">
<span className="inline-flex items-center gap-2.5">
<span className="text-2xs text-ink-400"></span>
<Switch
value={status}
onChange={setStatus}
items={[
{ v: 'scheduled', label: '未开赛' },
{ v: 'finished', label: '已完赛' },
{ v: '', label: '全部' },
]}
/>
</span>
<span className="inline-flex items-center gap-2.5">
<span className="text-2xs text-ink-400"></span>
<Switch
value={mode}
onChange={v => setMode(v as 'single' | 'multi')}
items={[
{ v: 'single', label: '单次', title: '单次调用,快但只有一个模型看全部数据' },
{ v: 'multi', label: '多专家', title: '5 个专家并行分析后由终裁汇总,质量更高' },
]}
/>
</span>
<span className="ml-auto inline-flex items-center gap-3">
<span className="tabular-nums"> {matches.length} </span>
<button onClick={load} disabled={loading} className="btn btn-sm">
{loading ? (<><Spinner /> </>) : '刷新'}
</button>
</span>
</div>
{/* ── 错误提示 ── */}
{error && (
<div className="bg-red-50 border border-red-200 text-red-700 px-4 py-2 rounded text-sm flex items-center justify-between">
<span> {error}</span>
<button onClick={() => setError(null)} className="text-red-500 hover:text-red-700 text-xs"></button>
<div className="flex items-start justify-between gap-3 border border-press bg-press-wash px-4 py-3">
<div>
<p className="text-sm font-medium text-press"></p>
<p className="mt-0.5 text-xs text-ink-600">{error}</p>
</div>
<button onClick={() => setError(null)} className="text-ink-400 transition-colors hover:text-ink-900" aria-label="关闭">
<svg viewBox="0 0 20 20" className="h-4 w-4" fill="currentColor" aria-hidden="true">
<path d="M6.3 5.3a1 1 0 011.4 0L10 7.6l2.3-2.3a1 1 0 111.4 1.4L11.4 9l2.3 2.3a1 1 0 01-1.4 1.4L10 10.4l-2.3 2.3a1 1 0 01-1.4-1.4L8.6 9 6.3 6.7a1 1 0 010-1.4z" />
</svg>
</button>
</div>
)}
{/* 比赛表 */}
<div className="bg-white rounded border overflow-hidden">
<table className="w-full text-sm">
<thead className="bg-gray-100 text-gray-600">
<tr>
<th className="text-left px-4 py-2"></th>
<th className="text-left px-4 py-2"></th>
<th className="text-left px-4 py-2"></th>
<th className="text-center px-4 py-2"></th>
<th className="text-center px-4 py-2"></th>
<th className="text-center px-4 py-2"></th>
</tr>
</thead>
<tbody>
{loading && (
<tr><td colSpan={6} className="text-center text-gray-400 py-8">
<span className="inline-block animate-spin mr-2"></span>...
</td></tr>
)}
{!loading && matches.length === 0 && (
<tr><td colSpan={6} className="text-center text-gray-400 py-8">,</td></tr>
)}
{matches.map(m => (
<tr key={m.id} className="border-t hover:bg-gray-50">
<td className="px-4 py-2 text-gray-600">{fmtDate(m.match_date)}</td>
<td className="px-4 py-2 font-medium">{m.home_team_zh || m.home_team}</td>
<td className="px-4 py-2 font-medium">{m.away_team_zh || m.away_team}</td>
<td className="px-4 py-2 text-center">
{m.home_goals !== null ? `${m.home_goals} - ${m.away_goals}` : '-'}
</td>
<td className="px-4 py-2 text-center">
<span className={`text-xs px-2 py-0.5 rounded ${
m.match_status === 'finished' ? 'bg-green-100 text-green-700' :
m.match_status === 'scheduled' ? 'bg-blue-100 text-blue-700' : 'bg-gray-100'
}`}>
{m.match_status === 'finished' ? '完赛' : m.match_status === 'scheduled' ? '未开赛' : m.match_status}
</span>
</td>
<td className="px-4 py-2 text-center">
<button onClick={() => predict(m.id)}
disabled={predictingId === m.id}
className="text-blue-600 hover:underline text-xs disabled:opacity-50">
{predictingId === m.id ? '预测中...' : 'LLM 预测'}
{/* ── 赛程栏:表格化,行间细线 ── */}
<section aria-label="赛程">
{loading && <SkeletonRows n={4} />}
{!loading && matches.length === 0 && (
<div className="border-y border-ink-200 py-14 text-center">
<p className="font-serif text-sm text-ink-600"></p>
<p className="mt-1.5 text-xs text-ink-400"> {leagueName} </p>
</div>
)}
{!loading && matches.map(m => {
const st = STATUS_META[m.match_status] ?? { label: m.match_status, cls: 'text-ink-400' }
const homeName = m.home_team_zh || m.home_team
const awayName = m.away_team_zh || m.away_team
const busy = predictingId === m.id
const active = predictionFor?.id === m.id
return (
<div
key={m.id}
className={`border-b border-ink-200 px-1 py-3 transition-colors hover:bg-paper-100 ${
active ? 'bg-press-wash/50' : ''
}`}
>
<div className="flex flex-col gap-2 sm:grid sm:grid-cols-[88px_minmax(0,1fr)_64px_minmax(0,1fr)_56px_auto] sm:items-center sm:gap-x-3 sm:gap-y-0">
{/* 日期 + 状态:移动端同行,桌面端日期单独归列 */}
<div className="flex items-center justify-between sm:contents">
<span className="text-2xs tabular-nums text-ink-400">{fmtDate(m.match_date)}</span>
<span className={`text-2xs sm:hidden ${st.cls}`}>{st.label}</span>
</div>
{/* 对阵:移动端主队/比分/客队同一行,桌面端 sm:contents 拆回 grid 列 */}
<div className="flex items-center gap-2 sm:contents">
{/* 主队(右对齐) */}
<div className="flex min-w-0 flex-1 items-center justify-end">
<span className="truncate text-sm font-medium text-ink-900">{homeName}</span>
</div>
{/* 比分 / VS */}
<div className="flex w-14 flex-shrink-0 flex-col items-center sm:w-auto">
{m.home_goals !== null && m.away_goals !== null ? (
<span className="font-serif text-base font-bold tabular-nums text-ink-900">
{m.home_goals}<span className="mx-0.5 font-normal text-ink-300">:</span>{m.away_goals}
</span>
) : (
<span className="text-2xs tracking-widest text-ink-400">VS</span>
)}
{m.home_xg !== null && m.away_xg !== null && (
<span className="text-2xs tabular-nums text-ink-400">
xG {m.home_xg.toFixed(1)}-{m.away_xg.toFixed(1)}
</span>
)}
</div>
{/* 客队(左对齐) */}
<div className="flex min-w-0 flex-1 items-center">
<span className="truncate text-sm font-medium text-ink-900">{awayName}</span>
</div>
</div>
{/* 状态列(桌面) */}
<span className={`hidden text-right text-2xs sm:block ${st.cls}`}>{st.label}</span>
{/* 预测按钮 */}
<div className="flex justify-end">
<button
onClick={() => predict(m)}
disabled={busy}
className="btn btn-sm w-[76px]"
title={`${mode === 'multi' ? '多专家' : '单次'}模式预测这场`}
>
{busy ? (<><Spinner /> </>) : '预测'}
</button>
</td>
</tr>
))}
</tbody>
</table>
</div>
{/* 预测中指示 */}
{predictingId && (
<div className="bg-blue-50 border border-blue-200 text-blue-700 px-4 py-3 rounded text-sm flex items-center gap-2">
<span className="inline-block animate-spin"></span>
LLM ,...
</div>
)}
{/* 预测结果 */}
{prediction && (
<div className="bg-white rounded border p-5 space-y-3">
<h3 className="font-bold text-lg">🤖 LLM </h3>
<div className="grid grid-cols-2 md:grid-cols-4 gap-4 text-sm">
<div className="bg-blue-50 rounded p-3">
<div className="text-gray-500 text-xs"></div>
<div className="text-xl font-bold">{prediction.pred_home_goals ?? '-'}</div>
</div>
<div className="bg-blue-50 rounded p-3">
<div className="text-gray-500 text-xs"></div>
<div className="text-xl font-bold">{prediction.pred_away_goals ?? '-'}</div>
</div>
<div className="bg-amber-50 rounded p-3">
<div className="text-gray-500 text-xs"></div>
<div className="text-xl font-bold">{prediction.pred_1x2 ?? '-'}</div>
</div>
<div className="bg-green-50 rounded p-3">
<div className="text-gray-500 text-xs"></div>
<div className="text-xl font-bold">
{prediction.confidence !== null ? `${(prediction.confidence * 100).toFixed(0)}%` : '-'}
</div>
</div>
</div>
</div>
<div className="text-xs text-gray-400">
{prediction.provider} / {prediction.model} · {prediction.latency_ms}ms
{prediction.mode === 'multi' && ' · 多 Agent 模式'}
</div>
)
})}
{/* 各专家 agent 报告 */}
{prediction.agent_outputs && prediction.agent_outputs.length > 0 && (
<div className="space-y-2">
<div className="text-sm font-medium text-gray-700"> Agent </div>
{prediction.agent_outputs.map((r) => (
<details key={r.agent} className="bg-white border rounded">
<summary className="cursor-pointer px-3 py-2 text-sm flex items-center justify-between">
<span className="font-medium">
{AGENT_LABELS[r.agent] || r.agent}
{r.status !== 'ok' && (
<span className={`ml-2 text-xs px-1.5 py-0.5 rounded ${
r.status === 'no_data' ? 'bg-gray-100 text-gray-500' : 'bg-red-100 text-red-600'
}`}>
{r.status === 'no_data' ? '无数据' : '失败'}
</span>
)}
</span>
<span className="flex gap-3 text-xs text-gray-500">
{r.home_edge !== null && (
<span className={r.home_edge > 0 ? 'text-blue-600' : r.home_edge < 0 ? 'text-amber-600' : ''}>
{r.home_edge > 0 ? '+' : ''}{r.home_edge.toFixed(2)}
</span>
)}
{r.confidence !== null && <span> {(r.confidence * 100).toFixed(0)}%</span>}
{r.probable_score && <span> {r.probable_score}</span>}
</span>
</summary>
<div className="px-3 pb-3 pt-1 space-y-2 text-sm">
{r.analysis && <p className="text-gray-700">{r.analysis}</p>}
{r.key_evidence.length > 0 && (
<ul className="text-xs text-gray-500 list-disc pl-4">
{r.key_evidence.map((e, i) => <li key={i}>{e}</li>)}
</ul>
)}
{r.exp_home_goals !== null && r.exp_away_goals !== null && (
<div className="text-xs text-gray-500">
: {r.exp_home_goals.toFixed(1)} - {r.exp_away_goals.toFixed(1)}
</div>
)}
<div className="text-xs text-gray-400">
{r.data_sufficiency} · {r.model} · {r.latency_ms}ms
</div>
</div>
</details>
))}
</div>
)}
{!loading && nextCursor && (
<div className="flex justify-center pt-4">
<button onClick={loadMore} disabled={loadingMore} className="btn btn-sm">
{loadingMore ? (<><Spinner /> </>) : '载入更多'}
</button>
</div>
)}
</section>
{prediction.reasoning && (
<div className="bg-gray-50 rounded p-3">
<div className="text-xs text-gray-500 mb-1"></div>
<div className="text-sm whitespace-pre-wrap">{prediction.reasoning}</div>
{/* ── 预测中占位 ── */}
{predictingId && !prediction && (
<div className="border border-ink-900">
<div className="flex items-center gap-2 border-b border-ink-900 bg-paper-100 px-4 py-2.5">
<Spinner className="text-press" />
<span className="text-sm font-medium text-ink-800"></span>
<span className="text-2xs text-ink-500">
{mode === 'multi' ? '五路专家并行分析后终裁,约需 20-60 秒' : '单次调用,约需 5-15 秒'}
</span>
</div>
<div className="space-y-4 px-4 py-6">
<div className="flex items-center justify-center gap-6">
<div className="skeleton h-4 w-20" />
<div className="skeleton h-10 w-24" />
<div className="skeleton h-4 w-20" />
</div>
)}
<details className="text-xs">
<summary className="cursor-pointer text-gray-500 hover:text-gray-700"></summary>
<pre className="mt-2 bg-gray-900 text-green-300 p-3 rounded overflow-x-auto text-xs">
{prediction.context}
</pre>
</details>
<div className="skeleton mx-auto h-px w-64" />
<div className="skeleton h-16 w-full" />
</div>
</div>
)}
{/* ── 预测版 ── */}
{prediction && predictionFor && (
<PredictionPanel prediction={prediction} match={predictionFor} mode={mode} />
)}
</div>
)
}
function PredictionPanel({
prediction,
match,
mode,
}: {
prediction: Prediction
match: Match
mode: 'single' | 'multi'
}) {
const homeName = match.home_team_zh || match.home_team
const awayName = match.away_team_zh || match.away_team
const okReports = (prediction.agent_outputs ?? []).filter(r => r.status === 'ok')
return (
<article className="border border-ink-900 bg-paper-50">
{/* 版头 */}
<div className="flex flex-wrap items-baseline justify-between gap-2 border-b border-ink-900 bg-paper-100 px-4 py-2.5 sm:px-5">
<h3 className="font-serif text-sm font-bold text-ink-900">
· {homeName} {awayName}
</h3>
<span className="text-2xs tabular-nums text-ink-500">
{prediction.provider} / {prediction.model}
{prediction.latency_ms !== null && ` · ${(prediction.latency_ms / 1000).toFixed(1)}s`}
</span>
</div>
<div className="space-y-7 px-4 py-6 sm:px-5">
{/* ── 预测比分:版面核心,大号宋体 ── */}
<div className="text-center">
<p className="font-serif text-5xl font-bold tabular-nums leading-none text-ink-900 sm:text-6xl">
{prediction.pred_home_goals ?? '-'}
<span className="mx-3 font-normal text-ink-300">:</span>
{prediction.pred_away_goals ?? '-'}
</p>
<p className="mt-3 text-2xs tracking-[0.5em] text-ink-400"></p>
</div>
{/* ── 胜平负 ── */}
<div className="border-y border-ink-200 py-4">
<OutcomeLine pick={prediction.pred_1x2} confidence={prediction.subjective_confidence} />
</div>
{/* ── 元信息一行 ── */}
<p className="text-center text-2xs text-ink-500">
{mode === 'multi' ? `多专家模式 · ${okReports.length}/${prediction.agent_outputs?.length ?? 0} 路有效` : '单次模式'}
{prediction.prompt_version && ` · prompt ${prediction.prompt_version}`}
{prediction.latency_ms !== null && ` · 终裁耗时 ${(prediction.latency_ms / 1000).toFixed(1)}`}
</p>
{/* ── 专家意见 ── */}
{prediction.agent_outputs && prediction.agent_outputs.length > 0 && (
<section>
<div className="section-head flex flex-wrap items-baseline justify-between gap-1">
<span></span>
{prediction.agent_weights && (
<span className="font-sans text-2xs font-normal text-ink-500">
:{Object.entries(prediction.agent_weights)
.sort((a, b) => b[1] - a[1])
.map(([k, v]) => `${AGENT_LABELS[k] ?? k} ${Math.round(v * 100)}%`)
.join(' / ')}
</span>
)}
</div>
<div>
{prediction.agent_outputs.map((r, i) => (
<AgentCard key={r.agent} report={r} no={CN_NUM[i] ?? String(i + 1)} />
))}
</div>
</section>
)}
{/* ── 终裁意见:引文式,红竖线 ── */}
{prediction.reasoning && (
<section>
<h4 className="section-head mb-3"></h4>
<blockquote className="border-l-2 border-press pl-4">
<p className="whitespace-pre-wrap font-serif text-sm leading-loose text-ink-700">
{prediction.reasoning}
</p>
</blockquote>
</section>
)}
{/* ── 原始上下文 ── */}
<details className="group">
<summary className="flex cursor-pointer list-none items-center gap-1.5 text-xs text-ink-500 transition-colors hover:text-ink-800">
<svg viewBox="0 0 20 20" className="h-3 w-3 transition-transform group-open:rotate-90" fill="currentColor" aria-hidden="true">
<path d="M7.3 5.3a1 1 0 011.4 0l4 4a1 1 0 010 1.4l-4 4a1 1 0 01-1.4-1.4L10.6 10 7.3 6.7a1 1 0 010-1.4z" />
</svg>
</summary>
<pre className="mt-2 max-h-80 overflow-auto border border-ink-200 bg-paper-100 p-3 font-mono text-2xs leading-relaxed text-ink-600">
{prediction.context}
</pre>
</details>
</div>
</article>
)
}
const STATUS_BADGE: Record<string, { label: string; cls: string }> = {
ok: { label: '正常', cls: 'text-ink-500' },
no_data: { label: '无数据', cls: 'text-ink-400' },
error: { label: '调用失败', cls: 'text-press' },
parse_error: { label: '解析失败', cls: 'text-press' },
}
const SUFFICIENCY_LABEL: Record<string, string> = {
high: '充分',
medium: '一般',
low: '偏少',
none: '无',
}
/** 单路专家意见:汉字编号 + 细线行 */
function AgentCard({ report: r, no }: { report: AgentReport; no: string }) {
const badge = STATUS_BADGE[r.status] ?? { label: r.status, cls: 'text-ink-400' }
const inactive = r.status !== 'ok'
return (
<details className="group border-b border-ink-200">
<summary className="flex cursor-pointer list-none items-baseline gap-2.5 px-1 py-3">
<span className="font-serif text-sm text-ink-400">{no}</span>
<span className="text-sm font-medium text-ink-900">{AGENT_LABELS[r.agent] ?? r.agent}</span>
<span className={`text-2xs ${badge.cls}`}>{badge.label}</span>
<span className="ml-auto flex items-baseline gap-3 text-2xs tabular-nums text-ink-500">
{r.status === 'ok' && r.subjective_confidence !== null && (
<span> {Math.round(r.subjective_confidence * 100)}%</span>
)}
{r.status === 'ok' && r.probable_score && (
<span className="font-serif font-bold text-ink-800">{r.probable_score}</span>
)}
<svg viewBox="0 0 20 20" className="h-3 w-3 self-center text-ink-300 transition-transform group-open:rotate-90" fill="currentColor" aria-hidden="true">
<path d="M7.3 5.3a1 1 0 011.4 0l4 4a1 1 0 010 1.4l-4 4a1 1 0 01-1.4-1.4L10.6 10 7.3 6.7a1 1 0 010-1.4z" />
</svg>
</span>
</summary>
<div className="space-y-3 px-1 pb-4 pl-7">
{/* 无数据 / 失败时给出明确说明,避免用户以为是空白 bug */}
{inactive && (
<p className="text-xs leading-relaxed text-ink-500">
{r.status === 'no_data' && '该维度没有可用数据,已跳过 LLM 分析以节省额度(不影响其他专家)。'}
{r.status === 'error' && '该专家调用失败,本次结论未纳入其视角(fail-open 设计,不阻断整体预测)。'}
{r.status === 'parse_error' && '模型输出未通过格式校验,该报告已丢弃。'}
</p>
)}
{!inactive && r.home_edge !== null && (
<div>
<div className="mb-1.5 flex items-baseline justify-between text-2xs">
<span className="text-ink-500"></span>
<span className={`font-semibold tabular-nums ${r.home_edge > 0 ? 'text-press' : r.home_edge < 0 ? 'text-ink-700' : 'text-ink-500'}`}>
{r.home_edge > 0 ? '+' : ''}{r.home_edge.toFixed(2)}
</span>
</div>
<EdgeBar value={r.home_edge} />
<div className="mt-1 flex justify-between text-2xs text-ink-400">
<span></span>
<span></span>
</div>
</div>
)}
{r.analysis && (
<p className="font-serif text-sm leading-loose text-ink-700">{r.analysis}</p>
)}
{r.key_evidence.length > 0 && (
<ul className="space-y-1.5">
{r.key_evidence.map((e, i) => (
<li key={i} className="flex gap-2 text-xs leading-relaxed text-ink-600">
<span className="flex-shrink-0 text-ink-300" aria-hidden="true"></span>
<span>{e}</span>
</li>
))}
</ul>
)}
{r.exp_home_goals !== null && r.exp_away_goals !== null && (
<p className="text-xs text-ink-500">
<span className="font-serif font-bold tabular-nums text-ink-900">{r.exp_home_goals.toFixed(1)} - {r.exp_away_goals.toFixed(1)}</span>
</p>
)}
{!inactive && (
<p className="border-t border-ink-100 pt-2.5 text-2xs text-ink-400">
{SUFFICIENCY_LABEL[r.data_sufficiency] ?? r.data_sufficiency}
<span className="mx-2 text-ink-200">|</span>
<span className="font-mono">{r.model}</span>
{r.latency_ms !== null && <span className="ml-2 tabular-nums">{r.latency_ms}ms</span>}
</p>
)}
</div>
</details>
)
}
+56 -1
View File
@@ -1,6 +1,61 @@
/** @type {import('tailwindcss').Config} */
export default {
content: ['./index.html', './src/**/*.{ts,tsx}'],
theme: { extend: {} },
theme: {
extend: {
colors: {
// 纸白:微暖底色,像新闻纸而不是纯白画布
paper: {
50: '#FDFCF8',
100: '#F7F4EC',
200: '#EDE9DE',
300: '#DDD7C7',
},
// 墨色:暖黑灰阶,替代冷调 slate
ink: {
50: '#FAF9F7',
100: '#F0EEE9',
200: '#E2DFD7',
300: '#C9C4B8',
400: '#9C9587',
500: '#6E675B',
600: '#524C42',
700: '#3B362E',
800: '#282420',
900: '#17140F',
},
// 印报红:全站唯一强调色,克制使用
press: {
DEFAULT: '#9E1B1B',
dark: '#7C1414',
wash: '#F7E9E4',
},
},
fontFamily: {
// 标题与比分:宋体血统,报纸版面的骨架
serif: [
'Georgia',
'"Times New Roman"',
'"Songti SC"',
'STSong',
'SimSun',
'"Noto Serif CJK SC"',
'serif',
],
// 正文:中文黑体栈
sans: [
'"PingFang SC"',
'"Hiragino Sans GB"',
'"Microsoft YaHei"',
'"Noto Sans CJK SC"',
'sans-serif',
],
mono: ['ui-monospace', 'SFMono-Regular', 'Menlo', 'Consolas', 'monospace'],
},
fontSize: {
'2xs': ['11px', { lineHeight: '16px' }],
},
},
},
plugins: [],
}
+20 -3
View File
@@ -15,7 +15,10 @@ from src.core.config import settings
async def lifespan(app: FastAPI) -> AsyncIterator[None]:
from src.db.base import init_db
from src.core.http_client import close_client
await init_db()
await init_db() # 验证连接,不建表
# P2-5: 启动时执行一次 ingest failure 重试清理
from src.data.retry_worker import run_retry_worker
await run_retry_worker()
yield
await close_client()
@@ -29,12 +32,14 @@ def create_app() -> FastAPI:
)
origins = [o.strip() for o in settings.CORS_ORIGINS.split(",") if o.strip()]
methods = [m.strip() for m in settings.CORS_METHODS.split(",") if m.strip()]
headers = [h.strip() for h in settings.CORS_HEADERS.split(",") if h.strip()]
app.add_middleware(
CORSMiddleware,
allow_origins=origins,
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
allow_methods=methods,
allow_headers=headers,
)
from src.api.routes.matches import router as matches_router
@@ -53,6 +58,18 @@ def create_app() -> FastAPI:
async def health():
return {"status": "healthy", "service": "profeto"}
@app.get("/health/ready")
async def health_ready():
"""就绪检查: 验证数据库连接。"""
from src.db.base import engine
try:
async with engine.begin() as conn:
await conn.run_sync(lambda conn: None)
return {"status": "ready"}
except Exception:
return {"status": "not_ready"}
return app
+37
View File
@@ -0,0 +1,37 @@
"""API 依赖:鉴权等横切关注点。
审查报告 P2-7:ingest / backtest / settle 这类「写入型或高成本」接口此前
完全无鉴权 —— 任何能访问到服务的人都可触发采集、或直接烧掉 LLM 额度。
策略(渐进式,不破坏本地开发):
- `ADMIN_API_KEY` 未配置 → 直接放行,并打一次 warning。
这样本地 `docker compose up` 无需额外配置即可用。
- 已配置 → 必须带匹配的 `X-API-Key` 请求头,否则 401。
"""
from __future__ import annotations
import logging
import secrets
from fastapi import Header, HTTPException
from src.core.config import settings
logger = logging.getLogger(__name__)
async def require_admin_key(x_api_key: str | None = Header(None, alias="X-API-Key")) -> None:
"""保护「写入型 / 高成本」接口的依赖。
用法: `@router.post("/ingest/bzzoiro", dependencies=[Depends(require_admin_key)])`
"""
expected = settings.ADMIN_API_KEY
if not expected:
logger.warning(
"ADMIN_API_KEY 未设置,采集/回测接口当前【无鉴权】。"
"生产环境请设置该环境变量。"
)
return
if not x_api_key or not secrets.compare_digest(x_api_key, expected):
raise HTTPException(status_code=401, detail="无效或缺失的 X-API-Key")
+14 -5
View File
@@ -1,11 +1,16 @@
"""回测路由。"""
from __future__ import annotations
from fastapi import APIRouter, HTTPException
import logging
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel, Field
from src.api.deps import require_admin_key
from src.llm.backtest import run_backtest
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/v1", tags=["backtest"])
@@ -18,10 +23,13 @@ class BacktestRequest(BaseModel):
model: str | None = Field(None, description="指定模型 (空=默认)")
@router.post("/backtest")
@router.post("/backtest", dependencies=[Depends(require_admin_key)])
async def backtest(req: BacktestRequest):
"""对历史比赛运行回测。
该接口会对每场已完赛比赛各发起一次 LLM 预测,成本高 —— 因此需要
`X-API-Key` 鉴权(见审查报告 P2-7)。
对每场已完赛比赛:
1. 用比赛之前的数据构建上下文 (防未来信息泄漏)
2. 调 LLM 预测
@@ -38,7 +46,8 @@ async def backtest(req: BacktestRequest):
model=req.model,
)
except Exception as e:
raise HTTPException(500, f"backtest failed: {e}")
logger.exception("backtest failed")
raise HTTPException(500, "回测执行失败,请查看服务器日志")
return {
"summary": {
@@ -46,7 +55,7 @@ async def backtest(req: BacktestRequest):
"scored": summary.scored,
"accuracy_1x2": summary.accuracy_1x2,
"avg_score_rmse": summary.avg_score_rmse,
"avg_confidence": summary.avg_confidence,
"avg_subjective_confidence": summary.avg_subjective_confidence,
"calibration": summary.calibration,
},
"results": [
@@ -61,7 +70,7 @@ async def backtest(req: BacktestRequest):
"pred_home": r.pred_home,
"pred_away": r.pred_away,
"pred_1x2": r.pred_1x2,
"confidence": r.confidence,
"subjective_confidence": r.subjective_confidence,
"correct_1x2": r.correct_1x2,
}
for r in summary.results
+11 -2
View File
@@ -1,23 +1,32 @@
"""评估路由。"""
from __future__ import annotations
import logging
from fastapi import APIRouter, Depends, HTTPException
from src.api.deps import require_admin_key
from src.api.schemas import EvalSummaryOut, SettleRequest
from src.db.base import AsyncSession, get_db, get_db_read
from src.llm.eval import get_eval_summary, settle_prediction
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/v1", tags=["eval"])
@router.post("/eval/settle")
@router.post("/eval/settle", dependencies=[Depends(require_admin_key)])
async def settle(req: SettleRequest, db: AsyncSession = Depends(get_db)):
"""回填实际结果。"""
try:
pred = await settle_prediction(req.prediction_id, req.home_goals, req.away_goals)
return {"id": pred.id, "settled": pred.settled}
except ValueError as e:
raise HTTPException(404, str(e))
logger.warning("settle failed: %s", e)
raise HTTPException(404, "预测记录不存在")
except Exception as e:
logger.exception("settle error")
raise HTTPException(500, "回填失败,请查看服务器日志")
@router.get("/eval/summary", response_model=EvalSummaryOut)
+31 -27
View File
@@ -1,56 +1,60 @@
"""采集路由。"""
from __future__ import annotations
from fastapi import APIRouter, HTTPException
import logging
from fastapi import APIRouter, Depends, HTTPException
from src.api.deps import require_admin_key
from src.api.schemas import IngestBzzoiroRequest, IngestResponse, IngestUnderstatRequest, IngestInjuriesRequest, IngestSimpleResponse
from src.data.sources import get_source
from src.data.injuries import ingest_injuries
from src.db.base import AsyncSessionLocal
from src.db.unit_of_work import get_uow
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/v1", tags=["ingest"])
@router.post("/ingest/bzzoiro", response_model=IngestResponse)
@router.post("/ingest/bzzoiro", response_model=IngestResponse, dependencies=[Depends(require_admin_key)])
async def ingest_bzzoiro_route(req: IngestBzzoiroRequest):
"""触发 bzzoiro 采集。"""
source = get_source("bzzoiro")
async with AsyncSessionLocal() as db:
try:
try:
async with get_uow() as session:
result = await source.ingest(
db,
session,
leagues=req.leagues,
date_from=req.date_from,
date_to=req.date_to,
status=req.status,
)
await db.commit()
return IngestResponse(**result)
except Exception as e:
await db.rollback()
raise HTTPException(500, str(e))
return IngestResponse(**result)
except Exception as e:
logger.exception("bzzoiro ingest failed")
raise HTTPException(500, "数据采集失败,请查看服务器日志")
@router.post("/ingest/understat", response_model=IngestSimpleResponse)
@router.post("/ingest/understat", response_model=IngestSimpleResponse, dependencies=[Depends(require_admin_key)])
async def ingest_understat_route(req: IngestUnderstatRequest):
"""触发 understat xG 回填。"""
source = get_source("understat")
async with AsyncSessionLocal() as db:
try:
result = await source.ingest(db, league=req.league, season=req.season)
return IngestSimpleResponse(**result)
except Exception as e:
await db.rollback()
raise HTTPException(500, str(e))
try:
async with get_uow() as session:
result = await source.ingest(session, league=req.league, season=req.season)
return IngestSimpleResponse(**result)
except Exception as e:
logger.exception("understat ingest failed")
raise HTTPException(500, "xG 回填失败,请查看服务器日志")
@router.post("/ingest/injuries", response_model=IngestSimpleResponse)
@router.post("/ingest/injuries", response_model=IngestSimpleResponse, dependencies=[Depends(require_admin_key)])
async def ingest_injuries_route(req: IngestInjuriesRequest):
"""触发伤停采集。"""
async with AsyncSessionLocal() as db:
try:
result = await ingest_injuries(db, date=req.date)
return IngestSimpleResponse(**result)
except Exception as e:
await db.rollback()
raise HTTPException(500, str(e))
try:
async with get_uow() as session:
result = await ingest_injuries(session, date=req.date)
return IngestSimpleResponse(**result)
except Exception as e:
logger.exception("injuries ingest failed")
raise HTTPException(500, "伤停采集失败,请查看服务器日志")
+16 -7
View File
@@ -1,7 +1,9 @@
"""预测路由。"""
from __future__ import annotations
from fastapi import APIRouter, Depends, HTTPException
import logging
from fastapi import APIRouter, Depends, HTTPException, Query
from sqlalchemy import select
from sqlalchemy.orm import selectinload
@@ -10,6 +12,8 @@ from src.db.base import AsyncSession, get_db, get_db_read
from src.db.models import Prediction
from src.llm.predict import predict_match, PredictResult
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/v1", tags=["predict"])
@@ -24,9 +28,14 @@ async def predict(req: PredictRequest, db: AsyncSession = Depends(get_db)):
mode=req.mode,
)
except ValueError as e:
raise HTTPException(404, str(e))
logger.warning("predict validation error: %s", e)
raise HTTPException(404, "比赛不存在")
except RuntimeError as e:
raise HTTPException(502, str(e))
logger.error("predict LLM error: %s", e)
raise HTTPException(502, "LLM 预测失败,请查看服务器日志")
except Exception as e:
logger.exception("predict unexpected error")
raise HTTPException(500, "预测失败,请查看服务器日志")
# single / multi 两种结果统一映射
return PredictOut(
@@ -38,7 +47,7 @@ async def predict(req: PredictRequest, db: AsyncSession = Depends(get_db)):
pred_home_goals=result.pred_home_goals,
pred_away_goals=result.pred_away_goals,
pred_1x2=result.pred_1x2,
confidence=result.confidence,
subjective_confidence=result.subjective_confidence,
reasoning=result.reasoning,
agent_outputs=getattr(result, "agent_outputs", None),
agent_weights=getattr(result, "agent_weights", None),
@@ -50,7 +59,7 @@ async def predict(req: PredictRequest, db: AsyncSession = Depends(get_db)):
@router.get("/predictions", response_model=list[PredictionOut])
async def list_predictions(
match_id: int | None = None,
limit: int = 50,
limit: int = Query(50, ge=1, le=200),
db: AsyncSession = Depends(get_db_read),
):
stmt = select(Prediction).options(selectinload(Prediction.match))
@@ -69,7 +78,7 @@ async def list_predictions(
pred_home_goals=p.pred_home_goals,
pred_away_goals=p.pred_away_goals,
pred_1x2=p.pred_1x2,
confidence=p.confidence,
subjective_confidence=p.subjective_confidence,
reasoning=p.reasoning,
agent_outputs=p.agent_outputs,
created_at=p.created_at,
@@ -96,7 +105,7 @@ async def get_prediction(prediction_id: int, db: AsyncSession = Depends(get_db_r
pred_home_goals=p.pred_home_goals,
pred_away_goals=p.pred_away_goals,
pred_1x2=p.pred_1x2,
confidence=p.confidence,
subjective_confidence=p.subjective_confidence,
reasoning=p.reasoning,
agent_outputs=p.agent_outputs,
created_at=p.created_at,
+2 -2
View File
@@ -54,7 +54,7 @@ class PredictOut(BaseModel):
pred_home_goals: float | None
pred_away_goals: float | None
pred_1x2: str | None
confidence: float | None
subjective_confidence: float | None
reasoning: str | None
agent_outputs: list[dict] | None = None
agent_weights: dict | None = None
@@ -72,7 +72,7 @@ class PredictionOut(BaseModel):
pred_home_goals: float | None
pred_away_goals: float | None
pred_1x2: str | None
confidence: float | None
subjective_confidence: float | None
reasoning: str | None
agent_outputs: list[dict] | None = None
created_at: datetime
+17
View File
@@ -32,6 +32,23 @@ class Settings(BaseSettings):
# --- CORS ---
CORS_ORIGINS: str = "http://localhost:5173,http://localhost:3000"
CORS_METHODS: str = "GET,POST,PUT,DELETE,OPTIONS"
CORS_HEADERS: str = "Authorization,Content-Type,X-API-Key,Accept"
# --- HTTP ---
HTTP_DEFAULT_TIMEOUT: int = 30
# --- database pool ---
DB_POOL_SIZE: int = 5
DB_MAX_OVERFLOW: int = 10
DB_POOL_TIMEOUT: int = 30
DB_POOL_RECYCLE: int = 1800
# --- 管理接口鉴权 ---
# 采集 / 回测等高成本或写入型接口需要此 Key(请求头 X-API-Key)。
# 留空表示「未启用鉴权」(本地开发默认),生产环境必须设置。
# 见审查报告 P2-7:ingest/backtest 无鉴权可被任意调用并烧掉 LLM 额度。
ADMIN_API_KEY: str = ""
settings = Settings()
+5 -2
View File
@@ -2,24 +2,27 @@
使用方:
- src/llm/provider.py: LLM 调用
- src/data/bzzoiro.py: bzzoiro 比赛数据
- src/data/understat.py: xG 抓取
- src/data/injuries.py: 伤停抓取
生命周期由 FastAPI lifespan 管理(关闭时 aclose)
调用方可通过 `timeout` 参数覆盖 per-request 超时
"""
from __future__ import annotations
import httpx
from src.core.config import settings
_shared_client: httpx.AsyncClient | None = None
_default_timeout = 30
def get_client() -> httpx.AsyncClient:
"""获取共享客户端(懒初始化)。"""
global _shared_client
if _shared_client is None or _shared_client.is_closed:
_shared_client = httpx.AsyncClient(timeout=_default_timeout)
_shared_client = httpx.AsyncClient(timeout=settings.HTTP_DEFAULT_TIMEOUT)
return _shared_client
-82
View File
@@ -1,82 +0,0 @@
"""重试工具:带指数退避的瞬态错误重试。"""
from __future__ import annotations
import asyncio
import functools
import logging
import random
import time
from typing import Callable, Iterable, TypeVar
logger = logging.getLogger(__name__)
T = TypeVar("T")
def with_retry(
*,
max_retries: int = 3,
base_delay: float = 1.0,
max_delay: float = 30.0,
retryable_exceptions: Iterable[type[BaseException]] = (Exception,),
on_retry: Callable[[Exception, int], None] | None = None,
) -> Callable:
"""重试装饰器(同步/异步通用,指数退避 + 抖动)。
Args:
max_retries: 最大重试次数
base_delay: 基础延迟()
max_delay: 最大延迟()
retryable_exceptions: 触发重试的异常类型
on_retry: 重试回调(exception, attempt)
"""
retryable = tuple(retryable_exceptions)
def decorator(func: Callable) -> Callable:
@functools.wraps(func)
async def async_wrapper(*args, **kwargs):
last_exc: Exception | None = None
for attempt in range(max_retries + 1):
try:
return await func(*args, **kwargs)
except retryable as e:
last_exc = e
if attempt == max_retries:
break
delay = min(base_delay * (2 ** attempt), max_delay)
delay += random.uniform(0, delay * 0.1) # 抖动
logger.warning(
"%s failed (attempt %d/%d), retry in %.1fs: %s",
func.__name__, attempt + 1, max_retries, delay, e,
)
if on_retry:
on_retry(e, attempt + 1)
await asyncio.sleep(delay)
raise last_exc # type: ignore[misc]
@functools.wraps(func)
def sync_wrapper(*args, **kwargs):
last_exc: Exception | None = None
for attempt in range(max_retries + 1):
try:
return func(*args, **kwargs)
except retryable as e:
last_exc = e
if attempt == max_retries:
break
delay = min(base_delay * (2 ** attempt), max_delay)
delay += random.uniform(0, delay * 0.1)
logger.warning(
"%s failed (attempt %d/%d), retry in %.1fs: %s",
func.__name__, attempt + 1, max_retries, delay, e,
)
if on_retry:
on_retry(e, attempt + 1)
time.sleep(delay)
raise last_exc # type: ignore[misc]
if asyncio.iscoroutinefunction(func):
return async_wrapper
return sync_wrapper
return decorator
+383 -71
View File
@@ -1,6 +1,13 @@
"""Bzzoiro 数据源:抓取 + 入库。
迁移自旧项目 app/data/sources/bzzoiro/,改成 async + 简化入库
使用 Repository 模式进行数据访问,不直接控制事务
改进(P1):
- Bronze 层集成:采集后先存 RawEvent,再规范化
- 死信表集成:采集/规范化失败写 IngestFailure
- 令牌桶限流:替换固定 REQUEST_INTERVAL sleep
- 数据血缘:记录 ETL 全过程
"""
from __future__ import annotations
@@ -8,61 +15,98 @@ import asyncio
import json as _json
import logging
import random
import time as _time
import urllib.error
import urllib.parse
import urllib.request
import uuid
from collections.abc import Iterable
from datetime import datetime, timezone
from sqlalchemy import select
from src.core.config import settings
from src.core.http_client import get_client
from src.data.config import BZZOIRO_LEAGUE_IDS, LEAGUE_COUNTRIES, LEAGUE_NAMES, REQUEST_INTERVAL
from src.data.match_lookup import find_existing_match, get_or_create_team
from src.data.normalize import normalize_bzzoiro
from src.data.rate_limiter import RateLimitedClient, TokenBucket
from src.data.sources import register
from src.db.models import League, Match, MatchStats
from src.db.models import DataLineage, DataQualityCheck, IngestFailure, League, Match, MatchStats, RawEvent, Team
logger = logging.getLogger(__name__)
# 令牌桶限流器:替代固定 sleep,rate 根据 REQUEST_INTERVAL 计算
_bzzoiro_limiter = TokenBucket(rate=1.0 / REQUEST_INTERVAL, capacity=3)
# P2-3: RateLimitedClient 包装,在 HTTP 调用层面透明限流
_bzzoiro_rl_client: RateLimitedClient | None = None
def _fetch_json_sync(path: str, params: dict | None = None, max_retries: int = 3) -> dict | list:
"""同步 HTTP(bzzoiro 客户端保持同步,在 async 函数里 run_in_executor)。"""
def _get_bzzoiro_rl_client() -> RateLimitedClient:
"""惰性初始化 RateLimitedClient(避免模块导入时创建 client)。"""
global _bzzoiro_rl_client
if _bzzoiro_rl_client is None:
client = get_client()
_bzzoiro_rl_client = RateLimitedClient(client, _bzzoiro_limiter)
return _bzzoiro_rl_client
def _to_date(value):
"""把 datetime / date / str 统一成 `date`。"""
if value is None:
return None
if hasattr(value, "date") and callable(value.date):
return value.date()
return value
def _match_key(home_team_id: int, away_team_id: int, match_date) -> tuple[int, int, str]:
"""比赛去重键:(主队, 客队, 天级日期 ISO 字符串)。
统一在这里构造,避免"预加载时用 str(date)、写入时用 isoformat()"这类
隐式格式依赖 两者当前恰好相等,但一旦有人改动其一就会静默失配,
导致所有比赛被判为不存在而重复插入
"""
d = _to_date(match_date)
return (home_team_id, away_team_id, d.isoformat() if d is not None else "")
async def _fetch_json_async(path: str, params: dict | None = None, max_retries: int = 3) -> dict | list:
"""异步 HTTP(bzzoiro 使用 httpx,不再阻塞事件循环线程池)。"""
base = settings.BZZOIRO_BASE.rstrip("/")
url = f"{base}/{path.lstrip('/')}"
if params:
url += "?" + urllib.parse.urlencode(params)
key = settings.BZZOIRO_KEY
if not key:
raise RuntimeError("BZZOIRO_KEY 未设置")
headers = {
"Authorization": f"Token {key}",
"Accept": "application/json",
}
last_exc: Exception | None = None
for attempt in range(max_retries):
try:
req = urllib.request.Request(url)
req.add_header("Authorization", f"Token {key}")
req.add_header("Accept", "application/json")
with urllib.request.urlopen(req, timeout=30) as resp:
return _json.loads(resp.read().decode("utf-8"))
except urllib.error.HTTPError as e:
# P2-3: 使用 RateLimitedClient 包装 HTTP 调用,透明限流
rl_client = _get_bzzoiro_rl_client()
resp = await rl_client.get(url, headers=headers, params=params, timeout=30)
resp.raise_for_status()
return resp.json()
except Exception as e:
last_exc = e
if e.code == 429:
# 指数退避: 429 通常意味着限速
status = getattr(getattr(e, "response", None), "status_code", None)
if status == 429:
delay = min(2 ** attempt, 16) + random.uniform(0, 1)
logger.warning("bzzoiro 429, retry %d in %.1fs", attempt + 1, delay)
_time.sleep(delay)
await asyncio.sleep(delay)
continue
if 500 <= e.code < 600:
if 500 <= (status or 0) < 600:
delay = min(2 ** attempt, 16) + random.uniform(0, 1)
logger.warning("bzzoiro %d, retry %d in %.1fs", e.code, attempt + 1, delay)
_time.sleep(delay)
logger.warning("bzzoiro %d, retry %d in %.1fs", status, attempt + 1, delay)
await asyncio.sleep(delay)
continue
raise # 4xx 直接抛
except (urllib.error.URLError, TimeoutError, ConnectionError) as e:
last_exc = e
delay = min(2 ** attempt, 16) + random.uniform(0, 1)
logger.warning("bzzoiro network error, retry %d in %.1fs: %s", attempt + 1, delay, e)
_time.sleep(delay)
# 网络错误(连接失败/超时)也退避重试
if isinstance(e, (TimeoutError, ConnectionError, OSError)):
delay = min(2 ** attempt, 16) + random.uniform(0, 1)
logger.warning("bzzoiro network error, retry %d in %.1fs: %s", attempt + 1, delay, e)
await asyncio.sleep(delay)
continue
raise
raise RuntimeError(f"bzzoiro request failed after {max_retries} attempts: {last_exc}")
@@ -74,15 +118,13 @@ async def fetch_bzzoiro_events(
date_to: str | None = None,
limit: int = 200,
) -> list[dict]:
"""抓取 bzzoiro 原始事件(异步包装)。"""
"""抓取 bzzoiro 原始事件(异步,无需 run_in_executor)。"""
league_id = BZZOIRO_LEAGUE_IDS.get(league_code)
if league_id is None:
raise ValueError(f"未知联赛代码: {league_code}")
loop = asyncio.get_event_loop()
rows: list[dict] = []
offset = 0
payload: dict | list = {}
while True:
params: dict = {
"league_id": league_id,
@@ -94,8 +136,7 @@ async def fetch_bzzoiro_events(
params["date_from"] = str(date_from)[:10]
if date_to:
params["date_to"] = str(date_to)[:10]
# 显式位置参数,避免 lambda 闭包捕获循环变量
payload = await loop.run_in_executor(None, _fetch_json_sync, "/events/", params)
payload = await _fetch_json_async("/events/", params)
batch = payload.get("results") or []
if not batch:
break
@@ -106,10 +147,141 @@ async def fetch_bzzoiro_events(
break
if len(batch) < limit:
break
await asyncio.sleep(REQUEST_INTERVAL)
# 令牌桶已在 _fetch_json_async 内部处理,这里不再需要固定 sleep
return rows
async def _write_bronze_events(db, raw_events: list[dict], batch_id: str) -> None:
"""将原始事件写入 Bronze 层(RawEvent 表)。
P2-1: 写入血缘记录,追踪 Bronze 层摄取过程
"""
for raw in raw_events:
try:
source_record_id = str(raw.get("id", ""))
if not source_record_id:
# P1-fix: 不再静默跳过,记录到死信表
logger.warning("bzzoiro raw event missing id: %s", str(raw)[:200])
await _write_ingest_failure(
db, "bzzoiro", "match", "",
"normalize_error", "raw event missing id", raw,
)
continue
bronze = RawEvent(
source_system="bzzoiro",
source_record_id=source_record_id,
raw_payload=raw,
ingest_batch_id=batch_id,
)
db.add(bronze)
except Exception as e:
logger.warning("bzzoiro raw_event write failed: %s", e)
try:
await db.flush()
except Exception as e:
logger.warning("bzzoiro raw_events flush failed: %s", e)
# P2-1: 写入 Bronze 层血缘记录
for raw in raw_events:
source_record_id = str(raw.get("id", ""))
if not source_record_id:
continue
await _write_data_lineage(
db,
source_system="bzzoiro",
source_record_id=source_record_id,
target_table="raw_events",
transform_name="bronze_ingest",
batch_id=batch_id,
)
async def _write_ingest_failure(
db,
source_system: str,
entity_type: str,
source_record_id: str,
error_type: str,
error_detail: str,
raw_payload: dict | None,
) -> None:
"""写入采集失败到死信表(IngestFailure)。"""
try:
from datetime import timedelta
failure = IngestFailure(
source_system=source_system,
entity_type=entity_type,
source_record_id=source_record_id,
error_type=error_type,
error_detail=error_detail[:2000] if error_detail else None,
raw_payload=raw_payload,
status="pending",
next_retry_at=datetime.now(timezone.utc) + timedelta(minutes=5),
)
db.add(failure)
await db.flush()
except Exception:
logger.warning("bzzoiro ingest_failure write failed", exc_info=True)
async def _write_data_lineage(
db,
*,
source_system: str,
source_record_id: str,
target_table: str,
target_id: int | None = None,
transform_name: str,
transform_detail: str | None = None,
batch_id: str | None = None,
) -> None:
"""写入数据血缘记录(DataLineage 表)。"""
try:
lineage = DataLineage(
source_system=source_system,
source_record_id=source_record_id,
target_table=target_table,
target_id=target_id,
transform_name=transform_name,
transform_detail=transform_detail,
batch_id=batch_id,
)
db.add(lineage)
await db.flush()
except Exception:
logger.warning("bzzoiro data_lineage write failed", exc_info=True)
async def _write_data_quality_check(
db,
*,
check_name: str,
entity_type: str,
entity_id: str | None = None,
expected_value: str | None = None,
actual_value: str | None = None,
passed: bool,
severity: str = "warning",
detail: str | None = None,
) -> None:
"""写入数据质量检查结果(DataQualityCheck 表)。"""
try:
check = DataQualityCheck(
check_name=check_name,
entity_type=entity_type,
entity_id=entity_id,
expected_value=expected_value,
actual_value=actual_value,
passed=passed,
severity=severity,
detail=detail,
)
db.add(check)
await db.flush()
except Exception:
logger.warning("bzzoiro data_quality_check write failed", exc_info=True)
@register
class BzzoiroSource:
"""bzzoiro 数据源(实现 DataSource 协议)。"""
@@ -125,8 +297,12 @@ class BzzoiroSource:
date_to: str | None = None,
status: str = "finished",
) -> dict:
"""采集 bzzoiro → 入库。返回统计。"""
"""采集 bzzoiro → 入库。返回统计。
注意: 本方法不控制事务(commit/rollback),由调用方通过 UnitOfWork 控制
"""
result: dict = {"leagues": {}, "total_inserted": 0, "total_updated": 0, "errors": []}
batch_id = uuid.uuid4().hex[:16]
for code in leagues:
league_r: dict = {"inserted": 0, "updated": 0, "errors": []}
@@ -135,9 +311,17 @@ class BzzoiroSource:
except Exception as e:
logger.exception("bzzoiro fetch failed for %s", code)
league_r["errors"].append(f"fetch failed: {e}")
# 写死信表
await _write_ingest_failure(
db, "bzzoiro", "match", f"fetch_{code}_{batch_id}",
"fetch_error", str(e), {"league": code, "date_from": date_from, "date_to": date_to},
)
result["leagues"][code] = league_r
continue
# === Bronze 层:先存 RawEvent ===
await _write_bronze_events(db, raw_events, batch_id)
# 获取或创建联赛
stmt = select(League).where(League.code == code)
league = (await db.execute(stmt)).scalar_one_or_none()
@@ -146,32 +330,89 @@ class BzzoiroSource:
db.add(league)
await db.flush()
for raw in raw_events:
try:
# === 批量优化: 预加载球队和已有比赛到内存 ===
team_name_to_id: dict[str, int] = {}
existing_matches: dict[tuple[int, int, str], Match] = {} # 完整对象,避免重复查询
# (NormalizedMatch, 原始 event) 成对保存:后续写 source_record_id 时
# 必须用配对的那条 event,不能依赖外层循环变量残留值。
normalized_matches: list[tuple] = []
if raw_events:
# 一次遍历: 收集球队名 + 规范化
all_team_names = set()
for raw in raw_events:
nm = normalize_bzzoiro(raw, code)
if nm is None:
continue
nm.validate()
except Exception as e:
logger.debug("normalize skip: %s", e)
league_r["errors"].append(f"normalize: {e}")
continue
if nm is not None:
try:
nm.validate()
except Exception as e:
# P1-3: 统一使用 warning,不追加到 errors(仅运行时错误入 errors)
logger.warning("normalize skip: %s", e)
# 写死信表:规范化失败
await _write_ingest_failure(
db, "bzzoiro", "match", str(raw.get("id", "")),
"normalize_error", str(e), raw,
)
continue
normalized_matches.append((nm, raw))
all_team_names.add(nm.home_team)
all_team_names.add(nm.away_team)
# 球队
home_team = await get_or_create_team(db, nm.home_team)
away_team = await get_or_create_team(db, nm.away_team)
if all_team_names:
stmt = select(Team).where(Team.name.in_(all_team_names))
teams = (await db.execute(stmt)).scalars().all()
team_name_to_id = {t.name: t.id for t in teams}
# 查找已有比赛(天级匹配)
existing = await find_existing_match(db, league.id, nm.home_team, nm.away_team, nm.date)
# P1-2: 按需加载,只加载 raw_events 涉及日期范围的比赛(加 30 天缓冲)
# 避免加载联赛全部历史比赛到内存(多赛季采集时内存溢出)
if normalized_matches:
from datetime import timedelta
dates = [nm.date for nm in normalized_matches if nm.date is not None]
if dates:
min_dt = min(dates) - timedelta(days=30)
max_dt = max(dates) + timedelta(days=30)
stmt = (
select(Match)
.where(Match.league_id == league.id)
.where(Match.match_date >= min_dt)
.where(Match.match_date <= max_dt)
)
existing_matches = {
_match_key(m.home_team_id, m.away_team_id, m.match_date_date): m
for m in (await db.execute(stmt)).scalars()
}
# else: existing_matches 保持空 dict(全量新比赛)
if existing is None:
for nm, raw in normalized_matches:
# 球队: 内存查找 + 按需创建
home_team_id = team_name_to_id.get(nm.home_team)
if home_team_id is None:
home = Team(name=nm.home_team)
db.add(home)
await db.flush()
home_team_id = home.id
team_name_to_id[nm.home_team] = home_team_id
away_team_id = team_name_to_id.get(nm.away_team)
if away_team_id is None:
away = Team(name=nm.away_team)
db.add(away)
await db.flush()
away_team_id = away.id
team_name_to_id[nm.away_team] = away_team_id
# 查找已有比赛: 内存查找
match_key = _match_key(home_team_id, away_team_id, nm.date)
existing_match = existing_matches.get(match_key)
if existing_match is None:
m = Match(
league_id=league.id,
season=nm.season_label or None,
home_team_id=home_team.id,
away_team_id=away_team.id,
home_team_id=home_team_id,
away_team_id=away_team_id,
match_date=nm.date,
match_date_date=nm.date.date() if hasattr(nm.date, "date") else nm.date,
match_date_date=_to_date(nm.date),
match_status=nm.match_status,
home_goals=nm.home_goals,
away_goals=nm.away_goals,
@@ -181,7 +422,19 @@ class BzzoiroSource:
)
db.add(m)
await db.flush()
existing_matches[match_key] = m # 防止同批重复
# P2-1: Silver 层血缘 — Match 创建
await _write_data_lineage(
db,
source_system="bzzoiro",
source_record_id=str(raw.get("id", "")),
target_table="matches",
target_id=m.id,
transform_name="silver_upsert",
batch_id=batch_id,
)
if nm.home_xg is not None or nm.away_xg is not None:
now = datetime.now(timezone.utc)
stats = MatchStats(
match_id=m.id,
home_xg=nm.home_xg,
@@ -197,45 +450,104 @@ class BzzoiroSource:
away_yellow_cards=nm.away_yellow_cards,
home_red_cards=nm.home_red_cards,
away_red_cards=nm.away_red_cards,
source="bzzoiro",
source_record_id=str(raw.get("id", "")),
retrieved_at=now,
available_at=now,
)
db.add(stats)
await db.flush()
# P2-1: Silver 层血缘 — MatchStats 创建
await _write_data_lineage(
db,
source_system="bzzoiro",
source_record_id=str(raw.get("id", "")),
target_table="match_stats",
target_id=m.id,
transform_name="silver_upsert",
batch_id=batch_id,
)
league_r["inserted"] += 1
else:
# 更新(只补空 / 状态升级)
# 已有比赛: 直接从内存获取对象更新(无需再查询)
changed = False
if existing.match_status != nm.match_status and nm.match_status == "finished":
existing.match_status = nm.match_status
if existing_match.match_status != nm.match_status and nm.match_status == "finished":
existing_match.match_status = nm.match_status
changed = True
if existing.home_goals is None and nm.home_goals is not None:
existing.home_goals = nm.home_goals
existing.away_goals = nm.away_goals
existing.home_ht_goals = nm.home_ht_goals
existing.away_ht_goals = nm.away_ht_goals
if existing_match.home_goals is None and nm.home_goals is not None:
existing_match.home_goals = nm.home_goals
existing_match.away_goals = nm.away_goals
existing_match.home_ht_goals = nm.home_ht_goals
existing_match.away_ht_goals = nm.away_ht_goals
changed = True
if existing.match_stage is None and nm.match_stage:
existing.match_stage = nm.match_stage
if existing_match.match_stage is None and nm.match_stage:
existing_match.match_stage = nm.match_stage
changed = True
# stats 只补空
if existing.stats is None and (nm.home_xg is not None or nm.away_xg is not None):
existing.stats = MatchStats(match_id=existing.id)
db.add(existing.stats)
if existing_match.stats is None and (nm.home_xg is not None or nm.away_xg is not None):
now = datetime.now(timezone.utc)
existing_match.stats = MatchStats(
match_id=existing_match.id,
home_xg=nm.home_xg,
away_xg=nm.away_xg,
home_shots=nm.home_shots,
away_shots=nm.away_shots,
home_shots_on_target=nm.home_shots_on_target,
away_shots_on_target=nm.away_shots_on_target,
home_corners=nm.home_corners,
away_corners=nm.away_corners,
home_possession=nm.home_possession,
home_yellow_cards=nm.home_yellow_cards,
away_yellow_cards=nm.away_yellow_cards,
home_red_cards=nm.home_red_cards,
away_red_cards=nm.away_red_cards,
source="bzzoiro",
source_record_id=str(raw.get("id", "")),
retrieved_at=now,
available_at=now,
)
db.add(existing_match.stats)
await db.flush()
if existing.stats is not None:
# P2-1: Silver 层血缘 — MatchStats 创建(更新路径)
await _write_data_lineage(
db,
source_system="bzzoiro",
source_record_id=str(raw.get("id", "")),
target_table="match_stats",
target_id=existing_match.id,
transform_name="silver_upsert",
batch_id=batch_id,
)
if existing_match.stats is not None:
for fld in ("home_xg", "away_xg", "home_shots", "away_shots",
"home_shots_on_target", "away_shots_on_target",
"home_corners", "away_corners", "home_possession",
"home_yellow_cards", "away_yellow_cards",
"home_red_cards", "away_red_cards"):
if getattr(existing.stats, fld, None) is None:
if getattr(existing_match.stats, fld, None) is None:
v = getattr(nm, fld, None)
if v is not None:
setattr(existing.stats, fld, v)
setattr(existing_match.stats, fld, v)
changed = True
if changed:
league_r["updated"] += 1
await db.commit()
# 注意: 不在此处 commit,由调用方 UnitOfWork 控制事务
result["leagues"][code] = league_r
result["total_inserted"] += league_r["inserted"]
result["total_updated"] += league_r["updated"]
# P2-2: 数据质量检查 — 行计数合理性(某联赛比赛数不应为 0)
total_records = league_r["inserted"] + league_r["updated"]
await _write_data_quality_check(
db,
check_name="bzzoiro_row_count",
entity_type="match",
entity_id=code,
expected_value=">=1",
actual_value=str(total_records),
passed=total_records > 0,
severity="critical" if total_records == 0 else "info",
detail=f"league={code} inserted={league_r['inserted']} updated={league_r['updated']}",
)
return result
+362 -47
View File
@@ -1,15 +1,25 @@
"""伤停数据采集器(api-football / api-sports.io)。
采集伤停数据并入库(injuries ), injuries agent 使用
改进(P0):
- 原子写缓存(tempfile + os.replace)
- TTL 分级:未来比赛 1h,当天 5min,历史 7day
- 文件锁防止并发写缓存冲突
- Bronze 层集成(RawEvent)
- 死信表集成(IngestFailure)
"""
from __future__ import annotations
import asyncio
import json
import logging
import os
import random
import tempfile
import time
from datetime import datetime, timezone
import uuid
from datetime import datetime, timedelta, timezone
from pathlib import Path
from typing import Any
@@ -17,14 +27,147 @@ import httpx
from src.core.config import settings
from src.core.http_client import get_client
from src.db.models import DataQualityCheck
logger = logging.getLogger(__name__)
API_BASE = "https://v3.football.api-sports.io"
DEFAULT_HOST = "v3.football.api-sports.io"
# 缓存目录
_CACHE_DIR = Path(__file__).resolve().parent.parent.parent / "data" / "injuries_cache"
# P2-3: 缓存目录改用系统临时目录,避免源码树内写入
_CACHE_DIR = Path(tempfile.gettempdir()) / "profeto_injuries"
# 模块级锁:防止并发写同一缓存文件
_cache_lock = asyncio.Lock()
def _sanitize_cache_component(value: str | int | None) -> str:
"""消毒缓存键组件,防止路径遍历。
只允许 [A-Za-z0-9_.-],其余字符替换为 '_'
"""
import re
s = str(value) if value is not None else "none"
return re.sub(r'[^A-Za-z0-9_.]', '_', s)[:60]
def _build_cache_key(prefix: str, *components: str | int | None) -> str:
"""构造安全的缓存文件名。"""
parts = [_sanitize_cache_component(c) for c in components]
return f"{prefix}_{'_'.join(parts)}.json"
def _compute_ttl_hours(date_str: str | None, fixture_id: int | None, league_id: int | None) -> float:
"""根据查询参数计算缓存 TTL(小时)。
TTL 分级策略:
- 未来比赛(date > now): 1 小时(赛前伤停变化频繁)
- 当天比赛(date == today): 5 分钟(赛中实时更新)
- 历史比赛(date < now): 7 (历史数据不变)
- 无日期参数: 1 小时(保守策略)
"""
if date_str is None:
# 无日期参数(按 fixture_id 或 league_id 查询),保守 TTL
return 1.0
try:
query_date = datetime.strptime(date_str, "%Y-%m-%d").replace(tzinfo=timezone.utc)
now = datetime.now(timezone.utc)
today = now.date()
if query_date.date() > today:
return 1.0 # 未来
elif query_date.date() == today:
return 5.0 / 60 # 当天:5 分钟
else:
return 168.0 # 历史:7 天
except (ValueError, TypeError):
return 1.0 # 解析失败,保守 TTL
def _write_cache_atomic(cache_file: Path, data: Any) -> None:
"""原子写缓存文件。
使用 tempfile + os.replace 实现原子写,防止读到写了一半的文件
配合模块级 asyncio.Lock,杜绝并发写冲突
"""
cache_file.parent.mkdir(parents=True, exist_ok=True)
# 在同一目录创建临时文件(保证 os.replace 是原子操作)
fd, tmp_path = tempfile.mkstemp(
dir=str(cache_file.parent),
prefix=f".{cache_file.name}.tmp_",
)
try:
with os.fdopen(fd, "w", encoding="utf-8") as f:
json.dump(data, f, ensure_ascii=False, default=str)
os.replace(tmp_path, cache_file)
except BaseException:
# 失败时清理临时文件
try:
os.unlink(tmp_path)
except OSError:
pass
raise
def _cleanup_expired_cache() -> int:
"""P2-4: 扫描缓存目录,删除过期的 .json 文件。
TTL 策略(保守取最大值,避免误删有效缓存):
- 最短 TTL 5 分钟(当天比赛),但清理阈值用 1 小时
- 超过 1 小时的缓存文件视为过期
Returns:
删除的文件数
"""
import glob as _glob
cache_dir = _CACHE_DIR
if not cache_dir.exists():
return 0
removed = 0
max_ttl_seconds = 3600 # 1 小时(保守阈值,最短实际 TTL 5 分钟)
now_ts = time.time()
for cache_file in _glob.glob(str(cache_dir / "*.json")):
try:
file_age = now_ts - os.path.getmtime(cache_file)
if file_age > max_ttl_seconds:
os.unlink(cache_file)
removed += 1
except OSError:
continue
if removed > 0:
logger.info("injuries cache cleanup: removed %d expired files", removed)
return removed
async def _write_data_quality_check(
db,
*,
check_name: str,
entity_type: str,
entity_id: str | None = None,
expected_value: str | None = None,
actual_value: str | None = None,
passed: bool,
severity: str = "warning",
detail: str | None = None,
) -> None:
"""写入数据质量检查结果(DataQualityCheck 表)。"""
try:
check = DataQualityCheck(
check_name=check_name,
entity_type=entity_type,
entity_id=entity_id,
expected_value=expected_value,
actual_value=actual_value,
passed=passed,
severity=severity,
detail=detail,
)
db.add(check)
await db.flush()
except Exception:
logger.warning("injuries data_quality_check write failed", exc_info=True)
async def fetch_injuries(*, date: str | None = None, fixture_id: int | None = None, league_id: int | None = None) -> list[dict]:
@@ -42,20 +185,25 @@ async def fetch_injuries(*, date: str | None = None, fixture_id: int | None = No
if not api_key:
raise RuntimeError("API_FOOTBALL_KEY 未设置")
# P2-4: 清理过期缓存文件
_cleanup_expired_cache()
cache_dir = _CACHE_DIR
cache_dir.mkdir(parents=True, exist_ok=True)
# 缓存命中 (7 天内有效)
cache_key = f"injuries_{date}_{fixture_id}_{league_id}.json"
# TTL 分级缓存
ttl_hours = _compute_ttl_hours(date, fixture_id, league_id)
# P0-fix: 消毒缓存键,防止路径遍历(date 参数可能含 '../' 等)
cache_key = _build_cache_key("injuries", date, fixture_id, league_id)
cache_file = cache_dir / cache_key
if cache_file.exists():
age_hours = (time.time() - cache_file.stat().st_mtime) / 3600
if age_hours < 168: # 7 天
logger.debug("injuries cache hit: %s (%.1fh old)", cache_key, age_hours)
if age_hours < ttl_hours:
logger.debug("injuries cache hit: %s (%.1fh old, ttl=%.2fh)", cache_key, age_hours, ttl_hours)
with open(cache_file, encoding="utf-8") as f:
return json.load(f)
else:
logger.debug("injuries cache expired: %s (%.1fh old)", cache_key, age_hours)
logger.debug("injuries cache expired: %s (%.1fh old, ttl=%.2fh)", cache_key, age_hours, ttl_hours)
headers = {
"x-apisports-key": api_key,
@@ -92,35 +240,85 @@ async def fetch_injuries(*, date: str | None = None, fixture_id: int | None = No
data = resp.json()
injuries = data.get("response", [])
# 写缓存
with open(cache_file, "w", encoding="utf-8") as f:
json.dump(injuries, ensure_ascii=False, default=str, fp=f)
# 原子写缓存 + 文件锁(防止并发写冲突)
async with _cache_lock:
_write_cache_atomic(cache_file, injuries)
return injuries
async def ingest_injuries(db, *, date: str | None = None) -> dict:
"""采集伤停数据并入库(injuries 表)。"""
"""采集伤停数据并入库(injuries 表)。
注意: 本方法不控制事务(commit/rollback),由调用方通过 UnitOfWork 控制
P1-4: 批量幂等检查,避免逐条查询的竞态条件(并发采集时 IntegrityError)
P1 Bronze: 采集成功后先存 RawEvent,再规范化
P1 死信: 采集/规范化失败时写 IngestFailure
"""
from sqlalchemy import select
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import selectinload
from src.data.team_names import normalize as normalize_name
from src.db.models import Injury, Team
from src.db.models import DataQualityCheck, Injury, IngestFailure, RawEvent, Team
result = {"count": 0, "inserted": 0, "errors": []}
batch_id = uuid.uuid4().hex[:16]
try:
raw_injuries = await fetch_injuries(date=date)
except Exception as e:
logger.exception("injuries fetch failed")
result["errors"].append(f"fetch failed: {e}")
# 写死信表(IngestFailure)
try:
failure = IngestFailure(
source_system="api-football",
entity_type="injury",
source_record_id=f"fetch_{batch_id}",
error_type="fetch_error",
error_detail=str(e)[:2000],
raw_payload={"date": date},
status="pending",
next_retry_at=datetime.now(timezone.utc) + timedelta(minutes=5),
)
db.add(failure)
await db.flush()
except Exception:
logger.warning("failed to write fetch error to ingest_failures", exc_info=True)
return result
result["count"] = len(raw_injuries)
# === Bronze 层:写 RawEvent ===
for raw in raw_injuries:
try:
player = raw.get("player", {}) or {}
fixture = raw.get("fixture", {}) or {}
source_record_id = f"inj_{player.get('id')}_{fixture.get('id')}"
bronze = RawEvent(
source_system="api-football",
source_record_id=source_record_id,
raw_payload=raw,
ingest_batch_id=batch_id,
)
db.add(bronze)
except Exception as e:
logger.warning("raw_event write failed for injury: %s", e)
try:
await db.flush()
except Exception as e:
logger.warning("raw_events flush failed: %s", e)
# 预加载所有球队(用于按名匹配)
teams = (await db.execute(select(Team))).scalars().all()
team_by_name = {t.name: t.id for t in teams}
# P1-4: 收集所有待插入记录的键,批量查询已存在的记录
# 避免逐条查询 + 插入的竞态条件(两个并发请求同时通过检查 → IntegrityError)
pending_records: list[dict] = []
parse_failures: list[dict] = [] # 规范化失败的原始数据,用于写死信表
for raw in raw_injuries:
try:
player = raw.get("player", {}) or {}
@@ -141,58 +339,175 @@ async def ingest_injuries(db, *, date: str | None = None) -> dict:
except (ValueError, AttributeError):
pass
# 强制 int 转换,API 可能返回字符串
player_id = player.get("id")
try:
player_id = int(player_id) if player_id is not None else None
except (ValueError, TypeError):
player_id = None
fixture_id = fixture.get("id")
try:
fixture_id = int(fixture_id) if fixture_id is not None else None
except (ValueError, TypeError):
fixture_id = None
# 幂等: 已存在则跳过
existing = (
await db.execute(
select(Injury).where(
Injury.player_id == player_id,
Injury.fixture_id == fixture_id,
Injury.injury_type == player.get("type"),
)
)
).scalar_one_or_none()
if existing is not None:
continue
injury = Injury(
player_id=player_id,
player_name=player_name,
team_id=team_id,
fixture_id=fixture_id,
league_id=(raw.get("league") or {}).get("id"),
injury_type=player.get("type"),
reason=player.get("reason"),
injury_date=injury_date,
)
db.add(injury)
result["inserted"] += 1
pending_records.append({
"player_id": player_id,
"player_name": player_name,
"team_id": team_id,
"fixture_id": fixture_id,
"league_id": (raw.get("league") or {}).get("id"),
"injury_type": player.get("type"),
"reason": player.get("reason"),
"injury_date": injury_date,
})
except Exception as e:
result["errors"].append(f"parse error: {e}")
parse_failures.append({"raw": raw, "error": str(e)})
await db.commit()
# 写规范化失败到死信表
for fail in parse_failures:
try:
raw = fail["raw"]
player = raw.get("player", {}) or {}
failure = IngestFailure(
source_system="api-football",
entity_type="injury",
source_record_id=f"inj_{player.get('id', 'unknown')}",
error_type="normalize_error",
error_detail=fail["error"][:2000],
raw_payload=raw,
status="pending",
next_retry_at=datetime.now(timezone.utc) + timedelta(minutes=5),
)
db.add(failure)
except Exception:
logger.warning("failed to write parse error to ingest_failures", exc_info=True)
if parse_failures:
try:
await db.flush()
except Exception:
logger.warning("ingest_failures flush failed", exc_info=True)
# P1-4: 批量查询已存在的记录(1 次 DB 往返)
existing_keys: set[tuple] = set()
if pending_records:
# 构造查询条件:所有 (player_id, fixture_id, injury_type) 组合
# 使用 OR 条件批量查询
conditions = []
for rec in pending_records:
conditions.append(
(Injury.player_id == rec["player_id"])
& (Injury.fixture_id == rec["fixture_id"])
& (Injury.injury_type == rec["injury_type"])
)
if conditions:
from sqlalchemy import or_
stmt = select(Injury.player_id, Injury.fixture_id, Injury.injury_type).where(or_(*conditions))
rows = (await db.execute(stmt)).all()
existing_keys = {(r[0], r[1], r[2]) for r in rows}
# P1-4: 批量插入(跳过已存在的)
for rec in pending_records:
key = (rec["player_id"], rec["fixture_id"], rec["injury_type"])
if key in existing_keys:
continue
injury = Injury(**rec)
db.add(injury)
result["inserted"] += 1
# 每 50 条 flush 一次,减少内存压力,同时捕获 IntegrityError
if result["inserted"] % 50 == 0:
try:
await db.flush()
except IntegrityError:
# P1-4: 并发采集时可能仍有竞态,回退到逐条插入
await db.rollback()
logger.warning("injuries batch IntegrityError, falling back to per-record insert")
return await _ingest_injuries_fallback(db, pending_records, result)
# 最终 flush
try:
await db.flush()
except IntegrityError:
await db.rollback()
logger.warning("injuries final flush IntegrityError, falling back to per-record insert")
return await _ingest_injuries_fallback(db, pending_records, result)
# P2-2: 数据质量检查 — 行计数 / 新增比例
total = result["count"]
inserted = result["inserted"]
# 某日期伤停数不应为 0(除非历史日期)
await _write_data_quality_check(
db,
check_name="injuries_row_count",
entity_type="injury",
entity_id=date or "unknown",
expected_value=">=0",
actual_value=str(total),
passed=True,
severity="info",
detail=f"date={date} fetched={total} inserted={inserted}",
)
# 注意: 不在此处 commit,由调用方 UnitOfWork 控制事务
logger.info("injuries: fetched %d, inserted %d for %s", result["count"], result["inserted"], date)
return result
async def get_injuries_for_match(db, team_id: int, match_date) -> list[Injury]:
"""查询某场比赛前某队的伤停名单(比赛日仍缺阵的)"""
from sqlalchemy import and_, or_, select
async def _ingest_injuries_fallback(db, pending_records: list[dict], result: dict) -> dict:
"""P1-4: 逐条插入回退,捕获每条 IntegrityError 避免整批回滚"""
from sqlalchemy.exc import IntegrityError
from src.db.models import Injury
if hasattr(match_date, "date"):
inserted = 0
for rec in pending_records:
injury = Injury(**rec)
db.add(injury)
try:
await db.flush()
inserted += 1
except IntegrityError:
await db.rollback()
# 已存在或其他冲突,跳过
continue
result["inserted"] = inserted
logger.info("injuries fallback: inserted %d records", inserted)
return result
async def get_injuries_for_match(db, team_id: int, match_date, as_of=None) -> list[Injury]:
"""查询某场比赛前某队的伤停名单(比赛日仍缺阵的)。
Args:
db: 数据库 session
team_id: 球队 ID
match_date: 比赛日期
as_of: 数据截止时间(用于回测防泄漏)
Returns:
伤停记录列表
"""
from sqlalchemy import select
from src.db.models import Injury
if hasattr(match_date, "date") and callable(match_date.date):
match_date = match_date.date()
stmt = (
select(Injury)
.where(Injury.team_id == team_id)
.where(Injury.injury_date <= match_date)
.where(or_(Injury.return_date.is_(None), Injury.return_date >= match_date))
.order_by(Injury.injury_date.desc())
.where(
(Injury.return_date.is_(None)) | (Injury.return_date >= match_date)
)
)
if as_of is not None:
if hasattr(as_of, "date") and callable(as_of.date):
as_of = as_of.date()
stmt = stmt.where(Injury.retrieved_at <= as_of)
result = await db.execute(stmt)
return list(result.scalars().all())
-40
View File
@@ -1,40 +0,0 @@
"""比赛匹配辅助函数(多数据源共用)。
bzzoiro / understat 等数据源在入库时都需要:
- 按队名获取或创建球队(get_or_create_team)
- 按联赛+主队+客队+日期找已有比赛(find_existing_match)
"""
from __future__ import annotations
from sqlalchemy import func, select
from src.db.models import Match, Team
async def get_or_create_team(db, name: str) -> Team:
"""按名获取球队,不存在则创建。"""
stmt = select(Team).where(Team.name == name)
team = (await db.execute(stmt)).scalar_one_or_none()
if team is None:
team = Team(name=name)
db.add(team)
await db.flush()
return team
async def find_existing_match(db, league_id: int, home_name: str, away_name: str, date) -> Match | None:
"""按联赛+主队+客队+日期找已有比赛(天级匹配,避免时间精度差异)。"""
home_team = (await db.execute(select(Team).where(Team.name == home_name))).scalar_one_or_none()
away_team = (await db.execute(select(Team).where(Team.name == away_name))).scalar_one_or_none()
if home_team is None or away_team is None:
return None
date_only = date.date() if hasattr(date, "date") else date
stmt = (
select(Match)
.where(Match.league_id == league_id)
.where(Match.home_team_id == home_team.id)
.where(Match.away_team_id == away_team.id)
.where(func.date(Match.match_date) == date_only)
)
return (await db.execute(stmt)).scalar_one_or_none()
+11 -6
View File
@@ -1,9 +1,9 @@
"""数据规范化:任意数据源原始记录 → NormalizedMatch。
迁移自旧项目 app/data/normalize.py,简化:
- 去掉 XGBackfill 双轨(不再需要独立回填)
- 去掉 PIT 时间契约(无训练集要防泄漏)
- 保留核心清洗契约(队名归一日期解析数值范围)
- 去掉 XGBackoff 双轨(不再需要独立回填)
- 去掉 PIT 时间契约(无训练集要防泄漏)
- 保留核心清洗契约(队名归一日期解析数值范围)
"""
from __future__ import annotations
@@ -90,7 +90,10 @@ def derive_season_label(date: datetime) -> str:
def _parse_date(value) -> datetime | None:
"""日期解析 → UTC datetime(带 tzinfo)。"""
"""日期解析 → UTC datetime(带 tzinfo)。
P2-2: 解析失败时记录 warning,避免静默丢数据而无感知
"""
if value in (None, ""):
return None
if isinstance(value, (int, float)):
@@ -111,6 +114,8 @@ def _parse_date(value) -> datetime | None:
return datetime.strptime(s[:19], fmt).replace(tzinfo=timezone.utc)
except ValueError:
continue
# P2-2 修复: 记录被丢弃的原始值,便于排查数据源格式变更
logger.warning("_parse_date failed, dropping record: %r", value)
return None
@@ -204,6 +209,6 @@ def normalize_understat(raw: dict, league_type: str) -> NormalizedMatch | None:
away_team=away,
match_status="finished",
season_label=derive_season_label(dt),
home_xg=_to_float(home_xg),
away_xg=_to_float(away_xg),
home_xg=home_xg,
away_xg=away_xg,
)
+98
View File
@@ -0,0 +1,98 @@
"""令牌桶限流器。
替代 bzzoiro 中固定的 REQUEST_INTERVAL sleep,提供更精细的速率控制
支持突发流量和平滑限流
"""
from __future__ import annotations
import asyncio
import logging
import time
logger = logging.getLogger(__name__)
class TokenBucket:
"""异步令牌桶限流器。
用法:
limiter = TokenBucket(rate=5.0, capacity=10)
await limiter.acquire() # 等待直到有可用令牌
Args:
rate: 每秒补充的令牌数
capacity: 桶容量(允许的最大突发量)
"""
def __init__(self, rate: float, capacity: int | None = None) -> None:
if rate <= 0:
raise ValueError(f"rate must be positive, got {rate}")
self._rate = rate
self._capacity = capacity or max(1, int(rate * 2))
self._tokens: float = self._capacity
self._last_refill = time.monotonic()
self._lock = asyncio.Lock()
async def acquire(self, tokens: int = 1) -> None:
"""获取指定数量的令牌,不足时等待。
Args:
tokens: 需要获取的令牌数
"""
if tokens <= 0:
return
if tokens > self._capacity:
raise ValueError(f"requested {tokens} exceeds capacity {self._capacity}")
while True:
async with self._lock:
self._refill()
if self._tokens >= tokens:
self._tokens -= tokens
return
# 计算需要等待的时间
wait_time = (tokens - self._tokens) / self._rate
logger.debug("token bucket: waiting %.2fs for %d tokens", wait_time, tokens)
await asyncio.sleep(wait_time)
def _refill(self) -> None:
"""补充令牌(基于经过的时间)。"""
now = time.monotonic()
elapsed = now - self._last_refill
if elapsed > 0:
self._tokens = min(self._capacity, self._tokens + elapsed * self._rate)
self._last_refill = now
@property
def tokens(self) -> float:
"""当前可用令牌数(近似)。"""
self._refill()
return self._tokens
class RateLimitedClient:
"""HTTP 客户端限流包装器。
httpx 客户端之上添加令牌桶限流,透明地控制请求速率
用法:
client = get_client()
limiter = TokenBucket(rate=5.0)
wrapper = RateLimitedClient(client, limiter)
resp = await wrapper.get(url)
"""
def __init__(self, client, limiter: TokenBucket) -> None:
self._client = client
self._limiter = limiter
async def get(self, url: str, **kwargs):
"""限流的 GET 请求。"""
await self._limiter.acquire()
return await self._client.get(url, **kwargs)
async def post(self, url: str, **kwargs):
"""限流的 POST 请求。"""
await self._limiter.acquire()
return await self._client.post(url, **kwargs)
+126
View File
@@ -0,0 +1,126 @@
"""P2-5: IngestFailure 重试 Worker。
查询死信表中 status='pending' AND next_retry_at <= now() 的记录,
根据 source_system error_type 决定重试策略
作为 FastAPI startup 事件注册,在应用启动时执行一次清理
"""
from __future__ import annotations
import logging
from datetime import datetime, timedelta, timezone
from sqlalchemy import select
from src.db.base import AsyncSessionLocal
logger = logging.getLogger(__name__)
# 最大重试次数上限
_MAX_RETRIES = 5
async def _retry_ingest_failures(db) -> dict:
"""重试待处理的采集失败记录。
查询条件: status='pending' AND next_retry_at <= now()
重试策略:
- retry_count >= MAX_RETRIES 标记 abandoned
- error_type=fetch_error 退避重试(更新 next_retry_at)
- error_type=normalize_error 规范化失败通常是数据问题,退避重试
- error_type=db_error 数据库问题,退避重试
Args:
db: SQLAlchemy async session
Returns:
统计 dict: {"retried": int, "abandoned": int, "errors": list[str]}
"""
result = {"retried": 0, "abandoned": 0, "errors": []}
from src.db.models import IngestFailure
now = datetime.now(timezone.utc)
# 查询待重试记录
stmt = (
select(IngestFailure)
.where(IngestFailure.status == "pending")
.where(IngestFailure.next_retry_at <= now)
.order_by(IngestFailure.next_retry_at)
.limit(50) # 每批最多处理 50 条,避免长时间持有事务
)
try:
rows = (await db.execute(stmt)).scalars().all()
except Exception as e:
logger.warning("retry_worker: query failed: %s", e)
result["errors"].append(f"query failed: {e}")
return result
if not rows:
return result
logger.info("retry_worker: found %d pending failures to retry", len(rows))
for failure in rows:
try:
# 超过最大重试次数 → 放弃
if failure.retry_count >= _MAX_RETRIES:
failure.status = "abandoned"
failure.resolved_at = now
await db.flush()
result["abandoned"] += 1
logger.info(
"retry_worker: abandoned %s/%s after %d retries",
failure.source_system, failure.source_record_id, failure.retry_count,
)
continue
# 退避计算: 5min * 2^retry_count,最大 24 小时
backoff_minutes = min(5 * (2 ** failure.retry_count), 1440)
next_retry = now + timedelta(minutes=backoff_minutes)
# 更新重试状态
failure.retry_count += 1
failure.next_retry_at = next_retry
failure.status = "retrying"
await db.flush()
result["retried"] += 1
logger.info(
"retry_worker: scheduled retry %d/%d for %s/%s (next: %s)",
failure.retry_count, _MAX_RETRIES,
failure.source_system, failure.source_record_id,
next_retry.isoformat(),
)
except Exception as e:
await db.rollback()
error_msg = f"retry {failure.source_system}/{failure.source_record_id}: {e}"
result["errors"].append(error_msg)
logger.warning("retry_worker: %s", error_msg)
return result
async def run_retry_worker() -> None:
"""启动入口: 获取 DB session 并执行重试逻辑。
设计为幂等操作 多次运行不会产生副作用( next_retry_at 约束)
"""
logger.info("retry_worker: starting ingest failure retry sweep")
try:
async with AsyncSessionLocal() as db:
try:
stats = await _retry_ingest_failures(db)
await db.commit()
logger.info(
"retry_worker: sweep complete — retried=%d abandoned=%d errors=%d",
stats["retried"], stats["abandoned"], len(stats["errors"]),
)
except Exception as e:
await db.rollback()
logger.warning("retry_worker: session failed: %s", e)
except Exception as e:
logger.warning("retry_worker: failed to get DB session: %s", e)
+294 -20
View File
@@ -1,6 +1,13 @@
"""Understat xG 数据源。
迁移自旧项目 app/data/sources/understat.py,改成 async
使用 Repository 模式进行数据访问,不直接控制事务
改进(P1):
- Bronze 层集成:采集后先存 RawEvent,再规范化
- 死信表集成:采集/规范化失败写 IngestFailure
- xG 覆盖更新: understat 数据更新时覆盖旧值(全量覆盖模式)
- xG 追踪字段:xg_source / xg_updated_at / xg_source_record_id
"""
from __future__ import annotations
@@ -9,13 +16,17 @@ import json
import logging
import random
import re
import uuid
from datetime import datetime, timedelta, timezone
from sqlalchemy import select
from sqlalchemy.orm import selectinload
from src.core.http_client import get_client
from src.data.config import FDCO_TO_UNDERSTAT, LEAGUE_NAMES
from src.data.match_lookup import find_existing_match
from src.data.normalize import normalize_understat
from src.data.sources import register
from src.db.models import League, Match, MatchStats
from src.db.models import DataLineage, DataQualityCheck, IngestFailure, League, Match, MatchStats, RawEvent, Team
logger = logging.getLogger(__name__)
@@ -43,6 +54,7 @@ async def fetch_understat(league_code: str, season: int) -> list[dict]:
"Referer": f"https://understat.com/league/{understat_league}/{season}",
}
# 重试:网络错误 / 5xx / 429
last_exc: Exception | None = None
for attempt in range(3):
@@ -54,12 +66,10 @@ async def fetch_understat(league_code: str, season: int) -> list[dict]:
except Exception as e:
last_exc = e
if attempt == 2:
raise
raise RuntimeError(f"understat fetch failed: {e}") from e
delay = min(2 ** attempt, 8) + random.uniform(0, 1)
logger.warning("understat fetch failed, retry %d in %.1fs: %s", attempt + 1, delay, e)
await asyncio.sleep(delay)
else:
raise RuntimeError(f"understat fetch failed: {last_exc}")
# understat 返回 JS 对象,需要提取 JSON
text = resp.text
@@ -73,6 +83,148 @@ async def fetch_understat(league_code: str, season: int) -> list[dict]:
return data
def _match_key(home_team_id: int, away_team_id: int, match_date) -> tuple[int, int, str]:
"""比赛去重键:(主队, 客队, 天级日期 ISO 字符串)。
统一在这里构造,避免"预加载时用 str(date)、写入时用 isoformat()"这类
隐式格式依赖 两者当前恰好相等,但一旦有人改动其一就会静默失配,
导致所有比赛被判为不存在而重复插入
"""
if hasattr(match_date, "date") and callable(match_date.date):
match_date = match_date.date()
return (home_team_id, away_team_id, match_date.isoformat() if match_date is not None else "")
async def _write_understat_bronze(db, raw_matches: list[dict], batch_id: str) -> None:
"""将 understat 原始数据写入 Bronze 层(RawEvent 表)。
P2-1: 写入血缘记录,追踪 Bronze 层摄取过程
"""
for raw in raw_matches:
try:
source_record_id = str(raw.get("id", ""))
if not source_record_id:
# P1-fix: 不再静默跳过,记录到死信表
logger.warning("understat raw match missing id: %s", str(raw)[:200])
await _write_ingest_failure(
db, "understat", "match", "",
"normalize_error", "raw match missing id", raw,
)
continue
bronze = RawEvent(
source_system="understat",
source_record_id=source_record_id,
raw_payload=raw,
ingest_batch_id=batch_id,
)
db.add(bronze)
except Exception as e:
logger.warning("understat raw_event write failed: %s", e)
try:
await db.flush()
except Exception as e:
logger.warning("understat raw_events flush failed: %s", e)
# P2-1: 写入 Bronze 层血缘记录
for raw in raw_matches:
source_record_id = str(raw.get("id", ""))
if not source_record_id:
continue
await _write_data_lineage(
db,
source_system="understat",
source_record_id=source_record_id,
target_table="raw_events",
transform_name="bronze_ingest",
batch_id=batch_id,
)
async def _write_ingest_failure(
db,
source_system: str,
entity_type: str,
source_record_id: str,
error_type: str,
error_detail: str,
raw_payload: dict | None,
) -> None:
"""写入采集失败到死信表(IngestFailure)。"""
try:
failure = IngestFailure(
source_system=source_system,
entity_type=entity_type,
source_record_id=source_record_id,
error_type=error_type,
error_detail=error_detail[:2000] if error_detail else None,
raw_payload=raw_payload,
status="pending",
next_retry_at=datetime.now(timezone.utc) + timedelta(minutes=5),
)
db.add(failure)
await db.flush()
except Exception:
logger.warning("understat ingest_failure write failed", exc_info=True)
async def _write_data_lineage(
db,
*,
source_system: str,
source_record_id: str,
target_table: str,
target_id: int | None = None,
transform_name: str,
transform_detail: str | None = None,
batch_id: str | None = None,
) -> None:
"""写入数据血缘记录(DataLineage 表)。"""
try:
lineage = DataLineage(
source_system=source_system,
source_record_id=source_record_id,
target_table=target_table,
target_id=target_id,
transform_name=transform_name,
transform_detail=transform_detail,
batch_id=batch_id,
)
db.add(lineage)
await db.flush()
except Exception:
logger.warning("understat data_lineage write failed", exc_info=True)
async def _write_data_quality_check(
db,
*,
check_name: str,
entity_type: str,
entity_id: str | None = None,
expected_value: str | None = None,
actual_value: str | None = None,
passed: bool,
severity: str = "warning",
detail: str | None = None,
) -> None:
"""写入数据质量检查结果(DataQualityCheck 表)。"""
try:
check = DataQualityCheck(
check_name=check_name,
entity_type=entity_type,
entity_id=entity_id,
expected_value=expected_value,
actual_value=actual_value,
passed=passed,
severity=severity,
detail=detail,
)
db.add(check)
await db.flush()
except Exception:
logger.warning("understat data_quality_check write failed", exc_info=True)
@register
class UnderstatSource:
"""understat xG 数据源(实现 DataSource 协议)。"""
@@ -80,25 +232,46 @@ class UnderstatSource:
name = "understat"
async def ingest(self, db, *, league: str, season: int) -> dict:
"""采集 understat xG → 回填到现有 Match。只回填 xG 字段,不创建新 Match。"""
from sqlalchemy import select
"""采集 understat xG → 回填到现有 Match。
注意: 本方法不控制事务(commit/rollback),由调用方通过 UnitOfWork 控制
P1-3: 批量查询优化,将单赛季 380 × 3 DB 往返降为 3 次查询
P1-4: xG 覆盖更新模式, understat 数据更新时覆盖旧值
"""
from src.db.repositories import LeagueRepository, TeamRepository
result = {"updated": 0, "skipped": 0, "unmatched": 0, "errors": []}
batch_id = uuid.uuid4().hex[:16]
try:
raw_matches = await fetch_understat(league, season)
except Exception as e:
logger.exception("understat fetch failed for %s %s", league, season)
result["errors"].append(f"fetch failed: {e}")
# 写死信表
await _write_ingest_failure(
db, "understat", "match", f"fetch_{league}_{season}_{batch_id}",
"fetch_error", str(e), {"league": league, "season": season},
)
return result
# === Bronze 层:先存 RawEvent ===
await _write_understat_bronze(db, raw_matches, batch_id)
# 使用 Repository
league_repo = LeagueRepository(db)
team_repo = TeamRepository(db)
# 查联赛
stmt = select(League).where(League.code == league)
league_obj = (await db.execute(stmt)).scalar_one_or_none()
league_obj = await league_repo.get_by_code(league)
if league_obj is None:
result["errors"].append(f"league {league} not found in DB")
return result
# === 批量优化: 一次规范化,收集球队名和日期 ===
normalized_matches: list = []
all_team_names: set[str] = set()
for raw in raw_matches:
if not raw.get("isResult"):
continue
@@ -109,25 +282,126 @@ class UnderstatSource:
continue
except Exception as e:
result["errors"].append(f"normalize: {e}")
# 写死信表:规范化失败
await _write_ingest_failure(
db, "understat", "match", str(raw.get("id", "")),
"normalize_error", str(e), raw,
)
continue
normalized_matches.append((nm, raw))
all_team_names.add(nm.home_team)
all_team_names.add(nm.away_team)
if not normalized_matches:
return result
# === 批量查询球队(1 次 DB 往返) ===
team_name_to_id = {}
if all_team_names:
teams = await team_repo.get_all_by_names(list(all_team_names))
team_name_to_id = {name: team.id for name, team in teams.items()}
# === 批量查询已有比赛(1 次 DB 往返,按日期范围) ===
match_dict: dict[tuple, Match] = {}
dates = [nm.date for nm, _ in normalized_matches if nm.date is not None]
if dates:
min_dt = min(dates) - timedelta(days=30)
max_dt = max(dates) + timedelta(days=30)
stmt = (
select(Match)
.options(selectinload(Match.stats))
.where(Match.league_id == league_obj.id)
.where(Match.match_date >= min_dt)
.where(Match.match_date <= max_dt)
)
for m in (await db.execute(stmt)).scalars():
key = _match_key(m.home_team_id, m.away_team_id, m.match_date_date)
match_dict[key] = m
# === 内存匹配 + 回填 xG(覆盖模式) ===
for nm, raw in normalized_matches:
home_team_id = team_name_to_id.get(nm.home_team)
away_team_id = team_name_to_id.get(nm.away_team)
if home_team_id is None or away_team_id is None:
result["unmatched"] += 1
continue
# 匹配已有 Match(天级)
existing = await find_existing_match(db, league_obj.id, nm.home_team, nm.away_team, nm.date)
match_key = _match_key(home_team_id, away_team_id, nm.date)
existing = match_dict.get(match_key)
if existing is None:
result["unmatched"] += 1
continue
# 回填 xG
now = datetime.now(timezone.utc)
source_record_id = str(raw.get("id", ""))
# 创建 stats 记录(如果不存在)
if existing.stats is None and (nm.home_xg is not None or nm.away_xg is not None):
existing.stats = MatchStats(match_id=existing.id)
existing.stats = MatchStats(
match_id=existing.id,
source="understat",
source_record_id=source_record_id,
retrieved_at=now,
available_at=now,
xg_source="understat",
xg_updated_at=now,
xg_source_record_id=source_record_id,
)
db.add(existing.stats)
await db.flush()
if existing.stats is not None:
if existing.stats.home_xg is None and nm.home_xg is not None:
existing.stats.home_xg = nm.home_xg
result["updated"] += 1
if existing.stats.away_xg is None and nm.away_xg is not None:
existing.stats.away_xg = nm.away_xg
# P2-1: Silver 层血缘 — MatchStats 创建
await _write_data_lineage(
db,
source_system="understat",
source_record_id=source_record_id,
target_table="match_stats",
target_id=existing.id,
transform_name="silver_upsert",
batch_id=batch_id,
)
await db.commit()
# xG 覆盖更新模式:当 understat 数据更新时覆盖旧值
if existing.stats is not None:
xg_updated = False
if nm.home_xg is not None:
existing.stats.home_xg = nm.home_xg
existing.stats.xg_source = "understat"
existing.stats.xg_updated_at = now
existing.stats.xg_source_record_id = source_record_id
result["updated"] += 1
xg_updated = True
if nm.away_xg is not None:
existing.stats.away_xg = nm.away_xg
existing.stats.xg_source = "understat"
existing.stats.xg_updated_at = now
existing.stats.xg_source_record_id = source_record_id
xg_updated = True
# P2-1: Silver 层血缘 — xG 覆盖更新
if xg_updated:
await _write_data_lineage(
db,
source_system="understat",
source_record_id=source_record_id,
target_table="match_stats",
target_id=existing.id,
transform_name="silver_xg_update",
batch_id=batch_id,
transform_detail=f"home_xg={nm.home_xg} away_xg={nm.away_xg}",
)
# P2-2: 数据质量检查 — 行计数合理性
total_processed = result["updated"] + result["skipped"] + result["unmatched"]
await _write_data_quality_check(
db,
check_name="understat_row_count",
entity_type="match",
entity_id=f"{league}_{season}",
expected_value=">=1",
actual_value=str(total_processed),
passed=total_processed > 0,
severity="critical" if total_processed == 0 else "info",
detail=f"league={league} season={season} updated={result['updated']} skipped={result['skipped']} unmatched={result['unmatched']}",
)
# 注意: 不在此处 commit,由调用方 UnitOfWork 控制事务
return result
+16 -3
View File
@@ -17,8 +17,10 @@ engine = create_async_engine(
settings.DATABASE_URL,
echo=False,
pool_pre_ping=True,
pool_size=10,
max_overflow=20,
pool_size=settings.DB_POOL_SIZE,
max_overflow=settings.DB_MAX_OVERFLOW,
pool_timeout=settings.DB_POOL_TIMEOUT,
pool_recycle=settings.DB_POOL_RECYCLE,
)
AsyncSessionLocal = async_sessionmaker(
@@ -53,6 +55,17 @@ async def get_db_read() -> AsyncIterator[AsyncSession]:
async def init_db() -> None:
"""开发/测试用:建表。生产建议用 alembic。"""
"""验证数据库连接(不建表)。
生产环境 schema Alembic 管理
本地开发/测试需要建表时调用 `create_all()`
"""
async with engine.begin() as conn:
# 只验证连接,不自动建表
await conn.run_sync(lambda conn: None)
async def create_all() -> None:
"""创建所有表(仅用于本地开发/测试)。"""
async with engine.begin() as conn:
await conn.run_sync(Base.metadata.create_all)
+160 -6
View File
@@ -1,10 +1,11 @@
"""5 张表 ORM: leagues / teams / matches / match_stats / predictions。"""
"""ORM 模型: leagues / teams / matches / match_stats / predictions / injuries / raw_events / ingest_failures / data_quality_checks / data_lineage"""
from __future__ import annotations
from datetime import date, datetime, timezone
from sqlalchemy import (
Boolean,
CheckConstraint,
Date,
DateTime,
Float,
@@ -13,6 +14,7 @@ from sqlalchemy import (
Integer,
String,
Text,
UniqueConstraint,
func,
)
from sqlalchemy.dialects.postgresql import JSONB
@@ -74,10 +76,20 @@ class Match(Base):
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_utcnow)
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_utcnow, onupdate=_utcnow)
league: Mapped[League] = relationship(back_populates="matches")
home_team: Mapped[Team] = relationship(foreign_keys=[home_team_id], back_populates="home_matches")
away_team: Mapped[Team] = relationship(foreign_keys=[away_team_id], back_populates="away_matches")
stats: Mapped["MatchStats | None"] = relationship(back_populates="match", cascade="all, delete-orphan")
league: Mapped[League] = relationship(back_populates="matches", lazy="selectin")
# lazy="selectin": 这些关系在业务里几乎总是一起读取(切片/回测/展示)。
# 默认的 lazy="select" 在 async SQLAlchemy 下,于 session 之外或未显式
# eager-load 时访问会抛 MissingGreenlet —— 已因此导致 form/stats/h2h
# 三个专家切片静默失败。统一改为预加载,从根上消除这类问题。
home_team: Mapped[Team] = relationship(
foreign_keys=[home_team_id], back_populates="home_matches", lazy="selectin"
)
away_team: Mapped[Team] = relationship(
foreign_keys=[away_team_id], back_populates="away_matches", lazy="selectin"
)
stats: Mapped["MatchStats | None"] = relationship(
back_populates="match", cascade="all, delete-orphan", lazy="selectin"
)
predictions: Mapped[list["Prediction"]] = relationship(back_populates="match", cascade="all, delete-orphan")
__table_args__ = (
@@ -114,9 +126,25 @@ class MatchStats(Base):
home_red_cards: Mapped[int | None] = mapped_column(Integer)
away_red_cards: Mapped[int | None] = mapped_column(Integer)
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_utcnow, onupdate=_utcnow)
# 数据血缘:追踪统计数据的来源和可用时间
source: Mapped[str | None] = mapped_column(String(30)) # bzzoiro / understat
source_record_id: Mapped[str | None] = mapped_column(String(100))
retrieved_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
available_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# xG 数据追踪:支持 understat 纠正旧 xG 值
xg_source: Mapped[str | None] = mapped_column(String(30)) # 具体 xG 数据源
xg_updated_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True)) # xG 最后更新时间
xg_source_record_id: Mapped[str | None] = mapped_column(String(100)) # xG 对应的源记录 ID
match: Mapped[Match] = relationship(back_populates="stats")
__table_args__ = (
# 数据血缘时间过滤查询用(按 available_at 取「赛前已可得」的统计)
Index("ix_match_stats_available_at", "available_at"),
# xG 数据源追踪查询用
Index("ix_match_stats_xg_source", "xg_source", "xg_updated_at"),
)
class Injury(Base):
"""球员伤停记录(api-football 数据源)。"""
@@ -156,12 +184,19 @@ class Prediction(Base):
pred_home_goals: Mapped[float | None] = mapped_column(Float)
pred_away_goals: Mapped[float | None] = mapped_column(Float)
pred_1x2: Mapped[str | None] = mapped_column(String(3))
confidence: Mapped[float | None] = mapped_column(Float)
subjective_confidence: Mapped[float | None] = mapped_column(Float) # LLM 主观置信度,非概率
reasoning: Mapped[str | None] = mapped_column(Text)
raw_response: Mapped[dict | None] = mapped_column(JSONB)
# multi-agent 模式: 各专家报告
mode: Mapped[str] = mapped_column(String(20), nullable=False, default="single")
agent_outputs: Mapped[dict | None] = mapped_column(JSONB)
# 预测状态: success / failed / degraded
status: Mapped[str] = mapped_column(String(20), nullable=False, default="success")
# 时间语义:区分比赛时间、预测创建时间、数据截止时间
match_kickoff_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
prediction_created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_utcnow)
prediction_cutoff_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
input_hash: Mapped[str | None] = mapped_column(String(64))
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_utcnow)
actual_home_goals: Mapped[int | None] = mapped_column(Integer)
actual_away_goals: Mapped[int | None] = mapped_column(Integer)
@@ -170,6 +205,125 @@ class Prediction(Base):
match: Mapped[Match] = relationship(back_populates="predictions")
__table_args__ = (
# P1-6: 数据库级唯一约束,防止同一 match+provider+model 产生重复预测
UniqueConstraint(
"match_id", "provider", "model",
name="uq_predictions_match_provider_model",
),
Index("ix_predictions_match", "match_id"),
Index("ix_predictions_provider_model", "provider", "model"),
# 数据截止时间过滤查询用(按 prediction_cutoff_at 取「赛前已生成」的预测)
Index("ix_predictions_cutoff_at", "prediction_cutoff_at"),
# 数据库级约束:最后一道防线
CheckConstraint("pred_home_goals >= 0", name="ck_pred_home_goals_nonneg"),
CheckConstraint("pred_away_goals >= 0", name="ck_pred_away_goals_nonneg"),
CheckConstraint("subjective_confidence >= 0 AND subjective_confidence <= 1", name="ck_confidence_range"),
CheckConstraint("pred_1x2 IN ('1', 'X', '2')", name="ck_pred_1x2_enum"),
CheckConstraint("mode IN ('single', 'multi')", name="ck_mode_enum"),
CheckConstraint("status IN ('success', 'failed', 'degraded')", name="ck_status_enum"),
)
# ── Bronze 层:原始事件记录 ──────────────────────────────────────────────
class RawEvent(Base):
"""Bronze 层:不可变的原始采集记录。
每个采集到的原始事件先写入此表,再规范化到 Silver (matches / match_stats)
提供完整的数据血缘回溯能力
"""
__tablename__ = "raw_events"
id: Mapped[int] = mapped_column(Integer, primary_key=True)
source_system: Mapped[str] = mapped_column(String(30), nullable=False) # bzzoiro / understat / api-football
source_record_id: Mapped[str] = mapped_column(String(100), nullable=False) # 源系统记录 ID
raw_payload: Mapped[dict] = mapped_column(JSONB, nullable=False) # 完整原始 JSON
ingested_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_utcnow, nullable=False)
ingest_batch_id: Mapped[str | None] = mapped_column(String(64)) # 批次 ID,用于关联同次采集
__table_args__ = (
UniqueConstraint("source_system", "source_record_id", name="uq_raw_events_source_record"),
Index("ix_raw_events_batch", "ingest_batch_id"),
Index("ix_raw_events_source_ingested", "source_system", "ingested_at"),
)
# ── 死信表:采集失败记录 ─────────────────────────────────────────────────
class IngestFailure(Base):
"""采集失败死信表。
当采集或规范化失败时,写入此表而非仅内存 dict
支持自动重试和人工排查
"""
__tablename__ = "ingest_failures"
id: Mapped[int] = mapped_column(Integer, primary_key=True)
source_system: Mapped[str] = mapped_column(String(30), nullable=False)
entity_type: Mapped[str] = mapped_column(String(30), nullable=False) # match / injury / team
source_record_id: Mapped[str | None] = mapped_column(String(100))
error_type: Mapped[str] = mapped_column(String(50), nullable=False) # fetch_error / normalize_error / db_error / validation_error
error_detail: Mapped[str | None] = mapped_column(Text)
raw_payload: Mapped[dict | None] = mapped_column(JSONB) # 失败时的原始数据,用于重试
retry_count: Mapped[int] = mapped_column(Integer, default=0)
next_retry_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
status: Mapped[str] = mapped_column(String(20), default="pending") # pending / retrying / resolved / abandoned
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_utcnow)
resolved_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
__table_args__ = (
Index("ix_ingest_failures_status_next_retry", "status", "next_retry_at"),
Index("ix_ingest_failures_source", "source_system", "entity_type"),
CheckConstraint("status IN ('pending', 'retrying', 'resolved', 'abandoned')", name="ck_ingest_failures_status"),
CheckConstraint("retry_count >= 0", name="ck_ingest_failures_retry_nonneg"),
)
# ── 数据质量检查表 ──────────────────────────────────────────────────────
class DataQualityCheck(Base):
"""数据质量检查结果记录。
每次运行数据质量检查时,将结果写入此表用于趋势分析和告警
"""
__tablename__ = "data_quality_checks"
id: Mapped[int] = mapped_column(Integer, primary_key=True)
check_name: Mapped[str] = mapped_column(String(100), nullable=False) # 检查名称
entity_type: Mapped[str] = mapped_column(String(30), nullable=False) # match / team / prediction
entity_id: Mapped[str | None] = mapped_column(String(50)) # 具体实体 ID
expected_value: Mapped[str | None] = mapped_column(Text) # 期望值(描述)
actual_value: Mapped[str | None] = mapped_column(Text) # 实际值
passed: Mapped[bool] = mapped_column(Boolean, nullable=False)
severity: Mapped[str] = mapped_column(String(10), nullable=False, default="warning") # info / warning / critical
detail: Mapped[str | None] = mapped_column(Text) # 详细描述
checked_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_utcnow, nullable=False)
__table_args__ = (
Index("ix_dqc_check_time", "check_name", "checked_at"),
Index("ix_dqc_entity", "entity_type", "entity_id"),
Index("ix_dqc_severity_passed", "severity", "passed"),
CheckConstraint("severity IN ('info', 'warning', 'critical')", name="ck_dqc_severity"),
)
# ── 数据血缘表 ──────────────────────────────────────────────────────────
class DataLineage(Base):
"""ETL 全过程元数据记录。
追踪从 Bronze Silver Gold 的完整转换链路
"""
__tablename__ = "data_lineage"
id: Mapped[int] = mapped_column(Integer, primary_key=True)
source_system: Mapped[str] = mapped_column(String(30), nullable=False)
source_record_id: Mapped[str] = mapped_column(String(100), nullable=False)
target_table: Mapped[str] = mapped_column(String(50), nullable=False) # matches / match_stats / predictions
target_id: Mapped[int | None] = mapped_column(Integer) # 目标表记录 ID
transform_name: Mapped[str] = mapped_column(String(100), nullable=False) # 转换步骤名称
transform_detail: Mapped[str | None] = mapped_column(Text) # 转换详情
batch_id: Mapped[str | None] = mapped_column(String(64)) # 批次 ID
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_utcnow, nullable=False)
__table_args__ = (
Index("ix_lineage_source", "source_system", "source_record_id"),
Index("ix_lineage_target", "target_table", "target_id"),
Index("ix_lineage_batch", "batch_id"),
)
+138
View File
@@ -0,0 +1,138 @@
"""Repository 层:封装数据访问。
Repository 只负责查询,不负责事务提交
事务由 Application Service 通过 UnitOfWork 控制
"""
from __future__ import annotations
from datetime import datetime
from sqlalchemy import select
from sqlalchemy.orm import selectinload
from sqlalchemy.ext.asyncio import AsyncSession
from src.db.models import League, Match, Prediction, Team
class MatchRepository:
"""比赛数据访问。"""
def __init__(self, session: AsyncSession) -> None:
self._session = session
async def get_by_id(self, match_id: int) -> Match | None:
return await self._session.get(Match, match_id)
async def get_with_relations(self, match_id: int) -> Match | None:
stmt = (
select(Match)
.options(
selectinload(Match.league),
selectinload(Match.home_team),
selectinload(Match.away_team),
selectinload(Match.stats),
)
.where(Match.id == match_id)
)
return (await self._session.execute(stmt)).scalar_one_or_none()
async def find_by_teams_and_date(
self, league_id: int, home_team_id: int, away_team_id: int, date
) -> Match | None:
"""按联赛+主队+客队+日期查找比赛(天级匹配)。
预加载 stats:调用方(understat 回填)会读取 existing.stats,
async session 下惰性加载会抛 MissingGreenlet
P2-3: 使用 match_date_date(已建索引)做等值匹配,避免 func.date()
导致的全表扫描
"""
if isinstance(date, datetime):
date = date.date()
elif hasattr(date, "date"):
date = date.date()
else:
# 字符串等其它格式,尝试转换
date = datetime.fromisoformat(str(date)).date()
stmt = (
select(Match)
.options(selectinload(Match.stats))
.where(Match.league_id == league_id)
.where(Match.home_team_id == home_team_id)
.where(Match.away_team_id == away_team_id)
.where(Match.match_date_date == date)
)
return (await self._session.execute(stmt)).scalar_one_or_none()
async def add(self, match: Match) -> None:
self._session.add(match)
await self._session.flush()
class TeamRepository:
"""球队数据访问。"""
def __init__(self, session: AsyncSession) -> None:
self._session = session
async def get_by_name(self, name: str) -> Team | None:
stmt = select(Team).where(Team.name == name)
return (await self._session.execute(stmt)).scalar_one_or_none()
async def get_or_create(self, name: str) -> Team:
"""按名获取球队,不存在则创建。"""
team = await self.get_by_name(name)
if team is None:
team = Team(name=name)
self._session.add(team)
await self._session.flush()
return team
async def get_all_by_names(self, names: list[str]) -> dict[str, Team]:
"""批量获取球队,返回 name → Team 映射。"""
if not names:
return {}
stmt = select(Team).where(Team.name.in_(names))
teams = (await self._session.execute(stmt)).scalars().all()
return {t.name: t for t in teams}
async def add(self, team: Team) -> None:
self._session.add(team)
await self._session.flush()
class LeagueRepository:
"""联赛数据访问。"""
def __init__(self, session: AsyncSession) -> None:
self._session = session
async def get_by_code(self, code: str) -> League | None:
stmt = select(League).where(League.code == code)
return (await self._session.execute(stmt)).scalar_one_or_none()
async def get_or_create(self, code: str, name: str, country: str | None = None) -> League:
league = await self.get_by_code(code)
if league is None:
league = League(code=code, name=name, country=country)
self._session.add(league)
await self._session.flush()
return league
async def add(self, league: League) -> None:
self._session.add(league)
await self._session.flush()
class PredictionRepository:
"""预测记录数据访问。"""
def __init__(self, session: AsyncSession) -> None:
self._session = session
async def get_by_id(self, prediction_id: int) -> Prediction | None:
return await self._session.get(Prediction, prediction_id)
async def add(self, prediction: Prediction) -> None:
self._session.add(prediction)
await self._session.flush()
+37
View File
@@ -0,0 +1,37 @@
"""工作单元(Unit of Work):统一事务边界。
使用方式:
async with get_uow() as uow:
await uow.session.get(Match, 1)
await uow.commit()
"""
from __future__ import annotations
from collections.abc import AsyncIterator
from contextlib import asynccontextmanager
from typing import TYPE_CHECKING
from src.db.base import AsyncSessionLocal
if TYPE_CHECKING:
from sqlalchemy.ext.asyncio import AsyncSession
@asynccontextmanager
async def get_uow() -> AsyncIterator[AsyncSession]:
"""创建新的工作单元(用于非路由上下文)。
用法:
async with get_uow() as session:
await session.get(...)
# 退出时自动 commit(无异常) 或 rollback(有异常)
"""
session = AsyncSessionLocal()
try:
yield session
await session.commit()
except Exception:
await session.rollback()
raise
finally:
await session.close()
+48 -32
View File
@@ -12,7 +12,7 @@ import logging
from dataclasses import dataclass, field
from pathlib import Path
from src.llm.context_builder import MatchHeader
from src.llm.context_builder import MatchHeader, SliceResult
from src.llm.provider import LLMProvider, LLMResponse
logger = logging.getLogger(__name__)
@@ -48,7 +48,7 @@ class AgentReport:
data_sufficiency: str = "medium" # high | medium | low | none
analysis: str = ""
home_edge: float | None = None # -1.0 ~ 1.0, 正=利主队
confidence: float | None = None # 0.0 ~ 1.0
subjective_confidence: float | None = None # 0.0 ~ 1.0
key_evidence: list[str] = field(default_factory=list)
# xg agent 专属
exp_home_goals: float | None = None
@@ -67,7 +67,7 @@ class AgentReport:
"data_sufficiency": self.data_sufficiency,
"analysis": self.analysis,
"home_edge": self.home_edge,
"confidence": self.confidence,
"subjective_confidence": self.subjective_confidence,
"key_evidence": self.key_evidence,
"exp_home_goals": self.exp_home_goals,
"exp_away_goals": self.exp_away_goals,
@@ -80,7 +80,12 @@ class AgentReport:
def _is_no_data(slice_text: str) -> bool:
"""切片是否全无数据(除了标题行全是无数据)。"""
"""兜底: 判断纯字符串切片是否全无数据。
仅用于 slice_fn 返回 `str`(未升级为 SliceResult)的场景
新代码应让切片返回 SliceResult 并显式声明 has_data 字符串子串匹配
依赖具体文案(无比分数据无伤停数据等变体会漏判),不可靠
"""
body = [ln.strip() for ln in slice_text.splitlines() if ln.strip()]
# 去掉标题行(── 开头)
content = [ln for ln in body if not ln.startswith("──")]
@@ -89,6 +94,18 @@ def _is_no_data(slice_text: str) -> bool:
return all(any(s in ln for s in NO_DATA_SENTINELS) for ln in content)
def _slice_has_data(slice_result) -> tuple[str, bool]:
"""把切片返回值统一成 (text, has_data)。
优先使用 SliceResult.has_data(结构化,可信);若切片函数仍返回 str,
则回退到文案子串匹配(向后兼容)
"""
if isinstance(slice_result, SliceResult):
return slice_result.text, slice_result.has_data
text = str(slice_result)
return text, not _is_no_data(text)
def _stub_no_data(agent: str) -> AgentReport:
return AgentReport(
agent=agent,
@@ -99,36 +116,34 @@ def _stub_no_data(agent: str) -> AgentReport:
def _parse_report(agent: str, parsed: dict, resp: LLMResponse, model: str) -> AgentReport:
"""把 LLM JSON 输出解析为 AgentReport,字段宽容处理"""
def _f(v, default=None):
try:
return float(v) if v is not None else default
except (TypeError, ValueError):
return default
"""把 LLM JSON 输出解析为 AgentReport,经过严格校验"""
from src.llm.validation import validate_agent_output
suff = str(parsed.get("data_sufficiency", "medium")).lower()
if suff not in ("high", "medium", "low", "none"):
suff = "medium"
evidence = parsed.get("key_evidence") or []
if isinstance(evidence, str):
evidence = [evidence]
score = parsed.get("probable_score")
if isinstance(score, dict):
score = f"{score.get('home', '?')}-{score.get('away', '?')}"
try:
validated = validate_agent_output(parsed)
except Exception as e:
# 校验失败 → 返回 parse_error 而非静默降级
return AgentReport(
agent=agent,
status="parse_error",
analysis=f"输出校验失败: {e}",
model=model,
latency_ms=resp.latency_ms,
prompt_tokens=resp.prompt_tokens,
completion_tokens=resp.completion_tokens,
)
return AgentReport(
agent=agent,
status="ok",
data_sufficiency=suff,
analysis=str(parsed.get("analysis", ""))[:600],
home_edge=_f(parsed.get("home_edge")),
confidence=_f(parsed.get("confidence")),
key_evidence=[str(e)[:120] for e in evidence[:5]],
exp_home_goals=_f(parsed.get("exp_home_goals")),
exp_away_goals=_f(parsed.get("exp_away_goals")),
probable_score=score if isinstance(score, str) else None,
data_sufficiency=validated.data_sufficiency,
analysis=validated.analysis,
home_edge=validated.home_edge,
subjective_confidence=validated.subjective_confidence,
key_evidence=validated.key_evidence,
exp_home_goals=validated.exp_home_goals,
exp_away_goals=validated.exp_away_goals,
probable_score=validated.probable_score,
model=model,
latency_ms=resp.latency_ms,
prompt_tokens=resp.prompt_tokens,
@@ -147,13 +162,14 @@ async def run_agent(
"""执行单个专家 agent: 切片 → no_data 门控 → 调 LLM → 解析报告。"""
# 1. 数据切片
try:
slice_text = await spec.slice_fn(header, before=before)
slice_result = await spec.slice_fn(header, before=before)
except Exception as e:
logger.exception("agent %s slice failed", spec.name)
return AgentReport(agent=spec.name, status="error", analysis=f"数据切片失败: {e}")
# 2. no_data 门控: 切片无数据 → 不调 LLM
if _is_no_data(slice_text):
# 2. no_data 门控: 切片显式声明无数据 → 不调 LLM
slice_text, has_data = _slice_has_data(slice_result)
if not has_data:
logger.debug("agent %s: slice is no_data, skipping LLM", spec.name)
return _stub_no_data(spec.name)
+37 -14
View File
@@ -2,14 +2,17 @@
from __future__ import annotations
import asyncio
import hashlib
import json
import logging
import time
from dataclasses import dataclass
from datetime import datetime, timezone
from src.core.config import settings
from src.db.base import AsyncSessionLocal
from src.db.models import Match, Prediction
from src.db.unit_of_work import get_uow
from src.llm.agents.base import AgentReport, AgentSpec, load_agent_prompt
from src.llm.context_builder import (
MatchHeader,
@@ -68,7 +71,7 @@ class MultiPredictResult:
pred_home_goals: float | None
pred_away_goals: float | None
pred_1x2: str | None
confidence: float | None
subjective_confidence: float | None
reasoning: str | None
agent_outputs: list[dict]
agent_weights: dict | None
@@ -164,6 +167,9 @@ async def predict_match_multi(
# 1. 比赛头(各 agent 共享;不存在则 404)
header = await load_match_header(match_id)
match_kickoff_at = header.match_dt
prediction_cutoff_at = header.match_dt # 默认:比赛时间作为数据截止
now = datetime.now(timezone.utc)
# 2. 并行专家
specialist_provider = _get_specialist_provider()
@@ -177,13 +183,26 @@ async def predict_match_multi(
latency_ms = int((time.perf_counter() - start) * 1000)
# 4. 存库
async with AsyncSessionLocal() as db:
m = await db.get(Match, match_id)
# 3.5 计算输入 hash(基于终裁报告)
input_hash = hashlib.sha256(
_reports_to_json(reports).encode("utf-8")
).hexdigest()
# 4. 存库(使用 UnitOfWork)
async with get_uow() as session:
m = await session.get(Match, match_id)
if m is None:
raise ValueError(f"match {match_id} not found")
agent_weights = final.get("agent_weights")
# 严格校验终裁输出
from src.llm.validation import validate_agent_weights, validate_prediction_output
try:
validated = validate_prediction_output(final)
except Exception as e:
raise RuntimeError(f"终裁输出校验失败: {e}")
# agent_weights 同样必须过校验(旧实现直接取 raw 值落库,未做任何检查)
agent_weights = validate_agent_weights(final.get("agent_weights"))
pred = Prediction(
match_id=match_id,
provider=settings.LLM_PROVIDER,
@@ -193,17 +212,21 @@ async def predict_match_multi(
prompt_tokens=sum(r.prompt_tokens or 0 for r in reports) + agg_prompt_tokens,
completion_tokens=sum(r.completion_tokens or 0 for r in reports) + agg_completion_tokens,
latency_ms=latency_ms,
pred_home_goals=final.get("pred_home_goals"),
pred_away_goals=final.get("pred_away_goals"),
pred_1x2=final.get("1x2"),
confidence=final.get("confidence"),
reasoning=final.get("reasoning"),
pred_home_goals=validated.pred_home_goals,
pred_away_goals=validated.pred_away_goals,
pred_1x2=validated.pred_1x2,
subjective_confidence=validated.subjective_confidence,
reasoning=validated.reasoning,
raw_response=final,
agent_outputs=[r.to_dict() for r in reports],
status="success",
match_kickoff_at=match_kickoff_at,
prediction_cutoff_at=prediction_cutoff_at,
prediction_created_at=now,
input_hash=input_hash,
)
db.add(pred)
await db.commit()
await db.refresh(pred)
session.add(pred)
await session.refresh(pred)
return MultiPredictResult(
prediction_id=pred.id,
@@ -214,7 +237,7 @@ async def predict_match_multi(
pred_home_goals=pred.pred_home_goals,
pred_away_goals=pred.pred_away_goals,
pred_1x2=pred.pred_1x2,
confidence=pred.confidence,
subjective_confidence=pred.subjective_confidence,
reasoning=pred.reasoning,
agent_outputs=pred.agent_outputs,
agent_weights=agent_weights,
+93 -55
View File
@@ -3,18 +3,23 @@
核心机制:
- build_context 已内置 before=match_date,天然防未来信息泄漏
- 对历史比赛跑预测 用实际比分 settle 统计准确率
- 并发控制: asyncio.Semaphore 限制同时 LLM 调用数
"""
from __future__ import annotations
import asyncio
import logging
from dataclasses import dataclass, field
from datetime import datetime
from sqlalchemy import and_, select
from sqlalchemy import select
from sqlalchemy.orm import selectinload
from src.db.base import AsyncSessionLocal
from src.db.models import League, Match, Prediction
from src.db.models import Match
from src.db.unit_of_work import get_uow
from src.llm.eval import settle_prediction
from src.llm.predict import predict_match
from src.llm.utils import actual_1x2
logger = logging.getLogger(__name__)
@@ -33,11 +38,28 @@ class BacktestMatchResult:
pred_home: float | None
pred_away: float | None
pred_1x2: str | None
confidence: float | None
subjective_confidence: float | None
correct_1x2: bool
prediction_id: int
@dataclass
class BacktestCandidate:
"""回测候选比赛(字段快照,不持有 ORM 对象)。
session 关闭后仍可安全读取:所有需要的关系字段已在查询时物化为普通值,
避免在 session 之外访问惰性加载的关系属性(会抛 MissingGreenlet)
"""
match_id: int
league_code: str | None
home_team: str
away_team: str
match_date: datetime
home_goals: int
away_goals: int
@dataclass
class BacktestSummary:
"""回测汇总统计。"""
@@ -45,20 +67,11 @@ class BacktestSummary:
scored: int
accuracy_1x2: float | None = None
avg_score_rmse: float | None = None
avg_confidence: float | None = None
avg_subjective_confidence: float | None = None
calibration: list[dict] = field(default_factory=list)
results: list[BacktestMatchResult] = field(default_factory=list)
def _actual_1x2(home: int, away: int) -> str:
"""实际比分 → 胜平负。"""
if home > away:
return "1"
if home == away:
return "X"
return "2"
async def _get_historical_matches(
db,
*,
@@ -66,10 +79,20 @@ async def _get_historical_matches(
date_from: str | None = None,
date_to: str | None = None,
limit: int = 50,
) -> list[Match]:
"""查询已完赛且有比分的比赛(回测候选)。"""
) -> list[BacktestCandidate]:
"""查询已完赛且有比分的比赛(回测候选)。
返回普通值快照而非 ORM 对象:调用方在 session 关闭后仍需使用这些字段,
league / home_team / away_team 是惰性加载关系, async 下于 session
之外访问会抛 MissingGreenlet这里用 selectinload 预加载后立即物化
"""
stmt = (
select(Match)
.options(
selectinload(Match.league),
selectinload(Match.home_team),
selectinload(Match.away_team),
)
.where(Match.match_status == "finished")
.where(Match.home_goals.is_not(None))
.where(Match.away_goals.is_not(None))
@@ -83,7 +106,19 @@ async def _get_historical_matches(
stmt = stmt.order_by(Match.match_date.desc()).limit(limit)
result = await db.execute(stmt)
return list(result.scalars().all())
# 在 session 内物化为纯数据,切断与 ORM 会话的耦合
return [
BacktestCandidate(
match_id=m.id,
league_code=m.league.code if m.league else None,
home_team=m.home_team.name if m.home_team else "?",
away_team=m.away_team.name if m.away_team else "?",
match_date=m.match_date,
home_goals=m.home_goals,
away_goals=m.away_goals,
)
for m in result.scalars().all()
]
async def run_backtest(
@@ -108,46 +143,49 @@ async def run_backtest(
Returns:
BacktestSummary 含逐场结果 + 汇总统计
"""
async with AsyncSessionLocal() as db:
matches = await _get_historical_matches(
db, league_id=league_id, date_from=date_from, date_to=date_to, limit=limit
async with get_uow() as session:
candidates = await _get_historical_matches(
session, league_id=league_id, date_from=date_from, date_to=date_to, limit=limit
)
summary = BacktestSummary(total=len(matches), scored=0)
summary = BacktestSummary(total=len(candidates), scored=0)
for m in matches:
try:
# 预测 (build_context 内部已用 before=match_date 防泄漏)
result = await predict_match(m.id, mode=mode, model=model)
# P2: 并发控制,同时最多 3 场预测(避免 LLM API 限流,保护下游服务)
sem = asyncio.Semaphore(3)
# 用实际比分 settle
await settle_prediction(result.prediction_id, m.home_goals, m.away_goals)
async def _one(c: BacktestCandidate) -> BacktestMatchResult | None:
async with sem:
try:
result = await predict_match(c.match_id, mode=mode, model=model, use_cache=False, backtest=True)
await settle_prediction(result.prediction_id, c.home_goals, c.away_goals)
actual = actual_1x2(c.home_goals, c.away_goals)
return BacktestMatchResult(
match_id=c.match_id,
league_code=c.league_code,
home_team=c.home_team,
away_team=c.away_team,
match_date=c.match_date.strftime("%Y-%m-%d") if c.match_date else "?",
actual_home=c.home_goals,
actual_away=c.away_goals,
actual_1x2=actual,
pred_home=result.pred_home_goals,
pred_away=result.pred_away_goals,
pred_1x2=result.pred_1x2,
subjective_confidence=result.subjective_confidence,
correct_1x2=result.pred_1x2 == actual,
prediction_id=result.prediction_id,
)
except Exception:
logger.exception("backtest match %s failed", c.match_id)
return None
actual = _actual_1x2(m.home_goals, m.away_goals)
correct = result.pred_1x2 == actual
bt = BacktestMatchResult(
match_id=m.id,
league_code=m.league.code if m.league else None,
home_team=m.home_team.name if m.home_team else "?",
away_team=m.away_team.name if m.away_team else "?",
match_date=m.match_date.strftime("%Y-%m-%d") if m.match_date else "?",
actual_home=m.home_goals,
actual_away=m.away_goals,
actual_1x2=actual,
pred_home=result.pred_home_goals,
pred_away=result.pred_away_goals,
pred_1x2=result.pred_1x2,
confidence=result.confidence,
correct_1x2=correct,
prediction_id=result.prediction_id,
)
summary.results.append(bt)
# 并行执行,保持结果顺序
results = await asyncio.gather(*[_one(c) for c in candidates])
for r in results:
if r is not None:
summary.results.append(r)
summary.scored += 1
except Exception as e:
logger.warning("backtest match %s failed: %s", m.id, e)
# 汇总统计
if summary.scored > 0:
correct_count = sum(1 for r in summary.results if r.correct_1x2)
@@ -162,10 +200,10 @@ async def run_backtest(
if errors:
summary.avg_score_rmse = round(sum(errors) / len(errors), 2)
# 平均置信度
confs = [r.confidence for r in summary.results if r.confidence is not None]
# 平均主观置信度
confs = [r.subjective_confidence for r in summary.results if r.subjective_confidence is not None]
if confs:
summary.avg_confidence = round(sum(confs) / len(confs), 2)
summary.avg_subjective_confidence = round(sum(confs) / len(confs), 2)
# 校准:按置信度分桶,看实际准确率是否匹配
summary.calibration = _compute_calibration(summary.results)
@@ -183,11 +221,11 @@ def _compute_calibration(results: list[BacktestMatchResult]) -> list[dict]:
"0.0-0.3": {"range": (0.0, 0.3), "total": 0, "correct": 0},
}
for r in results:
if r.confidence is None:
if r.subjective_confidence is None:
continue
for key, b in buckets.items():
lo, hi = b["range"]
if lo <= r.confidence <= hi:
if lo <= r.subjective_confidence <= hi:
b["total"] += 1
if r.correct_1x2:
b["correct"] += 1
+188 -69
View File
@@ -1,16 +1,22 @@
"""上下文构建器:数据切片 + 拼接。
架构:
- match_header: 比赛基础信息(对阵双方/联赛/时间)
- 切片函数: 每个领域 agent 一个数据切片(h2h / form / standings / injuries / xg)
- build_context: agent 路径,拼接全部切片(行为与旧版一致)
- match_header: 比赛基础信息(对阵双方/联赛/时间)
- 切片函数: 每个领域 agent 一个数据切片(h2h / form / standings / injuries / xg)
- build_context: agent 路径,拼接全部切片(行为与旧版一致)
multi-agent 路径由 agents/orchestrator.py 调用切片函数,每个专家只拿自己的切片
性能说明:
build_context 创建一个共享 session 并传给所有切片函数,
避免每个切片独立创建 session 回测 20 场并发时,
5 个切片 × 20 = 100 个连接会耗尽连接池(pool_size=15)
"""
from __future__ import annotations
import logging
from dataclasses import dataclass
from typing import TYPE_CHECKING
from sqlalchemy import select
from sqlalchemy.orm import selectinload
@@ -18,6 +24,9 @@ from sqlalchemy.orm import selectinload
from src.db.base import AsyncSessionLocal
from src.db.models import Match
if TYPE_CHECKING:
from sqlalchemy.ext.asyncio import AsyncSession
logger = logging.getLogger(__name__)
@@ -30,12 +39,38 @@ def _outcome(home_goals: int, away_goals: int, side: str) -> str:
return "W" if away_goals > home_goals else ("D" if away_goals == home_goals else "L")
def _is_stats_available(stats, before) -> bool:
"""检查统计数据在 cutoff 时间是否已可用。"""
if before is None:
return True
if stats.available_at is None:
return True # 无时间信息时保守处理:允许使用
return stats.available_at <= before
@dataclass
class SliceResult:
"""数据切片的显式结果(替代「靠文案子串猜有无数据」)。
旧实现用 `"无数据" in slice_text` 判断,依赖具体文案 一旦某个切片
写成无比分数据无伤停数据这类变体,判断就会静默失配
(见审查报告 P2-1)这里让切片函数直接声明 `has_data`,不再猜
"""
text: str
has_data: bool
n_records: int = 0
def __str__(self) -> str: # 让老调用点可直接当 str 用
return self.text
@dataclass
class MatchContext:
match_id: int
text: str
has_stats: bool
has_injuries: bool
match_dt: object | None = None # 比赛时间(回测防泄漏 + 快照用)
@dataclass
@@ -54,11 +89,19 @@ class MatchHeader:
league_id: int
async def load_match_header(match_id: int) -> MatchHeader:
"""加载比赛头信息(各 agent 共用)。"""
async with AsyncSessionLocal() as db:
async def load_match_header(match_id: int, db: AsyncSession | None = None) -> MatchHeader:
"""加载比赛头信息(各 agent 共用)。
Args:
match_id: 比赛 ID
db: 可选的共享 session不传则自建(向后兼容)
"""
if db is not None:
m = await _load_match(db, match_id)
return _to_header(m)
async with AsyncSessionLocal() as new_db:
m = await _load_match(new_db, match_id)
return _to_header(m)
def _to_header(m: Match) -> MatchHeader:
@@ -88,16 +131,24 @@ def header_text(h: MatchHeader) -> str:
# 切片函数: 每个领域 agent 一个
# ============================================================
async def h2h_slice(header: MatchHeader, *, limit: int = 8, before=None) -> str:
"""E - 历史交锋切片: 过去数年 + 近期交手数据,提取交手规律。before=match_date 用于回测。"""
async with AsyncSessionLocal() as db:
async def h2h_slice(header: MatchHeader, *, limit: int = 8, before=None, db: AsyncSession | None = None) -> SliceResult:
"""E - 历史交锋切片: 过去数年 + 近期交手数据,提取交手规律。before=match_date 用于回测。
db: 可选共享 session,避免每个切片独立建连(见模块 docstring)
"""
if db is not None:
h2h = await _get_h2h(db, header.home_team_id, header.away_team_id, before=before, limit=limit)
else:
async with AsyncSessionLocal() as new_db:
h2h = await _get_h2h(new_db, header.home_team_id, header.away_team_id, before=before, limit=limit)
lines = [f"── 历史交锋(近 {limit} 次) ──"]
n_with_score = 0
if h2h:
home_wins = draws = away_wins = 0
for hm in h2h:
d = hm.match_date.strftime("%Y-%m") if hm.match_date else "?"
if hm.home_goals is not None:
n_with_score += 1
if hm.home_goals > hm.away_goals: home_wins += 1
elif hm.home_goals == hm.away_goals: draws += 1
else: away_wins += 1
@@ -109,15 +160,24 @@ async def h2h_slice(header: MatchHeader, *, limit: int = 8, before=None) -> str:
lines.append(f" 总计 {total} 场: 主队 {home_wins}{draws}{away_wins}")
else:
lines.append(" 无数据")
return "\n".join(lines)
# has_data 以「有比分的交锋」为准:仅有对阵无比分时不足以支撑分析
return SliceResult(text="\n".join(lines), has_data=n_with_score > 0, n_records=n_with_score)
async def form_slice(header: MatchHeader, *, limit: int = 5, before=None) -> str:
"""A - 近期状态切片: 两队近 N 场赛果、关键事件、走势判断。before=match_date 用于回测。"""
async with AsyncSessionLocal() as db:
async def form_slice(header: MatchHeader, *, limit: int = 5, before=None, db: AsyncSession | None = None) -> SliceResult:
"""A - 近期状态切片: 两队近 N 场赛果、关键事件、走势判断。before=match_date 用于回测。
db: 可选共享 session,避免每个切片独立建连(见模块 docstring)
"""
if db is not None:
home_form = await _get_form(db, header.home_team_id, before=before, limit=limit)
away_form = await _get_form(db, header.away_team_id, before=before, limit=limit)
else:
async with AsyncSessionLocal() as new_db:
home_form = await _get_form(new_db, header.home_team_id, before=before, limit=limit)
away_form = await _get_form(new_db, header.away_team_id, before=before, limit=limit)
lines = []
n_scored = 0
for label, name, form, side in (
("主队", header.home_name, home_form, "home"),
("客队", header.away_name, away_form, "away"),
@@ -130,9 +190,11 @@ async def form_slice(header: MatchHeader, *, limit: int = 5, before=None) -> str
if o == "W": wins += 1
elif o == "D": draws += 1
else: losses += 1
if fm.home_goals is not None:
n_scored += 1
score = f"{fm.home_goals}-{fm.away_goals}" if fm.home_goals is not None else "vs"
xg = ""
if fm.stats and fm.stats.home_xg is not None:
if fm.stats and _is_stats_available(fm.stats, before) and fm.stats.home_xg is not None:
own = fm.stats.home_xg if side == "home" else fm.stats.away_xg
xg = f" (xG {own:.1f})"
opp = fm.away_team.name if side == "home" else fm.home_team.name
@@ -140,15 +202,23 @@ async def form_slice(header: MatchHeader, *, limit: int = 5, before=None) -> str
lines.append(f"{len(form)} 场: {wins}{draws}{losses}")
else:
lines.append(" 无数据")
return "\n".join(lines)
return SliceResult(text="\n".join(lines), has_data=n_scored > 0, n_records=n_scored)
async def stats_slice(header: MatchHeader, *, limit: int = 10, before=None) -> str:
"""B - 攻防数据切片: 进球、射门、控球,评估攻防强度。before=match_date 用于回测。"""
async with AsyncSessionLocal() as db:
async def stats_slice(header: MatchHeader, *, limit: int = 10, before=None, db: AsyncSession | None = None) -> SliceResult:
"""B - 攻防数据切片: 进球、射门、控球,评估攻防强度。before=match_date 用于回测。
db: 可选共享 session,避免每个切片独立建连(见模块 docstring)
"""
if db is not None:
home_form = await _get_form(db, header.home_team_id, before=before, limit=limit)
away_form = await _get_form(db, header.away_team_id, before=before, limit=limit)
else:
async with AsyncSessionLocal() as new_db:
home_form = await _get_form(new_db, header.home_team_id, before=before, limit=limit)
away_form = await _get_form(new_db, header.away_team_id, before=before, limit=limit)
lines = [f"── 攻防数据(近 {limit} 场) ──"]
n_total = 0
for label, name, form, side in (
("主队", header.home_name, home_form, "home"),
("客队", header.away_name, away_form, "away"),
@@ -161,7 +231,8 @@ async def stats_slice(header: MatchHeader, *, limit: int = 10, before=None) -> s
gf += fm.home_goals if side == "home" else fm.away_goals
ga += fm.away_goals if side == "home" else fm.home_goals
n += 1
if fm.stats:
# 只使用 cutoff 之前已可用的统计数据
if fm.stats and _is_stats_available(fm.stats, before):
if fm.stats.home_shots is not None:
shots += fm.stats.home_shots if side == "home" else fm.stats.away_shots
sot += fm.stats.home_shots_on_target if side == "home" else fm.stats.away_shots_on_target
@@ -173,6 +244,7 @@ async def stats_slice(header: MatchHeader, *, limit: int = 10, before=None) -> s
xg += fm.stats.home_xg if side == "home" else fm.stats.away_xg
xga += fm.stats.away_xg if side == "home" else fm.stats.home_xg
n_xg += 1
n_total += n
if n > 0:
lines.append(f" {label} {name}:")
lines.append(f" 场均进球 {gf/n:.2f}, 场均失球 {ga/n:.2f}")
@@ -183,15 +255,23 @@ async def stats_slice(header: MatchHeader, *, limit: int = 10, before=None) -> s
lines.append(f" {label} {name}: 无比分数据")
else:
lines.append(f" {label} {name}: 无数据")
return "\n".join(lines)
return SliceResult(text="\n".join(lines), has_data=n_total > 0, n_records=n_total)
async def home_away_slice(header: MatchHeader, *, limit: int = 10, before=None) -> str:
"""C - 主客因素切片: 主场战绩 vs 客场战绩,评估地理优势影响。before=match_date 用于回测。"""
async with AsyncSessionLocal() as db:
async def home_away_slice(header: MatchHeader, *, limit: int = 10, before=None, db: AsyncSession | None = None) -> SliceResult:
"""C - 主客因素切片: 主场战绩 vs 客场战绩,评估地理优势影响。before=match_date 用于回测。
db: 可选共享 session,避免每个切片独立建连(见模块 docstring)
"""
if db is not None:
home_home = await _get_home_away(db, header.home_team_id, "home", before=before, limit=limit)
away_away = await _get_home_away(db, header.away_team_id, "away", before=before, limit=limit)
else:
async with AsyncSessionLocal() as new_db:
home_home = await _get_home_away(new_db, header.home_team_id, "home", before=before, limit=limit)
away_away = await _get_home_away(new_db, header.away_team_id, "away", before=before, limit=limit)
lines = ["── 主客因素 ──"]
n_total = 0
for label, name, matches, side in (
("主队主场", header.home_name, home_home, "home"),
("客队客场", header.away_name, away_away, "away"),
@@ -207,6 +287,7 @@ async def home_away_slice(header: MatchHeader, *, limit: int = 10, before=None)
gf += m.home_goals if side == "home" else m.away_goals
ga += m.away_goals if side == "home" else m.home_goals
n = wins + draws + losses
n_total += n
if n > 0:
pct = wins / n * 100
lines.append(f" {label} {name}(近 {n} 场): {wins}{draws}{losses}负, 胜率 {pct:.0f}%")
@@ -215,22 +296,31 @@ async def home_away_slice(header: MatchHeader, *, limit: int = 10, before=None)
lines.append(f" {label} {name}: 无比分数据")
else:
lines.append(f" {label} {name}: 无数据")
return "\n".join(lines)
return SliceResult(text="\n".join(lines), has_data=n_total > 0, n_records=n_total)
async def injuries_slice(header: MatchHeader, *, before=None) -> str:
"""D - 阵容完整性切片: 伤停与停赛名单,评估战力缺失程度。"""
async def injuries_slice(header: MatchHeader, *, before=None, db: AsyncSession | None = None) -> SliceResult:
"""D - 阵容完整性切片: 伤停与停赛名单,评估战力缺失程度。
before=cutoff: 只使用 cutoff 之前已采集的伤停数据,防回测泄漏
db: 可选共享 session(见模块 docstring)
"""
from src.data.injuries import get_injuries_for_match
async with AsyncSessionLocal() as db:
home_injuries = await get_injuries_for_match(db, header.home_team_id, before or header.match_dt)
away_injuries = await get_injuries_for_match(db, header.away_team_id, before or header.match_dt)
cutoff = before or header.match_dt
if db is not None:
home_injuries = await get_injuries_for_match(db, header.home_team_id, cutoff, as_of=cutoff)
away_injuries = await get_injuries_for_match(db, header.away_team_id, cutoff, as_of=cutoff)
else:
async with AsyncSessionLocal() as new_db:
home_injuries = await get_injuries_for_match(new_db, header.home_team_id, cutoff, as_of=cutoff)
away_injuries = await get_injuries_for_match(new_db, header.away_team_id, cutoff, as_of=cutoff)
lines = ["── 阵容完整性 ──"]
has_data = False
n_records = 0
for label, injuries in (("主队", home_injuries), ("客队", away_injuries)):
if injuries:
has_data = True
n_records += len(injuries)
lines.append(f" {label}伤停({len(injuries)}人):")
for inj in injuries[:8]: # 最多显示 8 条
reason = inj.reason or inj.injury_type or "未知"
@@ -240,54 +330,61 @@ async def injuries_slice(header: MatchHeader, *, before=None) -> str:
else:
lines.append(f" {label}: 无伤停数据")
if not has_data:
return "── 阵容完整性 ──\n 无数据"
if n_records == 0:
return SliceResult(text="── 阵容完整性 ──\n 无数据", has_data=False, n_records=0)
return "\n".join(lines)
return SliceResult(text="\n".join(lines), has_data=True, n_records=n_records)
# ============================================================
# 单 agent 路径: 拼接全部切片(行为与旧版一致)
# ============================================================
async def build_context(match_id: int, *, form_last: int = 5, h2h_last: int = 5) -> MatchContext:
"""单 agent 路径的完整上下文: 拼接全部切片(before=比赛时间,防未来信息)。"""
header = await load_match_header(match_id)
parts = [header_text(header), ""]
has_stats = False
has_injuries = False
async def build_context(match_id: int, *, form_last: int = 5, h2h_last: int = 5, backtest: bool = False) -> MatchContext:
"""单 agent 路径的完整上下文: 拼接全部切片(before=比赛时间,防未来信息)。
form_text = await form_slice(header, limit=form_last, before=header.match_dt)
if "无数据" not in form_text:
has_stats = True
parts.append(form_text)
parts.append("")
has_stats / has_injuries 直接取切片显式声明的 has_data,
不再靠文案子串匹配(见审查报告 P2-1)
h2h_text = await h2h_slice(header, limit=h2h_last, before=header.match_dt)
parts.append(h2h_text)
parts.append("")
P2-6: backtest=True cutoff = match_date - 1,确保只用赛前数据
stats_text = await stats_slice(header, before=header.match_dt)
if "无数据" not in stats_text:
has_stats = True
parts.append(stats_text)
parts.append("")
P1-1: 使用单个共享 session 贯穿所有切片查询,避免连接池耗尽
"""
async with AsyncSessionLocal() as db:
header = await load_match_header(match_id, db=db)
# P2-6: 回测模式下 cutoff 提前 1 天,防止比赛日数据泄漏
cutoff = header.match_dt
if backtest and header.match_dt:
from datetime import timedelta
cutoff = header.match_dt - timedelta(days=1)
parts = [header_text(header), ""]
home_away_text = await home_away_slice(header, before=header.match_dt)
parts.append(home_away_text)
parts.append("")
form_res = await form_slice(header, limit=form_last, before=cutoff, db=db)
parts.append(form_res.text)
parts.append("")
injuries_text = await injuries_slice(header, before=header.match_dt)
if "无数据" not in injuries_text:
has_injuries = True
parts.append(injuries_text)
h2h_res = await h2h_slice(header, limit=h2h_last, before=cutoff, db=db)
parts.append(h2h_res.text)
parts.append("")
return MatchContext(
match_id=match_id,
text="\n".join(parts),
has_stats=has_stats,
has_injuries=has_injuries,
)
stats_res = await stats_slice(header, before=cutoff, db=db)
parts.append(stats_res.text)
parts.append("")
home_away_res = await home_away_slice(header, before=cutoff, db=db)
parts.append(home_away_res.text)
parts.append("")
injuries_res = await injuries_slice(header, before=cutoff, db=db)
parts.append(injuries_res.text)
return MatchContext(
match_id=match_id,
text="\n".join(parts),
has_stats=form_res.has_data or stats_res.has_data,
has_injuries=injuries_res.has_data,
match_dt=header.match_dt,
)
# ============================================================
@@ -312,9 +409,19 @@ async def _load_match(db, match_id: int) -> Match:
async def _get_form(db, team_id: int, before, *, limit: int = 5) -> list[Match]:
"""某队近 N 场(已完赛)。before=None 表示不限制(预测赛前的场景由调用方保证)。"""
"""某队近 N 场(已完赛)。before=None 表示不限制(预测赛前的场景由调用方保证)。
必须预加载 stats / home_team / away_team:切片函数会读取这些关系,
async session 下惰性加载会抛 MissingGreenlet
(models.py 已声明 lazy="selectin",此处显式声明以固化查询意图)
"""
stmt = (
select(Match)
.options(
selectinload(Match.stats),
selectinload(Match.home_team),
selectinload(Match.away_team),
)
.where(Match.match_status == "finished")
.where(Match.home_goals.is_not(None))
.where((Match.home_team_id == team_id) | (Match.away_team_id == team_id))
@@ -328,9 +435,13 @@ async def _get_form(db, team_id: int, before, *, limit: int = 5) -> list[Match]:
async def _get_h2h(db, home_id: int, away_id: int, before, *, limit: int = 5) -> list[Match]:
"""两队交锋史。"""
"""两队交锋史。需预加载 home_team / away_team(切片输出队名)。"""
stmt = (
select(Match)
.options(
selectinload(Match.home_team),
selectinload(Match.away_team),
)
.where(Match.match_status == "finished")
.where(Match.home_goals.is_not(None))
.where(
@@ -347,9 +458,17 @@ async def _get_h2h(db, home_id: int, away_id: int, before, *, limit: int = 5) ->
async def _get_home_away(db, team_id: int, side: str, before, *, limit: int = 10) -> list[Match]:
"""某队主场/客场近 N 场。side='home' 取主场,'away' 取客场。"""
"""某队主场/客场近 N 场。side='home' 取主场,'away' 取客场。
当前只用标量字段,但统一预加载以免后续扩展时踩坑
"""
stmt = (
select(Match)
.options(
selectinload(Match.stats),
selectinload(Match.home_team),
selectinload(Match.away_team),
)
.where(Match.match_status == "finished")
.where(Match.home_goals.is_not(None))
.order_by(Match.match_date.desc())
+8 -10
View File
@@ -5,23 +5,21 @@ import logging
from sqlalchemy import select
from src.db.base import AsyncSessionLocal
from src.db.models import Prediction
from src.db.unit_of_work import get_uow
logger = logging.getLogger(__name__)
async def settle_prediction(prediction_id: int, home_goals: int, away_goals: int) -> Prediction:
"""回填实际结果。"""
async with AsyncSessionLocal() as db:
pred = await db.get(Prediction, prediction_id)
async with get_uow() as session:
pred = await session.get(Prediction, prediction_id)
if pred is None:
raise ValueError(f"prediction {prediction_id} not found")
pred.actual_home_goals = home_goals
pred.actual_away_goals = away_goals
pred.settled = True
await db.commit()
await db.refresh(pred)
return pred
@@ -36,12 +34,12 @@ def _actual_1x2(home: int, away: int) -> str:
async def get_eval_summary() -> dict:
"""按 provider × 模型聚合评估。"""
async with AsyncSessionLocal() as db:
async with get_uow() as session:
stmt = (
select(Prediction)
.where(Prediction.settled == True)
)
result = await db.execute(stmt)
result = await session.execute(stmt)
rows = list(result.scalars().all())
from collections import defaultdict
@@ -61,8 +59,8 @@ async def get_eval_summary() -> dict:
err = ((p.pred_home_goals - p.actual_home_goals) ** 2 +
(p.pred_away_goals - p.actual_away_goals) ** 2) ** 0.5
b["score_errors"].append(err)
if p.confidence is not None:
b["conf_sum"] += p.confidence
if p.subjective_confidence is not None:
b["conf_sum"] += p.subjective_confidence
b["conf_count"] += 1
summary = []
@@ -76,6 +74,6 @@ async def get_eval_summary() -> dict:
"total": b["total"],
"accuracy_1x2": round(acc, 1),
"avg_score_rmse": round(avg_err, 2) if avg_err is not None else None,
"avg_confidence": round(avg_conf, 2) if avg_conf is not None else None,
"avg_subjective_confidence": round(avg_conf, 2) if avg_conf is not None else None,
})
return {"summary": summary}
+101 -39
View File
@@ -2,15 +2,18 @@
from __future__ import annotations
import functools
import hashlib
import logging
import time
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
from threading import Lock
from src.core.config import settings
from src.db.base import AsyncSessionLocal
from src.db.models import Match, Prediction
from src.db.unit_of_work import get_uow
from src.llm.context_builder import build_context
from src.llm.provider import LLMProvider, get_default_provider
@@ -20,29 +23,47 @@ _PROMPT_DIR = Path(__file__).resolve().parent / "prompts"
# ── LLM 响应缓存(match+provider+model+version → 结果) ──
_CACHE_TTL_SEC = 300 # 5 分钟
# P1-5: 缓存仅在 asyncio 协程内同步访问(dict 操作 GIL 原子),无需 threading.Lock。
# 删除 _cache_lock,避免同步锁阻塞事件循环;dict 的 get/set 在 CPython 下原子。
_cache: dict[str, tuple[float, PredictResult]] = {}
_cache_lock = Lock()
def _cache_key(match_id: int, provider: str, model: str, version: str) -> str:
return f"{match_id}:{provider}:{model}:{version}"
def _cache_key(match_id: int, provider: str, model: str, version: str, tpl_hash: str) -> str:
"""缓存键:含 prompt 模板内容 hash。
仅用 version 做键不够 编辑器里改动 `match_prediction_v1.md` 而版本号
不变时,进程内缓存仍会返回旧模板产生的旧结果(见审查报告 P2-6)
把模板内容 hash 纳入键,模板一改缓存自动失效
"""
return f"{match_id}:{provider}:{model}:{version}:{tpl_hash[:12]}"
def _get_cached(match_id: int, provider: str, model: str, version: str) -> PredictResult | None:
key = _cache_key(match_id, provider, model, version)
with _cache_lock:
if key in _cache:
ts, result = _cache[key]
if time.time() - ts < _CACHE_TTL_SEC:
return result
del _cache[key]
def _get_cached(match_id: int, provider: str, model: str, version: str, tpl_hash: str) -> PredictResult | None:
# P1-5: 无锁访问。dict get/del 在 CPython GIL 下原子,且无 await 穿插。
key = _cache_key(match_id, provider, model, version, tpl_hash)
entry = _cache.get(key)
if entry is not None:
ts, result = entry
if time.time() - ts < _CACHE_TTL_SEC:
return result
_cache.pop(key, None)
return None
def _set_cached(match_id: int, provider: str, model: str, version: str, result: PredictResult) -> None:
key = _cache_key(match_id, provider, model, version)
with _cache_lock:
_cache[key] = (time.time(), result)
def _set_cached(match_id: int, provider: str, model: str, version: str, tpl_hash: str, result: PredictResult) -> None:
# P1-5: 无锁写入。同上,dict set 原子。
key = _cache_key(match_id, provider, model, version, tpl_hash)
_cache[key] = (time.time(), result)
def clear_prompt_cache() -> None:
"""清空 prompt 模板缓存(供开发/热更新时手动调用)。
lru_cache 的模板缓存是进程级的,改完 .md 需要重启进程才能生效;
提供显式清理入口,避免"改了模板却看不到变化"的困惑(见审查报告 P2-5)
"""
_load_prompt_template.cache_clear()
logger.info("prompt 模板缓存已清空")
@functools.lru_cache(maxsize=8)
@@ -55,6 +76,11 @@ def _load_prompt_template(version: str = "v1") -> str:
return f.read()
def _prompt_template_hash(version: str) -> str:
"""prompt 模板内容 hash(用于缓存键,模板变更即失效)。"""
return hashlib.sha256(_load_prompt_template(version).encode("utf-8")).hexdigest()
@dataclass
class PredictResult:
prediction_id: int
@@ -64,7 +90,7 @@ class PredictResult:
pred_home_goals: float | None
pred_away_goals: float | None
pred_1x2: str | None
confidence: float | None
subjective_confidence: float | None
reasoning: str | None
context: str
latency_ms: int | None
@@ -78,11 +104,25 @@ async def predict_match(
model: str | None = None,
prompt_version: str | None = None,
mode: str = "multi",
use_cache: bool = True,
backtest: bool = False,
) -> "PredictResult | MultiPredictResult":
"""预测入口。mode=multi(默认)走多 agent;mode=single 走单次调用。"""
"""预测入口。mode=multi(默认)走多 agent;mode=single 走单次调用。
Args:
use_cache: 是否允许返回进程内缓存结果回测必须传 False
缓存命中不会新建 prediction ,调用方会对同一个 prediction_id
反复 settle,把不同比赛的真实比分覆盖到同一条记录上
backtest: 是否回测模式True build_context 使用 match_date-1 作为 cutoff
"""
if mode == "single":
return await _predict_single(
match_id, provider=provider, model=model, prompt_version=prompt_version
match_id,
provider=provider,
model=model,
prompt_version=prompt_version,
use_cache=use_cache,
backtest=backtest,
)
from src.llm.agents.orchestrator import predict_match_multi
@@ -95,6 +135,8 @@ async def _predict_single(
provider: LLMProvider | None = None,
model: str | None = None,
prompt_version: str | None = None,
use_cache: bool = True,
backtest: bool = False,
) -> PredictResult:
"""单次调用路径(原有实现)。"""
if provider is None:
@@ -102,15 +144,23 @@ async def _predict_single(
if model:
provider.model = model
version = prompt_version or "v1"
tpl_hash = _prompt_template_hash(version)
# 0. 查缓存(同 match+provider+model+version 5 分钟内直接返)
cached = _get_cached(match_id, settings.LLM_PROVIDER, provider.model, version)
if cached is not None:
logger.debug("predict cache hit match=%s", match_id)
return cached
# 0. 查缓存(同 match+provider+model+version+模板hash 5 分钟内直接返)
if use_cache:
cached = _get_cached(match_id, settings.LLM_PROVIDER, provider.model, version, tpl_hash)
if cached is not None:
logger.debug("predict cache hit match=%s", match_id)
return cached
# 1. 拼上下文
ctx = await build_context(match_id)
# 1. 拼上下文(P2-6: backtest 时使用 match_date-1天 作为 cutoff)
ctx = await build_context(match_id, backtest=backtest)
# 1.5 计算快照元数据(用于可复现性)
now = datetime.now(timezone.utc)
match_kickoff_at = ctx.match_dt
prediction_cutoff_at = ctx.match_dt # 默认:比赛时间作为数据截止
input_hash = hashlib.sha256(ctx.text.encode("utf-8")).hexdigest()
# 2. 拼 prompt(指定版本)
template = _load_prompt_template(version)
@@ -130,10 +180,17 @@ async def _predict_single(
parsed = resp.parsed or {}
# 4. 存预测(独立 session,因为 context 用的是自己的 session)
async with AsyncSessionLocal() as db:
# 3.5 严格校验 LLM 输出
from src.llm.validation import validate_prediction_output
try:
validated = validate_prediction_output(parsed)
except Exception as e:
raise RuntimeError(f"LLM 输出校验失败: {e}")
# 4. 存预测(使用 UnitOfWork 统一事务)
async with get_uow() as session:
# 验证 match 存在
m = await db.get(Match, match_id)
m = await session.get(Match, match_id)
if m is None:
raise ValueError(f"match {match_id} not found")
@@ -145,16 +202,20 @@ async def _predict_single(
prompt_tokens=resp.prompt_tokens,
completion_tokens=resp.completion_tokens,
latency_ms=resp.latency_ms,
pred_home_goals=parsed.get("pred_home_goals"),
pred_away_goals=parsed.get("pred_away_goals"),
pred_1x2=parsed.get("1x2"),
confidence=parsed.get("confidence"),
reasoning=parsed.get("reasoning"),
pred_home_goals=validated.pred_home_goals,
pred_away_goals=validated.pred_away_goals,
pred_1x2=validated.pred_1x2,
subjective_confidence=validated.subjective_confidence,
reasoning=validated.reasoning,
raw_response=resp.raw,
status="success",
match_kickoff_at=match_kickoff_at,
prediction_cutoff_at=prediction_cutoff_at,
prediction_created_at=now,
input_hash=input_hash,
)
db.add(pred)
await db.commit()
await db.refresh(pred)
session.add(pred)
await session.refresh(pred)
result = PredictResult(
prediction_id=pred.id,
@@ -164,13 +225,14 @@ async def _predict_single(
pred_home_goals=pred.pred_home_goals,
pred_away_goals=pred.pred_away_goals,
pred_1x2=pred.pred_1x2,
confidence=pred.confidence,
subjective_confidence=pred.subjective_confidence,
reasoning=pred.reasoning,
context=ctx.text,
latency_ms=resp.latency_ms,
raw=resp.raw,
)
# 5. 写入缓存
_set_cached(match_id, settings.LLM_PROVIDER, provider.model, version, result)
# 5. 写入缓存(仅当允许缓存时)
if use_cache:
_set_cached(match_id, settings.LLM_PROVIDER, provider.model, version, tpl_hash, result)
return result
+19
View File
@@ -0,0 +1,19 @@
"""LLM 模块共享工具函数。"""
from __future__ import annotations
def actual_1x2(home: int, away: int) -> str:
"""实际比分 → 胜平负。
单一权威源: backtest.py eval.py 共用,避免重复定义
"""
if home > away:
return "1"
if home == away:
return "X"
return "2"
def is_correct_1x2(pred: str | None, actual: str) -> bool:
"""预测是否命中胜平负。"""
return pred == actual
+205
View File
@@ -0,0 +1,205 @@
"""LLM 输出严格校验。
所有 LLM JSON 输出必须经过 Pydantic 校验 + 语义一致性检查后才能落库
"""
from __future__ import annotations
import logging
from pydantic import BaseModel, Field, field_validator, model_validator
logger = logging.getLogger(__name__)
# 单一权威源:从 orchestrator.SPECIALIST_SPECS 派生,避免两端独立定义导致静默偏离
from src.llm.agents.orchestrator import SPECIALIST_SPECS
KNOWN_AGENT_NAMES: tuple[str, ...] = tuple(spec.name for spec in SPECIALIST_SPECS)
class AgentReportSchema(BaseModel):
"""单个专家 Agent 输出的校验 schema。"""
data_sufficiency: str = "medium"
analysis: str = ""
home_edge: float | None = Field(None, ge=-1.0, le=1.0)
subjective_confidence: float | None = Field(None, ge=0.0, le=1.0)
key_evidence: list[str] = Field(default_factory=list)
exp_home_goals: float | None = Field(None, ge=0.0, le=10.0)
exp_away_goals: float | None = Field(None, ge=0.0, le=10.0)
probable_score: str | None = None
@field_validator("data_sufficiency")
@classmethod
def validate_sufficiency(cls, v: str) -> str:
allowed = {"high", "medium", "low", "none"}
return v.lower() if v.lower() in allowed else "medium"
@field_validator("key_evidence", mode="before")
@classmethod
def normalize_evidence(cls, v) -> list[str]:
if v is None:
return []
if isinstance(v, str):
return [v]
if isinstance(v, list):
return [str(e)[:120] for e in v[:5]]
return []
@field_validator("analysis")
@classmethod
def truncate_analysis(cls, v: str) -> str:
return str(v)[:600]
class PredictionOutputSchema(BaseModel):
"""最终预测输出的校验 schema。"""
pred_home_goals: float = Field(ge=0.0, le=10.0)
pred_away_goals: float = Field(ge=0.0, le=10.0)
pred_1x2: str
subjective_confidence: float = Field(ge=0.0, le=1.0)
reasoning: str = ""
@field_validator("pred_1x2")
@classmethod
def validate_1x2(cls, v: str) -> str:
if v not in ("1", "X", "2"):
raise ValueError(f"pred_1x2 must be '1', 'X', or '2', got '{v}'")
return v
@model_validator(mode="after")
def check_consistency(self) -> "PredictionOutputSchema":
"""验证比分与胜平负一致。
不一致时以比分为准修正 pred_1x2(比分是更结构化的输出),
**必须告警** 静默修正会掩盖 LLM 的自相矛盾,让问题无法被发现
"""
expected = _score_to_1x2(self.pred_home_goals, self.pred_away_goals)
if self.pred_1x2 != expected:
logger.debug(
"1x2 与比分不一致: 比分 %.1f-%.1f 推出 '%s',但 LLM 给出 '%s';以比分修正",
self.pred_home_goals, self.pred_away_goals, expected, self.pred_1x2,
)
self.pred_1x2 = expected
return self
def _score_to_1x2(home: float, away: float) -> str:
"""从比分推导胜平负。"""
if home > away:
return "1"
if home < away:
return "2"
return "X"
class AgentWeightsSchema(BaseModel):
"""终裁给出的各专家权重校验 schema。
权重含义:各专家报告在最终决策中的相对影响力约束:
- key 必须是已知的 5 个专家名
- value [0, 1]
- 总和允许有 0.05 的浮点误差(LLM 常凑不到精确 1.0),
超出则归一化到 1.0 而不是直接拒收
"""
weights: dict[str, float] = Field(default_factory=dict)
@field_validator("weights", mode="before")
@classmethod
def coerce_weights(cls, v):
if v is None:
return {}
if not isinstance(v, dict):
raise ValueError(f"agent_weights 必须是 dict,得到 {type(v).__name__}")
out: dict[str, float] = {}
for k, raw in v.items():
key = str(k).strip().lower()
if key not in KNOWN_AGENT_NAMES:
logger.warning("agent_weights 含未知专家 '%s',已忽略", k)
continue
f = _safe_float(raw)
if f is None:
logger.warning("agent_weights['%s']=%r 非数值,已忽略", k, raw)
continue
# 负数直接钳到 0;超过 1 的钳到 1
out[key] = min(max(f, 0.0), 1.0)
return out
@model_validator(mode="after")
def normalize_sum(self) -> "AgentWeightsSchema":
"""权重和不为 1 时归一化(而非拒收),并在偏离较大时告警。"""
if not self.weights:
return self
total = sum(self.weights.values())
if total <= 0:
return self
if abs(total - 1.0) > 0.05:
logger.warning("agent_weights 总和为 %.3f,已归一化到 1.0", total)
self.weights = {k: v / total for k, v in self.weights.items()}
return self
def validate_agent_weights(raw) -> dict[str, float]:
"""校验并规范化终裁给出的 agent_weights。非法输入返回空 dict。"""
if raw is None:
return {}
try:
return AgentWeightsSchema(weights=raw).weights
except Exception as e:
logger.warning("agent_weights 校验失败,丢弃: %s", e)
return {}
def validate_agent_output(raw: dict) -> AgentReportSchema:
"""校验并规范化单个 Agent 输出。"""
return AgentReportSchema(
data_sufficiency=raw.get("data_sufficiency", "medium"),
analysis=raw.get("analysis", ""),
home_edge=_safe_float(raw.get("home_edge")),
subjective_confidence=_safe_float(raw.get("subjective_confidence") or raw.get("confidence")),
key_evidence=raw.get("key_evidence", []),
exp_home_goals=_safe_float(raw.get("exp_home_goals")),
exp_away_goals=_safe_float(raw.get("exp_away_goals")),
probable_score=_format_score(raw.get("probable_score")),
)
def validate_prediction_output(raw: dict) -> PredictionOutputSchema:
"""校验最终预测输出。"""
# 优先新字段,旧字段仅兼容并打日志
conf = raw.get("subjective_confidence")
if conf is None and "confidence" in raw:
logger.warning("Deprecated field 'confidence' used, prefer 'subjective_confidence'")
conf = raw["confidence"]
return PredictionOutputSchema(
pred_home_goals=float(raw.get("pred_home_goals", 0)),
pred_away_goals=float(raw.get("pred_away_goals", 0)),
pred_1x2=raw.get("1x2") or raw.get("pred_1x2", "X"),
subjective_confidence=float(conf if conf is not None else 0.5),
reasoning=str(raw.get("reasoning", ""))[:1000],
)
def _safe_float(v) -> float | None:
"""安全转 float,失败返回 None。"""
if v is None:
return None
try:
f = float(v)
if not (f == f): # NaN check
return None
return f
except (TypeError, ValueError):
return None
def _format_score(v) -> str | None:
"""格式化比分输出。"""
if v is None:
return None
if isinstance(v, str):
return v
if isinstance(v, dict):
return f"{v.get('home', '?')}-{v.get('away', '?')}"
return None
+112 -7
View File
@@ -59,14 +59,14 @@ class TestReportParsing:
"data_sufficiency": "high",
"analysis": "主队交锋占优",
"home_edge": 0.6,
"confidence": 0.8,
"subjective_confidence": 0.8,
"key_evidence": ["近5次交锋主队4胜", "主场交锋3连胜"],
}
resp = LLMResponse(content="{}", parsed=parsed, prompt_tokens=100, completion_tokens=50, latency_ms=500)
r = _parse_report("h2h", parsed, resp, "gpt-4o-mini")
assert r.status == "ok"
assert r.home_edge == 0.6
assert r.confidence == 0.8
assert r.subjective_confidence == 0.8
assert len(r.key_evidence) == 2
assert r.data_sufficiency == "high"
@@ -78,7 +78,7 @@ class TestReportParsing:
"data_sufficiency": "medium",
"analysis": "主队火力更强",
"home_edge": 0.4,
"confidence": 0.7,
"subjective_confidence": 0.7,
"key_evidence": ["场均xG 2.1"],
"exp_home_goals": 2.1,
"exp_away_goals": 1.2,
@@ -96,16 +96,24 @@ class TestReportParsing:
parsed = {
"data_sufficiency": "bogus", # 非法 → medium
"home_edge": "very strong", # 非法 → None
"confidence": None,
"home_edge": "very strong", # 非法 → None (宽容降级)
"subjective_confidence": None,
"key_evidence": "单字符串", # → [str]
}
resp = LLMResponse(content="{}", parsed=parsed)
r = _parse_report("form", parsed, resp, "m")
# 宽容降级: 不抛异常,非法字段变 None 或默认值
assert r.status == "ok"
assert r.data_sufficiency == "medium"
assert r.home_edge is None
assert r.key_evidence == ["单字符串"]
# 验证范围约束: confidence > 1 会被截断或拒绝
parsed2 = {"subjective_confidence": 1.5, "home_edge": 2.0}
r2 = _parse_report("form", parsed2, resp, "m")
# Pydantic 会拒绝越界值 → parse_error
assert r2.status == "parse_error"
class TestRunAgent:
"""run_agent 执行器: 门控 + fail-open。"""
@@ -167,7 +175,7 @@ class TestRunAgent:
assert "A 2-1 B" in user
return LLMResponse(
content="{}",
parsed={"data_sufficiency": "high", "analysis": "ok", "home_edge": 0.5, "confidence": 0.9},
parsed={"data_sufficiency": "high", "analysis": "ok", "home_edge": 0.5, "subjective_confidence": 0.9},
prompt_tokens=10, completion_tokens=5, latency_ms=100,
)
@@ -192,7 +200,7 @@ class TestOrchestratorAggregation:
import json
reports = [
AgentReport(agent="h2h", status="ok", home_edge=0.5, confidence=0.8, analysis="a"),
AgentReport(agent="h2h", status="ok", home_edge=0.5, subjective_confidence=0.8, analysis="a"),
AgentReport(agent="injuries", status="no_data", data_sufficiency="none"),
]
text = _reports_to_json(reports)
@@ -211,3 +219,100 @@ class TestOrchestratorAggregation:
)
assert "{{match_header}}" not in rendered
assert "{{agent_reports}}" not in rendered
class TestSliceResultGate:
"""P2-1: 结构化 has_data 门控(替代脆弱的文案子串匹配)。"""
def test_sliceresult_empty_is_no_data(self):
from src.llm.agents.base import _slice_has_data
from src.llm.context_builder import SliceResult
text, has = _slice_has_data(SliceResult(text="── x ──\n 无数据", has_data=False))
assert has is False
def test_sliceresult_with_data_beats_text(self):
"""即使文案里出现「无数据」字样,结构化 has_data=True 也应胜出。
这正是旧实现的漏洞:文案匹配会把主队: 无伤停数据 / 客队: 2人伤停
这类混合输出这里显式验证结构化声明优先
"""
from src.llm.agents.base import _slice_has_data
from src.llm.context_builder import SliceResult
tricky = SliceResult(
text="── 阵容完整性 ──\n 主队: 无伤停数据\n 客队伤停(1人):\n - X: 拉伤",
has_data=True,
)
_, has = _slice_has_data(tricky)
assert has is True
def test_str_fallback_still_works(self):
"""旧式 str 切片(测试 mock / 自定义切片)仍走文案回退,保持兼容。"""
from src.llm.agents.base import _slice_has_data
assert _slice_has_data("── 伤停 ──\n 无数据")[1] is False
assert _slice_has_data("── 交锋 ──\n A 2-1 B")[1] is True
def test_sliceresult_str_compat(self):
"""SliceResult 可当 str 用(老调用点无需改)。"""
from src.llm.context_builder import SliceResult
s = SliceResult(text="hello", has_data=True)
assert str(s) == "hello"
class TestAgentWeightsValidation:
"""P2-2: agent_weights 必须过校验才能落库。"""
def test_unknown_agent_dropped(self):
from src.llm.validation import validate_agent_weights
w = validate_agent_weights({"form": 0.4, "h2h": 0.4, "bogus": 0.2, "stats": 0.4})
assert "bogus" not in w
assert set(w) <= {"form", "stats", "home_away", "injuries", "h2h"}
def test_out_of_range_clamped(self):
from src.llm.validation import validate_agent_weights
w = validate_agent_weights({"form": 5.0, "h2h": -1.0})
assert w["form"] == 1.0
assert w["h2h"] == 0.0
def test_sum_normalized(self):
from src.llm.validation import validate_agent_weights
w = validate_agent_weights({"form": 2.0, "stats": 2.0})
assert abs(sum(w.values()) - 1.0) < 1e-9
def test_no_weights_returns_empty(self):
from src.llm.validation import validate_agent_weights
assert validate_agent_weights(None) == {}
assert validate_agent_weights("not a dict") == {}
class TestPredictionConsistencyWarn:
"""P2-3: 比分与 1x2 不一致 → 以比分修正(且告警)。"""
def test_mismatch_is_corrected_to_score(self):
from src.llm.validation import validate_prediction_output
v = validate_prediction_output({
"pred_home_goals": 2.0,
"pred_away_goals": 1.0,
"pred_1x2": "X", # 与 2-1 矛盾
"subjective_confidence": 0.7,
})
assert v.pred_1x2 == "1" # 按比分修正
def test_consistent_passes_through(self):
from src.llm.validation import validate_prediction_output
v = validate_prediction_output({
"pred_home_goals": 0.0,
"pred_away_goals": 0.0,
"pred_1x2": "X",
"subjective_confidence": 0.5,
})
assert v.pred_1x2 == "X"
+1 -1
View File
@@ -96,7 +96,7 @@ class TestProviderMock:
def raise_for_status(self): pass
def json(self):
return {
"choices": [{"message": {"content": '{"pred_home_goals": 1.5, "pred_away_goals": 1.0, "1x2": "1", "confidence": 0.7, "reasoning": "test"}'}}],
"choices": [{"message": {"content": '{"pred_home_goals": 1.5, "pred_away_goals": 1.0, "1x2": "1", "subjective_confidence": 0.7, "reasoning": "test"}'}}],
"usage": {"prompt_tokens": 100, "completion_tokens": 50},
}
return FakeResp()
+99
View File
@@ -0,0 +1,99 @@
"""回归测试:锁定 P0 三项「静默失效」缺陷不再复发。
这些用例不依赖数据库 它们用静态分析检查代码结构,
因为三个 P0 的本质都是集成方式错误, mock slice_fn /
provider 的单元测试里永远照不出来(这正是它们当初漏网的原因)
- P0-1/P0-2: 查询 Match 的函数必须 eager-load 切片会访问的关系
- P0-3: bzzoiro source_event_id 时用的 raw 必须与 nm 配对
"""
from __future__ import annotations
import re
from pathlib import Path
SRC = Path(__file__).resolve().parent.parent / "src"
# 切片函数会读取的关系属性 → 查询时必须 eager-load
MATCH_RELATIONS = ("stats", "home_team", "away_team", "league")
def _read(rel: str) -> str:
return (SRC / rel).read_text(encoding="utf-8")
class TestEagerLoadCoverage:
"""P0-1 / P0-2: 凡是 select(Match) 且后续访问关系的函数,必须有 eager-load。"""
def test_context_builder_getters_eager_load(self):
"""_get_form / _get_h2h / _get_home_away 必须预加载关系。
models.py 已声明 lazy="selectin" 兜底,但这里同时检查显式
selectinload 显式声明是查询意图的固化,也被 P0 修复所依赖
"""
src = _read("llm/context_builder.py")
for fn in ("_get_form", "_get_h2h", "_get_home_away"):
# 截取函数体
m = re.search(rf"async def {fn}\(.*?(?=\nasync def |\n# =|\Z)", src, re.S)
assert m, f"{fn} 未找到"
body = m.group(0)
assert "selectinload" in body, (
f"{fn} 查询 Match 但未 eager-load 关系 —— "
"this would raise MissingGreenlet in async SQLAlchemy (P0-2)"
)
def test_backtest_candidates_eager_load(self):
"""回测取历史比赛必须 eager-load(否则 session 关闭后访问关系必炸)。"""
src = _read("llm/backtest.py")
assert "selectinload" in src, "backtest 未 eager-load 关系 (P0-1)"
def test_relationship_default_is_selectin(self):
"""models.py 中 Match 的高频关系应声明 lazy='selectin' 作为兜底。"""
src = _read("db/models.py")
# 找到 Match 类定义段
m = re.search(r"class Match\(Base\):.*?(?=\nclass )", src, re.S)
assert m, "Match 类未找到"
body = m.group(0)
for rel in MATCH_RELATIONS:
# 关系声明可能跨多行(stats/home_team/away_team 都是),因此按
# 「从 `rel: Mapped` 到下一个 `xxx: Mapped` 之前」整段匹配。
m_rel = re.search(
rf"^\s*{rel}: Mapped.*?(?=^\s*\w+: Mapped|\Z)", body, re.M | re.S
)
assert m_rel, f"Match.{rel} 未找到"
assert 'lazy="selectin"' in m_rel.group(0), (
f"Match.{rel} 未声明 lazy='selectin' —— 兜底缺失 (P0-2)"
)
class TestBzzoiroLineage:
"""P0-3: source_event_id 必须取配对的 raw,不能是循环残留变量。"""
def test_normalized_matches_carries_raw(self):
src = _read("data/bzzoiro.py")
# 规范化结果必须与原始 event 成对保存
assert "normalized_matches.append((nm, raw))" in src, (
"normalized_matches 未携带 (nm, raw) 元组 —— raw 变量泄漏会回归 (P0-3)"
)
# 内层消费循环必须解包成对变量
assert re.search(r"for nm, raw in normalized_matches", src), (
"消费循环未解包 (nm, raw) —— 血缘字段会取到错误 event (P0-3)"
)
def test_no_orphan_raw_use(self):
"""source_event_id 所在行必须在解包循环内(用缩进 + 上下文粗判)。"""
src = _read("data/bzzoiro.py")
lines = src.splitlines()
# 找到 "for nm, raw in normalized_matches" 所在行号
start = next(
(i for i, ln in enumerate(lines) if "for nm, raw in normalized_matches" in ln),
None,
)
assert start is not None
# 该循环之后、下一个同/更低缩进的顶层语句之前的范围
seg = "\n".join(lines[start:])
uses = [ln for ln in seg.splitlines() if "source_event_id" in ln]
assert uses, "未找到 source_event_id 赋值"
assert all("raw.get(" in ln for ln in uses), (
"source_event_id 未使用配对的 raw (P0-3)"
)