Author SHA1 Message Date
shangfangjian 4b0d6ee58a chore(P3): baseline 落库下沉 + MatchPredictPanel 拆分 + 多 worker/CSRF 文档
P3-2 baseline 落库从路由下沉到服务层(predict_baseline 内直接落库),
删除路由层 _persist_baseline,三种模式统一 result.prediction_id,对外 JSON 不变。

P3-1 MatchPredictPanel.PredictionPanel 拆为 OutcomePanel/AgentsPanel/ReasoningPanel
三个子组件,本文件保留 PredictModal/PredictProgress/Spinner,对外导出路径不变。

P3-3 docs 加 ⚠️ 多 worker 陷阱红字 + STRICT_SINGLE_WORKER 环境变量(启动期强制拒绝多 worker)。
P3-4 docs 新增「同站部署 vs 跨站 CSRF」节。
2026-09-21 23:29:09 +08:00
shangfangjian 45497d2112 refactor: admin_settings.py 按职责拆分为 4 个模块
681 行单文件拆为(保留 admin_settings 为 include_router 聚合入口):
- admin_datasources  数据源列表/连通性测试/KeyRing/ingest status (5 路由)
- admin_config       settings CRUD + 运行日志 (4 路由)
- admin_llm          LLM agents/models/ping (3 路由)
- admin_quality      stats/data-completeness/data-quality (4 路由)

所有路由仍挂 /api/v1/admin 且带 dependencies=[Depends(require_admin)]。
app.py 注册方式不变(仍 import admin_settings.router)。
2026-09-21 23:28:58 +08:00
shangfangjian 66f0844798 feat: standings Bronze 血缘补齐 + 队名归一 MVP
standings 成功 upsert 后写入 RawEvent(source_record_id=standings:{league}:{season})
+ DataLineage(target_table=standings),与 events/stats 管线对称。

队名归一化收敛到 TeamRepository.get_or_create 唯一咽喉点,
创建新 Team 时 info 日志打出原始名与归一后名;
Admin 新增 GET /api/v1/admin/team-name-duplicates 只读接口,
启发式列出近似重名候选(大小写变体/子串/前缀碰撞),不做自动合并。
2026-09-21 23:28:46 +08:00
shangfangjian 317a5e338a refactor: bzzoiro.py 按管线拆分为 5 个模块
单文件 852 行按职责拆分,保持 BzzoiroSource 与 get_source("bzzoiro") 行为不变:
- bzzoiro_common  HTTP 抓取(多 key 轮换) + 字段转换原语
- bzzoiro_events   fetch_bzzoiro_events + BzzoiroSource.ingest + Bronze 补写
- bzzoiro_standings  standings 管线
- bzzoiro_stats     stats 回填
- pipeline_write    RawEvent/IngestFailure/DataLineage 写入助手

子模块运行期经聚合门面 src.data.bzzoiro 解析可替换协作者,
单文件时代的 bz.* monkeypatch 语义完全保留。
路由 import 已指向新模块(ingest.py / schedules.py)。
2026-09-21 23:27:48 +08:00
shangfangjian 7a5c695b89 Merge pull request '公开只读 API+ 全套文档对齐(P1-1)' (#11) from docs-public-api-fixes into main
Reviewed-on: #11
2026-09-21 21:38:58 +08:00
WorkBuddy 6bdb1f8ae6 docs(deploy): 补充生产上线检查清单(README + docs/06)
- docs/06 新增「生产上线检查清单」10 项:APP_ENV / SECRET_KEY 强随机 /
  管理鉴权二选一 / 数据库强密码 / HTTPS+Cookie Secure /
  TRUST_PROXY_HEADERS+X-Forwarded-For / Nginx limit_req 限流前置
  (应用内限流与 KeyRing 单进程限制) / uvicorn 单 worker /
  /health 与 /health/ready / alembic 迁移已内置
- README 在部署说明后加同名精简清单,链接到 docs/06 详情
- 全部条目均对齐真实实现(security_check 弱值黑名单、auth.py
  secure=production、deps.py TRUST_PROXY 解析、app.py 双健康检查、
  compose/Dockerfile 内置迁移);纯文档,未改业务逻辑
2026-09-21 21:02:16 +08:00
WorkBuddy 80616cf459 docs: 全套文档与代码对齐(P1-1,数据源收敛 bzzoiro 后的存量漂移清理)
- 数据源:全部文档统一为 bzzoiro 唯一来源,删除 understat/api-football
  作为现行数据源的表述(仅保留历史注记);.env.example/docs/06 移除
  失效的 API_FOOTBALL_KEY
- 多 Agent:injuries 阵容完整性 → standings 联赛排名
  (README/01/04 的专家表、数据流图、no_data 示例、agent_weights、
  prompt 清单全部对齐 form/stats/home_away/standings/h2h)
- 数据库:6 张表 → 12 张表,补 standings/app_settings/schedules 与
  4 张治理表说明(标注 raw_events/data_quality_checks/data_lineage
  为预留未启用,ingest_failures 已启用),predictions 补 agent_weights 列
- API 文档(03):新增鉴权模型三档表;context/standings/leagues 公开端点
  补全;predictions/eval/backtest 标注需管理员;ingest/bzzoiro 补
  task/limit/season 参数契约,删除 understat/injuries 端点小节
- 目录树(README/01/07):与真实 src/、frontend/src 结构一致
  (core 7 文件、data 7 文件、移除不存在的 retry.py)
- 采集命令(02/05):understat 回填 → task=stats/standings;
  补管理员凭据提示;docs/08 文首标注历史/过时
2026-09-21 20:55:48 +08:00
WorkBuddy a24017eb69 feat(api): 比赛上下文与联赛列表改公开只读(P1-2/P1-3)
- GET /matches/{id}/context 移除 require_admin:公开站详情页「近况/交锋」
  数据来源,匿名 401 会导致前端静默空态;响应结构不变,不触发 LLM
- GET /leagues 改公开只读:公开站联赛筛选动态加载;仅返回
  id/code/name/country 四个展示字段,不含配置或密钥信息
- 前端新增 useLeagues hook:优先请求 /api/v1/leagues,失败/空回退
  本地五大联赛常量(已知代码保留中文标签,新联赛按 API 名称追加)
- 新增 tests/test_public_readonly_api.py(6 项):匿名 200/404、响应形状、
  路由依赖声明检查(matches 路由无 require_admin)+ 判别力守卫
  (ingest 路由必须检出 require_admin,防检测器恒真)
2026-09-21 20:55:26 +08:00
shangfangjian 3056a95ef4 Merge pull request '工程债清理(Bronze 血缘/预测类型统一/Matches 拆分/Repository 治理/导航单源等)'
Reviewed-on: #10
2026-09-21 19:57:48 +08:00
WorkBuddy d9aeff2114 debt(D7): 限流/KeyRing 单进程限制说明 + 生产启动一次性告警
- app.py lifespan:APP_ENV=production 时 logger.warning 提醒
  进程内 rate-limit/KeyRing 仅单进程有效(不 sys.exit、不引入 Redis)
- key_ring.py docstring 补充多 worker 不共享的后果说明
- deps.py 注释已具备(单进程有效,生产前置 Nginx),无需改动
2026-09-21 19:51:41 +08:00
WorkBuddy 17e99a4f9a debt(D5): agent_outputs 类型标注对齐真实 JSON 形状(list[dict]|dict|None)
- Mapped[list[dict] | dict | None]:multi 模式存专家报告列表,
  历史数据/兼容路径可能存 dict;加注释说明形状来源
- 仅类型标注修正,列类型(JSONB)与数据、迁移均不变
2026-09-21 19:51:38 +08:00
WorkBuddy 4514ef4e92 debt(D4): bzzoiro Team/League/Match 查找经 Repository 层,纠正空壳表述
- repositories.py 新增 MatchRepository.find_by_league_and_date_range /
  find_finished_with_stats(含 stats 预加载),TeamRepository.get_or_create
  支持 name_zh
- bzzoiro events/standings/stats 三条管线的联赛、球队、比赛查找与批量预载
  全部改走 Repository;事务提交仍由调用方 UnitOfWork 控制
- Standing/RawEvent/Lineage 管线内私有读写保留在本模块(不强行 Repository 化)
- 模块 docstring 纠正为准确表述,移除「已全面 Repository 化」的误导性声明
2026-09-21 19:48:56 +08:00
WorkBuddy 18c89111d8 debt(D6): Admin 导航单源 —— NAV_ITEMS 派生侧栏/命令面板/面包屑
- 新增 admin/nav.ts: 唯一配置源 NAV_ITEMS(to/label/group/icon/hideFromSidebar)
- COMMAND_PALETTE_PAGES / ROUTE_LABELS / NAV_SECTIONS 全部由其派生
- 删除 AdminLayout 中三份平行清单(已出现漂移: data-pipeline 不在侧栏、
  「系统」vs「系统设置」命名不一致、monitoring/eval 顺序两处相反)
- 已知微差: 命令面板「评估与监控」组内顺序统一为侧栏口径(eval 在前)
- 行为不变: 路由/标签/分组标题均保持现状;data-pipeline 保持仅面板可达
2026-09-21 19:34:31 +08:00
WorkBuddy 52d67863d6 debt(D3): 拆分 Matches.tsx(~1400 行)为 hooks + 组件 + 共享层
- pages/matches/types.ts: Match/Prediction/AgentReport + 常量(LEAGUES/STATUS_META/AGENT_LABELS 等)
- pages/matches/ui.tsx: Spinner/SkeletonRows/Switch/日期分组工具(Switch 自组件体内提升到模块级)
- pages/matches/hooks/useMatchesList.ts: 列表筛选/游标分页/进行中比赛,竞态防护语义不变
- pages/matches/hooks/useMatchPredict.ts: 预测状态机(连点/竞态/中止/超时)+ 可读错误文案
- pages/matches/components/MatchPredictPanel.tsx: 预测弹窗全套(过程可视化/结果/专家意见)
- pages/matches/components/MatchDetailSection.tsx: MatchRow(原内联行 JSX 抽出)+ 详情面板
- Matches.tsx 组装页 279 行(< 300);error state 共享行为与拆分前一致
- 验证: tsc + vite build 通过,渲染逻辑原样搬迁
2026-09-21 19:31:42 +08:00
WorkBuddy 5c7fdce0a3 debt(D2): 统一预测结果类型为 PredictResult,路由去 dict 分支
- PredictResult 扩展可选字段 mode/agent_outputs/agent_weights/prompt_tokens/completion_tokens
- MultiPredictResult 变为 PredictResult 别名(保留 R4 守卫标记与 re-export)
- baseline 改返回 PredictResult(修复 backtest 对 baseline AttributeError 的潜伏 bug)
- 预测路由单一属性映射,删除全部 isinstance(result, dict) 分支
- _persist_baseline 属性化,baseline upsert 语义不变(prompt_version/token/latency 同前)
- TDD: 5 新测试 + test_baseline.py 属性化;全量 257 passed
2026-09-21 19:23:50 +08:00
WorkBuddy 3a9f3f5a0e debt(D1): events 成功路径补写 Bronze 层(RawEvent + DataLineage)
- 新增 _events_record_id: 上游 id 缺失时用 (league:home:away:date) 合成稳定幂等键
- 新增 _write_events_bronze: best-effort 写 RawEvent(幂等) + Lineage(matches/events_ingest)
- ingest 插入与变更更新后触发;同批 seen 集合防重复;基础设施失败只 warning
- TDD: 6 测试(插入/合成键/幂等跳过/变更更新/无变化不写/失败不拖垮),双变异验证通过
2026-09-21 19:23:37 +08:00
shangfangjian f6c0145c32 Merge pull request 'fix: 死信表真正接线 + 前端 HTTP 收敛与状态修正' (#9) from fix-deadletter-frontend-cleanup into main
Reviewed-on: #9
2026-09-21 18:39:08 +08:00
54 changed files with 4903 additions and 2907 deletions
-1
View File
@@ -40,7 +40,6 @@ LLM_TIMEOUT=60
# ---- 数据源 ----
BZZOIRO_KEY=
API_FOOTBALL_KEY=
# ---- CORS ----
CORS_ORIGINS=http://localhost:5173,http://localhost:3000
+74 -35
View File
@@ -12,28 +12,30 @@
│ REST API
┌──────────────────────▼──────────────────────────────┐
│ FastAPI │
│ ├── /api/v1/matches 比赛查询
│ ├── /api/v1/matches 比赛查询(公开只读)
│ ├── /api/v1/predict LLM 预测 (单/多 Agent) │
│ ├── /api/v1/ingest/* 数据采集
│ ├── /api/v1/eval/* 评估回填
│ └── /api/v1/backtest 回测
│ ├── /api/v1/ingest/* 数据采集(需管理员)
│ ├── /api/v1/eval/* 评估回填(需管理员)
│ └── /api/v1/backtest 回测(需管理员)
└──────────┬─────────────────────────────┬────────────┘
│ │
┌──────────▼──────────┐ ┌─────────────▼────────────┐
│ PostgreSQL │ │ LLM (OpenAI-compatible) │
6 张表 │ │ OpenAI / Deepseek / │
12 张表 │ │ OpenAI / Deepseek / │
│ leagues/teams/ │ │ Ollama / 任意网关 │
│ matches/match_ │ └──────────────────────────┘
│ stats/predictions/
injuries
│ stats/standings/
predictions/
│ app_settings/ │
│ schedules + │
│ raw_events 等 4 张 │
│ 数据治理表 │
└─────────────────────┘
│ 采集
┌──────────┴─────────────────────────────────────────┐
│ 数据源 (DataSource 协议 + 注册表) │
── bzzoiro 比分 / 统计 / xG
│ ├── understat xG 回填 │
│ └── injuries 伤停数据 (api-football) │
── bzzoiro 比分 / 赛程 / 统计 / 积分榜
└────────────────────────────────────────────────────┘
```
@@ -55,7 +57,7 @@ API Route → Application Service → Repository → UnitOfWork → DB
比赛数据 → 切片 ─┬─→ A 近期状态专家 ─┐
├─→ B 攻防数据专家 ─┤
├─→ C 主客因素专家 ─┼─→ 终裁专家 ─→ 最终预测
├─→ D 阵容完整专家 ─┤
├─→ D 联赛排名专家 ─┤
└─→ E 历史交锋专家 ─┘
```
@@ -66,8 +68,7 @@ API Route → Application Service → Repository → UnitOfWork → DB
### 数据正确性保障
- **Cutoff 机制**: 回测时只使用 `cutoff_at` 之前已采集的数据
- **Injury 防泄漏**: 伤停查询强制 `retrieved_at <= cutoff`
- **Cutoff 机制**: 回测时只使用 `cutoff_at` 之前已采集的数据(近况/交锋/统计/积分榜切片统一生效)
- **LLM 输出校验**: Pydantic 严格校验 + 语义一致性检查
- **数据库约束**: CHECK 约束作为最后一道防线
@@ -130,6 +131,21 @@ cd frontend && npm install && npm run dev
后端运行在 `http://localhost:8000`,前端在 `http://localhost:5173`
## 生产上线检查清单
公网部署前逐项确认(第 1–4 项由启动校验强制,不满足拒绝启动;详见 [docs/06-deployment.md](docs/06-deployment.md#生产上线检查清单)):
- [ ] `APP_ENV=production`(安全校验 / Cookie `Secure` / 管理端点 fail-closed 的总开关)
- [ ] `SECRET_KEY` 强随机:`openssl rand -base64 32`,禁止弱值
- [ ] `ADMIN_PASSWORD``ADMIN_API_KEY` 至少配置其一
- [ ] 数据库强密码,禁止 `football:football` 等示例弱密码
- [ ] HTTPS(反代终结 TLS;production 下会话 Cookie 自动 `Secure`)
- [ ] 反代后设 `TRUST_PROXY_HEADERS=True`,仅可信反代可达 API,并配置 `X-Forwarded-For` / `X-Real-IP`
- [ ] 限流前置到 Nginx `limit_req`;应用内限流与 KeyRing 仅单进程有效,多 worker 会放大配额
- [ ] uvicorn 单 worker(默认);需扩容先网关统一限流再起多实例
- [ ] 启动后验证 `/health``/health/ready` 均 200
- [ ] 数据库迁移已内置:compose/Dockerfile 启动即执行 `alembic upgrade head`
## 安全与限流
- `/api/v1/predict`: 内存滑动窗口限流(10 次/分钟/IP),多 worker 时每进程独立计数
@@ -141,20 +157,29 @@ cd frontend && npm install && npm run dev
## API 概览
**公开只读**(无需登录;`predict` 带内存限流):
| 方法 | 路径 | 说明 |
|---|---|---|
| GET | `/api/v1/matches` | 比赛查询(筛选/分页) |
| GET | `/api/v1/leagues` | 联赛列表 |
| POST | `/api/v1/predict` | LLM 预测 (`mode=single`/`multi`) |
| GET | `/api/v1/predictions` | 预测历史 |
| POST | `/api/v1/ingest/bzzoiro` | 采集比分/统计 |
| POST | `/api/v1/ingest/understat` | 回填 xG |
| POST | `/api/v1/ingest/injuries` | 采集伤停 |
| GET | `/api/v1/leagues` | 联赛列表(仅 id/code/name/country) |
| GET | `/api/v1/matches` | 比赛查询(筛选/游标分页) |
| GET | `/api/v1/matches/{id}` | 比赛详情(含统计与最近预测) |
| GET | `/api/v1/matches/{id}/context` | 比赛上下文(双方近况 + 历史交锋) |
| GET | `/api/v1/standings` | 联赛积分榜 |
| POST | `/api/v1/predict` | LLM 预测 (`mode=single`/`multi`/`baseline`) |
| GET | `/health``/health/ready` | 存活 / 就绪检查(含 DB) |
**需管理员**(Cookie 会话或 `X-API-Key`):
| 方法 | 路径 | 说明 |
|---|---|---|
| POST | `/api/v1/ingest/bzzoiro` | 采集赛果/赛程/统计/积分榜 |
| GET | `/api/v1/predictions` | 预测历史(列表) |
| GET | `/api/v1/predictions/{id}` | 单条预测详情 |
| POST | `/api/v1/eval/settle` | 回填实际结果 |
| GET | `/api/v1/eval/summary` | 准确率汇总 |
| POST | `/api/v1/backtest` | 历史回测 |
| GET | `/health` | 存活检查 |
| GET | `/health/ready` | 就绪检查(含 DB) |
| `/api/v1/admin/**` | 配置/采集状态/日志/定时任务/死信等 | 管理后台(router 级鉴权) |
## 项目结构
@@ -163,36 +188,46 @@ Profeto/
├── src/
│ ├── api/ # FastAPI 路由层
│ │ ├── app.py # 应用工厂 + lifespan
│ │ ├── deps.py # 依赖注入:鉴权 / 限流
│ │ ├── schemas.py # Pydantic 请求/响应模型
│ │ └── routes/
│ │ ├── matches.py # 比赛查询
│ │ ├── predict.py # 预测入口
│ │ ├── ingest.py # 数据采集
│ │ ├── eval.py # 评估回填
│ │ ── backtest.py # 回测
│ │ ├── matches.py # 比赛查询(公开只读)
│ │ ├── predict.py # 预测入口 + 预测历史
│ │ ├── ingest.py # 数据采集(需管理员)
│ │ ├── eval.py # 评估回填(需管理员)
│ │ ── backtest.py # 回测(需管理员)
│ │ ├── auth.py # 登录/登出/改密
│ │ ├── admin_settings.py # /admin/** 配置/日志/数据质量(router 级鉴权)
│ │ └── schedules.py # 定时任务 + 死信重试(router 级鉴权)
│ ├── core/ # 基础设施
│ │ ├── config.py # pydantic-settings 配置
│ │ ├── crypto.py # 加密/哈希
│ │ ├── http_client.py # 共享 httpx 客户端
│ │ ── retry.py # 重试工具(指数退避)
│ │ ── log_buffer.py # 内存日志缓冲(admin 日志页)
│ │ ├── runtime_config.py # DB 配置覆盖(.env → app_settings)
│ │ ├── scheduler.py # 进程内 cron 调度器
│ │ └── security_check.py # 启动安全校验
│ ├── data/ # 数据层
│ │ ├── sources.py # DataSource 协议 + 注册表
│ │ ├── bzzoiro.py # bzzoiro 数据源(events/standings/stats)
│ │ ├── normalize.py # 数据规范化契约
│ │ ├── bzzoiro.py # bzzoiro 数据源
│ │ ├── understat.py # understat xG 数据源
│ │ ├── injuries.py # 伤停数据
│ │ ├── config.py # 联赛映射常量
│ │ ── team_names.py # 队名归一化
│ │ ── key_ring.py # API Key 轮换环(429 冷却)
│ │ ├── team_names.py # 队名归一化
│ │ └── team_names_zh.py # 队名中文映射
│ ├── db/ # 数据库
│ │ ├── base.py # SQLAlchemy async engine
│ │ ├── models.py # ORM 模型 (6 表)
│ │ ├── models.py # ORM 模型 (12 表)
│ │ ├── unit_of_work.py # UnitOfWork 事务封装
│ │ └── repositories.py # Repository 数据访问
│ └── llm/ # LLM 预测核心
│ ├── predict.py # 预测服务 (缓存 + 单/多模式)
│ ├── predict.py # 预测服务 (缓存 + 单/多/基线模式)
│ ├── context_builder.py # 数据切片 + 上下文拼接
│ ├── baseline.py # 基线预测(均值模型)
│ ├── eval.py # 评估统计
│ ├── backtest.py # 回测框架
│ ├── provider.py # 多提供商 LLM 抽象
│ ├── utils.py # LLM 工具函数
│ ├── validation.py # LLM 输出校验
│ ├── agents/
│ │ ├── base.py # Agent 基础设施 + 解析
@@ -200,6 +235,11 @@ Profeto/
│ └── prompts/ # Prompt 模板
├── alembic/ # 数据库迁移
├── frontend/ # React 前端
│ └── src/
│ ├── pages/ # 公开站(赛程 Matches + 积分榜 Standings)
│ ├── admin/ # 管理后台(布局/页面/数据访问层 dal.ts)
│ ├── components/ # 共享组件
│ └── lib/http.ts # 唯一 HTTP 实现(带凭据/超时/错误处理)
├── docs/ # 详细文档
├── tests/ # 单元测试
├── docker-compose.yml
@@ -234,7 +274,6 @@ Profeto/
| `LLM_SPECIALIST_MODEL` | 专家模型 (空=回落 LLM_MODEL) | |
| `LLM_AGGREGATOR_MODEL` | 终裁模型 (空=回落 LLM_MODEL) | |
| `BZZOIRO_KEY` | bzzoiro API Key | *(必填)* |
| `API_FOOTBALL_KEY` | api-football Key (伤停) | |
| `CORS_ORIGINS` | 允许的跨域来源 | `http://localhost:5173` |
## 测试
+36 -28
View File
@@ -13,16 +13,15 @@
│ │
│ 数据查询 预测编排 采集(手动/cron 触发) │
│ ┌──────┐ ┌────────────┐ ┌───────────────────┐ │
│ │matches│ │ orchestrator│ │ bzzoiro (赛果) │ │
│ │matches│ │ orchestrator│ │ bzzoiro (唯一源) │ │
│ │leagues│ │ ┌─ 5 专家并行(便宜模型) │ │
│ └──┬───┘ │ │ h2h / form / stats / │ │
│ │ │ │ home_away / injuries │ │
│ │ │ │ home_away / standings │ │
│ │ │ └─ aggregator 终裁(强模型) │ │
│ │ └────────────┘ └───────────────────┘
│ │ └ understat (xG)
──────────────────┐ └ injuries (伤停)
│ PostgreSQL (6 张表) │ httpx → 外部 API │
│ └────────────────────┘ │
│ │ └────────────┘ │ events / standings │
┌──┴──────────────┴──┐ │ /stats 三条管线 │
│ PostgreSQL (12 张表)│ └───────────────────┘
└────────────────────┘ httpx → 外部 API │
└─────────────────────────────────────────────────────┘
```
@@ -31,7 +30,7 @@
1. `POST /predict {match_id}` → orchestrator
2. `load_match_header`: 查比赛 + 双方 + 联赛(一次 eager load)
3. **5 个专家 agent 并行**(`asyncio.gather`),每个:
- 各自的数据切片函数查库(近况/交锋/积分榜 SQL 聚合/伤停/xG)
- 各自的数据切片函数查库(近况/交锋/积分榜聚合/射门控球/xG)
- 切片无数据 → **跳过 LLM**,直接 `no_data` stub(省 token、防幻觉)
- 有数据 → 专属 prompt(专家模型,便宜快)→ 结构化 JSON 报告(`home_edge` 方向性评分 + 证据)
4. **终裁 agent**:5 份报告 + 比赛信息 → 权衡采信度(`agent_weights`)→ 最终预测 JSON
@@ -44,7 +43,7 @@
|---|---|
| **多专家并行而非单次大 prompt** | 每维度独立迭代 prompt;报告可归因(哪个维度分析错了);总延迟 ≈ 2 次串行调用 |
| **专家/终裁模型分档** | 专家用便宜模型快速分析,终裁用强模型汇总决策,成本与质量平衡(`LLM_SPECIALIST_MODEL` / `LLM_AGGREGATOR_MODEL`) |
| **no_data 门控** | 无数据维度(如伤停未接入)不调 LLM,终裁知道维度缺失,不编造 |
| **no_data 门控** | 无数据维度(如积分榜未采集)不调 LLM,终裁知道维度缺失,不编造 |
| **fail-open** | 单个专家失败只标记 `status=error`,其余照常;研究场景可用性优先 |
| **`match_date_date` 天级去重** | 不同源时间精度不同,秒级匹配会产生重复行;天级 + 数据库唯一约束 |
| **积分榜 SQL 聚合 + season 过滤** | `UNION ALL` 主客双视角 + `GROUP BY` 在库内算,只算当前赛季(修复过跨赛季 bug) |
@@ -57,33 +56,38 @@
Profeto/
├── src/
│ ├── api/
│ │ ├── app.py # FastAPI 工厂(lifespan 仅验证 DB 连接,不建表)
│ │ ├── deps.py # 依赖:管理接口鉴权(X-API-Key)
│ │ ├── app.py # FastAPI 工厂(lifespan:迁移校验/定时任务/生产限流提醒)
│ │ ├── deps.py # 依赖:管理接口鉴权(Cookie/X-API-Key)+ 限流
│ │ ├── schemas.py # Pydantic v2 请求/响应
│ │ └── routes/
│ │ ├── matches.py # 联赛/比赛查询(游标分页)
│ │ ├── predict.py # 预测 + 预测历史
│ │ ├── ingest.py # 采集触发(自管 session,需鉴权)
│ │ ├── eval.py # 赛后回填 + 准确率汇总
│ │ ── backtest.py # 历史回测(需鉴权)
│ │ ├── matches.py # 联赛/比赛/上下文/积分榜(公开只读)
│ │ ├── predict.py # 预测(限流)+ 预测历史(需鉴权)
│ │ ├── ingest.py # 采集触发(需鉴权)
│ │ ├── eval.py # 赛后回填 + 准确率汇总(需鉴权)
│ │ ── backtest.py # 历史回测(需鉴权)
│ │ ├── auth.py # 登录/登出/改密
│ │ ├── admin_settings.py # /admin/** 配置/日志/数据质量(router 级鉴权)
│ │ └── schedules.py # 定时任务 + 死信重试(router 级鉴权)
│ ├── db/
│ │ ├── base.py # async engine + get_db/get_db_read
│ │ ├── models.py # 6 张表 ORM
│ │ ├── models.py # 12 张表 ORM
│ │ ├── repositories.py # 仓储层
│ │ └── unit_of_work.py # 事务边界
│ ├── data/
│ │ ├── bzzoiro.py # 赛果采集 + 幂等入库
│ │ ├── understat.py # xG 回填
│ │ ├── injuries.py # 伤停采集(带文件缓存)
│ │ ├── bzzoiro.py # 唯一数据源:events/standings/stats 三管线 + Bronze 层
│ │ ├── normalize.py # NormalizedMatch 清洗契约
│ │ ├── team_names.py # 队名归一映射
│ │ ├── team_names_zh.py # 队名中文名映射
│ │ ├── key_ring.py # 多 key 轮换(429 冷却,进程内)
│ │ ├── sources.py # 数据源注册表
│ │ └── config.py # 联赛代码映射
│ ├── llm/
│ │ ├── provider.py # OpenAI-compatible 抽象(共享连接池/JSON 兜底解析)
│ │ ├── context_builder.py # 数据切片(h2h/form/stats/home_away/injuries)+ 单 agent 拼接
│ │ ├── context_builder.py # 数据切片(h2h/form/stats/home_away/standings)+ 单 agent 拼接
│ │ ├── predict.py # 预测入口(mode 分派 + 缓存)
│ │ ├── baseline.py # 基线预测(均值模型,mode=baseline)
│ │ ├── eval.py # 准确率统计
│ │ ├── utils.py # LLM 工具函数
│ │ ├── validation.py # LLM 输出严格校验(Pydantic)
│ │ ├── backtest.py # 回测执行
│ │ ├── agents/
@@ -91,16 +95,20 @@ Profeto/
│ │ │ └── orchestrator.py # 并行专家 → 终裁 → 存库
│ │ └── prompts/
│ │ ├── match_prediction_v1/v2.md # 单 agent 模板
│ │ └── agents/{h2h,form,stats,home_away,injuries,aggregator}_v1.md
│ │ └── agents/{form,stats,home_away,standings,h2h,aggregator}_v1.md
│ └── core/
│ ├── config.py # pydantic-settings
│ ├── crypto.py # 加密/哈希
│ ├── http_client.py # 共享 httpx 客户端
── retry.py # 重试工具
├── alembic/versions/ # 0001~0006(0001 建表 → 0006 漂移清理)
├── frontend/src/pages/Matches.tsx # 单页(预测面板 + 专家报告折叠区 + 游标分页)
├── tests/ # 核心 + agent 测试
── log_buffer.py # 内存日志缓冲(admin 日志页)
│ ├── runtime_config.py # DB 配置覆盖(app_settings)
│ ├── scheduler.py # 进程内 cron 调度器
│ └── security_check.py # 启动安全校验
├── alembic/versions/ # 0001~0018(建表 → Bronze 层 → 单一数据源 → 基线模式等)
├── frontend/src/ # pages/(公开站) + admin/(管理后台) + lib/http.ts(唯一 HTTP 实现)
├── tests/ # 核心 + agent 测试(250+ 项,自包含)
├── docker-compose.yml # api + postgres 两容器
└── docs/ # 本文档
└── docs/ # 本文档
```
## 技术栈
@@ -113,5 +121,5 @@ Profeto/
| HTTP | httpx(共享连接池)/ urllib(bzzoiro 同步限速) |
| LLM | OpenAI-compatible 接口(openai/deepseek/ollama 等任一) |
| 前端 | Vite + React 18 + TypeScript + Tailwind |
| 测试 | pytest + pytest-asyncio(33 项,自包含) |
| 测试 | pytest + pytest-asyncio(250+ 项,自包含) |
| 部署 | Docker Compose(api + postgres) |
+13 -4
View File
@@ -66,6 +66,9 @@ curl -X POST http://localhost:8000/api/v1/ingest/bzzoiro \
-d '{"leagues":["E0"],"date_from":"2026-08-01","date_to":"2026-09-08"}'
```
> 注:采集/评估端点需管理员凭据。本地开发环境(未配置鉴权、非 production)默认放行;
> 生产环境需先 `POST /auth/login` 取 Cookie,或带 `X-API-Key` 头。
数据量大时**直接拉整赛季**(约 380 场,含近几个赛季更好,近况/交锋/积分榜都需要历史):
```bash
@@ -74,14 +77,20 @@ curl -X POST http://localhost:8000/api/v1/ingest/bzzoiro \
-d '{"leagues":["E0"],"date_from":"2025-08-01","date_to":"2026-09-08"}'
```
### 回填 xG(可选,让攻防数据 agent 有数据)
### 回填积分榜与统计(让攻防/排名专家有数据)
```bash
curl -X POST http://localhost:8000/api/v1/ingest/understat \
curl -X POST http://localhost:8000/api/v1/ingest/bzzoiro \
-H "Content-Type: application/json" \
-d '{"league":"E0","season":2025}'
-d '{"task":"standings"}'
curl -X POST http://localhost:8000/api/v1/ingest/bzzoiro \
-H "Content-Type: application/json" \
-d '{"task":"stats","leagues":["E0"],"limit":300}'
```
`task=stats` 只补空字段(xG/射门/控球等),不创建比赛。
### 查比赛
浏览器打开 http://localhost:5173 ,选"英超 / 未开赛";
@@ -126,4 +135,4 @@ curl http://localhost:8000/api/v1/eval/summary
| predict 返回 502 | 看 uvicorn 日志的 LLM error;确认 `LLM_BASE_URL`/`LLM_API_KEY`;`response_format` 不兼容的网关会报错(改用支持 json mode 的模型) |
| 采集 0 场 | bzzoiro Key 失效或联赛代码写错;先 `GET /api/v1/leagues` 看库里有没有联赛 |
| 专家报告全是 no_data | 历史数据不够 —— 近况需要每队近 5 场、积分榜需要本赛季已完赛比赛,多拉几周数据 |
| xg agent 报无 xG 数据 | 先跑 understat 回填;注意 understat 只有五大联赛 |
| stats 专家报无 xG/统计 | 先跑 `task=stats` 回填(bzzoiro 统计管线,只补空字段) |
+80 -34
View File
@@ -4,18 +4,30 @@ Base URL: `http://localhost:8000` · 交互式文档: `/docs`(Swagger)与 `/redo
所有数据端点返回 JSON。错误统一为 `{"detail": "<message>"}` + 对应 HTTP 状态码。
## 鉴权模型
| 级别 | 端点 | 说明 |
|---|---|---|
| **公开只读** | `GET /leagues``GET /matches``GET /matches/{id}``GET /matches/{id}/context``GET /standings``GET /health*` | 无需任何凭据;公开站直接调用 |
| **公开 + 限流** | `POST /predict` | 内存滑动窗口限流(10 次/分钟/IP) |
| **需管理员** | `GET /predictions*``POST /ingest/bzzoiro``/eval/*``POST /backtest``/admin/**` | Cookie 会话(`POST /auth/login` 颁发)或 `X-API-Key` 头 |
管理端点在生产环境未配置鉴权时 fail-closed(503),不会静默放行。
---
## 数据查询
## 数据查询(公开只读)
### `GET /api/v1/leagues`
列出已入库联赛。
列出已入库联赛(P1-3: 公开站联赛筛选动态加载来源)
```json
[{"id": 1, "code": "E0", "name": "Premier League", "country": "England"}]
```
仅返回 `id/code/name/country` 四个展示字段,不含任何配置或密钥信息。
### `GET /api/v1/matches`
比赛列表,游标分页。
@@ -47,7 +59,44 @@ Base URL: `http://localhost:8000` · 交互式文档: `/docs`(Swagger)与 `/redo
### `GET /api/v1/matches/{id}`
单场比赛详情,字段同上。
单场比赛详情,字段同上,另含 `stats`(统计)与 `recent_predictions`(最近 5 条预测摘要)
### `GET /api/v1/matches/{id}/context`
比赛上下文(公开只读,P1-2: 公开站详情页「近况/交锋」数据来源;不触发 LLM):
- `home_recent`: 主队最近 5 场已完赛
- `away_recent`: 客队最近 5 场已完赛
- `h2h`: 双方最近 5 次交手
```json
{
"home_recent": [
{"match_date": "2026-09-12T14:00:00+00:00", "home_team": "阿森纳",
"away_team": "切尔西", "home_goals": 2, "away_goals": 1}
],
"away_recent": [],
"h2h": []
}
```
数据不足时对应列表为空(前端展示空态)。比赛不存在返回 404。
### `GET /api/v1/standings`
联赛积分榜(公开只读)。参数:`league`(联赛代码,空 = 全部)、`season`(空 = 各联赛最新赛季)。
```json
{"leagues": [
{"league_code": "E0", "league_name": "Premier League", "season": "2026-2027",
"retrieved_at": "2026-09-20T08:00:00+00:00",
"rows": [{"position": 1, "team": "阿森纳", "team_en": "Arsenal",
"played": 5, "won": 4, "drawn": 1, "lost": 0,
"goals_for": 11, "goals_against": 3, "goal_diff": 8,
"points": 13, "xg_for": 9.8, "xg_against": 3.9,
"form": "WWWDW", "zone": "UEFA Champions League"}]}
]}
```
---
@@ -96,13 +145,13 @@ Base URL: `http://localhost:8000` · 交互式文档: `/docs`(Swagger)与 `/redo
"exp_home_goals": null, "exp_away_goals": null, "probable_score": null,
"model": "gpt-4o-mini", "latency_ms": 2100,
"prompt_tokens": 380, "completion_tokens": 120},
{"agent": "injuries", "status": "no_data", "data_sufficiency": "none",
{"agent": "standings", "status": "no_data", "data_sufficiency": "none",
"analysis": "该维度无数据,跳过分析。", "home_edge": null, "subjective_confidence": null,
"key_evidence": [], "exp_home_goals": null, "exp_away_goals": null,
"probable_score": null, "model": "", "latency_ms": null,
"prompt_tokens": null, "completion_tokens": null}
],
"agent_weights": {"form": 0.9, "stats": 0.8, "home_away": 0.7, "injuries": 0.0, "h2h": 0.8},
"agent_weights": {"form": 0.9, "stats": 0.8, "home_away": 0.7, "standings": 0.8, "h2h": 0.8},
"context": "[5 份报告的 JSON 串]",
"latency_ms": 9800
}
@@ -112,11 +161,11 @@ Base URL: `http://localhost:8000` · 交互式文档: `/docs`(Swagger)与 `/redo
错误:404 比赛不存在;502 LLM 调用失败(终裁失败时整体失败,专家失败不会)。
### `GET /api/v1/predictions?match_id=&limit=`
### `GET /api/v1/predictions?match_id=&limit=`(需管理员)
预测历史(倒序),含 `settled` 与实际比分回填状态。
### `GET /api/v1/predictions/{id}`
### `GET /api/v1/predictions/{id}`(需管理员)
单条预测详情(含完整 `agent_outputs`)。
@@ -124,40 +173,34 @@ Base URL: `http://localhost:8000` · 交互式文档: `/docs`(Swagger)与 `/redo
## 数据采集
### `POST /api/v1/ingest/bzzoiro`
### `POST /api/v1/ingest/bzzoiro`(需管理员)
从 bzzoiro 采集赛果/赛程并入库(幂等,重复跑安全)。
从 bzzoiro(唯一数据源)采集数据并入库(幂等,重复跑安全)。任务在后台异步执行,请求立即返回。
```json
{"leagues": ["E0", "SP1"], "date_from": "2025-08-01", "date_to": "2026-09-08", "status": "finished"}
{"task": "all", "leagues": ["E0", "SP1"], "date_from": "2025-08-01", "date_to": "2026-09-08", "status": "finished"}
```
- `status` 还可传 `scheduled` 拉未来赛程
| 字段 | 默认 | 说明 |
|---|---|---|
| `task` | `events` | 采集任务:`events`(比赛)/ `standings`(积分榜)/ `stats`(统计回填,含 xG)/ `all` |
| `leagues` | 全部已知联赛 | 联赛代码列表,如 `["E0", "SP1"]` |
| `date_from` / `date_to` | 空 | 日期范围(`YYYY-MM-DD`) |
| `status` | 空(两者都采) | `finished`(已完赛)/ `scheduled`(未来赛程) |
| `season` | 当前赛季 | standings 赛季,如 `"2026-2027"` |
| `limit` | 100 | stats 回填单次最大比赛数(1500) |
- 响应含每联赛 `inserted`/`updated`/`errors` 统计
- `task=stats` 只补空字段、不创建比赛(xG/射门/控球等统计回填)
### `POST /api/v1/ingest/understat`
回填 xG(只补空字段,不创建比赛):
```json
{"league": "E0", "season": 2025}
```
`season=2025` 表示 2025-2026 赛季。仅支持五大联赛。
### `POST /api/v1/ingest/injuries`
采集伤停(需 `API_FOOTBALL_KEY`,当前只返回计数,尚未接入 context):
```json
{"date": "2026-09-10"}
```
> 历史版本曾有独立的 understat(xG)与 injuries(伤停)采集端点,
> 已随数据源收敛为 bzzoiro 唯一来源而移除。
---
## 评估
### `POST /api/v1/eval/settle`
### `POST /api/v1/eval/settle`(需管理员)
赛后回填实际比分:
@@ -165,7 +208,7 @@ Base URL: `http://localhost:8000` · 交互式文档: `/docs`(Swagger)与 `/redo
{"prediction_id": 7, "home_goals": 2, "away_goals": 1}
```
### `GET /api/v1/eval/summary`
### `GET /api/v1/eval/summary`(需管理员)
`provider × model` 聚合已结算预测:
@@ -183,7 +226,10 @@ Base URL: `http://localhost:8000` · 交互式文档: `/docs`(Swagger)与 `/redo
## 基础
| 端点 | 说明 |
|---|---|
| `GET /health` | 存活检查 |
| `GET /docs` | Swagger UI |
| 端点 | 权限 | 说明 |
|---|---|---|
| `GET /health` | 公开 | 存活检查 |
| `GET /docs``GET /redoc` | 公开 | Swagger UI / ReDoc |
| `POST /auth/login``POST /auth/logout``GET /auth/me` | 公开 | 管理员 Cookie 会话登录/登出/当前用户 |
| `POST /auth/change-password` | 需管理员 | 修改管理员密码 |
| `POST /backtest` | 需管理员 | 历史回测(对已完赛比赛批量预测并评估) |
+7 -7
View File
@@ -10,7 +10,7 @@ Profeto 的核心预测路径是 **5 个领域专家 Agent 并行分析 + 1 个
| `form` 近期状态 | 分析比分与关键事件,判断近期走势 | 两队近 N 场赛果(含 xG) | `home_edge` + 走势判断 |
| `stats` 攻防数据 | 评估进球、射门与控球,量化攻防强度 | 近 N 场进球/射门/控球/xG 统计 | `home_edge` + 攻防强度 |
| `home_away` 主客因素 | 对比主场与客场表现,评估地理优势影响 | 主队主场战绩 + 客队客场战绩 | `home_edge` + 地理优势 |
| `injuries` 阵容完整性 | 汇总伤停与停赛名单,评估战力缺失程度 | 伤停数据(当前无源 → no_data 门控) | `home_edge``no_data` |
| `standings` 联赛排名 | 结合积分榜排名、积分与分区(欧冠/欧联/降级),评估双方竞争位置 | 两队当前赛季积分榜行(排名/积分/分区/近期战绩) | `home_edge``no_data` |
| `h2h` 历史交锋 | 分析过去数年以及近期的交手数据,提取交手规律 | 近 N 次交锋(含主客方向 + 总计统计) | `home_edge` + 交手规律 |
| `aggregator` 终裁 | 权衡 5 份报告 → 最终结论 | 5 份结构化报告 + 比赛头信息 | 最终预测 + 各报告采信度 |
@@ -25,7 +25,7 @@ POST /predict {match_id, mode: "multi"}
│ ├─ form agent ─┐
│ ├─ stats agent │ 每个 agent 拿到专属数据切片
│ ├─ home_away agent │ → no_data 门控 → 调 LLM → 输出 JSON 报告
│ ├─ injuries agent │ (无数据 → 跳过 LLM,返回 stub)
│ ├─ standings agent │ (无数据 → 跳过 LLM,返回 stub)
│ └─ h2h agent ─┘
├─ aggregator agent(5 份报告 + 比赛头 → 最终 JSON)
@@ -39,12 +39,12 @@ POST /predict {match_id, mode: "multi"}
5 个专家通过 `asyncio.gather` 并发,总延迟 ≈ `max(专家延迟) + 终裁延迟` ≈ 2 次串行 LLM 调用。
### 2. no_data 门控(省 token、防幻觉)
数据切片为空时(如伤停数据源未接入),**跳过 LLM 调用**,直接返回:
数据切片为空时(如该场比赛的积分榜尚未采集),**跳过 LLM 调用**,直接返回:
```json
{"agent": "injuries", "status": "no_data", "data_sufficiency": "none",
{"agent": "standings", "status": "no_data", "data_sufficiency": "none",
"analysis": "该维度无数据,跳过分析。"}
```
终裁 Agent 会看到这个 `no_data` 状态,不会编造伤停分析。
终裁 Agent 会看到这个 `no_data` 状态,不会编造积分榜分析。
### 3. fail-open(单专家失败不阻断)
单个专家 LLM 调用失败 → 其报告标记 `status: error`,其余 4 份 + 终裁照常执行。
@@ -89,7 +89,7 @@ POST /predict {match_id, mode: "multi"}
"1x2": "1",
"subjective_confidence": 0.68,
"reasoning": "综合 stats 报告的攻防强度与 form 报告的三连胜势头……",
"agent_weights": {"form": 0.9, "stats": 0.8, "home_away": 0.7, "injuries": 0.0, "h2h": 0.8}
"agent_weights": {"form": 0.9, "stats": 0.8, "home_away": 0.7, "standings": 0.8, "h2h": 0.8}
}
```
@@ -120,7 +120,7 @@ agents/
├── form_v1.md # 近期状态专家
├── stats_v1.md # 攻防数据专家
├── home_away_v1.md # 主客因素专家
├── injuries_v1.md # 阵容完整性专家
├── standings_v1.md # 联赛排名专家
├── h2h_v1.md # 历史交锋专家
└── aggregator_v1.md # 终裁
```
+56 -14
View File
@@ -4,9 +4,10 @@
| 数据源 | 用途 | 必需 Key | 说明 |
|---|---|---|---|
| bzzoiro | 赛果/赛程(主源) | `BZZOIRO_KEY` | 五大联赛历史 + 实时 |
| understat | xG 回填 | 无(公开) | 仅五大联赛,补 `match_stats.xg` |
| api-football | 伤停 | `API_FOOTBALL_KEY` | 当前只采集计数,未接入 context |
| bzzoiro(唯一) | 赛果/赛程/积分榜/统计(xG、射门、控球等) | `BZZOIRO_KEY` | 五大联赛 + 欧战,历史 + 实时 |
> 历史版本曾有 understat(xG 回填)与 api-football(伤停)两个辅助源,
> 现已移除:数据源收敛为 bzzoiro 唯一来源,统计与积分榜均由 bzzoiro 管线采集。
### bzzoiro
@@ -28,12 +29,6 @@
> - 黄牌: `home_yellow_cards` / `away_yellow_cards`
> - 红牌: `home_red_cards` / `away_red_cards`
### understat
- 端点:`/getLeagueData/{league}/{season}`,返回 JS 包裹的 JSON(需正则提取)
- 只回填 xG(`match_stats.home_xg`/`away_xg`),**不创建新比赛**
- 通过"天级日期 + 队名归一"匹配已有比赛
## 数据清洗契约
所有数据源统一清洗为 `NormalizedMatch`(`src/data/normalize.py`),字段:
@@ -68,9 +63,39 @@
`src/data/team_names.py` 维护 `NORMALIZE_MAP`(如 `Man City` → `Manchester City`),未命中映射的队名原样返回。
归一前先做 Unicode NFKD 去重音。
**唯一键是归一后英文名**:`teams.name` 带 `UNIQUE` 约束,所有入库路径均经 `TeamRepository.get_or_create` 收敛归一化
(events / standings 管线在调用前归一,仓库层再做一次幂等归一作为兜底)。创建新 Team 时打 `info` 日志记录「原始名 → 归一后名」。
> ⚠️ **`normalize` 当前大小写敏感**:仅当入参大小写与 `NORMALIZE_MAP` 键完全匹配时才触发映射
>(如 `"Man City"` → `"Manchester City"`,但 `"man city"` 原样保留)。上游 bzzoiro 返回的队名首字母大写,
>实际命中无问题;若新增数据源返回全小写/全大写队名,需先 `title()` 再归一,否则会绕过映射产生重复 Team。
**改名 / 合并流程**(人工):
当发现两个 `teams` 行实际是同一球队(如 `Manchester City` 与 `Man City` 因历史数据大小写差异各占一行):
1. 确定**保留行**(通常选归一后规范名、且被更多 Match 引用的那行)。
2. 将被删行的所有引用指向保留行(`UPDATE matches SET home_team_id = 保留id WHERE home_team_id = 删行id`,客场同理;
`standings` / `match_stats` 按 `team_id` 同理)。
3. 删掉多余行:`DELETE FROM teams WHERE id = 删行id`。
> 此过程引入外键约束风险,务必在事务中执行并先 `BEGIN; ... ` 验证行数后再 `COMMIT`。
> 暂不做自动合并(避免误合相似名),仅通过下方 Admin 接口列出「近似重名」候选,由人工判定。
## Admin:近似重名候选
`GET /api/v1/admin/team-name-duplicates` 只读列出启发式相似候选(大小写差异、子串包含、前缀碰撞),不做自动合并。
典型用途:定期巡检,发现候选后走上方人工 SQL 合并。启发式规则:
- **大小写变体**:`lower(name)` 相同但 `name` 不同(如 `Arsenal FC` / `arsenal fc`)。
- **子串包含**:A 是 B 的子串且长度 ≥ 5(如 `Manchester` / `Manchester City`)。
- **前缀碰撞**:前 8 个字符相同的两队。
命中任一规则即列为候选,按相似度分组返回。
## 数据库 Schema
6 张表:
12 张表:核心业务表 5 张见下方 DDL,其余 7 张(积分榜/配置/调度/治理)见后文表格。
```sql
-- 联赛
@@ -144,12 +169,25 @@ CREATE TABLE predictions (
reasoning TEXT,
raw_response JSONB, -- LLM 完整原始响应
agent_outputs JSONB, -- multi 模式: 5 份专家报告
agent_weights JSONB, -- multi 模式: 终裁给出的各专家权重
created_at TIMESTAMPTZ,
actual_home_goals INT, actual_away_goals INT, -- 赛后回填
settled BOOLEAN DEFAULT FALSE
);
```
其余 7 张表(DDL 略,详见 `src/db/models.py` 与 alembic 迁移):
| 表 | 状态 | 用途 |
|---|---|---|
| `standings` | 已启用 | 联赛积分榜快照,按 `(league_id, season, team_id)` upsert,同联赛同赛季只保留最新快照;含排名/战绩/进失球/积分/分区(zone) |
| `app_settings` | 已启用 | 后台运行时设置(如数据源 API Key),读取时优先于 `.env` 默认值 |
| `schedules` | 已启用 | 定时采集任务配置(task/cron/leagues/enabled),供内置调度器执行 |
| `raw_events` | 预留未启用 | Bronze 层原始事件存档;规划中用于重放与审计 |
| `ingest_failures` | 已启用 | 采集失败死信:bzzoiro 三条管线(events/standings/stats)抓取失败时写入,admin 后台可查看与重试 |
| `data_quality_checks` | 预留未启用 | 数据质量检查结果;规划中定时检查比赛/统计/积分榜完整性 |
| `data_lineage` | 预留未启用 | ETL 血缘追踪;规划中记录源记录到目标表的映射 |
### 关键设计点
1. **`match_date_date`(天级日期)**: 用于天级去重。bzzoiro 返回的时间带时分秒,精确匹配不可靠,故拆出 `DATE` 列做唯一键。
@@ -170,7 +208,9 @@ CREATE TABLE predictions (
- 状态:只允许单向升级(`scheduled` → `finished`),防止完赛行被覆盖成赛程
- stats:只补空(`home_xg` 已有值时不覆盖)
`ingest_understat` 只回填 xG(也只补空),不创建比赛。
`task=stats` 只回填统计(xG/射门/控球等,也只补空),不创建比赛。
`task=standings` 按 `(league_id, season, team_id)` upsert 积分榜快照,同一联赛同一赛季只保留最新一份。
## 采集建议
@@ -183,9 +223,11 @@ curl -X POST /api/v1/ingest/bzzoiro \
curl -X POST /api/v1/ingest/bzzoiro \
-d '{"leagues":["E0"],"date_from":"2026-09-01","date_to":"2026-09-08"}'
# 3. xG 回填(可选,提升 xg agent 质量)
curl -X POST /api/v1/ingest/understat -d '{"league":"E0","season":2025}'
curl -X POST /api/v1/ingest/understat -d '{"league":"E0","season":2026}'
# 3. 积分榜 + 统计回填(xG/射门/控球,提升 stats/standings 专家质量)
curl -X POST /api/v1/ingest/bzzoiro -d '{"task":"standings"}'
curl -X POST /api/v1/ingest/bzzoiro -d '{"task":"stats","leagues":["E0"],"limit":300}'
# 注:采集端点需管理员凭据(Cookie 会话或 X-API-Key 头),下同
```
建议用外部 cron(如系统 crontab)定时触发,不引入 worker/redis。
+39 -2
View File
@@ -33,6 +33,25 @@ curl http://localhost:8000/health
> **注意**: `api` 服务启动时会先执行 `alembic upgrade head` 迁移数据库,再启动 uvicorn。
> 容器内数据库连接自动使用 `postgres` 服务名(通过 compose `environment` 覆盖 `.env` 中的 `DB_HOST`)。
## 生产上线检查清单
公网上线前逐项勾选。第 14 项由 `src/core/security_check.py``APP_ENV=production`
启动时**强制校验**,不满足直接拒绝启动(开发环境仅告警);管理鉴权另有请求期 fail-closed(503)。
- [ ] **1. `APP_ENV=production`** — 安全校验、Cookie `Secure`、管理端点 fail-closed 均以它为总开关
- [ ] **2. `SECRET_KEY` 强随机** — 用 `openssl rand -base64 32` 生成;禁止弱值/短值(弱值黑名单会拒绝启动,含把生成指令原样粘进去的情况)
- [ ] **3. 管理鉴权至少其一**`ADMIN_PASSWORD``ADMIN_API_KEY`(后台改过密码后以数据库哈希优先);两者皆空时管理接口 503
- [ ] **4. 数据库强密码** — 禁止 `football:football` 等示例弱密码(启动校验会拒绝);compose 的 `POSTGRES_PASSWORD` 必填,缺失时容器拒绝启动
- [ ] **5. HTTPS** — 由反代(Nginx/Caddy)终结 TLS;`APP_ENV=production` 下会话 Cookie 自动 `Secure`(且 HttpOnly + SameSite=Lax)
- [ ] **6. 反代信任头**`TRUST_PROXY_HEADERS=True`,且**仅可信反代可达 API**;反代需设置 `X-Forwarded-For`(`$proxy_add_x_forwarded_for`)与 `X-Real-IP`,否则限流/日志按反代 IP 计数
- [ ] **7. 限流前置到网关** — 推荐 Nginx `limit_req`(配置见[安全与限流](#安全与限流));应用内限流与 KeyRing 为**单进程内存实现**,多 worker 各自独立计数会把实际配额放大 N 倍(启动时会打印一次性告警)
- [ ] **8. uvicorn 单 worker** — compose/Dockerfile 默认单 worker,保持即可;需横向扩容时先在网关统一限流,再起多实例(每实例仍单 worker)
> ⚠️ **多 worker 陷阱**:应用内限流(`_RateLimiter`)与 KeyRing 均为**进程内纯内存状态**,多 worker 部署(如 `uvicorn --workers 4`)时各进程**各自独立计数、互不共享**——实际限流配额会被放大 N 倍、KeyRing 限流状态也不同步。
> 若确需多 worker,必须前置 Nginx/网关做**全局限流**(见[安全与限流](#安全与限流)),并设环境变量 `STRICT_SINGLE_WORKER=True`(见下)在启动期强制拒绝多 worker,避免静默配额漂移。
- [ ] **9. 启动后健康检查**`curl /health` 返回 200(存活);`curl /health/ready` 返回 200(就绪,校验数据库连通,不可达时 503)
- [ ] **10. 数据库迁移** — compose/Dockerfile 启动命令已内置 `alembic upgrade head && uvicorn …`,升级镜像重启即自动迁移,无需手动执行
## 本地开发部署
```bash
@@ -81,13 +100,31 @@ cd frontend && npm install && npm run dev
| `LLM_TIMEOUT` | ❌ | `60` | 单次调用超时(秒) |
| `LLM_SPECIALIST_MODEL` | ❌ | — | 专家模型(回落 `LLM_MODEL`) |
| `LLM_AGGREGATOR_MODEL` | ❌ | — | 终裁模型(回落 `LLM_MODEL`) |
| `BZZOIRO_KEY` | ✅ | — | bzzoiro 数据源 Key |
| `API_FOOTBALL_KEY` | ❌ | — | 伤停数据源 Key |
| `BZZOIRO_KEY` | ✅ | — | bzzoiro 数据源 Key(唯一数据源) |
| `CORS_ORIGINS` | ❌ | `http://localhost:5173,...` | 允许的跨域来源 |
| `STRICT_SINGLE_WORKER` | ❌ | `False` | `True` 时若以多 worker 启动则拒绝(防限流配额漂移) |
| `SECRET_KEY` | ❌ | — | 加密主密钥(生产环境必填) |
| `ADMIN_PASSWORD` | ❌ | — | 管理后台密码(留空=不启用) |
| `ADMIN_API_KEY` | ❌ | — | 机器/脚本调用的 API Key |
## 同站部署 vs 跨站 CSRF
Profeto 管理鉴权使用 **HttpOnly Cookie 会话**(登录后服务端写入),`allow_credentials=True` 的 CORS 配置允许浏览器跨域携带 Cookie——这也引入了 CSRF 面。部署拓扑决定风险等级:
**同站部署(推荐)**: 前端与 API 同域(反代把 `/``/api` 都转发到同一后端,或同源端口)。
- 浏览器视为 **same-origin**,CORS 不触发;`SameSite=Lax` 会话 Cookie 天然阻断跨站请求携带。
- 风险最低。`CORS_ORIGINS` 可设为空或同域来源,仅作兜底。
**跨站部署**: 前端与 API 不同域(如前端 `app.example.com`、API `api.example.com`,或开发时 `localhost:3000``localhost:8000`)。
- 必须把 API 域名列入 `CORS_ORIGINS`,且 `allow_credentials=True` 才能携带 Cookie。
- 此时任何被允许域下的页面都能构造带 Cookie 的请求 → **CSRF 面**:
- 状态变更接口(采集/回测/改密等写操作)要求**管理员 Cookie + 同域**,攻击者无法从第三方站点读取 Cookie,但可构造跨域表单/请求——`SameSite=Lax` 会阻断跨站 POST 表单提交(顶级导航 GET 仍放行),这是当前主要防线。
- `GET /api/v1/admin/*` 只读接口受 `SameSite=Lax` 下顶级导航可能被利用,但攻击者无法读取响应(CORS 不匹配时浏览器拦截)。
- **加固建议**:
1. 反代层加 `Origin`/`Referer` 校验,仅放行 `CORS_ORIGINS` 列表中的来源(即便 FastAPI CORS 已通过,反代校验是多一层纵深)。
2. 写操作要求自定义请求头(如 `X-Requested-With: XMLHttpRequest`),第三方站点无法在无预检下添加自定义头,天然阻断简单跨站 POST。
3. 生产强制 HTTPS(`APP_ENV=production` 下 Cookie 自动 `Secure`),防中间人窃 Cookie。
## LLM 提供商配置示例
### OpenAI
+11 -7
View File
@@ -83,16 +83,16 @@ Profeto/
│ │ └── app.py # FastAPI 工厂
│ ├── db/
│ │ ├── base.py # SQLAlchemy async engine + session
│ │ ├── models.py # 6 张表 ORM
│ │ ├── models.py # 12 张表 ORM
│ │ ├── repositories.py # 仓储层(查询封装)
│ │ └── unit_of_work.py # 事务边界
│ ├── data/
│ │ ├── bzzoiro.py # bzzoiro 采集 + 入库
│ │ ├── understat.py # understat xG 回填
│ │ ├── injuries.py # 伤停采集
│ │ ├── bzzoiro.py # bzzoiro 采集 + 入库(唯一数据源)
│ │ ├── normalize.py # 数据清洗契约
│ │ ├── team_names.py # 队名归一化映射
│ │ ├── team_names_zh.py # 队名中文名映射
│ │ ├── sources.py # 数据源注册表
│ │ ├── key_ring.py # 数据源 Key 读取(DB 设置优先于 env)
│ │ └── config.py # 联赛映射常量
│ ├── llm/
│ │ ├── provider.py # LLM 提供商抽象(OpenAI-compatible)
@@ -110,13 +110,17 @@ Profeto/
│ │ ├── form_v1.md
│ │ ├── stats_v1.md
│ │ ├── home_away_v1.md
│ │ ├── injuries_v1.md
│ │ ├── standings_v1.md
│ │ ├── h2h_v1.md
│ │ └── aggregator_v1.md
│ └── core/
│ ├── config.py # pydantic-settings 配置
│ ├── http_client.py # 共享 httpx 客户端
── retry.py # 重试工具
── crypto.py # 对称加密(Fernet)与密码哈希
│ ├── log_buffer.py # 内存日志缓冲(admin「系统日志」页)
│ ├── runtime_config.py # 运行时配置(数据库优先,回落 .env)
│ ├── scheduler.py # 定时任务调度器(cron 触发采集)
│ └── security_check.py # 生产启动安全校验(缺配置拒绝启动)
├── frontend/ # React 单页前端
├── alembic/ # 数据库迁移
│ └── versions/
@@ -194,7 +198,7 @@ cp src/llm/prompts/agents/h2h_v1.md src/llm/prompts/agents/h2h_v2.md
### 3. 新增数据源
1.`src/data/` 写采集模块(参考 `understat.py`)
1.`src/data/` 写采集模块(参考 `bzzoiro.py`)
2.`normalize.py` 加清洗函数
3.`context_builder.py` 加切片函数
4.`api/routes/ingest.py` 加端点
+6
View File
@@ -1,3 +1,9 @@
> ⚠️ **历史文档(已过时)**:伤停(injuries)数据源与 api-football 集成已移除,
> 数据源收敛为 bzzoiro 唯一来源。本文仅作历史决策记录保留,
> 现状请见 [05-data.md](05-data.md) 与 [01-architecture.md](01-architecture.md)。
---
"""检查 injuries 数据源 api-football 的响应结构。"""
API_FOOTBALL_INJURY_RESPONSE_EXAMPLE = """
{
+5 -53
View File
@@ -9,6 +9,7 @@ import { useState, useEffect, useCallback } from 'react'
import { NavLink, Outlet, useLocation } from 'react-router-dom'
import { fetchAuthState, logout, UNAUTHORIZED_EVENT } from './api'
import { fetchHealth } from './dal'
import { COMMAND_PALETTE_PAGES, ROUTE_LABELS, NAV_SECTIONS } from './nav'
import Login from './Login'
import { useCommandPalette, CommandPalette } from './useCommandPalette'
@@ -88,58 +89,9 @@ function Icon({ name }: { name: string }) {
}
}
const NAV_PAGES: Array<{ to: string; label: string; group: string }> = [
{ to: '/admin', label: '仪表盘', group: '概览' },
{ to: '/admin/collection', label: '数据采集', group: '数据流水线' },
{ to: '/admin/data-completeness', label: '数据完整性', group: '数据流水线' },
{ to: '/admin/data-pipeline', label: '数据管线', group: '数据流水线' },
{ to: '/admin/predictions', label: '预测历史', group: '数据流水线' },
{ to: '/admin/backtest', label: '回测', group: '数据流水线' },
{ to: '/admin/monitoring', label: '监控', group: '评估与监控' },
{ to: '/admin/eval', label: '评估', group: '评估与监控' },
{ to: '/admin/settings', label: '设置', group: '系统' },
{ to: '/admin/logs', label: '日志', group: '系统' },
]
// 路由 → 面包屑标签
const ROUTE_LABELS: Record<string, string> = {
'/admin': '仪表盘',
'/admin/collection': '数据采集',
'/admin/data-completeness': '数据完整性',
'/admin/data-pipeline': '数据管线',
'/admin/predictions': '预测历史',
'/admin/backtest': '回测',
'/admin/monitoring': '监控',
'/admin/settings': '设置',
'/admin/logs': '日志',
'/admin/eval': '评估',
}
const NAV_SECTIONS: { title: string; items: Array<{ to: string; label: string; icon: string }> }[] = [
{
title: '数据流水线',
items: [
{ to: '/admin/collection', label: '数据采集', icon: 'collection' },
{ to: '/admin/data-completeness', label: '数据完整性', icon: 'chart' },
{ to: '/admin/predictions', label: '预测历史', icon: 'logs' },
{ to: '/admin/backtest', label: '回测', icon: 'repeat' },
],
},
{
title: '评估与监控',
items: [
{ to: '/admin/eval', label: '评估', icon: 'eval' },
{ to: '/admin/monitoring', label: '监控', icon: 'monitor' },
],
},
{
title: '系统设置',
items: [
{ to: '/admin/settings', label: '设置', icon: 'settings' },
{ to: '/admin/logs', label: '日志', icon: 'logs' },
],
},
]
// D6: 导航三视图(侧栏/命令面板/面包屑)统一由 admin/nav.ts 的 NAV_ITEMS
// 单源派生 —— 此处的 NAV_PAGES/ROUTE_LABELS/NAV_SECTIONS 平行清单已删除,
// 新增页面/改标签只改 nav.ts 一处。
/** 报眉日期行,与前台同款式 */
function dateLine(): string {
@@ -156,7 +108,7 @@ export default function AdminLayout() {
const [healthOk, setHealthOk] = useState<boolean | null>(null)
const [authed, setAuthed] = useState<boolean | null>(null)
const location = useLocation()
const palette = useCommandPalette(NAV_PAGES)
const palette = useCommandPalette(COMMAND_PALETTE_PAGES)
// 登录门禁:挂载时探测会话,收到 401 事件(会话过期)自动切回登录页
useEffect(() => {
+67
View File
@@ -0,0 +1,67 @@
/**
* Admin 导航单源(D6)。
*
* 背景: 侧栏(NAV_SECTIONS)、命令面板(NAV_PAGES)、面包屑(ROUTE_LABELS)
* 此前各维护一份平行清单,已出现漂移 —— data-pipeline 不在侧栏、
* 「系统」vs「系统设置」命名不一致、monitoring/eval 两处顺序相反。
*
* 现在只允许维护 NAV_ITEMS 一份,其余视图一律由此派生;
* 禁止再新建平行导航清单(修改入口/新增页面只改这里)。
*/
export interface NavItem {
to: string
label: string
/** 命令面板中的分组名(展示原样) */
group: string
/** 侧栏图标名(见 AdminLayout 的 Icon) */
icon: string
/** 仅命令面板/面包屑可达,不进侧栏(深链页) */
hideFromSidebar?: boolean
}
/** 唯一的导航配置源。顺序 = 侧栏渲染顺序(命令面板分组内顺序与之相同)。 */
export const NAV_ITEMS: NavItem[] = [
{ to: '/admin', label: '仪表盘', group: '概览', icon: 'chart' },
{ to: '/admin/collection', label: '数据采集', group: '数据流水线', icon: 'collection' },
{ to: '/admin/data-completeness', label: '数据完整性', group: '数据流水线', icon: 'chart' },
{ to: '/admin/data-pipeline', label: '数据管线', group: '数据流水线', icon: 'chart', hideFromSidebar: true },
{ to: '/admin/predictions', label: '预测历史', group: '数据流水线', icon: 'logs' },
{ to: '/admin/backtest', label: '回测', group: '数据流水线', icon: 'repeat' },
{ to: '/admin/eval', label: '评估', group: '评估与监控', icon: 'eval' },
{ to: '/admin/monitoring', label: '监控', group: '评估与监控', icon: 'monitor' },
{ to: '/admin/settings', label: '设置', group: '系统', icon: 'settings' },
{ to: '/admin/logs', label: '日志', group: '系统', icon: 'logs' },
]
/** 命令面板条目(原 NAV_PAGES 的唯一来源) */
export const COMMAND_PALETTE_PAGES = NAV_ITEMS.map(({ to, label, group }) => ({ to, label, group }))
/** 路由 → 面包屑标签(原 ROUTE_LABELS 的唯一来源) */
export const ROUTE_LABELS: Record<string, string> = Object.fromEntries(
NAV_ITEMS.map(i => [i.to, i.label]),
)
/** 侧栏分组标题的历史显示名(仅侧栏使用;与面板分组名不同时在此映射) */
const SIDEBAR_GROUP_TITLES: Record<string, string> = {
: '系统设置',
}
/**
* 侧栏分组(原 NAV_SECTIONS 的唯一来源)。
* 仪表盘(group=概览)在 AdminLayout 中独立渲染于顶部,不进分组循环。
*/
export const NAV_SECTIONS = (() => {
const sidebarItems = NAV_ITEMS.filter(i => !i.hideFromSidebar && i.group !== '概览')
const titles: string[] = []
for (const i of sidebarItems) {
const title = SIDEBAR_GROUP_TITLES[i.group] ?? i.group
if (!titles.includes(title)) titles.push(title)
}
return titles.map(title => ({
title,
items: sidebarItems
.filter(i => (SIDEBAR_GROUP_TITLES[i.group] ?? i.group) === title)
.map(({ to, label, icon }) => ({ to, label, icon })),
}))
})()
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,171 @@
/**
* AgentsPanel: 五路专家意见 —— 可折叠 + 状态摘要 + 权重条形图 + 单路详情。
*
* P3-1: 从 MatchPredictPanel.PredictionPanel 拆出,渲染逻辑原样搬迁。
*/
import { useState } from 'react'
import type { AgentReport, Prediction } from '../../types'
import { AGENT_LABELS, CN_NUM } from '../../types'
const STATUS_BADGE: Record<string, { label: string; cls: string }> = {
ok: { label: '正常', cls: 'text-ink-500' },
no_data: { label: '无数据', cls: 'text-ink-400' },
error: { label: '调用失败', cls: 'text-press' },
parse_error: { label: '解析失败', cls: 'text-press' },
}
const SUFFICIENCY_LABEL: Record<string, string> = {
high: '充分',
medium: '一般',
low: '偏少',
none: '无',
}
/** home_edge(-1~1,正=利主队)的可视化:以中线为原点的双向细条 */
function EdgeBar({ value }: { value: number }) {
const v = Math.max(-1, Math.min(1, value))
const half = Math.abs(v) * 50
return (
<div className="relative h-px w-full bg-ink-200" role="presentation">
<span className="absolute left-1/2 top-1/2 h-2 w-px -translate-x-1/2 -translate-y-1/2 bg-ink-400" />
<span
className={`absolute top-0 h-px transition-all duration-500 ${v >= 0 ? 'bg-press' : 'bg-ink-600'}`}
style={
v >= 0
? { left: '50%', width: `${half}%` }
: { right: '50%', width: `${half}%` }
}
/>
</div>
)
}
/** 单路专家意见:汉字编号 + 细线行 */
function AgentCard({ report: r, no }: { report: AgentReport; no: string }) {
const badge = STATUS_BADGE[r.status] ?? { label: r.status, cls: 'text-ink-400' }
const inactive = r.status !== 'ok'
return (
<details className="group border-b border-ink-200">
<summary className="flex cursor-pointer list-none items-baseline gap-2.5 px-1 py-3">
<span className="font-serif text-sm text-ink-400">{no}</span>
<span className="text-sm font-medium text-ink-900">{AGENT_LABELS[r.agent] ?? r.agent}</span>
<span className={`text-2xs ${badge.cls}`}>{badge.label}</span>
<span className="ml-auto flex items-baseline gap-3 text-2xs tabular-nums text-ink-500">
{r.status === 'ok' && r.subjective_confidence !== null && (
<span> {Math.round(r.subjective_confidence * 100)}%</span>
)}
{r.status === 'ok' && r.probable_score && (
<span className="font-serif font-bold text-ink-800">{r.probable_score}</span>
)}
<svg viewBox="0 0 20 20" className="h-3 w-3 self-center text-ink-300 transition-transform group-open:rotate-90" fill="currentColor" aria-hidden="true">
<path d="M7.3 5.3a1 1 0 011.4 0l4 4a1 1 0 010 1.4l-4 4a1 1 0 01-1.4-1.4L10.6 10 7.3 6.7a1 1 0 010-1.4z" />
</svg>
</span>
</summary>
<div className="space-y-3 px-1 pb-4 pl-7">
{inactive && (
<p className="text-xs leading-relaxed text-ink-500">
{r.status === 'no_data' && '该维度没有可用数据,已跳过 LLM 分析以节省额度(不影响其他专家)。'}
{r.status === 'error' && '该专家调用失败,本次结论未纳入其视角(fail-open 设计,不阻断整体预测)。'}
{r.status === 'parse_error' && '模型输出未通过格式校验,该报告已丢弃。'}
</p>
)}
{!inactive && r.home_edge !== null && (
<div>
<div className="mb-1.5 flex items-baseline justify-between text-2xs">
<span className="text-ink-500"></span>
<span className={`font-semibold tabular-nums ${r.home_edge > 0 ? 'text-press' : r.home_edge < 0 ? 'text-ink-700' : 'text-ink-500'}`}>
{r.home_edge > 0 ? '+' : ''}{r.home_edge.toFixed(2)}
</span>
</div>
<EdgeBar value={r.home_edge} />
<div className="mt-1 flex justify-between text-2xs text-ink-400">
<span></span>
<span></span>
</div>
</div>
)}
{r.analysis && (
<p className="font-serif text-sm leading-loose text-ink-700">{r.analysis}</p>
)}
{r.key_evidence.length > 0 && (
<ul className="space-y-1.5">
{r.key_evidence.map((e, i) => (
<li key={i} className="flex gap-2 text-xs leading-relaxed text-ink-600">
<span className="flex-shrink-0 text-ink-300" aria-hidden="true"></span>
<span>{e}</span>
</li>
))}
</ul>
)}
{r.exp_home_goals !== null && r.exp_away_goals !== null && (
<p className="text-xs text-ink-500">
<span className="font-serif font-bold tabular-nums text-ink-900">{r.exp_home_goals.toFixed(1)} - {r.exp_away_goals.toFixed(1)}</span>
</p>
)}
{!inactive && (
<p className="border-t border-ink-100 pt-2.5 text-2xs text-ink-400">
{SUFFICIENCY_LABEL[r.data_sufficiency] ?? r.data_sufficiency}
<span className="mx-2 text-ink-200">|</span>
<span className="font-mono">{r.model}</span>
{r.latency_ms !== null && <span className="ml-2 tabular-nums">{r.latency_ms}ms</span>}
</p>
)}
</div>
</details>
)
}
export function AgentsPanel({ prediction }: { prediction: Prediction }) {
const [expertsOpen, setExpertsOpen] = useState(false)
const degraded = prediction.status === 'degraded' || prediction.status === 'failed'
const reports = prediction.agent_outputs ?? []
const okReports = reports.filter(r => r.status === 'ok')
if (reports.length === 0) return null
return (
<section>
<button
onClick={() => setExpertsOpen(o => !o)}
className="flex w-full items-center justify-between border-b border-ink-200 pb-2 text-left"
>
<span className="section-head mb-0">({okReports.length}/{reports.length} )</span>
<span className="text-2xs text-ink-400">{expertsOpen ? '收起' : '展开'}</span>
</button>
{!degraded && prediction.agent_weights && Object.keys(prediction.agent_weights).length > 0 && (
<div className="mt-3 space-y-1.5">
<span className="text-2xs text-ink-500"></span>
{Object.entries(prediction.agent_weights)
.sort((a, b) => b[1] - a[1])
.map(([k, v]) => (
<div key={k} className="grid grid-cols-[96px_minmax(0,1fr)_40px] items-center gap-2">
<span className="truncate text-2xs text-ink-500">{AGENT_LABELS[k] ?? k}</span>
<div className="h-1.5 bg-paper-100">
<div className="h-full bg-press" style={{ width: `${Math.round(v * 100)}%` }} />
</div>
<span className="text-right text-2xs tabular-nums text-ink-500">{Math.round(v * 100)}%</span>
</div>
))}
</div>
)}
{expertsOpen && (
<div className="mt-2">
{reports.map((r, i) => (
<AgentCard key={r.agent} report={r} no={CN_NUM[i] ?? String(i + 1)} />
))}
</div>
)}
</section>
)
}
@@ -0,0 +1,369 @@
/**
* MatchDetailSection: 赛程行(MatchRow)+ 展开详情面板(统计/近况/H2H/历史预测)。
*
* D3: 从 Matches.tsx 拆出,渲染逻辑原样搬迁。MatchRow 原先是主组件内
* group.map 的内联 JSX,现接收回调(onToggle/onPredict)保持行为一致;
* 展开懒加载的 state 仍由页面持有。
*/
import TeamSideTag from '../../../components/TeamSideTag'
import { fetchMatchDetail, fetchMatchContext } from '../../../admin/dal'
import type { MatchDetailOut, MatchContextOut, MatchRecentPrediction, TeamRecentMatch } from '../../../admin/types'
import type { MatchStatsDetail } from '../../../admin/types'
import { STATUS_META } from '../types'
import type { Match } from '../types'
import { Spinner } from '../ui'
/** 比赛详情面板:双方近况/H2H + 历史预测列表(只读) */
function MatchDetailPanel({
match, detail, ctx, loading,
}: {
match: Match
detail: MatchDetailOut | undefined
ctx: MatchContextOut | undefined
loading: boolean
}) {
const homeName = match.home_team_zh || match.home_team
const awayName = match.away_team_zh || match.away_team
const finished = match.match_status === 'finished'
return (
<div className="border-b border-ink-200 bg-paper-100/50 px-3 py-4">
{loading && (
<div className="flex items-center gap-2 text-xs text-ink-500"><Spinner /> </div>
)}
{!loading && !detail && !ctx && (
<p className="py-4 text-center text-xs text-ink-400"></p>
)}
{!loading && (detail || ctx) && (
<div className="space-y-5">
{/* 比分区(终场/当前比分 + 状态 + 预测按钮) */}
<div className="flex flex-wrap items-center justify-between gap-3">
<div className="text-center">
<p className="font-serif text-3xl font-bold tabular-nums leading-none text-ink-900">
{match.home_goals ?? '-'}{' '}<span className="text-ink-300">:</span>{' '}{match.away_goals ?? '-'}
</p>
<p className="mt-1 text-2xs text-ink-500">
{match.match_stage || ''} {match.match_status === 'finished' ? '· 已完赛' : match.match_status === 'scheduled' ? '· 未开赛' : `· ${match.match_status}`}
</p>
{match.home_xg != null && match.away_xg != null && (
<p className="text-2xs tabular-nums text-ink-400">xG {match.home_xg.toFixed(1)}{match.away_xg.toFixed(1)}</p>
)}
</div>
{!finished && (
<span className="text-2xs text-ink-500">
</span>
)}
</div>
{/* 比赛详细统计(bzzoiro /events/{id}/stats/) */}
{detail?.stats && (
<MatchStatsPanel stats={detail.stats} homeName={homeName} awayName={awayName} />
)}
{/* 双方近况 + H2H */}
{(ctx?.home_recent?.length || ctx?.away_recent?.length || ctx?.h2h?.length) ? (
<div className="grid gap-4 sm:grid-cols-3">
<RecentBlock title={`${homeName} 近况`} rows={ctx?.home_recent} side="home" />
<RecentBlock title={`${awayName} 近况`} rows={ctx?.away_recent} side="away" />
<RecentBlock title="历史交锋(H2H)" rows={ctx?.h2h} side="h2h" />
</div>
) : (
!loading && <p className="text-2xs text-ink-400"></p>
)}
{/* 历史预测列表 */}
<div>
<h4 className="section-head mb-2">({detail?.recent_predictions?.length ?? 0})</h4>
{detail?.recent_predictions?.length ? (
<div className="space-y-2">
{detail.recent_predictions.map(p => (
<PredictionHistoryRow key={p.id} p={p} />
))}
</div>
) : (
<p className="py-3 text-center text-2xs text-ink-400"></p>
)}
</div>
</div>
)}
</div>
)
}
/** 赛程表一行:可点击展开详情;展开时懒加载详情(只读,不触发 LLM) */
export function MatchRow({
m,
busy,
expanded,
detail,
ctx,
detailLoading,
onToggle,
onPredict,
}: {
m: Match
busy: boolean
expanded: boolean
detail: MatchDetailOut | undefined
ctx: MatchContextOut | undefined
detailLoading: boolean
onToggle: () => void
onPredict: (m: Match) => void
}) {
const st = STATUS_META[m.match_status] ?? { label: m.match_status, cls: 'text-ink-400' }
const homeName = m.home_team_zh || m.home_team
const awayName = m.away_team_zh || m.away_team
const finished = m.match_status === 'finished'
return (
<div>
{/* 行:可点击展开 */}
<div
className={`border-b border-ink-200 px-4 py-5 transition-colors hover:bg-paper-100/70 cursor-pointer sm:px-1 sm:py-4 ${
expanded ? 'bg-paper-100/60' : ''
}`}
onClick={onToggle}
role="button"
tabIndex={0}
onKeyDown={e => { if (e.key === 'Enter') onToggle() }}
aria-expanded={expanded}
>
{/* 桌面 grid: 日期 | 主队 | 比分 | 客队 | 状态 | 按钮 */}
<div className="flex flex-col gap-3 sm:grid sm:grid-cols-[96px_minmax(0,1fr)_72px_minmax(0,1fr)_64px_88px] sm:items-center sm:gap-x-4 sm:gap-y-0">
{/* 日期 + 状态:小屏同行;桌面 date 单独一列 */}
<div className="flex items-center justify-between text-xs sm:contents">
<span className="tabular-nums text-ink-500 sm:text-xs">{fmtTime(m.match_date)}</span>
<span className={`sm:hidden ${st.cls}`}>{st.label}</span>
</div>
{/* 主队 + 比分 + 客队:移动端 grid 三列(严格居中),桌面端 grid 分列 */}
<div className="grid grid-cols-[1fr_auto_1fr] items-center gap-4 sm:contents">
{/* 主队(右对齐) */}
<span className="flex min-w-0 items-center justify-end gap-2">
<TeamSideTag side="home" />
<span className="truncate text-sm font-medium text-ink-900">{homeName}</span>
</span>
{/* 比分 / VS(严格居中) */}
<span className="flex flex-col items-center justify-center">
{m.home_goals !== null && m.away_goals !== null ? (
<span className="font-serif text-xl font-bold tabular-nums leading-none text-ink-900 sm:text-xl">
{m.home_goals}<span className="mx-1 font-normal text-ink-300">:</span>{m.away_goals}
</span>
) : (
<span className="text-sm tracking-[0.2em] text-ink-500">VS</span>
)}
{m.home_xg !== null && m.away_xg !== null && (
<span className="mt-0.5 text-2xs tabular-nums text-ink-400">
xG {m.home_xg.toFixed(1)}{m.away_xg.toFixed(1)}
</span>
)}
</span>
{/* 客队(左对齐) */}
<span className="flex min-w-0 items-center gap-2">
<TeamSideTag side="away" />
<span className="truncate text-sm font-medium text-ink-900">{awayName}</span>
</span>
</div>
{/* 状态标签:小屏隐藏(已有);桌面用徽标样式 */}
<span className="hidden text-right sm:block">
<span className={`inline-block border px-1.5 py-0.5 text-2xs leading-tight ${st.cls} ${
m.match_status === 'finished'
? 'border-ink-200 text-ink-500'
: m.match_status === 'scheduled'
? 'border-ink-300 text-ink-600'
: 'border-press/30 text-press'
}`}>
{st.label}
</span>
</span>
{/* 预测按钮(统一,响应式尺寸) */}
{!finished && (
<div className="flex justify-end" onClick={e => e.stopPropagation()}>
<button
onClick={() => onPredict(m)}
disabled={busy}
className={`btn ${busy ? '' : 'btn-solid'} w-full min-h-[44px] sm:w-[84px] sm:min-h-0 sm:btn-sm`}
title="以多专家模式预测这场"
>
{busy ? (<><Spinner /> </>) : '预测'}
</button>
</div>
)}
</div>
</div>{/* 关闭可点击行 */}
{/* 展开详情面板 */}
{expanded && (
<MatchDetailPanel
match={m} detail={detail} ctx={ctx}
loading={detailLoading}
/>
)}
</div>
)
}
/** 行内时间展示:只显示 HH:mm(日期由分组头承担) */
function fmtTime(s: string): string {
const d = new Date(s)
return d.toLocaleTimeString('zh-CN', { hour: '2-digit', minute: '2-digit', hour12: false })
}
/** 比赛详细统计面板(bzzoiro /events/{id}/stats/) */
function MatchStatsPanel({
stats, homeName, awayName,
}: { stats: MatchStatsDetail; homeName: string; awayName: string }) {
const rows: Array<{ label: string; home: number | null; away: number | null; highlight?: 'high' | 'low' }> = [
{ label: '预期进球(xG)', home: stats.home_xg, away: stats.away_xg },
{ label: '射门', home: stats.home_shots, away: stats.away_shots },
{ label: '射正', home: stats.home_shots_on_target, away: stats.away_shots_on_target },
{ label: '角球', home: stats.home_corners, away: stats.away_corners },
{ label: '犯规', home: stats.home_fouls, away: stats.away_fouls },
{ label: '绝佳机会', home: stats.home_big_chances, away: stats.away_big_chances },
{ label: '黄牌', home: stats.home_yellow_cards, away: stats.away_yellow_cards },
{ label: '红牌', home: stats.home_red_cards, away: stats.away_red_cards },
]
const hasAny = rows.some(r => r.home != null || r.away != null)
if (!hasAny) return null
// 控球率用横条展示
const possHome = stats.home_possession
const possAway = possHome != null ? Math.max(0, 100 - possHome) : null
return (
<div>
<h4 className="section-head mb-2"></h4>
{/* 控球率横条 */}
{possHome != null && possAway != null && (
<div className="mb-3">
<div className="mb-1 flex justify-between text-2xs text-ink-500">
<span>{possHome.toFixed(0)}%</span>
<span className="text-ink-400"></span>
<span>{possAway.toFixed(0)}%</span>
</div>
<div className="flex h-1.5 overflow-hidden rounded-full bg-ink-200">
<div className="bg-ink-700 transition-[width] duration-500" style={{ width: `${possHome}%` }} />
<div className="bg-ink-300 transition-[width] duration-500" style={{ width: `${possAway}%` }} />
</div>
</div>
)}
{/* 主客对比表 */}
<div className="overflow-x-auto">
<table className="w-full text-xs">
<thead>
<tr className="border-b border-ink-200 text-ink-400">
<th className="py-1.5 text-left font-medium">{homeName}</th>
<th className="py-1.5 text-center font-medium text-ink-500"></th>
<th className="py-1.5 text-right font-medium">{awayName}</th>
</tr>
</thead>
<tbody>
{rows.filter(r => r.home != null || r.away != null).map(r => {
const h = r.home ?? 0
const a = r.away ?? 0
const winner = h > a ? 'home' : h < a ? 'away' : 'tie'
return (
<tr key={r.label} className="border-b border-ink-100">
<td className={`py-1.5 text-right tabular-nums ${winner === 'home' ? 'font-bold text-ink-900' : 'text-ink-500'}`}>
{r.home ?? '—'}
</td>
<td className="py-1.5 text-center text-ink-500">{r.label}</td>
<td className={`py-1.5 text-left tabular-nums ${winner === 'away' ? 'font-bold text-ink-900' : 'text-ink-500'}`}>
{r.away ?? '—'}
</td>
</tr>
)
})}
</tbody>
</table>
</div>
</div>
)
}
/** 近况/H2H 单区块 */
function RecentBlock({ title, rows, side }: { title: string; rows?: TeamRecentMatch[]; side: 'home' | 'away' | 'h2h' }) {
return (
<div>
<h5 className="mb-1.5 text-2xs font-medium text-ink-500">{title}</h5>
{rows && rows.length > 0 ? (
<ul className="space-y-1">
{rows.map((r, i) => {
const date = r.match_date ? new Date(r.match_date).toLocaleDateString('zh-CN', { month: '2-digit', day: '2-digit' }) : '—'
const score = (r.home_goals != null && r.away_goals != null) ? `${r.home_goals}-${r.away_goals}` : 'vs'
const label = side === 'h2h'
? `${r.home_team ?? '?'} ${score} ${r.away_team ?? '?'}`
: `${score}`
return (
<li key={i} className="flex items-center justify-between text-2xs tabular-nums text-ink-600">
<span className="text-ink-400">{date}</span>
<span className="truncate">{label}</span>
</li>
)
})}
</ul>
) : (
<p className="text-2xs text-ink-300"></p>
)}
</div>
)
}
/** 历史预测单行(含专家报告入口) */
function PredictionHistoryRow({ p }: { p: MatchRecentPrediction }) {
const badge = p.status === 'degraded'
? { label: 'degraded', cls: 'text-press' }
: p.settled
? { label: p.correct_1x2 === undefined ? '已结算' : p.correct_1x2 ? '命中' : '未中', cls: p.correct_1x2 ? 'text-ink-900' : 'text-ink-400' }
: { label: p.status === 'success' ? '成功' : p.status, cls: 'text-ink-600' }
const score = (p.pred_home_goals != null && p.pred_away_goals != null)
? `${p.pred_home_goals.toFixed(1)}-${p.pred_away_goals.toFixed(1)}`
: '—'
const alt = (p.alt_pred_home_goals != null && p.alt_pred_away_goals != null)
? `${p.alt_pred_home_goals.toFixed(1)}-${p.alt_pred_away_goals.toFixed(1)}` : null
const hasAgents = p.agent_outputs && p.agent_outputs.length > 0
return (
<div className="border-b border-ink-200 pb-2 last:border-b-0">
<div className="flex items-center justify-between text-xs">
<span className="tabular-nums text-ink-600">
{score} {p.pred_1x2 ? `(${p.pred_1x2})` : ''}
{alt && <span className="ml-1 text-ink-400"> {alt}</span>}
</span>
<span className="flex items-center gap-2">
{p.subjective_confidence != null && (
<span className="text-2xs tabular-nums text-ink-400"> {Math.round(p.subjective_confidence * 100)}%</span>
)}
<span className={`text-2xs ${badge.cls}`}>{badge.label}</span>
</span>
</div>
<div className="mt-0.5 flex items-center justify-between text-2xs text-ink-400">
<span className="truncate">{p.model} · {p.mode} · {p.created_at ? new Date(p.created_at).toLocaleString('zh-CN', { month: '2-digit', day: '2-digit', hour: '2-digit', minute: '2-digit', hour12: false }) : '—'}</span>
{hasAgents && <span className="text-press">{p.agent_outputs!.length} </span>}
</div>
{p.reasoning && (
<p className="mt-1 line-clamp-2 font-serif text-2xs leading-relaxed text-ink-500">{p.reasoning}</p>
)}
</div>
)
}
// 详情懒加载取数在此文件内聚:页面只需切换 expandedId 并缓存结果
export async function loadMatchDetailBundle(
matchId: number,
): Promise<{ detail: MatchDetailOut | null; ctx: MatchContextOut | null }> {
const [d, c] = await Promise.all([
fetchMatchDetail(matchId).catch(() => null),
fetchMatchContext(matchId).catch(() => null),
])
return { detail: d, ctx: c }
}
@@ -0,0 +1,247 @@
/**
* MatchPredictPanel: 预测弹窗全套(过程可视化 / 结果版面 / 专家意见)。
*
* D3: 从 Matches.tsx 拆出,渲染逻辑原样搬迁。对外只导出 PredictModal;
* PredictionPanel 复用 Prediction 的 embedded 模式由弹窗内渲染。
*/
import { useEffect, useState } from 'react'
import TeamSideTag from '../../../components/TeamSideTag'
import type { Match, Prediction } from '../types'
import { AGENT_LABELS } from '../types'
import { AgentsPanel } from './AgentsPanel'
import { OutcomePanel } from './OutcomePanel'
import { ReasoningPanel } from './ReasoningPanel'
function Spinner({ className = '' }: { className?: string }) {
return (
<svg
viewBox="0 0 20 20"
className={`h-3.5 w-3.5 animate-spin ${className}`}
fill="none"
aria-hidden="true"
>
<circle cx="10" cy="10" r="7.5" stroke="currentColor" strokeWidth="1.5" strokeOpacity="0.25" />
<path d="M17.5 10A7.5 7.5 0 0010 2.5" stroke="currentColor" strokeWidth="1.5" strokeLinecap="round" />
</svg>
)
}
/** 胜平负一行文字:选中的红字加方块标记,未选中的退灰 */
.**
* P3-1:PredictionPanel ,:
* OutcomePanel(//) / AgentsPanel() / ReasoningPanel(/)
* ( + + )
*/
function PredictionPanel({
prediction,
match,
embedded = false,
}: {
prediction: Prediction
match: Match
/** 弹窗嵌入模式:弹窗已提供报头,这里省略自带版头 */
embedded?: boolean
}) {
const homeName = match.home_team_zh || match.home_team
const awayName = match.away_team_zh || match.away_team
const degraded = prediction.status === 'degraded' || prediction.status === 'failed'
const reports = prediction.agent_outputs ?? []
const okReports = reports.filter(r => r.status === 'ok')
return (
<article className={embedded ? 'bg-paper-50' : 'border border-ink-900 bg-paper-50'}>
{!embedded && (
<div className="flex flex-wrap items-baseline justify-between gap-2 border-b border-ink-900 bg-paper-100 px-4 py-2.5 sm:px-5">
<h3 className="flex flex-wrap items-center gap-1.5 font-serif text-sm font-bold text-ink-900">
·
<TeamSideTag side="home" />
{homeName}
<span></span>
<TeamSideTag side="away" />
{awayName}
</h3>
<span className="text-2xs tabular-nums text-ink-500">
{prediction.provider} / {prediction.model}
{prediction.latency_ms !== null && ` · ${(prediction.latency_ms / 1000).toFixed(1)}s`}
</span>
</div>
)}
<div className="space-y-7 px-4 py-6 sm:px-5">
{degraded && (
<div className="border-l-2 border-press bg-press-wash/40 px-4 py-3">
<p className="font-serif text-sm font-bold text-press-dark">
{prediction.status === 'failed' ? '预测失败' : '预测降级(degraded)'}
</p>
<p className="mt-1.5 whitespace-pre-wrap text-xs leading-relaxed text-ink-600">
{prediction.reasoning || '所有专家均无有效数据或调用失败,无法生成可靠比分。'}
</p>
</div>
)}
{!degraded && <OutcomePanel prediction={prediction} match={match} />}
<p className="text-center text-2xs text-ink-500">
`多专家模式 · ${okReports.length}/${reports.length} 路有效`
{prediction.prompt_version && ` · prompt ${prediction.prompt_version}`}
</p>
<AgentsPanel prediction={prediction} />
<ReasoningPanel prediction={prediction} />
</div>
</article>
)
}
function PredictProgress() {
const [elapsed, setElapsed] = useState(0)
useEffect(() => {
const t = setInterval(() => setElapsed(e => e + 0.5), 500)
return () => clearInterval(t)
}, [])
// 阶段阈值(秒): 切片 → 专家(各路依次点亮) → 终裁
const SLICE_END = 3
const AGENT_START = 4
const AGENT_STEP = 8 // 每路专家约 8s 点亮一路
const AGG_START = AGENT_START + AGENT_STEP * 5
const agents = ['form', 'stats', 'home_away', 'standings', 'h2h']
const phase = elapsed < SLICE_END ? 'slice'
: elapsed < AGG_START ? 'agents' : 'agg'
const pct = Math.min(95, Math.round((elapsed / 70) * 100))
return (
<div className="px-5 py-8 sm:px-8">
{/* 阶段标题 */}
<div className="flex items-center justify-center gap-2">
<Spinner className="text-press" />
<span className="font-serif text-sm font-bold text-ink-900">
{phase === 'slice' && '正在组装比赛数据切片'}
{phase === 'agents' && '五路专家并行分析中'}
{phase === 'agg' && '终裁专家汇总裁定中'}
</span>
<span className="text-2xs tabular-nums text-ink-400">{elapsed.toFixed(0)}s</span>
</div>
{/* 进度条:渐进式,不封顶到 100% */}
<div className="mx-auto mt-5 h-1 w-full max-w-md overflow-hidden bg-ink-100" role="progressbar" aria-valuenow={pct}>
<div
className={`h-full bg-press transition-all duration-500 ${phase === 'agg' ? 'animate-pulse' : ''}`}
style={{ width: `${pct}%` }}
/>
</div>
{/* 专家灯序(多专家模式) */}
<ul className="mx-auto mt-6 max-w-md space-y-1.5">
{agents.map((a, i) => {
const lit = elapsed >= AGENT_START + AGENT_STEP * (i + 1)
const activeNow = !lit && elapsed >= AGENT_START + AGENT_STEP * i
return (
<li
key={a}
className={`flex items-center justify-between border-b border-ink-200 pb-1.5 text-xs transition-colors ${
lit ? 'text-ink-800' : activeNow ? 'text-ink-900' : 'text-ink-300'
}`}
>
<span className="flex items-center gap-2">
<span
aria-hidden="true"
className={`inline-block h-1.5 w-1.5 ${lit ? 'bg-ink-900' : activeNow ? 'bg-press animate-pulse' : 'bg-ink-200'}`}
/>
{AGENT_LABELS[a] ?? a}
</span>
{lit && <span className="text-2xs text-ink-400"> </span>}
{activeNow && <span className="text-2xs text-press"></span>}
</li>
)
})}
</ul>
<p className="mt-6 text-center text-2xs text-ink-400">
, 30-90 ; token,使
</p>
<p className="mt-1 text-center text-2xs text-ink-300">
提示:每分钟限 10 ,
</p>
</div>
)
}
export function PredictModal({
match,
predicting,
prediction,
error,
onClose,
}: {
match: Match
predicting: boolean
prediction: Prediction | null
error: string | null
onClose: () => void
}) {
const homeName = match.home_team_zh || match.home_team
const awayName = match.away_team_zh || match.away_team
useEffect(() => {
const h = (e: KeyboardEvent) => {
if (e.key === 'Escape') onClose()
}
document.addEventListener('keydown', h)
return () => document.removeEventListener('keydown', h)
}, [onClose])
return (
<div
className="fixed inset-0 z-50 flex items-start justify-center overflow-y-auto bg-ink-900/50 p-4 sm:items-center"
role="dialog"
aria-modal="true"
aria-label={`预测 ${homeName}${awayName}`}
onClick={e => {
if (e.target === e.currentTarget) onClose()
}}
>
<div className="relative flex max-h-[92vh] w-full max-w-2xl flex-col overflow-hidden bg-paper-50 shadow-2xl">
{/* 弹窗报头 */}
<div className="flex flex-shrink-0 items-center justify-between border-b border-ink-900 bg-paper-100 px-4 py-2.5 sm:px-5">
<h3 className="flex flex-wrap items-center gap-1.5 font-serif text-sm font-bold text-ink-900">
·
<TeamSideTag side="home" />
{homeName}
<span></span>
<TeamSideTag side="away" />
{awayName}
</h3>
<button
onClick={onClose}
className="flex h-11 w-11 flex-shrink-0 items-center justify-center text-ink-400 transition-colors hover:text-ink-900"
aria-label="关闭"
>
<svg className="h-4 w-4" viewBox="0 0 20 20" fill="currentColor" aria-hidden="true">
<path d="M6.3 5.3a1 1 0 011.4 0L10 7.6l2.3-2.3a1 1 0 111.4 1.4L11.4 9l2.3 2.3a1 1 0 01-1.4 1.4L10 10.4l-2.3 2.3a1 1 0 01-1.4-1.4L8.6 9 6.3 6.7a1 1 0 010-1.4z" />
</svg>
</button>
</div>
{/* 弹窗体(小屏可滚动) */}
<div className="flex-1 overflow-y-auto">
{predicting ? (
<PredictProgress />
) : error ? (
<div className="px-5 py-10 text-center sm:px-8">
<p className="font-serif text-sm font-bold text-press"></p>
<p className="mx-auto mt-3 max-w-md whitespace-pre-wrap text-left text-xs leading-relaxed text-ink-600">
{error}
</p>
<button onClick={onClose} className="btn btn-sm mt-6"></button>
</div>
) : prediction ? (
<PredictionPanel prediction={prediction} match={match} embedded />
) : null}
</div>
</div>
</div>
)
}
@@ -0,0 +1,126 @@
/**
* OutcomePanel: 预测主结论 —— 比分 / 胜平负 / 置信度 / 成本。
*
* P3-1: 从 MatchPredictPanel.PredictionPanel 拆出,渲染逻辑原样搬迁。
*/
import TeamSideTag from '../../../../components/TeamSideTag'
import type { Match, Prediction } from '../../types'
import { OUTCOME_LABEL } from '../../types'
/** 置信度细线:0~1 数值的低调可视化 */
function Meter({ value }: { value: number }) {
const pct = Math.max(0, Math.min(100, Math.round(value * 100)))
return (
<div className="h-px w-full bg-ink-200" role="presentation">
<div className="h-px bg-press transition-[width] duration-500" style={{ width: `${pct}%` }} />
</div>
)
}
/** 胜平负一行文字:选中的红字加方块标记,未选中的退灰 */
function OutcomeLine({
pick,
confidence,
}: {
pick: string | null
confidence: number | null
}) {
const options = ['1', 'X', '2'] as const
return (
<div>
<div className="flex items-baseline justify-center gap-6 sm:gap-10">
{options.map(o => {
const on = pick === o
return (
<div key={o} className="flex flex-col items-center gap-1">
<span className={`flex items-center gap-1.5 text-sm ${on ? 'font-semibold text-press' : 'text-ink-400'}`}>
{on && <span className="inline-block h-2 w-2 bg-press" aria-hidden="true" />}
{OUTCOME_LABEL[o]}
</span>
{on && confidence !== null && (
<span className="text-2xs tabular-nums text-ink-500">
{Math.round(confidence * 100)}%
</span>
)}
</div>
)
})}
</div>
{pick && confidence !== null && (
<div className="mx-auto mt-3 max-w-xs">
<Meter value={confidence} />
<p className="mt-1 text-center text-2xs text-ink-400">,</p>
</div>
)}
</div>
)
}
/** 预测成本展示:耗时 + token + 限流余量 */
function PredictionCost({ prediction }: { prediction: Prediction }) {
const latency = prediction.latency_ms != null ? `${(prediction.latency_ms / 1000).toFixed(1)}s` : null
const tokens = prediction.prompt_tokens != null || prediction.completion_tokens != null
? `${prediction.prompt_tokens ?? '?'}/${prediction.completion_tokens ?? '?'}`
: null
if (!latency && !tokens && prediction.rate_limit_remaining == null) return null
return (
<div className="border-t border-ink-200 pt-3 text-2xs text-ink-500">
<div className="flex flex-wrap items-center justify-center gap-x-4 gap-y-1">
{latency && (
<span className="inline-flex items-center gap-1">
<span aria-hidden="true" className="opacity-60"></span> {latency}
</span>
)}
{tokens && (
<span className="inline-flex items-center gap-1">
<span aria-hidden="true" className="opacity-60">Tok</span>prompt/completion: {tokens}
</span>
)}
{prediction.rate_limit_remaining != null && prediction.rate_limit_remaining <= 3 && (
<span className="text-press" title="每分钟最多 10 次预测">
: {prediction.rate_limit_remaining}/10()
</span>
)}
</div>
</div>
)
}
export function OutcomePanel({ prediction, match }: { prediction: Prediction; match: Match }) {
const homeName = match.home_team_zh || match.home_team
const awayName = match.away_team_zh || match.away_team
const degraded = prediction.status === 'degraded' || prediction.status === 'failed'
return (
<>
{!degraded && (
<div className="text-center">
<p className="font-serif text-5xl font-bold tabular-nums leading-none text-ink-900 sm:text-6xl">
{prediction.pred_home_goals ?? '-'}
<span className="mx-3 font-normal text-ink-300">:</span>
{prediction.pred_away_goals ?? '-'}
</p>
<p className="mt-3 text-2xs tracking-[0.5em] text-ink-400"></p>
{prediction.alt_pred_home_goals != null && prediction.alt_pred_away_goals != null && (
<p className="mt-2 text-2xs tabular-nums text-ink-400">
{' '}
<span className="font-serif text-sm font-bold tabular-nums text-ink-600">
{prediction.alt_pred_home_goals}<span className="mx-0.5 font-normal text-ink-300">:</span>{prediction.alt_pred_away_goals}
</span>
</p>
)}
</div>
)}
{!degraded && (
<div className="border-y border-ink-200 py-4">
<OutcomeLine pick={prediction.pred_1x2} confidence={prediction.subjective_confidence} />
</div>
)}
{!degraded && <PredictionCost prediction={prediction} />}
</>
)
}
@@ -0,0 +1,34 @@
/**
* ReasoningPanel: 终裁意见 / 降级原因 —— 预测的文本解释。
*
* P3-1: 从 MatchPredictPanel.PredictionPanel 拆出,渲染逻辑原样搬迁。
*/
import type { Prediction } from '../../types'
export function ReasoningPanel({ prediction }: { prediction: Prediction }) {
const degraded = prediction.status === 'degraded' || prediction.status === 'failed'
if (!prediction.reasoning) return null
// 降级态:reasoning 展示为「降级原因」
if (degraded) {
return (
<section>
<h4 className="section-head mb-2"></h4>
<blockquote className="border-l-2 border-press pl-4">
<p className="whitespace-pre-wrap font-serif text-sm leading-loose text-ink-700">{prediction.reasoning}</p>
</blockquote>
</section>
)
}
// 成功态:reasoning 展示为「终裁意见」
return (
<section>
<h4 className="section-head mb-3"></h4>
<blockquote className="border-l-2 border-press pl-4">
<p className="whitespace-pre-wrap font-serif text-sm leading-loose text-ink-700">{prediction.reasoning}</p>
</blockquote>
</section>
)
}
@@ -0,0 +1,38 @@
/**
* 公开站联赛列表(P1-3):优先请求 GET /api/v1/leagues,
* 失败或返回空数组则回退本地五大联赛常量(LEAGUES)。
*
* 显示名规则:常量里已有的 code 沿用中文标签(保持现有 UI 语言不变),
* 新增联赛用 API 返回的 name;排序按常量顺序优先、新联赛按 API 返回序追加。
* API 仅返回 {id, code, name, country},无敏感配置字段。
*/
import { useEffect, useState } from 'react'
import { fetchLeagues } from '../../../admin/dal'
import { LEAGUES } from '../types'
export function useLeagues(): { code: string; name: string }[] {
const [leagues, setLeagues] = useState(LEAGUES)
useEffect(() => {
let alive = true
;(async () => {
// dal.fetchLeagues 已兜底:网络/权限异常时返回 []
const rows = await fetchLeagues()
if (!alive || rows.length === 0) return
const zhName = new Map(LEAGUES.map(l => [l.code, l.name] as const))
const rank = new Map(LEAGUES.map((l, i) => [l.code, i] as const))
const merged = rows
.slice()
.sort(
(a, b) => (rank.get(a.code) ?? LEAGUES.length) - (rank.get(b.code) ?? LEAGUES.length),
)
.map(l => ({ code: l.code, name: zhName.get(l.code) ?? l.name }))
setLeagues(merged)
})()
return () => {
alive = false
}
}, [])
return leagues
}
@@ -0,0 +1,93 @@
/**
* useMatchPredict: 预测流程状态机(发起/进行中/结果/失败/关闭中止)。
*
* D3: 从 Matches.tsx 拆出。语义不变:
* - 连点防护:同一场比赛预测中再次点击直接忽略
* - 竞态防护:递增序号,过期响应丢弃
* - 关闭弹窗 = 中止在途请求 + 序号失效(catch/then 不再写入)
* - 5 分钟超时,与 nginx 代理 300s 对齐
*/
import { useRef, useState } from 'react'
import { http } from '../../../lib/http'
import type { Match, Prediction } from '../types'
/** 把后端/网络错误翻译成用户可读文案 */
function readablePredictError(e: unknown): string {
if (e instanceof Error) {
const m = e.message
if (/429/.test(m)) {
// 429 来自后端限流(每分钟 10 次),非上游 LLM
return '操作过于频繁:每分钟最多 10 次预测。为保护 LLM 额度,请稍后再试。'
}
if (/502/.test(m)) return 'LLM 服务暂时不可用(502),请稍后重试'
if (/402|Payment Required|额度|余额/.test(m)) return 'LLM 额度不足(402),请检查 API Key 余额'
if (/400|已完赛/.test(m)) return '该比赛已完赛,不再支持预测'
if (/409|已结算/.test(m)) return '该预测已结算,不能重新预测'
if (/timeout|超时|timed out/i.test(m)) return '请求超时,请稍后重试'
return m
}
return String(e)
}
interface UseMatchPredictOptions {
/** 共享 error state(拆分前列表与预测共用同一个 error,行为保持一致) */
onError: (msg: string | null) => void
}
export function useMatchPredict({ onError }: UseMatchPredictOptions) {
const [predictingId, setPredictingId] = useState<number | null>(null)
const [prediction, setPrediction] = useState<Prediction | null>(null)
const [predictionFor, setPredictionFor] = useState<Match | null>(null)
const predictSeq = useRef(0)
// 预测请求控制器:关闭弹窗时中止
const predictAbort = useRef<AbortController | null>(null)
function closePredict() {
predictAbort.current?.abort()
predictSeq.current++ // 令中止请求的 catch/then 全部失效,不再写入错误
setPredictingId(null)
setPrediction(null)
setPredictionFor(null)
onError(null)
}
const predict = async (m: Match) => {
// 防连点:若该场比赛已在预测中,直接忽略
if (predictingId === m.id) return
const seq = ++predictSeq.current
setPredictingId(m.id)
onError(null)
setPrediction(null)
setPredictionFor(m)
// LLM 多专家预测耗时可达数分钟,给足超时(与 nginx 代理 300s 对齐)
const controller = new AbortController()
predictAbort.current = controller
const timer = setTimeout(() => controller.abort(), 300_000)
try {
const data = await http.post<Prediction>('/predict', { match_id: m.id, mode: 'multi' }, {
timeoutMs: 300_000,
signal: controller.signal,
})
if (seq !== predictSeq.current) return
setPrediction(data)
} catch (e) {
if (seq !== predictSeq.current) return
onError(
e instanceof DOMException && e.name === 'AbortError'
? '预测超时(5 分钟),请稍后重试'
: readablePredictError(e),
)
} finally {
clearTimeout(timer)
if (seq === predictSeq.current) setPredictingId(null)
}
}
return {
predictingId,
prediction,
predictionFor,
predict,
closePredict,
}
}
@@ -0,0 +1,94 @@
/**
* useMatchesList: 赛程列表数据获取(筛选/游标分页/进行中比赛)。
*
* D3: 从 Matches.tsx 拆出。竞态防护语义不变 —— 递增序号只认最后一次请求;
* loadMore 不自增序号(切换筛选才自增,翻页跟随当前序列)。
*/
import { useCallback, useEffect, useRef, useState } from 'react'
import { http } from '../../../lib/http'
import type { Match } from '../types'
interface UseMatchesListOptions {
/** 共享 error state(拆分前列表与预测共用同一个 error,行为保持一致) */
onError: (msg: string | null) => void
}
export function useMatchesList({ onError }: UseMatchesListOptions) {
const [league, setLeague] = useState('E0')
const [status, setStatus] = useState('scheduled')
const [matches, setMatches] = useState<Match[]>([])
const [nextCursor, setNextCursor] = useState<string | null>(null)
const [loadingMore, setLoadingMore] = useState(false)
const [loading, setLoading] = useState(false)
const [showAllUpcoming, setShowAllUpcoming] = useState(false) // 默认仅展示未来 3 天;true 展开全部
const [liveMatches, setLiveMatches] = useState<Match[]>([]) // 进行中比赛(顶部独立区块)
// 请求竞态防护:切换联赛/状态很快时,先发的慢请求可能后返回,
// 把旧结果覆盖到新筛选上。用递增序号只认最后一次请求的响应。
const loadSeq = useRef(0)
const load = useCallback(async () => {
const seq = ++loadSeq.current
setLoading(true)
setLoadingMore(false)
setShowAllUpcoming(false) // 切换筛选重置为「未来 3 天」视图
onError(null)
try {
const params = new URLSearchParams({ league, status, limit: '50' })
const data = await http.get<{ items: Match[]; next_cursor: string | null }>(`/matches?${params}`)
if (seq !== loadSeq.current) return // 已有更新的请求,丢弃本次结果
setMatches(data.items)
setNextCursor(data.next_cursor ?? null)
} catch (e) {
if (seq !== loadSeq.current) return
onError(e instanceof Error ? e.message : String(e))
} finally {
if (seq === loadSeq.current) setLoading(false)
}
}, [league, status, onError])
// 加载下一页(游标分页)
const loadMore = async () => {
if (!nextCursor || loadingMore) return
const seq = loadSeq.current // 不做自增:切换筛选会自增,这里只跟随当前序列
setLoadingMore(true)
try {
const params = new URLSearchParams({ league, status, limit: '50', cursor: nextCursor })
const data = await http.get<{ items: Match[]; next_cursor: string | null }>(`/matches?${params}`)
if (seq !== loadSeq.current) return
setMatches(prev => [...prev, ...data.items])
setNextCursor(data.next_cursor ?? null)
} catch (e) {
if (seq !== loadSeq.current) return
onError(e instanceof Error ? e.message : String(e))
} finally {
if (seq === loadSeq.current) setLoadingMore(false)
}
}
// 加载进行中比赛(顶部独立区块)
const loadLive = useCallback(async () => {
try {
const params = new URLSearchParams({ league, status: 'in_play', limit: '20' })
const data = await http.get<{ items: Match[] }>(`/matches?${params}`)
setLiveMatches(data.items ?? [])
} catch {
/* ignore:进行中非核心功能 */
}
}, [league])
useEffect(() => { load(); loadLive() }, [load, loadLive])
return {
league, setLeague,
status, setStatus,
matches,
nextCursor,
loading,
loadingMore,
showAllUpcoming, setShowAllUpcoming,
liveMatches,
load,
loadMore,
}
}
+98
View File
@@ -0,0 +1,98 @@
/**
* Matches 页面族共享类型与常量。
*
* D3(工程债): Matches.tsx 原本 ~1400 行,类型/常量/预测面板/详情面板/
* 列表逻辑全部内联。本文件是拆分后的共享层 —— 只放数据契约与纯常量,
* 不含 React 组件。
*/
/** /matches 列表项(公开接口) */
export interface Match {
id: number
league_code: string | null
season: string | null
home_team: string
away_team: string
home_team_zh: string | null
away_team_zh: string | null
match_date: string
match_status: string
home_goals: number | null
away_goals: number | null
match_stage: string | null
home_xg: number | null
away_xg: number | null
}
/** POST /predict 响应 */
export interface Prediction {
prediction_id: number
provider: string
model: string
prompt_version: string | null
mode: string
pred_home_goals: number | null
pred_away_goals: number | null
alt_pred_home_goals: number | null
alt_pred_away_goals: number | null
pred_1x2: string | null
subjective_confidence: number | null
reasoning: string | null
status: string
agent_outputs: AgentReport[] | null
agent_weights: Record<string, number> | null
context: string
latency_ms: number | null
prompt_tokens: number | null
completion_tokens: number | null
rate_limit_remaining: number | null
}
/** 多专家单路报告 */
export interface AgentReport {
agent: string
status: string
data_sufficiency: string
analysis: string
home_edge: number | null
subjective_confidence: number | null
key_evidence: string[]
exp_home_goals: number | null
exp_away_goals: number | null
probable_score: string | null
model: string
latency_ms: number | null
}
export const AGENT_LABELS: Record<string, string> = {
h2h: '历史交锋分析专家',
form: '近期状态分析专家',
stats: '攻防数据分析专家',
home_away: '主客因素分析专家',
standings: '联赛排名分析专家',
}
export const LEAGUES = [
{ code: 'E0', name: '英超' },
{ code: 'SP1', name: '西甲' },
{ code: 'D1', name: '德甲' },
{ code: 'I1', name: '意甲' },
{ code: 'F1', name: '法甲' },
]
/** 汉字编号,给专家意见排版用 */
export const CN_NUM = ['一', '二', '三', '四', '五', '六', '七', '八']
export const STATUS_META: Record<string, { label: string; cls: string }> = {
finished: { label: '已完赛', cls: 'text-ink-400' },
scheduled: { label: '未开赛', cls: 'text-ink-600' },
// 键与 normalize.py 的 VALID_STATUS 对齐: 库里存的是 in_play(上游 live 被归一化),不存在 'live' 状态
in_play: { label: '进行中', cls: 'text-press font-medium' },
paused: { label: '暂停', cls: 'text-press font-medium' },
postponed: { label: '延期', cls: 'text-ink-400' },
cancelled: { label: '取消', cls: 'text-ink-400' },
suspended: { label: '中止', cls: 'text-ink-400' },
}
/** 1x2 → 中文标签 */
export const OUTCOME_LABEL: Record<string, string> = { '1': '主胜', X: '平局', '2': '客胜' }
+114
View File
@@ -0,0 +1,114 @@
/**
* Matches 页面族共享的原子 UI 小件(无业务状态)。
*
* D3: 从 Matches.tsx 内联定义上移到模块级 —— Switch 原先定义在组件函数
* 体内(每次渲染重建组件对象),它没有内部 state,提升后渲染结果一致。
*/
import type { Match } from './types'
export function Spinner({ className = '' }: { className?: string }) {
return (
<svg
viewBox="0 0 20 20"
className={`h-3.5 w-3.5 animate-spin ${className}`}
fill="none"
aria-hidden="true"
>
<circle cx="10" cy="10" r="7.5" stroke="currentColor" strokeWidth="1.5" strokeOpacity="0.25" />
<path d="M17.5 10A7.5 7.5 0 0010 2.5" stroke="currentColor" strokeWidth="1.5" strokeLinecap="round" />
</svg>
)
}
/** 骨架占位行:低调脉动灰块 */
export function SkeletonRows({ n = 4 }: { n?: number }) {
return (
<>
{Array.from({ length: n }).map((_, i) => (
<div key={i} className="flex items-center gap-4 border-b border-ink-200 px-1 py-3.5">
<div className="skeleton h-3 w-16" />
<div className="skeleton h-3 flex-1" />
<div className="skeleton h-3 w-10" />
<div className="skeleton h-3 flex-1" />
<div className="skeleton h-3 w-16" />
</div>
))}
</>
)
}
/** 状态/模式一组的文字切换 */
export function Switch({ value, onChange, items }: {
value: string
onChange: (v: string) => void
items: { v: string; label: string; title?: string }[]
}) {
return (
<span className="inline-flex items-center gap-2.5">
{items.map((it, i) => (
<span key={it.v} className="inline-flex items-center gap-2.5">
{i > 0 && <span className="text-ink-300" aria-hidden="true">/</span>}
<button
onClick={() => onChange(it.v)}
title={it.title}
className={`relative tab ${value === it.v ? 'tab-on' : ''} text-xs`}
>
{it.label}
</button>
</span>
))}
</span>
)
}
/** 日期分组头显示:今日/明天/周几 · 年月日 */
export function formatDateHeader(dateKey: string): string {
if (!dateKey) return '未开赛'
const d = new Date(dateKey + 'T00:00:00')
if (isNaN(d.getTime())) return dateKey
const today = new Date()
const todayKey = `${today.getFullYear()}-${String(today.getMonth() + 1).padStart(2, '0')}-${String(today.getDate()).padStart(2, '0')}`
const tmr = new Date(today)
tmr.setDate(tmr.getDate() + 1)
const tmrKey = `${tmr.getFullYear()}-${String(tmr.getMonth() + 1).padStart(2, '0')}-${String(tmr.getDate()).padStart(2, '0')}`
const weekday = ['周日', '周一', '周二', '周三', '周四', '周五', '周六'][d.getDay()]
if (dateKey === todayKey) return `今日 ${weekday}`
if (dateKey === tmrKey) return `明日 ${weekday}`
return `${d.getMonth() + 1}${d.getDate()}${weekday}`
}
/** UTC ISO → 本地日期 YYYY-MM-DD(用于分组) */
export function toLocalDateKey(iso: string): string {
const d = new Date(iso)
return `${d.getFullYear()}-${String(d.getMonth() + 1).padStart(2, '0')}-${String(d.getDate()).padStart(2, '0')}`
}
/** 按本地日期分组(非 UTC),保持时间序 */
export function groupByDate(list: Match[]): Array<[string, Match[]]> {
const map = new Map<string, Match[]>()
for (const m of list) {
const key = toLocalDateKey(m.match_date)
const arr = map.get(key)
if (arr) arr.push(m)
else map.set(key, [m])
}
return [...map.entries()]
}
/** 日期 key 辅助:YYYY-MM-DD(本地时区) */
function dateKey(d: Date): string {
return `${d.getFullYear()}-${String(d.getMonth() + 1).padStart(2, '0')}-${String(d.getDate()).padStart(2, '0')}`
}
/** 未来 3 天窗口:今天 00:00 → 第 3 天 00:00(即今天/明天/后天) */
function addDays(d: Date, n: number): string {
const x = new Date(d)
x.setFullYear(x.getFullYear(), x.getMonth(), x.getDate() + n)
return dateKey(x)
}
/** 比赛是否在未来 3 天内(用于默认视图过滤) */
export function withinNext3Days(matchDate: string): boolean {
const key = toLocalDateKey(matchDate)
return key >= dateKey(new Date()) && key < addDays(new Date(), 3)
}
+29
View File
@@ -2,6 +2,7 @@
from __future__ import annotations
import logging
import os
from collections.abc import AsyncIterator
from contextlib import asynccontextmanager
@@ -30,6 +31,34 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
await ensure_admin_password_hashed() # .env 明文密码 → scrypt 哈希(幂等)
await assert_security_on_startup() # 启动安全校验(生产拒绝/开发警告)
# D7(工程债): 进程内限流(_RateLimiter)与 KeyRing 均为单进程状态;
# 多 worker 部署时各进程独立计数,限流阈值会按 worker 数放大、KeyRing 不共享。
# 生产环境应将限流前置到 Nginx/网关,或以单 worker 运行(见 src/api/deps.py 注释)。
# 此处仅提醒一次,不阻断启动,也不引入 Redis 等外部依赖。
if settings.APP_ENV == "production":
logger.warning(
"APP_ENV=production: 进程内 rate-limit 与 KeyRing 仅单进程有效;"
"多 worker 部署请将限流前置到 Nginx/网关,或以单 worker 运行"
)
# P3-3:STRICT_SINGLE_WORKER 启动期强制校验,拒绝多 worker 静默配额漂移。
# uvicorn 通过 --workers 传入;此处以环境变量 UVICORN_WORKERS 或启动参数判定。
# 为避免耦合 uvicorn 内部,仅校验一个显式传入的标记:当 STRICT_SINGLE_WORKER=True 时,
# 要求环境变量 UVICORN_WORKERS 不为空且 <=1,否则拒绝启动。
if settings.STRICT_SINGLE_WORKER:
workers = os.environ.get("UVICORN_WORKERS", "1")
try:
n_workers = int(workers)
except ValueError:
n_workers = 1
if n_workers > 1:
raise RuntimeError(
f"STRICT_SINGLE_WORKER=True 但以 {n_workers} worker 启动会被拒绝 "
f"(应用内限流/KeyRing 多 worker 下各自独立计数,配额放大 {n_workers} 倍)。"
f"请前置 Nginx/网关全局限流后再启用多 worker,或保持单 worker。"
)
logger.info("STRICT_SINGLE_WORKER=True:已确认单 worker 启动,限流配额不会漂移")
# 注册默认定时任务(如果数据库中没有)
from src.db.base import AsyncSessionLocal
from sqlalchemy import select
+88
View File
@@ -0,0 +1,88 @@
"""后台管理:配置项 CRUD(settings)与运行日志查询。
所有接口需管理员鉴权(require_admin)。路由前缀 /api/v1/admin。
配置项白名单见 src/core/runtime_config.py SETTING_DEFS,之外的 key 一律拒绝。
"""
from __future__ import annotations
import logging
from fastapi import APIRouter, Depends, HTTPException, Query
from pydantic import BaseModel
from src.api.deps import require_admin
from src.core.log_buffer import get_entries
from src.core.runtime_config import (
SETTING_DEFS,
clear_runtime_value,
get_setting_origin,
mask_value,
set_runtime_value,
)
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/v1/admin", tags=["admin"], dependencies=[Depends(require_admin)])
class SettingUpdateIn(BaseModel):
value: str
@router.get("/settings")
async def list_settings():
"""全部可配置项(脱敏),供后台各配置页渲染。"""
out = []
for key, defn in SETTING_DEFS.items():
origin, value = await get_setting_origin(key)
out.append(
{
"key": key,
"label": defn.label,
"description": defn.description,
"sensitive": defn.sensitive,
"configured": origin != "none",
"masked": mask_value(value, defn.sensitive),
"origin": origin,
}
)
return out
@router.get("/logs")
async def read_logs(
level: str | None = Query(None, description="最低级别: DEBUG/INFO/WARNING/ERROR"),
keyword: str | None = Query(None, description="消息或 logger 关键字"),
limit: int = Query(200, ge=1, le=1000),
):
"""查询应用运行日志(内存环形缓冲,最新在前;进程重启后清零)。"""
entries = get_entries(level, keyword, limit)
return {"entries": entries, "count": len(entries)}
@router.put("/settings/{key}")
async def update_setting(key: str, body: SettingUpdateIn):
"""更新配置项(写入 app_settings 覆盖 .env)。传空值请改用 DELETE。"""
if key not in SETTING_DEFS:
raise HTTPException(404, f"不支持的配置项: {key}")
value = body.value.strip()
if not value:
raise HTTPException(400, "值不能为空;如需回落 .env 请调用清除接口")
await set_runtime_value(key, value)
defn = SETTING_DEFS[key]
return {"key": key, "masked": mask_value(value, defn.sensitive), "origin": "db"}
@router.delete("/settings/{key}")
async def clear_setting(key: str):
"""清除 DB 覆盖值,回落 .env 默认。"""
if key not in SETTING_DEFS:
raise HTTPException(404, f"不支持的配置项: {key}")
await clear_runtime_value(key)
origin, value = await get_setting_origin(key)
defn = SETTING_DEFS[key]
return {
"key": key,
"masked": mask_value(value, defn.sensitive),
"origin": origin,
}
+228
View File
@@ -0,0 +1,228 @@
"""后台管理:数据源列表/连通性测试、KeyRing 状态、采集健康概览。
所有接口需管理员鉴权(require_admin)。路由前缀 /api/v1/admin。
"""
from __future__ import annotations
import logging
import time
from datetime import date, datetime
from fastapi import APIRouter, Depends, HTTPException
from sqlalchemy import func, select
from src.api.deps import require_admin
from src.core.config import settings
from src.core.http_client import get_client
from src.core.runtime_config import (
SETTING_DEFS,
get_runtime_value,
get_setting_origin,
mask_value,
)
from src.db.base import AsyncSession, get_db_read
from src.db.models import Match, MatchStats, Standing
from src.data.key_ring import get_key_ring
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/v1/admin", tags=["admin"], dependencies=[Depends(require_admin)])
# ── 数据源元数据(bzzoiro 单一数据源) ────────────────────────────
_SOURCES: list[dict] = [
{
"name": "bzzoiro",
"label": "Bzzoiro",
"description": "唯一数据源:赛程比分 + 积分榜 + 比赛详细统计(xG/射门/控球等)",
"setting_keys": ["BZZOIRO_KEY", "BZZOIRO_BASE"],
},
]
async def _last_ingestion(db: AsyncSession, source: str) -> datetime | None:
"""源最近一次采集时间(取自数据血缘字段,无记录返回 None)。"""
return (
await db.execute(
select(func.max(MatchStats.retrieved_at)).where(MatchStats.source == source)
)
).scalar()
@router.get("/datasources")
async def list_datasources(db: AsyncSession = Depends(get_db_read)):
"""数据源列表:各配置项的脱敏值、来源(db/env/none)与最近采集时间。"""
result = []
for src in _SOURCES:
settings_out = []
for key in src["setting_keys"]:
origin, value = await get_setting_origin(key)
defn = SETTING_DEFS[key]
settings_out.append(
{
"key": key,
"label": defn.label,
"description": defn.description,
"sensitive": defn.sensitive,
"configured": origin != "none",
"masked": mask_value(value, defn.sensitive),
"origin": origin,
}
)
key_configured = all(s["configured"] for s in settings_out) if settings_out else True
last = await _last_ingestion(db, src["name"])
result.append(
{
"name": src["name"],
"label": src["label"],
"description": src["description"],
"key_configured": key_configured,
"last_ingestion": last.isoformat() if last else None,
"settings": settings_out,
}
)
return result
# ── 连通性测试 ──────────────────────────────────────────────────
_TEST_TIMEOUT = 15
async def _probe(url: str, headers: dict | None = None, params: dict | None = None) -> dict:
"""单次 HTTP 探测,返回 (ok, status, latency_ms, detail)。不重试。"""
client = get_client()
start = time.monotonic()
try:
resp = await client.get(url, headers=headers, params=params, timeout=_TEST_TIMEOUT)
except Exception as e:
return {
"ok": False,
"status": None,
"latency_ms": int((time.monotonic() - start) * 1000),
"detail": f"无法连接: {e}",
}
latency = int((time.monotonic() - start) * 1000)
status = resp.status_code
if status == 200:
detail = "连接成功"
elif status in (401, 403):
detail = "服务可达,但密钥无效或无权限"
else:
detail = f"服务返回 HTTP {status}"
return {"ok": status == 200, "status": status, "latency_ms": latency, "detail": detail}
@router.post("/datasources/{name}/test")
async def test_datasource(name: str):
"""轻量连通性测试:真实请求上游一次,不触发任何入库。"""
src = next((s for s in _SOURCES if s["name"] == name), None)
if src is None:
raise HTTPException(404, f"未知数据源: {name}")
if name == "bzzoiro":
key = await get_runtime_value("BZZOIRO_KEY")
if not key:
return {"ok": False, "status": None, "latency_ms": 0, "detail": "BZZOIRO_KEY 未配置"}
base = (await get_runtime_value("BZZOIRO_BASE")).rstrip("/")
today = date.today().isoformat()
return await _probe(
f"{base}/events/",
headers={"Authorization": f"Token {key}", "Accept": "application/json"},
params={"date_from": today, "date_to": today},
)
raise HTTPException(404, f"未知数据源: {name}")
# ── 数据源健康/最近采集状态(只读,不触发采集) ──────────────────────
@router.get("/ingest/status")
async def ingest_status(db: AsyncSession = Depends(get_db_read)):
"""数据源采集健康概览(bzzoiro 单源;只读,不触发任何采集)。"""
bzzoiro_key = await get_runtime_value("BZZOIRO_KEY")
bzzoiro_base = await get_runtime_value("BZZOIRO_BASE")
# 比赛覆盖
match_row = (
await db.execute(
select(
func.count().label("cnt"),
func.max(Match.match_date).label("latest_match_date"),
func.max(Match.created_at).label("latest_row_at"),
).where(Match.match_status == "finished")
)
).one()
# 统计覆盖(精确 retrieved_at)
stats_row = (
await db.execute(
select(
func.count().label("cnt"),
func.max(MatchStats.retrieved_at).label("latest_retrieved"),
).where(MatchStats.source == "bzzoiro")
)
).one()
# 积分榜覆盖
standings_row = (
await db.execute(select(func.count()).select_from(Standing))
).scalar()
bzzoiro = {
"name": "bzzoiro",
"label": "Bzzoiro",
"key_configured": bool(bzzoiro_key),
"base_url": (bzzoiro_base.rstrip("/") if bzzoiro_base else None) or settings.BZZOIRO_BASE,
"reachable": None, # 不主动探测
"last_success_at": (stats_row.latest_retrieved or match_row.latest_row_at),
"last_success_at_iso": (
stats_row.latest_retrieved or match_row.latest_row_at
).isoformat() if (stats_row.latest_retrieved or match_row.latest_row_at) else None,
"latest_match_date": match_row.latest_match_date.isoformat() if match_row.latest_match_date else None,
"recent_count": match_row.cnt or 0,
"stats_count": stats_row.cnt or 0,
"standings_count": standings_row or 0,
"note": "last_success_at 取 match_stats.retrieved_at(统计回填)与 matches.created_at(比赛行)的较大者",
"last_failure": _last_failure_log("bzzoiro"),
}
return {"sources": [bzzoiro]}
@router.get("/keyring/status")
async def keyring_status():
"""KeyRing 运行状态:当前使用的 key、冷却状态、轮转信息(供管理后台展示)。"""
base = (await get_runtime_value("BZZOIRO_BASE")).rstrip("/")
raw_keys = await get_runtime_value("BZZOIRO_KEY")
ring = get_key_ring(base, raw_keys)
st = ring.stats()
st["base_url"] = base
st["cooldown_seconds"] = ring._cooldown
st["has_multiple"] = ring.has_multiple
st["active_key"] = ring.active_key
return st
@router.post("/keyring/cooldown/reset")
async def keyring_reset_cooldown():
"""手动重置所有 key 的冷却状态(用于紧急恢复)。"""
base = (await get_runtime_value("BZZOIRO_BASE")).rstrip("/")
raw_keys = await get_runtime_value("BZZOIRO_KEY")
ring = get_key_ring(base, raw_keys)
ring._blocked_until.clear()
return {"ok": True, "message": "已重置所有 key 冷却状态", "stats": ring.stats()}
def _last_failure_log(source: str) -> dict | None:
"""从系统日志缓冲中查找某数据源的最近一次错误(仅作参考,非专用失败表)。"""
from src.core.log_buffer import get_entries
entries = get_entries(min_level="ERROR", keyword=source, limit=5)
if not entries:
return None
e = entries[0]
return {
"at": datetime.fromtimestamp(e["ts"]).isoformat(),
"logger": e["logger"],
"detail": e["message"][:200],
"note": "approx:来自内存日志缓冲,非专用采集失败表;进程重启后清零",
}
+126
View File
@@ -0,0 +1,126 @@
"""后台管理:LLM 专家/终裁配置、可用模型探测、连通性测试。
所有接口需管理员鉴权(require_admin)。路由前缀 /api/v1/admin。
"""
from __future__ import annotations
import logging
import time
import httpx
from fastapi import APIRouter, Depends
from src.api.deps import require_admin
from src.core.config import settings
from src.core.http_client import get_client
from src.core.runtime_config import (
AGENT_META,
SETTING_DEFS,
get_runtime_value,
get_setting_origin,
mask_value,
)
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/v1/admin", tags=["admin"], dependencies=[Depends(require_admin)])
@router.get("/llm/agents")
async def list_llm_agents():
"""各专家/终裁的独立 LLM 配置状态(含当前生效模型的解析结果)。"""
out = []
for agent in AGENT_META:
aid = agent["id"].upper()
pfx = f"AGENT_{aid}_"
fields = {}
for suffix in ("MODEL", "BASE_URL", "API_KEY"):
origin, value = await get_setting_origin(f"{pfx}{suffix}")
defn = SETTING_DEFS[f"{pfx}{suffix}"]
fields[suffix.lower()] = {
"configured": origin != "none",
"masked": mask_value(value, defn.sensitive),
"origin": origin,
}
# 生效模型 = 覆盖 → 层级默认(专家/终裁 env) → 全局 LLM_MODEL
tier_default = (
settings.LLM_AGGREGATOR_MODEL if agent["id"] == "aggregator" else settings.LLM_SPECIALIST_MODEL
)
effective_model = (
fields["model"]["masked"]
if fields["model"]["configured"]
else (tier_default or await get_runtime_value("LLM_MODEL"))
)
out.append(
{
"id": agent["id"],
"label": agent["label"],
"fields": fields,
"effective_model": effective_model,
}
)
return out
@router.get("/llm/models")
async def list_llm_models():
"""探测当前 LLM 服务可用的模型列表(OpenAI 兼容 GET /models)。
只读探测,不产生费用;配置缺失或服务不可达时返回 ok=false 与原因。
"""
base_url = (await get_runtime_value("LLM_BASE_URL")).rstrip("/")
api_key = await get_runtime_value("LLM_API_KEY")
if not base_url or not api_key:
return {"ok": False, "models": [], "detail": "LLM_BASE_URL 或 LLM_API_KEY 未配置"}
client = get_client()
start = time.monotonic()
try:
resp = await client.get(
f"{base_url}/models",
headers={"Authorization": f"Bearer {api_key}"},
timeout=httpx.Timeout(connect=10.0, read=20.0, write=10.0, pool=10.0),
)
except Exception as e:
return {
"ok": False,
"models": [],
"latency_ms": int((time.monotonic() - start) * 1000),
"detail": f"无法连接 LLM 服务: {e}",
}
latency = int((time.monotonic() - start) * 1000)
if resp.status_code in (401, 403):
return {"ok": False, "models": [], "latency_ms": latency, "detail": "密钥无效或无权限(HTTP 401/403)"}
if resp.status_code != 200:
return {"ok": False, "models": [], "latency_ms": latency, "detail": f"服务返回 HTTP {resp.status_code}"}
try:
data = resp.json()
except Exception:
return {"ok": False, "models": [], "latency_ms": latency, "detail": "响应不是合法 JSON"}
models: list[str] = []
items = data.get("data") if isinstance(data, dict) else None
if isinstance(items, list):
models = sorted(
str(m.get("id")) for m in items if isinstance(m, dict) and m.get("id")
)
if not models:
return {"ok": False, "models": [], "latency_ms": latency, "detail": "服务未返回模型列表"}
return {"ok": True, "models": models, "latency_ms": latency, "detail": f"{len(models)} 个可用模型"}
@router.post("/llm/ping")
async def llm_ping():
"""LLM 连通性测试(不依赖比赛)。只发一次 chat 请求验证配置。"""
from src.llm.provider import get_default_provider
p = await get_default_provider()
resp = await p.chat(
system="你是测试助手。",
user="ping",
max_tokens=10,
)
if resp.error:
return {"ok": False, "message": resp.error}
return {"ok": True, "message": "LLM 连接正常", "model": p.model}
+343
View File
@@ -0,0 +1,343 @@
"""后台管理:管理区统计、数据完整性分析、数据质量检查。
所有接口需管理员鉴权(require_admin)。路由前缀 /api/v1/admin。
"""
from __future__ import annotations
import logging
from datetime import datetime, timedelta, timezone
from fastapi import APIRouter, Depends
from sqlalchemy import func, select
from src.api.deps import require_admin
from src.db.base import AsyncSession, get_db_read
from src.db.models import DataQualityCheck, IngestFailure, League, Match, MatchStats, Prediction, Standing
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/v1/admin", tags=["admin"], dependencies=[Depends(require_admin)])
@router.get("/stats")
async def admin_stats(db: AsyncSession = Depends(get_db_read)):
"""管理区统计(只读):预测次数 + 比赛覆盖。轻量聚合,无 LLM 调用。"""
day_ago = datetime.now(timezone.utc) - timedelta(days=1)
week_ago = datetime.now(timezone.utc) - timedelta(days=7)
r = (
await db.execute(
select(
func.count().label("total"),
func.count().filter(Prediction.created_at >= day_ago).label("last_24h"),
func.count().filter(Prediction.created_at >= week_ago).label("last_7d"),
)
)
).one()
# F3 修复: 补充真实比赛计数(非 limit=100 近似)
match_cnt = (await db.execute(select(func.count()).select_from(Match))).scalar() or 0
finished_cnt = (await db.execute(select(func.count()).where(Match.match_status == "finished"))).scalar() or 0
stats_cnt = (await db.execute(select(func.count()).select_from(MatchStats))).scalar() or 0
standings_cnt = (await db.execute(select(func.count()).select_from(Standing))).scalar() or 0
return {
"predictions": {"total": r.total, "last_24h": r.last_24h, "last_7d": r.last_7d},
"matches": {"total": match_cnt, "finished": finished_cnt},
"stats": {"total": stats_cnt},
"standings": {"total": standings_cnt},
}
# ── 数据完整性分析(可视化数据源) ────────────────────────────────
@router.get("/data-completeness")
async def data_completeness(db: AsyncSession = Depends(get_db_read)):
"""按联赛统计数据完整性:比赛覆盖、字段覆盖、积分榜覆盖。
前端「数据完整性」页据此渲染,回答三个问题:
1. 数据是否齐全(各联赛比赛/统计/积分榜量级)
2. 字段是否齐全(每张统计表各字段非空率)
3. 覆盖是否新鲜(最近一场/最近一次采集)
"""
from src.data.config import BZZOIRO_LEAGUE_IDS, LEAGUE_NAMES, LEAGUE_COUNTRIES
out_leagues: list[dict] = []
for code, bzz_id in BZZOIRO_LEAGUE_IDS.items():
# 比赛覆盖
m = (
await db.execute(
select(
func.count().label("total"),
func.count().filter(Match.match_status == "finished").label("finished"),
func.count().filter(Match.match_status == "scheduled").label("scheduled"),
func.count().filter(Match.source_event_id.is_not(None)).label("with_source_id"),
func.max(Match.match_date).label("latest_match"),
func.min(Match.match_date).label("earliest_match"),
)
.select_from(Match)
.join(League, League.id == Match.league_id)
.where(League.code == code)
)
).one()
# 统计字段覆盖(联表 matches)
s = (
await db.execute(
select(
func.count().label("rows"),
func.count(MatchStats.home_xg).label("xg"),
func.count(MatchStats.home_shots).label("shots"),
func.count(MatchStats.home_possession).label("possession"),
func.count(MatchStats.home_corners).label("corners"),
func.count(MatchStats.home_fouls).label("fouls"),
func.count(MatchStats.home_big_chances).label("big_chances"),
func.count(MatchStats.home_yellow_cards).label("cards"),
)
.select_from(MatchStats)
.join(Match, Match.id == MatchStats.match_id)
.join(League, League.id == Match.league_id)
.where(League.code == code)
)
).one()
# 积分榜覆盖
st = (
await db.execute(
select(
func.count().label("rows"),
func.max(Standing.retrieved_at).label("latest_retrieved"),
)
.select_from(Standing)
.join(League, League.id == Standing.league_id)
.where(League.code == code)
)
).one()
stats_rows = s.rows or 0
pct = lambda n: round(n / stats_rows * 100, 1) if stats_rows else 0.0 # noqa: E731
out_leagues.append(
{
"code": code,
"name": LEAGUE_NAMES.get(code, code),
"country": LEAGUE_COUNTRIES.get(code),
"matches": {
"total": m.total or 0,
"finished": m.finished or 0,
"scheduled": m.scheduled or 0,
"with_source_id": m.with_source_id or 0,
"earliest_match": m.earliest_match.isoformat() if m.earliest_match else None,
"latest_match": m.latest_match.isoformat() if m.latest_match else None,
},
"stats": {
"rows": stats_rows,
"fields": {
"xg": {"count": s.xg or 0, "pct": pct(s.xg or 0)},
"shots": {"count": s.shots or 0, "pct": pct(s.shots or 0)},
"possession": {"count": s.possession or 0, "pct": pct(s.possession or 0)},
"corners": {"count": s.corners or 0, "pct": pct(s.corners or 0)},
"fouls": {"count": s.fouls or 0, "pct": pct(s.fouls or 0)},
"big_chances": {"count": s.big_chances or 0, "pct": pct(s.big_chances or 0)},
"cards": {"count": s.cards or 0, "pct": pct(s.cards or 0)},
},
},
"standings": {
"rows": st.rows or 0,
"latest_retrieved": st.latest_retrieved.isoformat() if st.latest_retrieved else None,
},
}
)
# 整体健康信号
total_finished = sum(l["matches"]["finished"] for l in out_leagues)
total_stats = sum(l["stats"]["rows"] for l in out_leagues)
stats_coverage = round(total_stats / total_finished * 100, 1) if total_finished else 0.0
issues: list[str] = []
for l in out_leagues:
if l["matches"]["finished"] == 0:
issues.append(f"{l['name']}: 无已完赛比赛,请先运行「比赛数据」采集")
elif l["stats"]["rows"] == 0:
issues.append(f"{l['name']}: 已完赛 {l['matches']['finished']} 场但无统计回填,请运行「统计回填」采集")
elif stats_coverage < 80:
issues.append(f"{l['name']}: 统计覆盖率仅 {stats_coverage}%,建议增量回填")
if l["standings"]["rows"] == 0:
issues.append(f"{l['name']}: 无积分榜数据,请运行「积分榜」采集")
if not issues:
issues.append("各联赛数据完整度良好")
return {
"generated_at": datetime.now(timezone.utc).isoformat(),
"leagues": out_leagues,
"totals": {
"finished_matches": total_finished,
"stats_rows": total_stats,
"stats_coverage_pct": stats_coverage,
},
"issues": issues,
}
# ── 数据质量检查 API ────────────────────────────────────────────
@router.get("/data-quality")
async def data_quality_checks(db: AsyncSession = Depends(get_db_read)):
"""数据质量检查结果(只读)。"""
# 最近的失败记录
failures = (
await db.execute(
select(IngestFailure)
.where(IngestFailure.status.in_(["pending", "retrying"]))
.order_by(IngestFailure.created_at.desc())
.limit(20)
)
).scalars().all()
# 最近的质量检查
checks = (
await db.execute(
select(DataQualityCheck)
.order_by(DataQualityCheck.checked_at.desc())
.limit(20)
)
).scalars().all()
return {
"failures": [
{
"id": f.id,
"source": f.source_system,
"entity_type": f.entity_type,
"source_record_id": f.source_record_id,
"error_type": f.error_type,
"error_detail": f.error_detail,
"retry_count": f.retry_count,
"status": f.status,
"created_at": f.created_at.isoformat() if f.created_at else None,
}
for f in failures
],
"checks": [
{
"id": c.id,
"check_name": c.check_name,
"entity_type": c.entity_type,
"passed": c.passed,
"severity": c.severity,
"detail": c.detail,
"checked_at": c.checked_at.isoformat() if c.checked_at else None,
}
for c in checks
],
}
@router.post("/data-quality/run")
async def run_data_quality_check(db: AsyncSession = Depends(get_db_read)):
"""手动触发一次数据质量检查。"""
checks = []
# 检查1: 已完赛但无统计的比赛
finished_no_stats = (
await db.execute(
select(func.count())
.select_from(Match)
.outerjoin(MatchStats, Match.id == MatchStats.match_id)
.where(Match.match_status == "finished")
.where(MatchStats.id.is_(None))
)
).scalar() or 0
checks.append(DataQualityCheck(
check_name="finished_without_stats",
entity_type="match",
actual_value=float(finished_no_stats),
passed=finished_no_stats == 0,
severity="warning" if finished_no_stats > 0 else "info",
detail={"message": f"{finished_no_stats} 场已完赛比赛缺少统计数据"},
))
# 检查2: 积分榜缺失的联赛
leagues_without_standings = (
await db.execute(
select(func.count())
.select_from(League)
.outerjoin(Standing, League.id == Standing.league_id)
.where(Standing.id.is_(None))
)
).scalar() or 0
checks.append(DataQualityCheck(
check_name="league_without_standings",
entity_type="league",
actual_value=float(leagues_without_standings),
passed=leagues_without_standings == 0,
severity="warning" if leagues_without_standings > 0 else "info",
detail={"message": f"{leagues_without_standings} 个联赛缺少积分榜"},
))
for c in checks:
db.add(c)
await db.commit()
return {"ok": True, "checks": [{"name": c.check_name, "passed": c.passed} for c in checks]}
# ── 近似重名候选(只读,启发式,不做自动合并) ──────────────────────
@router.get("/team-name-duplicates")
async def team_name_duplicates(db: AsyncSession = Depends(get_db_read)):
"""只读列出近似重名候选(大小写变体/子串包含/前缀碰撞)。
启发式规则(命中任一即列为候选):
- 大小写变体: lower(name) 相同但 name 不同
- 子串包含: A 是 B 的子串且 len(A) ≥ 5
- 前缀碰撞: 前 8 字符相同(忽略大小写)
仅作排查参考,合并需走人工 SQL(见 docs/05-data.md)。
"""
teams = (await db.execute(select(Team.id, Team.name))).all()
by_lower: dict[str, list[dict]] = {}
for t in teams:
key = (t.name or "").lower()
by_lower.setdefault(key, []).append({"id": t.id, "name": t.name})
groups: list[dict] = []
# 规则1: 大小写变体(lower 相同但原名不同)
for key, members in by_lower.items():
if len(members) > 1:
groups.append({
"rule": "case_variant",
"key": key,
"members": members,
})
# 规则2 & 3: 子串包含 / 前缀碰撞(仅在 lower 名不同的组间比较)
distinct = [m for members in by_lower.values() for m in members]
seen_pairs: set[tuple[int, int]] = set()
for i, a in enumerate(distinct):
na = (a["name"] or "").lower()
for b in distinct[i + 1:]:
nb = (b["name"] or "").lower()
if na == nb:
continue # 已被规则1覆盖
pair = (min(a["id"], b["id"]), max(a["id"], b["id"]))
if pair in seen_pairs:
continue
hit = None
if len(na) >= 5 and na in nb:
hit = "substring"
elif len(nb) >= 5 and nb in na:
hit = "substring"
elif len(na) >= 8 and len(nb) >= 8 and na[:8] == nb[:8]:
hit = "prefix"
if hit:
seen_pairs.add(pair)
groups.append({
"rule": hit,
"members": [a, b],
})
return {
"count": len(groups),
"hint": "命中任一启发式仅表示'可疑',合并前请人工确认是否同一球队",
"groups": groups,
}
+21 -664
View File
@@ -1,668 +1,25 @@
"""后台管理路由:数据源配置的查看、修改与连通性测试
"""后台管理路由聚合入口:按职责拆分为四个子模块,统一挂载
所有接口需管理员鉴权(require_admin)。配置项白名单见
src/core/runtime_config.py SETTING_DEFS,之外的 key 一律拒绝
所有路由仍挂在 /api/v1/admin,且均带 dependencies=[Depends(require_admin)]
(鉴权由各子路由器声明,行为与拆分前完全一致)
子模块:
- admin_datasources 数据源列表/连通性测试、KeyRing、采集健康概览
- admin_config settings CRUD、运行日志
- admin_llm LLM agents/models/ping
- admin_quality stats、data-completeness、data-quality
"""
from __future__ import annotations
import logging
import time
from datetime import date, datetime, timedelta, timezone
from fastapi import APIRouter, Depends, HTTPException, Query
from pydantic import BaseModel
from sqlalchemy import func, select
import httpx
from src.api.deps import require_admin
from src.core.config import settings
from src.core.http_client import get_client
from src.core.log_buffer import get_entries
from src.core.runtime_config import (
AGENT_META,
SETTING_DEFS,
clear_runtime_value,
get_runtime_value,
get_setting_origin,
mask_value,
set_runtime_value,
)
from src.db.base import AsyncSession, get_db_read
from src.db.models import League, Match, MatchStats, Standing
from src.data.key_ring import get_key_ring, parse_keys
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/v1/admin", tags=["admin"], dependencies=[Depends(require_admin)])
# ── 数据源元数据(bzzoiro 单一数据源) ────────────────────────────
_SOURCES: list[dict] = [
{
"name": "bzzoiro",
"label": "Bzzoiro",
"description": "唯一数据源:赛程比分 + 积分榜 + 比赛详细统计(xG/射门/控球等)",
"setting_keys": ["BZZOIRO_KEY", "BZZOIRO_BASE"],
},
]
class SettingUpdateIn(BaseModel):
value: str
async def _last_ingestion(db: AsyncSession, source: str) -> datetime | None:
"""源最近一次采集时间(取自数据血缘字段,无记录返回 None)。"""
return (
await db.execute(
select(func.max(MatchStats.retrieved_at)).where(MatchStats.source == source)
)
).scalar()
@router.get("/datasources")
async def list_datasources(db: AsyncSession = Depends(get_db_read)):
"""数据源列表:各配置项的脱敏值、来源(db/env/none)与最近采集时间。"""
result = []
for src in _SOURCES:
settings_out = []
for key in src["setting_keys"]:
origin, value = await get_setting_origin(key)
defn = SETTING_DEFS[key]
settings_out.append(
{
"key": key,
"label": defn.label,
"description": defn.description,
"sensitive": defn.sensitive,
"configured": origin != "none",
"masked": mask_value(value, defn.sensitive),
"origin": origin,
}
)
key_configured = all(s["configured"] for s in settings_out) if settings_out else True
last = await _last_ingestion(db, src["name"])
result.append(
{
"name": src["name"],
"label": src["label"],
"description": src["description"],
"key_configured": key_configured,
"last_ingestion": last.isoformat() if last else None,
"settings": settings_out,
}
)
return result
@router.get("/settings")
async def list_settings():
"""全部可配置项(脱敏),供后台各配置页渲染。"""
out = []
for key, defn in SETTING_DEFS.items():
origin, value = await get_setting_origin(key)
out.append(
{
"key": key,
"label": defn.label,
"description": defn.description,
"sensitive": defn.sensitive,
"configured": origin != "none",
"masked": mask_value(value, defn.sensitive),
"origin": origin,
}
)
return out
# ── LLM 可用模型检测 ────────────────────────────────────────────
@router.get("/logs")
async def read_logs(
level: str | None = Query(None, description="最低级别: DEBUG/INFO/WARNING/ERROR"),
keyword: str | None = Query(None, description="消息或 logger 关键字"),
limit: int = Query(200, ge=1, le=1000),
):
"""查询应用运行日志(内存环形缓冲,最新在前;进程重启后清零)。"""
entries = get_entries(level, keyword, limit)
return {"entries": entries, "count": len(entries)}
@router.get("/llm/agents")
async def list_llm_agents():
"""各专家/终裁的独立 LLM 配置状态(含当前生效模型的解析结果)。"""
out = []
for agent in AGENT_META:
aid = agent["id"].upper()
pfx = f"AGENT_{aid}_"
fields = {}
for suffix in ("MODEL", "BASE_URL", "API_KEY"):
origin, value = await get_setting_origin(f"{pfx}{suffix}")
defn = SETTING_DEFS[f"{pfx}{suffix}"]
fields[suffix.lower()] = {
"configured": origin != "none",
"masked": mask_value(value, defn.sensitive),
"origin": origin,
}
# 生效模型 = 覆盖 → 层级默认(专家/终裁 env) → 全局 LLM_MODEL
tier_default = (
settings.LLM_AGGREGATOR_MODEL if agent["id"] == "aggregator" else settings.LLM_SPECIALIST_MODEL
)
effective_model = (
fields["model"]["masked"]
if fields["model"]["configured"]
else (tier_default or await get_runtime_value("LLM_MODEL"))
)
out.append(
{
"id": agent["id"],
"label": agent["label"],
"fields": fields,
"effective_model": effective_model,
}
)
return out
@router.get("/llm/models")
async def list_llm_models():
"""探测当前 LLM 服务可用的模型列表(OpenAI 兼容 GET /models)。
只读探测,不产生费用;配置缺失或服务不可达时返回 ok=false 与原因。
"""
base_url = (await get_runtime_value("LLM_BASE_URL")).rstrip("/")
api_key = await get_runtime_value("LLM_API_KEY")
if not base_url or not api_key:
return {"ok": False, "models": [], "detail": "LLM_BASE_URL 或 LLM_API_KEY 未配置"}
client = get_client()
start = time.monotonic()
try:
resp = await client.get(
f"{base_url}/models",
headers={"Authorization": f"Bearer {api_key}"},
timeout=httpx.Timeout(connect=10.0, read=20.0, write=10.0, pool=10.0),
)
except Exception as e:
return {
"ok": False,
"models": [],
"latency_ms": int((time.monotonic() - start) * 1000),
"detail": f"无法连接 LLM 服务: {e}",
}
latency = int((time.monotonic() - start) * 1000)
if resp.status_code in (401, 403):
return {"ok": False, "models": [], "latency_ms": latency, "detail": "密钥无效或无权限(HTTP 401/403)"}
if resp.status_code != 200:
return {"ok": False, "models": [], "latency_ms": latency, "detail": f"服务返回 HTTP {resp.status_code}"}
try:
data = resp.json()
except Exception:
return {"ok": False, "models": [], "latency_ms": latency, "detail": "响应不是合法 JSON"}
models: list[str] = []
items = data.get("data") if isinstance(data, dict) else None
if isinstance(items, list):
models = sorted(
str(m.get("id")) for m in items if isinstance(m, dict) and m.get("id")
)
if not models:
return {"ok": False, "models": [], "latency_ms": latency, "detail": "服务未返回模型列表"}
return {"ok": True, "models": models, "latency_ms": latency, "detail": f"{len(models)} 个可用模型"}
@router.put("/settings/{key}")
async def update_setting(key: str, body: SettingUpdateIn):
"""更新配置项(写入 app_settings 覆盖 .env)。传空值请改用 DELETE。"""
if key not in SETTING_DEFS:
raise HTTPException(404, f"不支持的配置项: {key}")
value = body.value.strip()
if not value:
raise HTTPException(400, "值不能为空;如需回落 .env 请调用清除接口")
await set_runtime_value(key, value)
defn = SETTING_DEFS[key]
return {"key": key, "masked": mask_value(value, defn.sensitive), "origin": "db"}
@router.delete("/settings/{key}")
async def clear_setting(key: str):
"""清除 DB 覆盖值,回落 .env 默认。"""
if key not in SETTING_DEFS:
raise HTTPException(404, f"不支持的配置项: {key}")
await clear_runtime_value(key)
origin, value = await get_setting_origin(key)
defn = SETTING_DEFS[key]
return {
"key": key,
"masked": mask_value(value, defn.sensitive),
"origin": origin,
}
# ── 连通性测试 ──────────────────────────────────────────────────
_TEST_TIMEOUT = 15
async def _probe(url: str, headers: dict | None = None, params: dict | None = None) -> dict:
"""单次 HTTP 探测,返回 (ok, status, latency_ms, detail)。不重试。"""
client = get_client()
start = time.monotonic()
try:
resp = await client.get(url, headers=headers, params=params, timeout=_TEST_TIMEOUT)
except Exception as e:
return {
"ok": False,
"status": None,
"latency_ms": int((time.monotonic() - start) * 1000),
"detail": f"无法连接: {e}",
}
latency = int((time.monotonic() - start) * 1000)
status = resp.status_code
if status == 200:
detail = "连接成功"
elif status in (401, 403):
detail = "服务可达,但密钥无效或无权限"
else:
detail = f"服务返回 HTTP {status}"
return {"ok": status == 200, "status": status, "latency_ms": latency, "detail": detail}
@router.post("/datasources/{name}/test")
async def test_datasource(name: str):
"""轻量连通性测试:真实请求上游一次,不触发任何入库。"""
src = next((s for s in _SOURCES if s["name"] == name), None)
if src is None:
raise HTTPException(404, f"未知数据源: {name}")
if name == "bzzoiro":
key = await get_runtime_value("BZZOIRO_KEY")
if not key:
return {"ok": False, "status": None, "latency_ms": 0, "detail": "BZZOIRO_KEY 未配置"}
base = (await get_runtime_value("BZZOIRO_BASE")).rstrip("/")
today = date.today().isoformat()
return await _probe(
f"{base}/events/",
headers={"Authorization": f"Token {key}", "Accept": "application/json"},
params={"date_from": today, "date_to": today},
)
raise HTTPException(404, f"未知数据源: {name}")
@router.post("/llm/ping")
async def llm_ping():
"""LLM 连通性测试(不依赖比赛)。只发一次 chat 请求验证配置。"""
from src.llm.provider import get_default_provider
p = await get_default_provider()
resp = await p.chat(
system="你是测试助手。",
user="ping",
max_tokens=10,
)
if resp.error:
return {"ok": False, "message": resp.error}
return {"ok": True, "message": "LLM 连接正常", "model": p.model}
# ── 数据源健康/最近采集状态(只读,不触发采集) ──────────────────────
@router.get("/ingest/status")
async def ingest_status(db: AsyncSession = Depends(get_db_read)):
"""数据源采集健康概览(bzzoiro 单源;只读,不触发任何采集)。"""
bzzoiro_key = await get_runtime_value("BZZOIRO_KEY")
bzzoiro_base = await get_runtime_value("BZZOIRO_BASE")
# 比赛覆盖
match_row = (
await db.execute(
select(
func.count().label("cnt"),
func.max(Match.match_date).label("latest_match_date"),
func.max(Match.created_at).label("latest_row_at"),
).where(Match.match_status == "finished")
)
).one()
# 统计覆盖(精确 retrieved_at)
stats_row = (
await db.execute(
select(
func.count().label("cnt"),
func.max(MatchStats.retrieved_at).label("latest_retrieved"),
).where(MatchStats.source == "bzzoiro")
)
).one()
# 积分榜覆盖
standings_row = (
await db.execute(select(func.count()).select_from(Standing))
).scalar()
bzzoiro = {
"name": "bzzoiro",
"label": "Bzzoiro",
"key_configured": bool(bzzoiro_key),
"base_url": (bzzoiro_base.rstrip("/") if bzzoiro_base else None) or settings.BZZOIRO_BASE,
"reachable": None, # 不主动探测
"last_success_at": (stats_row.latest_retrieved or match_row.latest_row_at),
"last_success_at_iso": (
stats_row.latest_retrieved or match_row.latest_row_at
).isoformat() if (stats_row.latest_retrieved or match_row.latest_row_at) else None,
"latest_match_date": match_row.latest_match_date.isoformat() if match_row.latest_match_date else None,
"recent_count": match_row.cnt or 0,
"stats_count": stats_row.cnt or 0,
"standings_count": standings_row or 0,
"note": "last_success_at 取 match_stats.retrieved_at(统计回填)与 matches.created_at(比赛行)的较大者",
"last_failure": _last_failure_log("bzzoiro"),
}
return {"sources": [bzzoiro]}
@router.get("/keyring/status")
async def keyring_status():
"""KeyRing 运行状态:当前使用的 key、冷却状态、轮转信息(供管理后台展示)。"""
base = (await get_runtime_value("BZZOIRO_BASE")).rstrip("/")
raw_keys = await get_runtime_value("BZZOIRO_KEY")
ring = get_key_ring(base, raw_keys)
st = ring.stats()
st["base_url"] = base
st["cooldown_seconds"] = ring._cooldown
st["has_multiple"] = ring.has_multiple
st["active_key"] = ring.active_key
return st
@router.post("/keyring/cooldown/reset")
async def keyring_reset_cooldown():
"""手动重置所有 key 的冷却状态(用于紧急恢复)。"""
base = (await get_runtime_value("BZZOIRO_BASE")).rstrip("/")
raw_keys = await get_runtime_value("BZZOIRO_KEY")
ring = get_key_ring(base, raw_keys)
ring._blocked_until.clear()
return {"ok": True, "message": "已重置所有 key 冷却状态", "stats": ring.stats()}
def _last_failure_log(source: str) -> dict | None:
"""从系统日志缓冲中查找某数据源的最近一次错误(仅作参考,非专用失败表)。"""
entries = get_entries(min_level="ERROR", keyword=source, limit=5)
if not entries:
return None
e = entries[0]
return {
"at": datetime.fromtimestamp(e["ts"]).isoformat(),
"logger": e["logger"],
"detail": e["message"][:200],
"note": "approx:来自内存日志缓冲,非专用采集失败表;进程重启后清零",
}
@router.get("/stats")
async def admin_stats(db: AsyncSession = Depends(get_db_read)):
"""管理区统计(只读):预测次数 + 比赛覆盖。轻量聚合,无 LLM 调用。"""
from sqlalchemy import func, text
from src.db.models import Prediction, Match, MatchStats, Standing
day_ago = datetime.now(timezone.utc) - timedelta(days=1)
week_ago = datetime.now(timezone.utc) - timedelta(days=7)
r = (
await db.execute(
select(
func.count().label("total"),
func.count().filter(Prediction.created_at >= day_ago).label("last_24h"),
func.count().filter(Prediction.created_at >= week_ago).label("last_7d"),
)
)
).one()
# F3 修复: 补充真实比赛计数(非 limit=100 近似)
match_cnt = (await db.execute(select(func.count()).select_from(Match))).scalar() or 0
finished_cnt = (await db.execute(select(func.count()).where(Match.match_status == "finished"))).scalar() or 0
stats_cnt = (await db.execute(select(func.count()).select_from(MatchStats))).scalar() or 0
standings_cnt = (await db.execute(select(func.count()).select_from(Standing))).scalar() or 0
return {
"predictions": {"total": r.total, "last_24h": r.last_24h, "last_7d": r.last_7d},
"matches": {"total": match_cnt, "finished": finished_cnt},
"stats": {"total": stats_cnt},
"standings": {"total": standings_cnt},
}
# ── 数据完整性分析(可视化数据源) ────────────────────────────────
@router.get("/data-completeness")
async def data_completeness(db: AsyncSession = Depends(get_db_read)):
"""按联赛统计数据完整性:比赛覆盖、字段覆盖、积分榜覆盖。
前端「数据完整性」页据此渲染,回答三个问题:
1. 数据是否齐全(各联赛比赛/统计/积分榜量级)
2. 字段是否齐全(每张统计表各字段非空率)
3. 覆盖是否新鲜(最近一场/最近一次采集)
"""
from src.data.config import BZZOIRO_LEAGUE_IDS, LEAGUE_NAMES, LEAGUE_COUNTRIES
out_leagues: list[dict] = []
for code, bzz_id in BZZOIRO_LEAGUE_IDS.items():
# 比赛覆盖
m = (
await db.execute(
select(
func.count().label("total"),
func.count().filter(Match.match_status == "finished").label("finished"),
func.count().filter(Match.match_status == "scheduled").label("scheduled"),
func.count().filter(Match.source_event_id.is_not(None)).label("with_source_id"),
func.max(Match.match_date).label("latest_match"),
func.min(Match.match_date).label("earliest_match"),
)
.select_from(Match)
.join(League, League.id == Match.league_id)
.where(League.code == code)
)
).one()
# 统计字段覆盖(联表 matches)
s = (
await db.execute(
select(
func.count().label("rows"),
func.count(MatchStats.home_xg).label("xg"),
func.count(MatchStats.home_shots).label("shots"),
func.count(MatchStats.home_possession).label("possession"),
func.count(MatchStats.home_corners).label("corners"),
func.count(MatchStats.home_fouls).label("fouls"),
func.count(MatchStats.home_big_chances).label("big_chances"),
func.count(MatchStats.home_yellow_cards).label("cards"),
)
.select_from(MatchStats)
.join(Match, Match.id == MatchStats.match_id)
.join(League, League.id == Match.league_id)
.where(League.code == code)
)
).one()
# 积分榜覆盖
st = (
await db.execute(
select(
func.count().label("rows"),
func.max(Standing.retrieved_at).label("latest_retrieved"),
)
.select_from(Standing)
.join(League, League.id == Standing.league_id)
.where(League.code == code)
)
).one()
stats_rows = s.rows or 0
pct = lambda n: round(n / stats_rows * 100, 1) if stats_rows else 0.0 # noqa: E731
out_leagues.append(
{
"code": code,
"name": LEAGUE_NAMES.get(code, code),
"country": LEAGUE_COUNTRIES.get(code),
"matches": {
"total": m.total or 0,
"finished": m.finished or 0,
"scheduled": m.scheduled or 0,
"with_source_id": m.with_source_id or 0,
"earliest_match": m.earliest_match.isoformat() if m.earliest_match else None,
"latest_match": m.latest_match.isoformat() if m.latest_match else None,
},
"stats": {
"rows": stats_rows,
"fields": {
"xg": {"count": s.xg or 0, "pct": pct(s.xg or 0)},
"shots": {"count": s.shots or 0, "pct": pct(s.shots or 0)},
"possession": {"count": s.possession or 0, "pct": pct(s.possession or 0)},
"corners": {"count": s.corners or 0, "pct": pct(s.corners or 0)},
"fouls": {"count": s.fouls or 0, "pct": pct(s.fouls or 0)},
"big_chances": {"count": s.big_chances or 0, "pct": pct(s.big_chances or 0)},
"cards": {"count": s.cards or 0, "pct": pct(s.cards or 0)},
},
},
"standings": {
"rows": st.rows or 0,
"latest_retrieved": st.latest_retrieved.isoformat() if st.latest_retrieved else None,
},
}
)
# 整体健康信号
total_finished = sum(l["matches"]["finished"] for l in out_leagues)
total_stats = sum(l["stats"]["rows"] for l in out_leagues)
stats_coverage = round(total_stats / total_finished * 100, 1) if total_finished else 0.0
issues: list[str] = []
for l in out_leagues:
if l["matches"]["finished"] == 0:
issues.append(f"{l['name']}: 无已完赛比赛,请先运行「比赛数据」采集")
elif l["stats"]["rows"] == 0:
issues.append(f"{l['name']}: 已完赛 {l['matches']['finished']} 场但无统计回填,请运行「统计回填」采集")
elif stats_coverage < 80:
issues.append(f"{l['name']}: 统计覆盖率仅 {stats_coverage}%,建议增量回填")
if l["standings"]["rows"] == 0:
issues.append(f"{l['name']}: 无积分榜数据,请运行「积分榜」采集")
if not issues:
issues.append("各联赛数据完整度良好")
return {
"generated_at": datetime.now(timezone.utc).isoformat(),
"leagues": out_leagues,
"totals": {
"finished_matches": total_finished,
"stats_rows": total_stats,
"stats_coverage_pct": stats_coverage,
},
"issues": issues,
}
# ── 数据质量检查 API ────────────────────────────────────────────
@router.get("/data-quality")
async def data_quality_checks(db: AsyncSession = Depends(get_db_read)):
"""数据质量检查结果(只读)。"""
from src.db.models import IngestFailure, DataQualityCheck
from sqlalchemy import func
# 最近的失败记录
failures = (
await db.execute(
select(IngestFailure)
.where(IngestFailure.status.in_(["pending", "retrying"]))
.order_by(IngestFailure.created_at.desc())
.limit(20)
)
).scalars().all()
# 最近的质量检查
checks = (
await db.execute(
select(DataQualityCheck)
.order_by(DataQualityCheck.checked_at.desc())
.limit(20)
)
).scalars().all()
return {
"failures": [
{
"id": f.id,
"source": f.source_system,
"entity_type": f.entity_type,
"source_record_id": f.source_record_id,
"error_type": f.error_type,
"error_detail": f.error_detail,
"retry_count": f.retry_count,
"status": f.status,
"created_at": f.created_at.isoformat() if f.created_at else None,
}
for f in failures
],
"checks": [
{
"id": c.id,
"check_name": c.check_name,
"entity_type": c.entity_type,
"passed": c.passed,
"severity": c.severity,
"detail": c.detail,
"checked_at": c.checked_at.isoformat() if c.checked_at else None,
}
for c in checks
],
}
@router.post("/data-quality/run")
async def run_data_quality_check(db: AsyncSession = Depends(get_db_read)):
"""手动触发一次数据质量检查。"""
from src.db.models import DataQualityCheck, Match, MatchStats, Standing, League
from sqlalchemy import func
checks = []
# 检查1: 已完赛但无统计的比赛
finished_no_stats = (
await db.execute(
select(func.count())
.select_from(Match)
.outerjoin(MatchStats, Match.id == MatchStats.match_id)
.where(Match.match_status == "finished")
.where(MatchStats.id.is_(None))
)
).scalar() or 0
checks.append(DataQualityCheck(
check_name="finished_without_stats",
entity_type="match",
actual_value=float(finished_no_stats),
passed=finished_no_stats == 0,
severity="warning" if finished_no_stats > 0 else "info",
detail={"message": f"{finished_no_stats} 场已完赛比赛缺少统计数据"},
))
# 检查2: 积分榜缺失的联赛
leagues_without_standings = (
await db.execute(
select(func.count())
.select_from(League)
.outerjoin(Standing, League.id == Standing.league_id)
.where(Standing.id.is_(None))
)
).scalar() or 0
checks.append(DataQualityCheck(
check_name="league_without_standings",
entity_type="league",
actual_value=float(leagues_without_standings),
passed=leagues_without_standings == 0,
severity="warning" if leagues_without_standings > 0 else "info",
detail={"message": f"{leagues_without_standings} 个联赛缺少积分榜"},
))
for c in checks:
db.add(c)
await db.commit()
return {"ok": True, "checks": [{"name": c.check_name, "passed": c.passed} for c in checks]}
from fastapi import APIRouter
from src.api.routes.admin_config import router as admin_config_router
from src.api.routes.admin_datasources import router as admin_datasources_router
from src.api.routes.admin_llm import router as admin_llm_router
from src.api.routes.admin_quality import router as admin_quality_router
router = APIRouter()
router.include_router(admin_datasources_router)
router.include_router(admin_config_router)
router.include_router(admin_llm_router)
router.include_router(admin_quality_router)
+2 -1
View File
@@ -16,7 +16,8 @@ from fastapi import APIRouter, Depends, HTTPException
from src.api.deps import require_admin
from src.api.schemas import IngestBzzoiroRequest
from src.data.config import BZZOIRO_LEAGUE_IDS
from src.data.bzzoiro import ingest_bzzoiro_event_stats, ingest_bzzoiro_standings
from src.data.bzzoiro_standings import ingest_bzzoiro_standings
from src.data.bzzoiro_stats import ingest_bzzoiro_event_stats
from src.data.sources import get_source
from src.db.unit_of_work import get_uow
+4 -4
View File
@@ -7,7 +7,6 @@ from fastapi import APIRouter, Depends, HTTPException, Query
from sqlalchemy import func, or_, select
from sqlalchemy.orm import selectinload
from src.api.deps import require_admin
from src.api.schemas import MatchListOut, MatchOut, PredictionOut
from src.db.base import AsyncSession, get_db_read
from src.db.models import League, Match, Prediction, Standing
@@ -32,8 +31,9 @@ def _stats_dict(stats) -> dict | None:
}
@router.get("/leagues", response_model=list[dict], dependencies=[Depends(require_admin)])
@router.get("/leagues", response_model=list[dict])
async def list_leagues(db: AsyncSession = Depends(get_db_read)):
"""联赛列表(公开只读,P1-3: 公开站联赛筛选需要;仅返回展示字段)。"""
stmt = select(League).order_by(League.name)
result = await db.execute(stmt)
leagues = result.scalars().all()
@@ -190,9 +190,9 @@ async def get_match(match_id: int, db: AsyncSession = Depends(get_db_read)):
)
@router.get("/matches/{match_id}/context", dependencies=[Depends(require_admin)])
@router.get("/matches/{match_id}/context")
async def match_context(match_id: int, db: AsyncSession = Depends(get_db_read)):
"""比赛上下文(只读,不触发 LLM):双方近况 + 历史交锋。
"""比赛上下文(公开只读,P1-2: 公开站详情页需要;不触发 LLM):双方近况 + 历史交锋。
全部基于现有数据聚合:
- recent_home / recent客队:该队最近 5 场已完赛(进球/结果)
+22 -57
View File
@@ -63,77 +63,42 @@ async def predict(req: PredictRequest, request: Request):
logger.exception("predict unexpected error")
raise HTTPException(500, "预测失败,请查看服务器日志")
# baseline 模式:结果已是 dict,需独立落库(prediction_id)
if req.mode == "baseline":
prediction_id = await _persist_baseline(req.match_id, result)
else:
prediction_id = result.prediction_id
# D2: 三种模式统一返回 PredictResult —— 字段映射单一化,无 dict 分支。
# P3-2:baseline 已在服务层(predict_baseline)落库并回填真实 prediction_id,
# 路由层不再需要特殊的 _persist_baseline,与 single/multi 路径统一。
prediction_id = result.prediction_id
# 3. 结果映射(无 DB 访问)
logger.info(
"预测完成 match=%s mode=%s pred=%s:%s (%s)",
req.match_id, req.mode,
result.get("pred_home_goals") if isinstance(result, dict) else result.pred_home_goals,
result.get("pred_away_goals") if isinstance(result, dict) else result.pred_away_goals,
result.get("pred_1x2") if isinstance(result, dict) else result.pred_1x2,
result.pred_home_goals, result.pred_away_goals, result.pred_1x2,
)
result_dict = result if isinstance(result, dict) else None
return PredictOut(
prediction_id=prediction_id,
provider=result.get("provider") if result_dict else result.provider,
model=result.get("model") if result_dict else result.model,
prompt_version=result.get("prompt_version") if result_dict else getattr(result, "prompt_version", None),
provider=result.provider,
model=result.model,
prompt_version=result.prompt_version,
mode=req.mode,
pred_home_goals=result.get("pred_home_goals") if result_dict else result.pred_home_goals,
pred_away_goals=result.get("pred_away_goals") if result_dict else result.pred_away_goals,
alt_pred_home_goals=result.get("alt_pred_home_goals") if result_dict else result.alt_pred_home_goals,
alt_pred_away_goals=result.get("alt_pred_away_goals") if result_dict else result.alt_pred_away_goals,
pred_1x2=result.get("pred_1x2") if result_dict else result.pred_1x2,
subjective_confidence=result.get("subjective_confidence") if result_dict else result.subjective_confidence,
reasoning=result.get("reasoning") if result_dict else result.reasoning,
status=result.get("status", "success") if result_dict else getattr(result, "status", "success"),
agent_outputs=result.get("agent_outputs") if result_dict else getattr(result, "agent_outputs", None),
agent_weights=result.get("agent_weights") if result_dict else getattr(result, "agent_weights", None),
context=result.get("context", "") if result_dict else result.context,
latency_ms=result.get("latency_ms", 0) if result_dict else result.latency_ms,
prompt_tokens=result.get("prompt_tokens") if result_dict else getattr(result, "prompt_tokens", None),
completion_tokens=result.get("completion_tokens") if result_dict else getattr(result, "completion_tokens", None),
pred_home_goals=result.pred_home_goals,
pred_away_goals=result.pred_away_goals,
alt_pred_home_goals=result.alt_pred_home_goals,
alt_pred_away_goals=result.alt_pred_away_goals,
pred_1x2=result.pred_1x2,
subjective_confidence=result.subjective_confidence,
reasoning=result.reasoning,
status=result.status,
agent_outputs=result.agent_outputs,
agent_weights=result.agent_weights,
context=result.context,
latency_ms=result.latency_ms,
prompt_tokens=result.prompt_tokens,
completion_tokens=result.completion_tokens,
rate_limit_remaining=get_predict_rate_limit_remaining(request),
)
async def _persist_baseline(match_id: int, baseline: dict) -> int:
"""将基线预测结果写入 prediction 表,复用 upsert 语义。"""
from src.db.unit_of_work import get_uow
from src.llm.predict import _upsert_prediction
async with get_uow() as session:
pred = await _upsert_prediction(
session,
match_id=match_id,
provider_name="baseline",
model="baseline",
mode="baseline",
run_type="live", # baseline 是 live 预测的变体,符合 ck_run_type_enum
values={
"prompt_version": "baseline_v1",
"prompt_tokens": 0,
"completion_tokens": 0,
"latency_ms": 0,
"pred_home_goals": baseline["pred_home_goals"],
"pred_away_goals": baseline["pred_away_goals"],
"pred_1x2": baseline["pred_1x2"],
"subjective_confidence": baseline["subjective_confidence"],
"reasoning": baseline["reasoning"],
"raw_response": baseline.get("raw", baseline),
"status": "success",
},
)
return pred.id
@router.get("/predictions", response_model=list[PredictionOut], dependencies=[Depends(require_admin)])
async def list_predictions(
match_id: int | None = None,
+2 -1
View File
@@ -10,7 +10,8 @@ from sqlalchemy import select, delete
from src.api.deps import require_admin
from src.api.schemas import ScheduleIn, ScheduleUpdate, ScheduleOut
from src.core.scheduler import scheduler
from src.data.bzzoiro import ingest_bzzoiro_event_stats, ingest_bzzoiro_standings
from src.data.bzzoiro_standings import ingest_bzzoiro_standings
from src.data.bzzoiro_stats import ingest_bzzoiro_event_stats
from src.data.sources import get_source
from src.data.config import BZZOIRO_LEAGUE_IDS
from src.db.base import AsyncSession, get_db_read
+4
View File
@@ -11,6 +11,10 @@ class Settings(BaseSettings):
# --- app ---
APP_ENV: str = "development"
LOG_LEVEL: str = "INFO"
# P3-3:多 worker 时应用内限流与 KeyRing 各自独立计数(配额放大 N 倍)。
# 设为 True 时若以多 worker 启动 uvicorn 则拒绝启动,避免静默配额漂移。
# 仅在你已前置 Nginx/网关做全局限流、确认不需要此守护时留空/False。
STRICT_SINGLE_WORKER: bool = False
# 生产环境强制要求管理鉴权配置,即使 APP_ENV=production 也生效。
# True 时若 auth_configured() 为 False 则拒绝(503),development 保持 fail-open。
REQUIRE_ADMIN_AUTH: bool = False
+66 -728
View File
@@ -1,733 +1,71 @@
"""Bzzoiro 数据源:抓取 + 入库(单一数据源)。
"""Bzzoiro 数据源:抓取 + 入库(单一数据源)—— 聚合门面
三条管线:
1. events — 比赛日程/比分(/events/),含 source_event_id 血缘
2. standings— 联赛积分榜快照(/leagues/{id}/standings/)
3. stats — 已完赛比赛详细统计回填(/events/{id}/stats/)
实现按管线拆分(单文件 → 多模块),本模块只做再导出,保持两个不变量:
1. sources._load_sources() 仍从本模块导入 BzzoiroSource(注册表入口不变);
2. 测试与脚本对 `bz.<名称>` 的 monkeypatch 语义不变 —— 子模块在运行期
经本门面解析可替换协作者(抓取函数 / Bronze 写入助手 / REQUEST_INTERVAL),
与拆分前的单文件行为一致。
使用 Repository 模式进行数据访问,不直接控制事务(由调用方 UnitOfWork 控制)。
三条管线(各自模块):
1. events — 比赛日程/比分(/events/),含 source_event_id 血缘 → bzzoiro_events.py
2. standings— 联赛积分榜快照(/leagues/{id}/standings/) → bzzoiro_standings.py
3. stats — 已完赛比赛详细统计回填(/events/{id}/stats/) → bzzoiro_stats.py
共享基础:HTTP 抓取(多 key 轮换)与字段转换 → bzzoiro_common.py;
Bronze 基础设施(RawEvent/IngestFailure/DataLineage)→ pipeline_write.py。
D4(工程债): Team/League/Match 的查找/创建经 Repository 层(src/db/repositories.py),
各管线不直接控制事务(commit/rollback 由调用方 UnitOfWork 控制,只 flush)。
Standing/RawEvent/Lineage 等管线内私有读写仍不强行 Repository 化。
"""
from __future__ import annotations
import asyncio
import logging
import random
from collections.abc import Iterable
from datetime import datetime, timedelta, timezone
from sqlalchemy import select
from sqlalchemy.orm import selectinload
import httpx
from src.core.runtime_config import get_runtime_value
from src.core.http_client import get_client
from src.data.config import BZZOIRO_LEAGUE_IDS, LEAGUE_COUNTRIES, LEAGUE_NAMES, REQUEST_INTERVAL
from src.data.key_ring import _mask, get_key_ring
from src.data.normalize import normalize_bzzoiro
from src.data.team_names_zh import zh_name
from src.data.sources import register
from src.db.models import League, Match, MatchStats, Standing, Team, RawEvent, IngestFailure, DataLineage
logger = logging.getLogger(__name__)
def _to_date(value):
"""把 datetime / date / str 统一成 `date`。"""
if value is None:
return None
if hasattr(value, "date") and callable(value.date):
return value.date()
return value
def _to_int_or_none(value) -> int | None:
"""宽松转 int(用于上游 ID 解析,失败返回 None 不抛错)。"""
if value is None:
return None
try:
return int(str(value).strip())
except (TypeError, ValueError):
return None
def _match_key(home_team_id: int, away_team_id: int, match_date) -> tuple[int, int, str]:
"""比赛去重键:(主队, 客队, 天级日期 ISO 字符串)。
统一在这里构造,避免"预加载时用 str(date)、写入时用 isoformat()"这类
隐式格式依赖 —— 两者当前恰好相等,但一旦有人改动其一就会静默失配,
导致所有比赛被判为不存在而重复插入。
"""
d = _to_date(match_date)
return (home_team_id, away_team_id, d.isoformat() if d is not None else "")
async def _fetch_json_async(path: str, params: dict | None = None, max_retries: int = 3) -> dict | list:
"""异步 HTTP(bzzoiro 使用 httpx,不再阻塞事件循环线程池)。
多 key 轮换:遇到 429 自动切换到下一个 key;全部 key 冷却时等待最早恢复。
"""
base = (await get_runtime_value("BZZOIRO_BASE")).rstrip("/")
raw_keys = await get_runtime_value("BZZOIRO_KEY")
ring = get_key_ring(base, raw_keys)
url = f"{base}/{path.lstrip('/')}"
key = ring.get()
if not key:
raise RuntimeError("BZZOIRO_KEY 未设置")
last_exc: Exception | None = None
for attempt in range(max_retries):
headers = {
"Authorization": f"Token {key}",
"Accept": "application/json",
}
try:
client = get_client()
# 整请求兜底: httpx 无 total 超时,用 wait_for 防「滴水式」限速挂死
resp = await asyncio.wait_for(
client.get(
url, headers=headers, params=params,
timeout=httpx.Timeout(connect=10.0, read=30.0, write=10.0, pool=10.0),
),
timeout=60.0,
)
resp.raise_for_status()
return resp.json()
except Exception as e:
last_exc = e
status = getattr(getattr(e, "response", None), "status_code", None)
if status == 429:
# 限流:标记当前 key 冷却,切换到下一个
new_key = ring.report_rate_limited(key)
if new_key and new_key != key:
logger.info("bzzoiro 429 → 切换 key: %s%s,立即重试", _mask(key), _mask(new_key))
key = new_key
continue # 立即重试,不等待
# 单 key 或全部冷却:等待最早恢复的 key
wait = ring.wait_if_all_blocked()
if wait > 0:
logger.warning("bzzoiro 全部 key 冷却,等待 %.1fs 后重试", wait)
await asyncio.sleep(min(wait, 30.0))
else:
delay = min(2 ** attempt, 16) + random.uniform(0, 1)
logger.warning("bzzoiro 429, retry %d in %.1fs", attempt + 1, delay)
await asyncio.sleep(delay)
key = ring.get() or key
continue
if 500 <= (status or 0) < 600:
delay = min(2 ** attempt, 16) + random.uniform(0, 1)
logger.warning("bzzoiro %d, retry %d in %.1fs", status, attempt + 1, delay)
await asyncio.sleep(delay)
continue
# 网络错误(连接失败/超时)也退避重试
if isinstance(e, (TimeoutError, ConnectionError, OSError)):
delay = min(2 ** attempt, 16) + random.uniform(0, 1)
logger.warning("bzzoiro network error, retry %d in %.1fs: %s", attempt + 1, delay, e)
await asyncio.sleep(delay)
continue
raise
raise RuntimeError(f"bzzoiro request failed after {max_retries} attempts: {last_exc}")
async def fetch_bzzoiro_events(
league_code: str,
*,
status: str = "finished",
date_from: str | None = None,
date_to: str | None = None,
limit: int = 200,
) -> list[dict]:
"""抓取 bzzoiro 原始事件(纯异步,无需 run_in_executor)。"""
league_id = BZZOIRO_LEAGUE_IDS.get(league_code)
if league_id is None:
raise ValueError(f"未知联赛代码: {league_code}")
rows: list[dict] = []
offset = 0
while True:
params: dict = {
"league_id": league_id,
"status": status,
"limit": limit,
"offset": offset,
}
if date_from:
params["date_from"] = str(date_from)[:10]
if date_to:
params["date_to"] = str(date_to)[:10]
payload = await _fetch_json_async("/events/", params)
batch = payload.get("results") or []
if not batch:
break
rows.extend(batch)
total = payload.get("total")
offset += limit
if total is not None and offset >= total:
break
if len(batch) < limit:
break
await asyncio.sleep(REQUEST_INTERVAL)
return rows
@register
class BzzoiroSource:
"""bzzoiro 数据源(实现 DataSource 协议)。"""
name = "bzzoiro"
async def ingest(
self,
db,
*,
leagues: Iterable[str],
date_from: str | None = None,
date_to: str | None = None,
status: str = "finished",
) -> dict:
"""采集 bzzoiro → 入库。返回统计。
注意: 本方法不控制事务(commit/rollback),由调用方通过 UnitOfWork 控制。
"""
result: dict = {"leagues": {}, "total_inserted": 0, "total_updated": 0, "errors": []}
for code in leagues:
league_r: dict = {"inserted": 0, "updated": 0, "errors": []}
try:
raw_events = await fetch_bzzoiro_events(code, status=status, date_from=date_from, date_to=date_to)
except Exception as e:
# 单联赛抓取失败隔离:记录错误后继续其余联赛,不拖垮整批
logger.exception("bzzoiro fetch failed for %s", code)
league_r["errors"].append(f"fetch failed: {e}")
await _safe_write_ingest_failure(
db,
entity_type="events",
source_record_id=None,
error=e,
raw_payload={"league": code, "status": status, "date_from": date_from, "date_to": date_to},
)
result["leagues"][code] = league_r
continue
# 获取或创建联赛
stmt = select(League).where(League.code == code)
league = (await db.execute(stmt)).scalar_one_or_none()
if league is None:
league = League(code=code, name=LEAGUE_NAMES.get(code, code), country=LEAGUE_COUNTRIES.get(code))
db.add(league)
await db.flush()
# === 批量优化: 预加载球队和已有比赛到内存 ===
team_name_to_id: dict[str, int] = {}
existing_matches: dict[tuple[int, int, str], Match] = {} # 完整对象,避免重复查询
# (NormalizedMatch, 原始 event) 成对保存:后续写 source_event_id 时
# 必须用配对的那条 event,不能依赖外层循环变量残留值。
normalized_matches: list[tuple] = []
if raw_events:
# 一次遍历: 收集球队名 + 规范化
all_team_names = set()
for raw in raw_events:
nm = normalize_bzzoiro(raw, code)
if nm is not None:
try:
nm.validate()
except Exception as e:
# P1-3: 统一使用 warning,不追加到 errors(仅运行时错误入 errors)
logger.warning("normalize skip: %s", e)
continue
normalized_matches.append((nm, raw))
all_team_names.add(nm.home_team)
all_team_names.add(nm.away_team)
if all_team_names:
stmt = select(Team).where(Team.name.in_(all_team_names))
teams = (await db.execute(stmt)).scalars().all()
team_name_to_id = {t.name: t.id for t in teams}
# P1-2: 按需加载,只加载 raw_events 涉及日期范围的比赛(加 30 天缓冲)
# 避免加载联赛全部历史比赛到内存(多赛季采集时内存溢出)
if normalized_matches:
# normalized_matches 存的是 (nm, raw) 元组,遍历需解包
dates = [nm.date for nm, _raw in normalized_matches if nm.date is not None]
if dates:
min_dt = min(dates) - timedelta(days=30)
max_dt = max(dates) + timedelta(days=30)
stmt = (
select(Match)
.where(Match.league_id == league.id)
.where(Match.match_date >= min_dt)
.where(Match.match_date <= max_dt)
)
existing_matches = {
_match_key(m.home_team_id, m.away_team_id, m.match_date_date): m
for m in (await db.execute(stmt)).scalars()
}
# else: existing_matches 保持空 dict(全量新比赛)
for nm, raw in normalized_matches:
# 球队: 内存查找 + 按需创建
home_team_id = team_name_to_id.get(nm.home_team)
if home_team_id is None:
home = Team(name=nm.home_team, name_zh=zh_name(nm.home_team))
db.add(home)
await db.flush()
home_team_id = home.id
team_name_to_id[nm.home_team] = home_team_id
away_team_id = team_name_to_id.get(nm.away_team)
if away_team_id is None:
away = Team(name=nm.away_team, name_zh=zh_name(nm.away_team))
db.add(away)
await db.flush()
away_team_id = away.id
team_name_to_id[nm.away_team] = away_team_id
# 查找已有比赛: 内存查找
match_key = _match_key(home_team_id, away_team_id, nm.date)
existing_match = existing_matches.get(match_key)
if existing_match is None:
m = Match(
league_id=league.id,
season=nm.season_label or None,
home_team_id=home_team_id,
away_team_id=away_team_id,
match_date=nm.date,
match_date_date=_to_date(nm.date),
match_status=nm.match_status,
home_goals=nm.home_goals,
away_goals=nm.away_goals,
home_ht_goals=nm.home_ht_goals,
away_ht_goals=nm.away_ht_goals,
match_stage=nm.match_stage,
source_event_id=_to_int_or_none(raw.get("id")),
)
db.add(m)
await db.flush()
existing_matches[match_key] = m # 防止同批重复
# 统计字段不在 /events/ 载荷中(单独由 stats 管线回填),
# 此处不再创建 MatchStats。
league_r["inserted"] += 1
else:
# 已有比赛: 直接从内存获取对象更新(无需再查询)
changed = False
if existing_match.match_status != nm.match_status and nm.match_status == "finished":
existing_match.match_status = nm.match_status
changed = True
if existing_match.home_goals is None and nm.home_goals is not None:
existing_match.home_goals = nm.home_goals
existing_match.away_goals = nm.away_goals
existing_match.home_ht_goals = nm.home_ht_goals
existing_match.away_ht_goals = nm.away_ht_goals
changed = True
if existing_match.match_stage is None and nm.match_stage:
existing_match.match_stage = nm.match_stage
changed = True
if existing_match.source_event_id is None:
eid = _to_int_or_none(raw.get("id"))
if eid is not None:
existing_match.source_event_id = eid
changed = True
if changed:
league_r["updated"] += 1
# 注意: 不在此处 commit,由调用方 UnitOfWork 控制事务
result["leagues"][code] = league_r
result["total_inserted"] += league_r["inserted"]
result["total_updated"] += league_r["updated"]
return result
# ============================================================
# 管线基础设施:RawEvent / IngestFailure / DataLineage
# ============================================================
async def _write_raw_event(db, source_system: str, source_record_id: str, raw_payload: dict, batch_id: str | None = None) -> None:
"""写入 Bronze 层原始事件(幂等:同 source_record_id 跳过)。"""
from sqlalchemy import select as _select
stmt = _select(RawEvent).where(
RawEvent.source_system == source_system,
RawEvent.source_record_id == source_record_id,
)
existing = (await db.execute(stmt)).scalar_one_or_none()
if existing is None:
db.add(RawEvent(
source_system=source_system,
source_record_id=source_record_id,
raw_payload=raw_payload,
ingest_batch_id=batch_id,
))
async def _write_ingest_failure(db, source_system: str, entity_type: str, source_record_id: str | None, error_type: str, error_detail: str | None, raw_payload: dict | None = None) -> None:
"""写入采集失败死信。"""
db.add(IngestFailure(
source_system=source_system,
entity_type=entity_type,
source_record_id=source_record_id,
error_type=error_type,
error_detail=error_detail,
raw_payload=raw_payload,
))
async def _safe_write_ingest_failure(
db,
*,
entity_type: str,
source_record_id: str | None,
error: Exception,
raw_payload: dict | None = None,
) -> None:
"""抓取失败时尽力写入死信表(失败不影响主流程)。
死信是「可观测性」基础设施,与 RawEvent/Lineage 同级:写入失败只记
warning,绝不能让原始抓取错误之外的新异常打断采集循环。
"""
try:
await _write_ingest_failure(
db, "bzzoiro", entity_type, source_record_id,
"fetch_error", str(error), raw_payload,
)
except Exception:
logger.warning(
"写入 ingest_failures 死信失败(entity=%s, record=%s): %s",
entity_type, source_record_id, error, exc_info=True,
)
async def _write_lineage(db, source_system: str, source_record_id: str, target_table: str, target_id: int | None, transform_name: str, transform_detail: dict | None = None, batch_id: str | None = None) -> None:
"""写入 ETL 血缘追踪。"""
db.add(DataLineage(
source_system=source_system,
source_record_id=source_record_id,
target_table=target_table,
target_id=target_id,
transform_name=transform_name,
transform_detail=transform_detail,
batch_id=batch_id,
))
# ============================================================
# 积分榜管线:/leagues/{id}/standings/ → standings 表
# ============================================================
async def fetch_bzzoiro_standings(league_code: str, season: str | None = None) -> dict:
"""抓取联赛积分榜(纯抓取,不入库)。season 为 None 时取当前赛季。"""
league_id = BZZOIRO_LEAGUE_IDS.get(league_code)
if league_id is None:
raise ValueError(f"未知联赛代码: {league_code}")
params: dict = {}
if season:
params["season"] = season
return await _fetch_json_async(f"/leagues/{league_id}/standings/", params)
def _season_label_from_dates(start_date, end_date) -> str:
"""从赛季起止日期推导赛季标签(与 derive_season_label 语义一致)。"""
try:
if isinstance(start_date, str):
start = datetime.fromisoformat(start_date[:10])
else:
start = start_date
y = start.year
return f"{y}-{y + 1}" if start.month >= 8 else f"{y - 1}-{y}"
except (TypeError, ValueError):
return "?"
async def ingest_bzzoiro_standings(db, *, leagues: Iterable[str], season: str | None = None) -> dict:
"""采集积分榜 → upsert standings 表。
season 为 None 时采集当前赛季(bzzoiro 默认返回 is_current 赛季)。
球队名与 events 管线使用同一 normalize 规则,保证 Team 匹配。
"""
from src.data.team_names import normalize as normalize_name
result: dict = {"leagues": {}, "total_upserted": 0, "errors": []}
for code in leagues:
league_r: dict = {"upserted": 0, "teams_created": 0, "rows": 0, "errors": []}
try:
payload = await fetch_bzzoiro_standings(code, season=season)
except Exception as e:
logger.exception("bzzoiro standings fetch failed for %s", code)
league_r["errors"].append(str(e))
await _safe_write_ingest_failure(
db,
entity_type="standings",
source_record_id=None,
error=e,
raw_payload={"league": code, "season": season},
)
result["leagues"][code] = league_r
result["errors"].append(f"{code}: {e}")
continue
rows = payload.get("standings") or []
if not rows:
result["leagues"][code] = {"error": "无积分榜数据(赛季未开始或未提供)"}
result["errors"].append(f"{code}: 无积分榜数据")
continue
# 联赛(get-or-create)
stmt = select(League).where(League.code == code)
league = (await db.execute(stmt)).scalar_one_or_none()
if league is None:
league = League(code=code, name=LEAGUE_NAMES.get(code, code), country=LEAGUE_COUNTRIES.get(code))
db.add(league)
await db.flush()
# 赛季标签:优先用返回的 season 对象推导
season_obj = payload.get("season") or {}
season_label = _season_label_from_dates(
season_obj.get("start_date"), season_obj.get("end_date")
)
if season_label == "?":
season_label = season or ""
# 批量预载球队(与 events 管线使用同一 normalize 规则,保证 Team 匹配)
names = {normalize_name(str(r.get("team_name", ""))) for r in rows}
names.discard("")
team_map: dict[str, Team] = {}
if names:
stmt = select(Team).where(Team.name.in_(names))
for t in (await db.execute(stmt)).scalars():
team_map[t.name] = t
now = datetime.now(timezone.utc)
for r in rows:
team_name = normalize_name(str(r.get("team_name", "")))
if not team_name:
continue
team = team_map.get(team_name)
if team is None:
team = Team(name=team_name, name_zh=zh_name(team_name))
db.add(team)
await db.flush()
team_map[team_name] = team
league_r["teams_created"] += 1
zone = r.get("zone") or {}
values = dict(
position=_to_int_or_none(r.get("position")) or 0,
played=_to_int_or_none(r.get("played")) or 0,
won=_to_int_or_none(r.get("won")) or 0,
drawn=_to_int_or_none(r.get("drawn")) or 0,
lost=_to_int_or_none(r.get("lost")) or 0,
goals_for=_to_int_or_none(r.get("gf")) or 0,
goals_against=_to_int_or_none(r.get("ga")) or 0,
goal_diff=_to_int_or_none(r.get("gd")) or 0,
points=_to_int_or_none(r.get("pts")) or 0,
xg_for=_to_float_or_none(r.get("xgf")),
xg_against=_to_float_or_none(r.get("xga")),
form=r.get("form") or None,
zone=zone.get("label") or zone.get("key") or None,
updated_at=now,
retrieved_at=now,
)
# 同一联赛同一赛季只保留最新快照:按 (league, season, team) upsert
stmt = select(Standing).where(
Standing.league_id == league.id,
Standing.season == season_label,
Standing.team_id == team.id,
)
standing = (await db.execute(stmt)).scalar_one_or_none()
if standing is None:
standing = Standing(
league_id=league.id, season=season_label, team_id=team.id, **values
)
db.add(standing)
else:
for k, v in values.items():
setattr(standing, k, v)
league_r["upserted"] += 1
league_r["rows"] = len(rows)
result["leagues"][code] = league_r
result["total_upserted"] += league_r["upserted"]
logger.info(
"bzzoiro standings 采集完成: %s 赛季 %s, upsert %d/%d",
code, season_label, league_r["upserted"], league_r["rows"],
)
return result
# ============================================================
# 统计回填管线:/events/{id}/stats/ → match_stats 表
# ============================================================
# bzzoiro stats 字段 → MatchStats 字段映射(stats.home / stats.away 下)
_STATS_FIELD_MAP = {
"xg": ("home_xg", "away_xg"), # 回退 expected_goals
"ball_possession": ("home_possession", None), # 只取主队值,客队=100-home
"total_shots": ("home_shots", "away_shots"),
"shots_on_target": ("home_shots_on_target", "away_shots_on_target"),
"corner_kicks": ("home_corners", "away_corners"),
"yellow_cards": ("home_yellow_cards", "away_yellow_cards"),
"red_cards": ("home_red_cards", "away_red_cards"),
"big_chances": ("home_big_chances", "away_big_chances"),
"fouls": ("home_fouls", "away_fouls"),
}
def _pick(d: dict, *keys):
"""按优先级取第一个非空字段值。"""
for k in keys:
v = d.get(k)
if v is not None:
return v
return None
def _stats_from_payload(payload: dict) -> dict:
"""把 /events/{id}/stats/ 响应映射成 MatchStats 字段 dict。
响应结构: {"event_id": ..., "stats": {"home": {...}, "away": {...}}}
"""
stats = (payload or {}).get("stats") or {}
home = stats.get("home") or {}
away = stats.get("away") or {}
out: dict = {}
xg_h = _pick(home, "xg", "expected_goals")
xg_a = _pick(away, "xg", "expected_goals")
if xg_h is not None:
out["home_xg"] = _to_float_or_none(xg_h)
if xg_a is not None:
out["away_xg"] = _to_float_or_none(xg_a)
poss = home.get("ball_possession")
if poss is not None:
p = _to_float_or_none(poss)
if p is not None:
out["home_possession"] = p
for src, (h_fld, a_fld) in _STATS_FIELD_MAP.items():
if src in ("xg", "ball_possession"):
continue # 已处理
hv = home.get(src)
av = away.get(src)
if hv is not None and h_fld:
out[h_fld] = _to_int_or_none(hv)
if av is not None and a_fld:
out[a_fld] = _to_int_or_none(av)
return out
def _to_float_or_none(value) -> float | None:
if value is None:
return None
try:
return float(str(value).strip())
except (TypeError, ValueError):
return None
async def ingest_bzzoiro_event_stats(
db,
*,
leagues: Iterable[str],
limit: int = 100,
only_missing: bool = True,
) -> dict:
"""回填已完赛比赛的详细统计(逐场调 /events/{id}/stats/)。
筛选条件: match_status=finished 且 source_event_id 非空。
only_missing=True 时跳过已有统计的比赛(增量);False 则全量刷新。
limit 控制单次最多处理的比赛数(上游限速 1.2s/请求,大批量需分次触发)。
"""
result: dict = {"fetched": 0, "created": 0, "updated": 0, "skipped": 0, "errors": []}
league_ids = [BZZOIRO_LEAGUE_IDS[c] for c in leagues if c in BZZOIRO_LEAGUE_IDS]
if not league_ids:
result["errors"].append("无有效联赛代码")
return result
stmt = (
select(Match)
.options(selectinload(Match.stats))
.where(Match.match_status == "finished")
.where(Match.source_event_id.is_not(None))
.where(Match.league_id.in_(league_ids))
.order_by(Match.match_date.desc())
.limit(limit * 3 if only_missing else limit)
)
matches = (await db.execute(stmt)).scalars().all()
now = datetime.now(timezone.utc)
processed = 0
for m in matches:
if processed >= limit:
break
if only_missing and m.stats is not None and m.stats.home_shots is not None:
result["skipped"] += 1
continue
processed += 1
try:
payload = await _fetch_json_async(f"/events/{m.source_event_id}/stats/")
except Exception as e:
logger.warning("stats fetch failed match=%s event=%s: %s", m.id, m.source_event_id, e)
result["errors"].append(f"match {m.id}: {e}")
await _safe_write_ingest_failure(
db,
entity_type="match_stats",
source_record_id=str(m.source_event_id),
error=e,
raw_payload={"match_id": m.id},
)
await asyncio.sleep(REQUEST_INTERVAL)
continue
result["fetched"] += 1
fields = _stats_from_payload(payload)
if not fields:
result["skipped"] += 1
await asyncio.sleep(REQUEST_INTERVAL)
continue
if m.stats is None:
available_at = m.match_date + timedelta(hours=2) if m.match_date else now
m.stats = MatchStats(
match_id=m.id,
source="bzzoiro",
source_record_id=str(m.source_event_id),
retrieved_at=now,
available_at=available_at,
)
db.add(m.stats)
result["created"] += 1
else:
result["updated"] += 1
if m.stats.source is None:
m.stats.source = "bzzoiro"
m.stats.source_record_id = str(m.source_event_id)
if m.stats.retrieved_at is None:
m.stats.retrieved_at = now
if m.stats.available_at is None and m.match_date:
m.stats.available_at = m.match_date + timedelta(hours=2)
for fld, v in fields.items():
if hasattr(m.stats, fld):
setattr(m.stats, fld, v)
# 管线基础设施:写入 RawEvent + DataLineage
batch_id = f"bzzoiro-stats-{m.source_event_id}-{now.strftime('%Y%m%d%H%M%S')}"
try:
await _write_raw_event(db, "bzzoiro", str(m.source_event_id), payload, batch_id)
await _write_lineage(db, "bzzoiro", str(m.source_event_id), "match_stats", m.stats.id if m.stats else None, "stats_backfill", {"match_id": m.id}, batch_id)
except Exception:
pass # 基础设施写入失败不影响主流程
await asyncio.sleep(REQUEST_INTERVAL)
logger.info(
"bzzoiro stats 回填完成: 抓取 %d, 新建 %d, 更新 %d, 跳过 %d, 错误 %d",
result["fetched"], result["created"], result["updated"],
result["skipped"], len(result["errors"]),
)
return result
# ── 配置常量(原文件即从 config 再导出,维持 bz.REQUEST_INTERVAL 等引用) ──
from src.data.config import ( # noqa: F401
BZZOIRO_LEAGUE_IDS,
LEAGUE_COUNTRIES,
LEAGUE_NAMES,
REQUEST_INTERVAL,
)
from src.data.key_ring import _mask # noqa: F401 (R1 测试引用 bz._mask)
from src.data.normalize import normalize_bzzoiro # noqa: F401
# ── 共享原语:HTTP 抓取 + 宽松字段转换 ──
from src.data.bzzoiro_common import ( # noqa: F401
_fetch_json_async,
_match_key,
_to_date,
_to_float_or_none,
_to_int_or_none,
)
# ── 管线基础设施:RawEvent / IngestFailure / DataLineage ──
from src.data.pipeline_write import ( # noqa: F401
_safe_write_ingest_failure,
_write_ingest_failure,
_write_lineage,
_write_raw_event,
)
# ── events 管线:BzzoiroSource(注册表入口)+ 抓取/入库 ──
from src.data.bzzoiro_events import ( # noqa: F401
BzzoiroSource,
_events_record_id,
_write_events_bronze,
fetch_bzzoiro_events,
)
# ── standings 管线 ──
from src.data.bzzoiro_standings import ( # noqa: F401
_season_label_from_dates,
_write_standings_bronze,
fetch_bzzoiro_standings,
ingest_bzzoiro_standings,
)
# ── stats 回填管线 ──
from src.data.bzzoiro_stats import ( # noqa: F401
_pick,
_stats_from_payload,
ingest_bzzoiro_event_stats,
)
+123
View File
@@ -0,0 +1,123 @@
"""bzzoiro 管线共享原语:HTTP 抓取(多 key 轮换)与宽松字段转换。
从 bzzoiro.py 拆出(单文件 → 多模块):仅放无业务语义的共享基础,
三条管线(events/standings/stats)与聚合门面见 bzzoiro.py。
"""
from __future__ import annotations
import asyncio
import logging
import random
import httpx
from src.core.http_client import get_client
from src.core.runtime_config import get_runtime_value
from src.data.key_ring import _mask, get_key_ring
logger = logging.getLogger(__name__)
def _to_date(value):
"""把 datetime / date / str 统一成 `date`。"""
if value is None:
return None
if hasattr(value, "date") and callable(value.date):
return value.date()
return value
def _to_int_or_none(value) -> int | None:
"""宽松转 int(用于上游 ID 解析,失败返回 None 不抛错)。"""
if value is None:
return None
try:
return int(str(value).strip())
except (TypeError, ValueError):
return None
def _to_float_or_none(value) -> float | None:
try:
return float(str(value).strip())
except (TypeError, ValueError):
return None
def _match_key(home_team_id: int, away_team_id: int, match_date) -> tuple[int, int, str]:
"""比赛去重键:(主队, 客队, 天级日期 ISO 字符串)。
统一在这里构造,避免"预加载时用 str(date)、写入时用 isoformat()"这类
隐式格式依赖 —— 两者当前恰好相等,但一旦有人改动其一就会静默失配,
导致所有比赛被判为不存在而重复插入。
"""
d = _to_date(match_date)
return (home_team_id, away_team_id, d.isoformat() if d is not None else "")
async def _fetch_json_async(path: str, params: dict | None = None, max_retries: int = 3) -> dict | list:
"""异步 HTTP(bzzoiro 使用 httpx,不再阻塞事件循环线程池)。
多 key 轮换:遇到 429 自动切换到下一个 key;全部 key 冷却时等待最早恢复。
"""
base = (await get_runtime_value("BZZOIRO_BASE")).rstrip("/")
raw_keys = await get_runtime_value("BZZOIRO_KEY")
ring = get_key_ring(base, raw_keys)
url = f"{base}/{path.lstrip('/')}"
key = ring.get()
if not key:
raise RuntimeError("BZZOIRO_KEY 未设置")
last_exc: Exception | None = None
for attempt in range(max_retries):
headers = {
"Authorization": f"Token {key}",
"Accept": "application/json",
}
try:
client = get_client()
# 整请求兜底: httpx 无 total 超时,用 wait_for 防「滴水式」限速挂死
resp = await asyncio.wait_for(
client.get(
url, headers=headers, params=params,
timeout=httpx.Timeout(connect=10.0, read=30.0, write=10.0, pool=10.0),
),
timeout=60.0,
)
resp.raise_for_status()
return resp.json()
except Exception as e:
last_exc = e
status = getattr(getattr(e, "response", None), "status_code", None)
if status == 429:
# 限流:标记当前 key 冷却,切换到下一个
new_key = ring.report_rate_limited(key)
if new_key and new_key != key:
logger.info("bzzoiro 429 → 切换 key: %s%s,立即重试", _mask(key), _mask(new_key))
key = new_key
continue # 立即重试,不等待
# 单 key 或全部冷却:等待最早恢复的 key
wait = ring.wait_if_all_blocked()
if wait > 0:
logger.warning("bzzoiro 全部 key 冷却,等待 %.1fs 后重试", wait)
await asyncio.sleep(min(wait, 30.0))
else:
delay = min(2 ** attempt, 16) + random.uniform(0, 1)
logger.warning("bzzoiro 429, retry %d in %.1fs", attempt + 1, delay)
await asyncio.sleep(delay)
key = ring.get() or key
continue
if 500 <= (status or 0) < 600:
delay = min(2 ** attempt, 16) + random.uniform(0, 1)
logger.warning("bzzoiro %d, retry %d in %.1fs", status, attempt + 1, delay)
await asyncio.sleep(delay)
continue
# 网络错误(连接失败/超时)也退避重试
if isinstance(e, (TimeoutError, ConnectionError, OSError)):
delay = min(2 ** attempt, 16) + random.uniform(0, 1)
logger.warning("bzzoiro network error, retry %d in %.1fs: %s", attempt + 1, delay, e)
await asyncio.sleep(delay)
continue
raise
raise RuntimeError(f"bzzoiro request failed after {max_retries} attempts: {last_exc}")
+310
View File
@@ -0,0 +1,310 @@
"""bzzoiro events 管线:比赛日程/比分抓取(/events/)与入库(matches 表)。
从 bzzoiro.py 拆出。比赛主数据唯一入口;Team/League/Match 查找/创建经
Repository 层,事务由调用方 UnitOfWork 控制(分批事务约定不变)。
可替换协作者(抓取函数 / Bronze 写入助手 / REQUEST_INTERVAL)在运行期
经聚合门面 src.data.bzzoiro 解析 —— 与拆分前的单文件 monkeypatch 语义一致。
"""
from __future__ import annotations
import asyncio
import logging
from collections.abc import Iterable
from datetime import datetime, timedelta, timezone
from src.data.bzzoiro_common import _match_key, _to_date, _to_int_or_none
from src.data.config import BZZOIRO_LEAGUE_IDS, LEAGUE_COUNTRIES, LEAGUE_NAMES
from src.data.normalize import normalize_bzzoiro
from src.data.sources import register
from src.data.team_names_zh import zh_name
from src.db.models import Match
from src.db.repositories import LeagueRepository, MatchRepository, TeamRepository
logger = logging.getLogger(__name__)
async def fetch_bzzoiro_events(
league_code: str,
*,
status: str = "finished",
date_from: str | None = None,
date_to: str | None = None,
limit: int = 200,
) -> list[dict]:
"""抓取 bzzoiro 原始事件(纯异步,无需 run_in_executor)。"""
from src.data import bzzoiro as bz
league_id = BZZOIRO_LEAGUE_IDS.get(league_code)
if league_id is None:
raise ValueError(f"未知联赛代码: {league_code}")
rows: list[dict] = []
offset = 0
while True:
params: dict = {
"league_id": league_id,
"status": status,
"limit": limit,
"offset": offset,
}
if date_from:
params["date_from"] = str(date_from)[:10]
if date_to:
params["date_to"] = str(date_to)[:10]
payload = await bz._fetch_json_async("/events/", params)
batch = payload.get("results") or []
if not batch:
break
rows.extend(batch)
total = payload.get("total")
offset += limit
if total is not None and offset >= total:
break
if len(batch) < limit:
break
await asyncio.sleep(bz.REQUEST_INTERVAL)
return rows
@register
class BzzoiroSource:
"""bzzoiro 数据源(实现 DataSource 协议)。"""
name = "bzzoiro"
async def ingest(
self,
db,
*,
leagues: Iterable[str],
date_from: str | None = None,
date_to: str | None = None,
status: str = "finished",
) -> dict:
"""采集 bzzoiro → 入库。返回统计。
注意: 本方法不控制事务(commit/rollback),由调用方通过 UnitOfWork 控制。
"""
from src.data import bzzoiro as bz
result: dict = {"leagues": {}, "total_inserted": 0, "total_updated": 0, "errors": []}
for code in leagues:
league_r: dict = {"inserted": 0, "updated": 0, "errors": []}
try:
raw_events = await bz.fetch_bzzoiro_events(code, status=status, date_from=date_from, date_to=date_to)
except Exception as e:
# 单联赛抓取失败隔离:记录错误后继续其余联赛,不拖垮整批
logger.exception("bzzoiro fetch failed for %s", code)
league_r["errors"].append(f"fetch failed: {e}")
await bz._safe_write_ingest_failure(
db,
entity_type="events",
source_record_id=None,
error=e,
raw_payload={"league": code, "status": status, "date_from": date_from, "date_to": date_to},
)
result["leagues"][code] = league_r
continue
# D4: 联赛查找/创建经 LeagueRepository(事务仍由调用方 UoW 提交)
league = await LeagueRepository(db).get_or_create(
code, LEAGUE_NAMES.get(code, code), LEAGUE_COUNTRIES.get(code)
)
team_r = TeamRepository(db)
match_r = MatchRepository(db)
# === 批量优化: 预加载球队和已有比赛到内存 ===
team_name_to_id: dict[str, int] = {}
existing_matches: dict[tuple[int, int, str], Match] = {} # 完整对象,避免重复查询
# (NormalizedMatch, 原始 event) 成对保存:后续写 source_event_id 时
# 必须用配对的那条 event,不能依赖外层循环变量残留值。
normalized_matches: list[tuple] = []
if raw_events:
# 一次遍历: 收集球队名 + 规范化
all_team_names = set()
for raw in raw_events:
nm = normalize_bzzoiro(raw, code)
if nm is not None:
try:
nm.validate()
except Exception:
continue
normalized_matches.append((nm, raw))
all_team_names.add(nm.home_team)
all_team_names.add(nm.away_team)
if all_team_names:
team_name_to_id = {
name: t.id
for name, t in (await team_r.get_all_by_names(list(all_team_names))).items()
}
# P1-2: 按需加载,只加载 raw_events 涉及日期范围的比赛(加 30 天缓冲)
# 避免加载联赛全部历史比赛到内存(多赛季采集时内存溢出)
if normalized_matches:
# normalized_matches 存的是 (nm, raw) 元组,遍历需解包
dates = [nm.date for nm, _raw in normalized_matches if nm.date is not None]
if dates:
min_dt = min(dates) - timedelta(days=30)
max_dt = max(dates) + timedelta(days=30)
matches_in_range = await match_r.find_by_league_and_date_range(
league.id, min_dt, max_dt
)
existing_matches = {
_match_key(m.home_team_id, m.away_team_id, m.match_date_date): m
for m in matches_in_range
}
# else: existing_matches 保持空 dict(全量新比赛)
# D1: Bronze 层批次信息(每联赛每批次一个 batch_id;seen 防同批重复写入)
now = datetime.now(timezone.utc)
bronze_batch_id = f"bzzoiro-events-{code}-{now:%Y%m%d%H%M%S}"
bronze_written: set[str] = set()
for nm, raw in normalized_matches:
# D1: RawEvent 幂等键(上游 id 或合成键),插入/变更更新共用
record_id = _events_record_id(code, nm, raw)
# 球队: 内存查找 + 按需创建(D4: 经 TeamRepository)
home_team_id = team_name_to_id.get(nm.home_team)
if home_team_id is None:
home = await team_r.get_or_create(nm.home_team, name_zh=zh_name(nm.home_team))
home_team_id = home.id
team_name_to_id[nm.home_team] = home_team_id
away_team_id = team_name_to_id.get(nm.away_team)
if away_team_id is None:
away = await team_r.get_or_create(nm.away_team, name_zh=zh_name(nm.away_team))
away_team_id = away.id
team_name_to_id[nm.away_team] = away_team_id
# 查找已有比赛: 内存查找
match_key = _match_key(home_team_id, away_team_id, nm.date)
existing_match = existing_matches.get(match_key)
if existing_match is None:
m = Match(
league_id=league.id,
season=nm.season_label or None,
home_team_id=home_team_id,
away_team_id=away_team_id,
match_date=nm.date,
match_date_date=_to_date(nm.date),
match_status=nm.match_status,
home_goals=nm.home_goals,
away_goals=nm.away_goals,
home_ht_goals=nm.home_ht_goals,
away_ht_goals=nm.away_ht_goals,
match_stage=nm.match_stage,
source_event_id=_to_int_or_none(raw.get("id")),
)
db.add(m)
await db.flush()
existing_matches[match_key] = m # 防止同批重复
# 统计字段不在 /events/ 载荷中(单独由 stats 管线回填),
# 此处不再创建 MatchStats。
league_r["inserted"] += 1
# D1: 成功插入 → 补写 Bronze 层(原始载荷 + 血缘)
if record_id not in bronze_written:
bronze_written.add(record_id)
await _write_events_bronze(
db,
source_record_id=record_id,
raw_payload=raw,
target_match_id=m.id,
league_code=code,
match_status=nm.match_status,
batch_id=bronze_batch_id,
)
else:
# 已有比赛: 直接从内存获取对象更新(无需再查询)
changed = False
if existing_match.match_status != nm.match_status and nm.match_status == "finished":
existing_match.match_status = nm.match_status
changed = True
if existing_match.home_goals is None and nm.home_goals is not None:
existing_match.home_goals = nm.home_goals
existing_match.away_goals = nm.away_goals
existing_match.home_ht_goals = nm.home_ht_goals
existing_match.away_ht_goals = nm.away_ht_goals
changed = True
if existing_match.match_stage is None and nm.match_stage:
existing_match.match_stage = nm.match_stage
changed = True
if existing_match.source_event_id is None:
eid = _to_int_or_none(raw.get("id"))
if eid is not None:
existing_match.source_event_id = eid
changed = True
if changed:
league_r["updated"] += 1
# D1: 变更更新 → 补写血缘(RawEvent 幂等键不变,重复采集自动跳过)
if record_id not in bronze_written:
bronze_written.add(record_id)
await _write_events_bronze(
db,
source_record_id=record_id,
raw_payload=raw,
target_match_id=existing_match.id,
league_code=code,
match_status=nm.match_status,
batch_id=bronze_batch_id,
)
# 注意: 不在此处 commit,由调用方 UnitOfWork 控制事务
result["leagues"][code] = league_r
result["total_inserted"] += league_r["inserted"]
result["total_updated"] += league_r["updated"]
return result
def _events_record_id(league_code: str, nm, raw: dict) -> str:
"""events 载荷的 RawEvent 幂等键。
优先用上游 event id;缺失时用 (league:home:away:date) 合成稳定键 ——
取 normalize 后的队名与天级日期(与 _match_key 同口径),不依赖 DB 自增 id,
保证同一来源比赛重复采集时命中同一条 RawEvent,不产生重复原始载荷。
"""
eid = _to_int_or_none(raw.get("id"))
if eid is not None:
return str(eid)
d = _to_date(nm.date)
date_part = d.isoformat() if d is not None else "na"
return f"{league_code}:{nm.home_team}:{nm.away_team}:{date_part}"
async def _write_events_bronze(
db,
*,
source_record_id: str,
raw_payload: dict,
target_match_id: int | None,
league_code: str,
match_status: str | None,
batch_id: str,
) -> None:
"""events 成功插入/更新单场比赛后的 Bronze 层补写:RawEvent(幂等) + DataLineage。
D1(工程债):此前只有 stats 回填写 RawEvent/Lineage,events 管线作为比赛
主数据的唯一入口反而不留溯源记录。幂等性由 _write_raw_event 的
source_record_id 查重保证;best-effort:基础设施写入失败只记 warning,
绝不拖垮采集主流程(与 _safe_write_ingest_failure 同级约束)。
"""
from src.data import bzzoiro as bz
try:
await bz._write_raw_event(db, "bzzoiro", source_record_id, raw_payload, batch_id)
await bz._write_lineage(
db, "bzzoiro", source_record_id,
"matches", target_match_id, "events_ingest",
{"league": league_code, "match_status": match_status},
batch_id,
)
except Exception:
logger.warning(
"events Bronze 写入失败(record=%s, match=%s),不影响采集主流程",
source_record_id, target_match_id, exc_info=True,
)
+215
View File
@@ -0,0 +1,215 @@
"""bzzoiro standings 管线:联赛积分榜快照(/leagues/{id}/standings/)→ standings 表。
从 bzzoiro.py 拆出。同一联赛同一赛季只保留最新快照(按 (league, season, team)
upsert);球队名与 events 管线使用同一 normalize 规则,保证 Team 匹配。
可替换协作者(抓取函数 / Bronze 写入助手)在运行期经聚合门面
src.data.bzzoiro 解析 —— 与拆分前的单文件 monkeypatch 语义一致。
"""
from __future__ import annotations
import logging
from collections.abc import Iterable
from datetime import datetime, timezone
from sqlalchemy import select
from src.data.bzzoiro_common import _to_float_or_none, _to_int_or_none
from src.data.config import BZZOIRO_LEAGUE_IDS, LEAGUE_COUNTRIES, LEAGUE_NAMES
from src.data.team_names_zh import zh_name
from src.db.models import Standing, Team
from src.db.repositories import LeagueRepository, TeamRepository
logger = logging.getLogger(__name__)
async def fetch_bzzoiro_standings(league_code: str, season: str | None = None) -> dict:
"""抓取联赛积分榜(纯抓取,不入库)。season 为 None 时取当前赛季。"""
from src.data import bzzoiro as bz
league_id = BZZOIRO_LEAGUE_IDS.get(league_code)
if league_id is None:
raise ValueError(f"未知联赛代码: {league_code}")
params: dict = {}
if season:
params["season"] = season
return await bz._fetch_json_async(f"/leagues/{league_id}/standings/", params)
def _season_label_from_dates(start_date, end_date) -> str:
"""从赛季起止日期推导赛季标签(与 derive_season_label 语义一致)。"""
try:
if isinstance(start_date, str):
start = datetime.fromisoformat(start_date[:10])
else:
start = start_date
if start is None:
return "?"
y = start.year
return f"{y}-{y + 1}" if start.month >= 8 else f"{y - 1}-{y}"
except (TypeError, ValueError):
return "?"
async def ingest_bzzoiro_standings(db, *, leagues: Iterable[str], season: str | None = None) -> dict:
"""采集积分榜 → upsert standings 表。
season 为 None 时采集当前赛季(bzzoiro 默认返回 is_current 赛季)。
球队名与 events 管线使用同一 normalize 规则,保证 Team 匹配。
"""
from src.data import bzzoiro as bz
from src.data.team_names import normalize as normalize_name
result: dict = {"leagues": {}, "total_upserted": 0, "errors": []}
for code in leagues:
league_r: dict = {"upserted": 0, "teams_created": 0, "rows": 0, "errors": []}
try:
payload = await bz.fetch_bzzoiro_standings(code, season=season)
except Exception as e:
logger.exception("bzzoiro standings fetch failed for %s", code)
league_r["errors"].append(str(e))
await bz._safe_write_ingest_failure(
db,
entity_type="standings",
source_record_id=None,
error=e,
raw_payload={"league": code, "season": season},
)
result["leagues"][code] = league_r
result["errors"].append(f"{code}: {e}")
continue
rows = payload.get("standings") or []
if not rows:
result["leagues"][code] = {"error": "无积分榜数据(赛季未开始或未提供)"}
result["errors"].append(f"{code}: 无积分榜数据")
continue
# 联赛(get-or-create,D4: 经 LeagueRepository)
league = await LeagueRepository(db).get_or_create(
code, LEAGUE_NAMES.get(code, code), LEAGUE_COUNTRIES.get(code)
)
team_r = TeamRepository(db)
# 赛季标签:优先用返回的 season 对象推导
season_obj = payload.get("season") or {}
season_label = _season_label_from_dates(
season_obj.get("start_date"), season_obj.get("end_date")
)
if season_label == "?":
season_label = season or ""
# 批量预载球队(与 events 管线使用同一 normalize 规则,保证 Team 匹配)
names = {normalize_name(str(r.get("team_name", ""))) for r in rows}
names.discard("")
team_map: dict[str, Team] = await team_r.get_all_by_names(list(names))
now = datetime.now(timezone.utc)
for r in rows:
team_name = normalize_name(str(r.get("team_name", "")))
if not team_name:
continue
team = team_map.get(team_name)
if team is None:
team = await team_r.get_or_create(team_name, name_zh=zh_name(team_name))
team_map[team_name] = team
league_r["teams_created"] += 1
zone = r.get("zone") or {}
values = dict(
position=_to_int_or_none(r.get("position")) or 0,
played=_to_int_or_none(r.get("played")) or 0,
won=_to_int_or_none(r.get("won")) or 0,
drawn=_to_int_or_none(r.get("drawn")) or 0,
lost=_to_int_or_none(r.get("lost")) or 0,
goals_for=_to_int_or_none(r.get("gf")) or 0,
goals_against=_to_int_or_none(r.get("ga")) or 0,
goal_diff=_to_int_or_none(r.get("gd")) or 0,
points=_to_int_or_none(r.get("pts")) or 0,
xg_for=_to_float_or_none(r.get("xgf")),
xg_against=_to_float_or_none(r.get("xga")),
form=r.get("form") or None,
zone=zone.get("label") or zone.get("key") or None,
updated_at=now,
retrieved_at=now,
)
# 同一联赛同一赛季只保留最新快照:按 (league, season, team) upsert
stmt = select(Standing).where(
Standing.league_id == league.id,
Standing.season == season_label,
Standing.team_id == team.id,
)
standing = (await db.execute(stmt)).scalar_one_or_none()
if standing is None:
standing = Standing(
league_id=league.id, season=season_label, team_id=team.id, **values
)
db.add(standing)
else:
for k, v in values.items():
setattr(standing, k, v)
league_r["upserted"] += 1
league_r["rows"] = len(rows)
# D1(对称 events/stats 管线): 联赛成功 upsert → 补写 Bronze 层。
# 幂等键 standings:{league}:{season}:积分榜是联赛级快照,一次成功
# 采集写一条 RawEvent(整份原始载荷)+ 一条血缘。season 用实际入库的
# 标签(由载荷推导,与 Standing.season 同口径),不依赖调用方传参,
# 保证不同调用方(season=None 或显式传参)对同一赛季命中同一条 RawEvent。
if league_r["upserted"] > 0:
bronze_batch_id = f"bzzoiro-standings-{code}-{now:%Y%m%d%H%M%S}"
await _write_standings_bronze(
db,
source_record_id=f"standings:{code}:{season_label}",
raw_payload=payload,
league_id=league.id,
league_code=code,
season_label=season_label,
rows_upserted=league_r["upserted"],
batch_id=bronze_batch_id,
)
result["leagues"][code] = league_r
result["total_upserted"] += league_r["upserted"]
logger.info(
"bzzoiro standings 采集完成: %s 赛季 %s, upsert %d/%d",
code, season_label, league_r["upserted"], league_r["rows"],
)
return result
async def _write_standings_bronze(
db,
*,
source_record_id: str,
raw_payload: dict,
league_id: int | None,
league_code: str,
season_label: str,
rows_upserted: int,
batch_id: str,
) -> None:
"""standings 成功 upsert 一个联赛后的 Bronze 层补写:RawEvent(幂等) + DataLineage。
与 _write_events_bronze 同级约束:幂等性由 _write_raw_event 的
source_record_id 查重保证(积分榜是联赛级快照,同联赛同赛季重复采集
命中同一条 RawEvent);best-effort:基础设施写入失败只记 warning,
绝不拖垮采集主流程。
"""
from src.data import bzzoiro as bz
try:
await bz._write_raw_event(db, "bzzoiro", source_record_id, raw_payload, batch_id)
await bz._write_lineage(
db, "bzzoiro", source_record_id,
"standings", league_id, "standings_ingest",
{"league": league_code, "season": season_label, "rows_upserted": rows_upserted},
batch_id,
)
except Exception:
logger.warning(
"standings Bronze 写入失败(record=%s, league=%s),不影响采集主流程",
source_record_id, league_code, exc_info=True,
)
+180
View File
@@ -0,0 +1,180 @@
"""bzzoiro stats 回填管线:已完赛比赛详细统计(/events/{id}/stats/)→ match_stats 表。
从 bzzoiro.py 拆出。上游限速(REQUEST_INTERVAL 秒/请求),大批量回填需分次触发;
只 add/flush 不 commit,事务由调用方 UnitOfWork 控制。
可替换协作者(_fetch_json_async / Bronze 写入助手 / REQUEST_INTERVAL)在运行期
经聚合门面 src.data.bzzoiro 解析 —— 与拆分前的单文件 monkeypatch 语义一致。
"""
from __future__ import annotations
import asyncio
import logging
from collections.abc import Iterable
from datetime import datetime, timedelta, timezone
from src.data.bzzoiro_common import _to_float_or_none, _to_int_or_none
from src.data.config import BZZOIRO_LEAGUE_IDS
from src.db.models import MatchStats
from src.db.repositories import MatchRepository
logger = logging.getLogger(__name__)
# bzzoiro stats 字段 → MatchStats 字段映射(stats.home / stats.away 下)
_STATS_FIELD_MAP = {
"xg": ("home_xg", "away_xg"), # 回退 expected_goals
"ball_possession": ("home_possession", None), # 只取主队值,客队=100-home
"total_shots": ("home_shots", "away_shots"),
"shots_on_target": ("home_shots_on_target", "away_shots_on_target"),
"corner_kicks": ("home_corners", "away_corners"),
"yellow_cards": ("home_yellow_cards", "away_yellow_cards"),
"red_cards": ("home_red_cards", "away_red_cards"),
"big_chances": ("home_big_chances", "away_big_chances"),
"fouls": ("home_fouls", "away_fouls"),
}
def _pick(d: dict, *keys):
"""按优先级取第一个非空字段值。"""
for k in keys:
v = d.get(k)
if v is not None:
return v
return None
def _stats_from_payload(payload: dict) -> dict:
"""把 /events/{id}/stats/ 响应映射成 MatchStats 字段 dict。
响应结构: {"event_id": ..., "stats": {"home": {...}, "away": {...}}}
"""
stats = (payload or {}).get("stats") or {}
home = stats.get("home") or {}
away = stats.get("away") or {}
out: dict = {}
xg_h = _pick(home, "xg", "expected_goals")
xg_a = _pick(away, "xg", "expected_goals")
if xg_h is not None:
out["home_xg"] = _to_float_or_none(xg_h)
if xg_a is not None:
out["away_xg"] = _to_float_or_none(xg_a)
poss = home.get("ball_possession")
if poss is not None:
p = _to_float_or_none(poss)
if p is not None:
out["home_possession"] = p
for src, (h_fld, a_fld) in _STATS_FIELD_MAP.items():
if src in ("xg", "ball_possession"):
continue # 已处理
hv = home.get(src)
av = away.get(src)
if hv is not None and h_fld:
out[h_fld] = _to_int_or_none(hv)
if av is not None and a_fld:
out[a_fld] = _to_int_or_none(av)
return out
async def ingest_bzzoiro_event_stats(
db,
*,
leagues: Iterable[str],
limit: int = 100,
only_missing: bool = True,
) -> dict:
"""回填已完赛比赛的详细统计(逐场调 /events/{id}/stats/)。
筛选条件: match_status=finished 且 source_event_id 非空。
only_missing=True 时跳过已有统计的比赛(增量);False 则全量刷新。
limit 控制单次最多处理的比赛数(上游限速 1.2s/请求,大批量需分次触发)。
"""
from src.data import bzzoiro as bz
result: dict = {"fetched": 0, "created": 0, "updated": 0, "skipped": 0, "errors": []}
league_ids = [BZZOIRO_LEAGUE_IDS[c] for c in leagues if c in BZZOIRO_LEAGUE_IDS]
if not league_ids:
result["errors"].append("无有效联赛代码")
return result
# D4: 候选比赛查询经 MatchRepository(含 stats 预加载,筛选/排序/limit 语义不变)
matches = await MatchRepository(db).find_finished_with_stats(
league_ids, limit=limit * 3 if only_missing else limit
)
now = datetime.now(timezone.utc)
processed = 0
for m in matches:
if processed >= limit:
break
if only_missing and m.stats is not None and m.stats.home_shots is not None:
result["skipped"] += 1
continue
processed += 1
try:
payload = await bz._fetch_json_async(f"/events/{m.source_event_id}/stats/")
except Exception as e:
logger.warning("stats fetch failed match=%s event=%s: %s", m.id, m.source_event_id, e)
result["errors"].append(f"match {m.id}: {e}")
await bz._safe_write_ingest_failure(
db,
entity_type="match_stats",
source_record_id=str(m.source_event_id),
error=e,
raw_payload={"match_id": m.id},
)
await asyncio.sleep(bz.REQUEST_INTERVAL)
continue
result["fetched"] += 1
fields = _stats_from_payload(payload)
if not fields:
result["skipped"] += 1
await asyncio.sleep(bz.REQUEST_INTERVAL)
continue
if m.stats is None:
available_at = m.match_date + timedelta(hours=2) if m.match_date else now
m.stats = MatchStats(
match_id=m.id,
source="bzzoiro",
source_record_id=str(m.source_event_id),
retrieved_at=now,
available_at=available_at,
)
db.add(m.stats)
result["created"] += 1
else:
result["updated"] += 1
if m.stats.source is None:
m.stats.source = "bzzoiro"
m.stats.source_record_id = str(m.source_event_id)
if m.stats.retrieved_at is None:
m.stats.retrieved_at = now
if m.stats.available_at is None and m.match_date:
m.stats.available_at = m.match_date + timedelta(hours=2)
for fld, v in fields.items():
if hasattr(m.stats, fld):
setattr(m.stats, fld, v)
# 管线基础设施:写入 RawEvent + DataLineage
batch_id = f"bzzoiro-stats-{m.source_event_id}-{now.strftime('%Y%m%d%H%M%S')}"
try:
await bz._write_raw_event(db, "bzzoiro", str(m.source_event_id), payload, batch_id)
await bz._write_lineage(db, "bzzoiro", str(m.source_event_id), "match_stats", m.stats.id if m.stats else None, "stats_backfill", {"match_id": m.id}, batch_id)
except Exception:
pass # 基础设施写入失败不影响主流程
await asyncio.sleep(bz.REQUEST_INTERVAL)
logger.info(
"bzzoiro stats 回填完成: 抓取 %d, 新建 %d, 更新 %d, 跳过 %d, 错误 %d",
result["fetched"], result["created"], result["updated"],
result["skipped"], len(result["errors"]),
)
return result
+2 -1
View File
@@ -1,7 +1,8 @@
"""API Key 轮换环:多 key 自动切换,遇到限流(429)自动跳过已冷却 key。
设计:
- 进程内纯内存状态(限速是短时状态,无需持久化)
- 进程内纯内存状态(限速是短时状态,无需持久化;D7: 多 worker 部署时各进程
独立计数、不共享,上游限速额度应按 worker 数分摊,或前置网关统一管理)
- 单 key 场景零开销:直接透传
- 多 key 场景:429 时把当前 key 标记冷却(默认 60s),轮转到下一个可用 key
- 全部 key 都在冷却时:使用最早冷却的那个 key 并等待(退化到单 key 重试)
+80
View File
@@ -0,0 +1,80 @@
"""管线基础设施写入助手:RawEvent(Bronze 原始载荷)/ IngestFailure(死信)/ DataLineage(血缘)。
从 bzzoiro.py 拆出。约定(与拆分前一致):
- 只 add 不 commit —— 事务由调用方 UnitOfWork 控制,分批事务约定不变;
- 死信与 Bronze 写入同为 best-effort:失败只记 warning,绝不拖垮采集主流程。
"""
from __future__ import annotations
import logging
from src.db.models import DataLineage, IngestFailure, RawEvent
logger = logging.getLogger(__name__)
async def _write_raw_event(db, source_system: str, source_record_id: str, raw_payload: dict, batch_id: str | None = None) -> None:
"""写入 Bronze 层原始事件(幂等:同 source_record_id 跳过)。"""
from sqlalchemy import select as _select
stmt = _select(RawEvent).where(
RawEvent.source_system == source_system,
RawEvent.source_record_id == source_record_id,
)
existing = (await db.execute(stmt)).scalar_one_or_none()
if existing is None:
db.add(RawEvent(
source_system=source_system,
source_record_id=source_record_id,
raw_payload=raw_payload,
ingest_batch_id=batch_id,
))
async def _write_ingest_failure(db, source_system: str, entity_type: str, source_record_id: str | None, error_type: str, error_detail: str | None, raw_payload: dict | None = None) -> None:
"""写入采集失败死信。"""
db.add(IngestFailure(
source_system=source_system,
entity_type=entity_type,
source_record_id=source_record_id,
error_type=error_type,
error_detail=error_detail,
raw_payload=raw_payload,
))
async def _safe_write_ingest_failure(
db,
*,
entity_type: str,
source_record_id: str | None,
error: Exception,
raw_payload: dict | None = None,
) -> None:
"""抓取失败时尽力写入死信表(失败不影响主流程)。
死信是「可观测性」基础设施,与 RawEvent/Lineage 同级:写入失败只记
warning,绝不能让原始抓取错误之外的新异常打断采集循环。
"""
try:
await _write_ingest_failure(
db, "bzzoiro", entity_type, source_record_id,
"fetch_error", str(error), raw_payload,
)
except Exception:
logger.warning(
"写入 ingest_failures 死信失败(entity=%s, record=%s): %s",
entity_type, source_record_id, error, exc_info=True,
)
async def _write_lineage(db, source_system: str, source_record_id: str, target_table: str, target_id: int | None, transform_name: str, transform_detail: dict | None = None, batch_id: str | None = None) -> None:
"""写入 ETL 血缘追踪。"""
db.add(DataLineage(
source_system=source_system,
source_record_id=source_record_id,
target_table=target_table,
target_id=target_id,
transform_name=transform_name,
transform_detail=transform_detail,
batch_id=batch_id,
))
+3 -1
View File
@@ -232,7 +232,9 @@ class Prediction(Base):
raw_response: Mapped[dict | None] = mapped_column(JSONB)
# multi-agent 模式: 各专家报告
mode: Mapped[str] = mapped_column(String(20), nullable=False, default="single")
agent_outputs: Mapped[dict | None] = mapped_column(JSONB)
# D5: multi-agent 模式存各专家报告列表(list[dict]);历史数据/兼容路径可能存 dict。
# 仅修正类型标注与真实 JSON 形状一致,列类型(JSONB)与数据不变。
agent_outputs: Mapped[list[dict] | dict | None] = mapped_column(JSONB)
# Fix: agent_weights 独立持久化到列(原本只在 raw_response 中)
agent_weights: Mapped[dict | None] = mapped_column(JSONB)
# 预测状态: success / failed / degraded
+48 -4
View File
@@ -5,6 +5,10 @@ Repository 只负责查询,不负责事务提交。
"""
from __future__ import annotations
import logging
logger = logging.getLogger(__name__)
from datetime import datetime
from sqlalchemy import select
from sqlalchemy.orm import selectinload
@@ -64,6 +68,34 @@ class MatchRepository:
)
return (await self._session.execute(stmt)).scalar_one_or_none()
async def find_by_league_and_date_range(
self, league_id: int, start, end
) -> list[Match]:
"""批量预加载某联赛日期范围内的比赛(ingest 管线内存去重用)。"""
stmt = (
select(Match)
.where(Match.league_id == league_id)
.where(Match.match_date >= start)
.where(Match.match_date <= end)
)
return (await self._session.execute(stmt)).scalars().all()
async def find_finished_with_stats(self, league_ids: list[int], *, limit: int) -> list[Match]:
"""已完赛且有上游 event id 的比赛(按日期倒序),供统计回填逐场拉取。
预加载 stats:调用方需读取 existing.stats 判断是否跳过。
"""
stmt = (
select(Match)
.options(selectinload(Match.stats))
.where(Match.match_status == "finished")
.where(Match.source_event_id.is_not(None))
.where(Match.league_id.in_(league_ids))
.order_by(Match.match_date.desc())
.limit(limit)
)
return (await self._session.execute(stmt)).scalars().all()
async def add(self, match: Match) -> None:
self._session.add(match)
await self._session.flush()
@@ -79,11 +111,23 @@ class TeamRepository:
stmt = select(Team).where(Team.name == name)
return (await self._session.execute(stmt)).scalar_one_or_none()
async def get_or_create(self, name: str) -> Team:
"""按名获取球队,不存在则创建"""
team = await self.get_by_name(name)
async def get_or_create(self, name: str, *, name_zh: str | None = None) -> Team:
"""按名获取球队,不存在则创建(name_zh 供 bzzoiro 管线写中文名)。
归一化咽喉:所有入库 Team.name 必须经过 team_names.normalize,
此处统一收敛,避免各调用点散落归一化逻辑导致重复 Team。
创建新 Team 时 info 打出原始名与归一后的规范名,便于排查重名。
"""
from src.data.team_names import normalize as normalize_name
normalized = normalize_name(name) or name.strip()
team = await self.get_by_name(normalized)
if team is None:
team = Team(name=name)
logger.info(
"创建新 Team: %s -> %s",
name, normalized,
)
team = Team(name=normalized, name_zh=name_zh)
self._session.add(team)
await self._session.flush()
return team
+7 -24
View File
@@ -6,14 +6,13 @@ import hashlib
import json
import logging
import time
from dataclasses import dataclass
from datetime import datetime, timezone
from src.core.config import settings
from src.db.base import AsyncSessionLocal
from src.db.models import Match, Prediction
from src.db.unit_of_work import get_uow
from src.llm.predict import _upsert_prediction
from src.llm.predict import PredictResult, _upsert_prediction
from src.llm.agents.base import AgentReport, AgentSpec, load_agent_prompt
from src.llm.context_builder import (
MatchHeader,
@@ -81,28 +80,12 @@ AGENT_LABELS_ZH: dict[str, str] = {
}
@dataclass
class MultiPredictResult:
prediction_id: int
provider: str
model: str
prompt_version: str
mode: str
pred_home_goals: float | None
pred_away_goals: float | None
alt_pred_home_goals: int | None
alt_pred_away_goals: int | None
pred_1x2: str | None
subjective_confidence: float | None
reasoning: str | None
context: str
agent_outputs: list[dict]
agent_weights: dict | None
status: str = "success"
latency_ms: int | None = None
prompt_tokens: int | None = None
completion_tokens: int | None = None
raw: dict | None = None
# D2(工程债): multi 结果类型与 single 统一 —— 扩展后的 PredictResult 用可选
# 字段(agent_outputs/agent_weights/prompt_tokens/completion_tokens/mode)承载
# 全部模式,此处仅保留别名。保留 `MultiPredictResult` 名字的原因:
# 1. predict_match_multi 签名 `-> MultiPredictResult:` 是 R4 源码守卫的标记;
# 2. src/llm/agents/__init__.py 对外 re-export 该名字。
MultiPredictResult = PredictResult
async def _agent_provider(agent_id: str, *, tier: str, model_override: str | None = None) -> LLMProvider:
+53 -17
View File
@@ -12,6 +12,7 @@ from sqlalchemy import case, func, select
from src.db.base import AsyncSession, AsyncSessionLocal
from src.db.models import Match
from src.llm.predict import PredictResult, _upsert_prediction
logger = logging.getLogger(__name__)
@@ -52,22 +53,28 @@ async def predict_baseline(
*,
backtest: bool = False,
cutoff_at: datetime | None = None,
) -> dict:
) -> PredictResult:
"""极简基线预测:主场场均进球 vs 客场场均进球。
返回 PredictResult 兼容的字典:
返回 PredictResult(D2 统一结果类型):
provider=model="baseline", 不调用 LLM,latency_ms0
P3-2:baseline 落库下沉到服务层 直接在服务层完成落库并回填真实
prediction_id,路由层不再需要特殊的 _persist_baseline, single/multi
路径统一(result.prediction_id 即可用)对外 JSON 不变
"""
from src.db.unit_of_work import get_uow
async with AsyncSessionLocal() as db:
match = await db.get(Match, match_id)
if match is None:
raise ValueError(f"match {match_id} not found")
before = None
if backtest and match.match_dt:
if backtest and match.match_date:
from datetime import timedelta
before = match.match_dt - timedelta(days=1)
before = match.match_date - timedelta(days=1)
elif cutoff_at is not None:
before = cutoff_at
@@ -90,24 +97,53 @@ async def predict_baseline(
else:
pred_1x2 = "X"
return {
"pred_home_goals": float(pred_home),
"pred_away_goals": float(pred_away),
"alt_pred_home_goals": None,
"alt_pred_away_goals": None,
"pred_1x2": pred_1x2,
"subjective_confidence": 0.5,
values = {
"prompt_version": "baseline_v1",
"prompt_tokens": 0,
"completion_tokens": 0,
"latency_ms": 0,
"pred_home_goals": float(pred_home),
"pred_away_goals": float(pred_away),
"pred_1x2": pred_1x2,
"subjective_confidence": 0.5,
"reasoning": (
f"基线估计(非投注建议): 主队主场场均进球 {home_avg:.2f} → 预测 {pred_home}; "
f"客队客场场均进球 {away_avg:.2f} → 预测 {pred_away}"
),
"provider": "baseline",
"model": "baseline",
"prompt_version": "baseline_v1",
"mode": "baseline",
"raw_response": {"home_avg": round(home_avg, 2), "away_avg": round(away_avg, 2)},
"status": "success",
"latency_ms": 0,
"raw": {"home_avg": round(home_avg, 2), "away_avg": round(away_avg, 2)},
}
# P3-2:服务层落库,回填真实 prediction_id(与 single/multi 统一)。
async with get_uow() as session:
pred = await _upsert_prediction(
session,
match_id=match_id,
provider_name="baseline",
model="baseline",
mode="baseline",
run_type="live",
values=values,
)
prediction_id = pred.id
return PredictResult(
prediction_id=prediction_id,
provider="baseline",
model="baseline",
prompt_version="baseline_v1",
mode="baseline",
pred_home_goals=float(pred_home),
pred_away_goals=float(pred_away),
alt_pred_home_goals=None,
alt_pred_away_goals=None,
pred_1x2=pred_1x2,
subjective_confidence=0.5,
reasoning=values["reasoning"],
context="", # baseline 不构建 LLM 上下文
status="success",
latency_ms=0,
prompt_tokens=0,
completion_tokens=0,
raw=values["raw_response"],
)
+18 -1
View File
@@ -89,6 +89,14 @@ def _prompt_template_hash(version: str) -> str:
@dataclass
class PredictResult:
"""三种预测模式(single/multi/baseline)的统一结果类型。
D2(工程债): 原本 single 返回本类multi 重复定义 MultiPredictResult
baseline 返回裸 dict,导致路由 isinstance(dict) 双分支 + backtest
baseline 直接 AttributeError现以可选字段扩展本类承载全部模式;
MultiPredictResult 是本类的别名( src/llm/agents/orchestrator.py)
"""
prediction_id: int
provider: str
model: str
@@ -101,8 +109,15 @@ class PredictResult:
subjective_confidence: float | None
reasoning: str | None
context: str
# 模式标识: single(默认) / multi / baseline
mode: str = "single"
# multi 专属: 各专家报告列表与融合权重;single/baseline 为 None
agent_outputs: list[dict] | None = None
agent_weights: dict | None = None
status: str = "success"
latency_ms: int | None = None
prompt_tokens: int | None = None
completion_tokens: int | None = None
raw: dict | None = None
@@ -158,9 +173,11 @@ async def predict_match(
use_cache: bool = True,
backtest: bool = False,
cutoff_at=None,
) -> "PredictResult | MultiPredictResult":
) -> PredictResult:
"""预测入口。mode=multi(默认)走多 agent;mode=single 走单次调用;mode=baseline 走无 LLM 基线。
三种模式统一返回 PredictResult(D2);multi MultiPredictResult 是其别名
Args:
mode: multi(默认,5 专家+终裁) / single(单次) / baseline(极简统计基线,不调用 LLM)
use_cache:是否允许返回进程内缓存结果回测必须传 False
+4 -4
View File
@@ -79,18 +79,18 @@ class TestWriteBufferStrategy:
def test_bzzoirot_new_match_available_at_is_two_hours_after_kickoff(self):
"""bzzoiro 新建比赛(stats 回填创建 MatchStats)时 available_at 应为开球 + 2 小时。"""
import inspect
from src.data import bzzoiro
from src.data import bzzoiro_stats
source = inspect.getsource(bzzoiro)
source = inspect.getsource(bzzoiro_stats)
assert 'timedelta(hours=2)' in source, \
"bzzoiro 应使用 match_date + timedelta(hours=2) 作为 available_at"
def test_bzzoirot_multiple_writes_use_two_hour_buffer(self):
"""bzzoiro 多处写入(创建/更新)都应使用 2 小时缓冲。"""
import inspect
from src.data import bzzoiro
from src.data import bzzoiro_stats
source = inspect.getsource(bzzoiro)
source = inspect.getsource(bzzoiro_stats)
count = source.count('timedelta(hours=2)')
assert count >= 2, f"期望至少 2 处 timedelta(hours=2),实际 {count}"
+43 -16
View File
@@ -5,6 +5,7 @@
"""
from __future__ import annotations
from types import SimpleNamespace
from unittest.mock import patch
import pytest
@@ -12,6 +13,24 @@ import pytest
from src.llm.baseline import _avg_goals, predict_baseline
class _FakeUoW:
"""P3-2:baseline 在服务层落库,测试需 mock get_uow。"""
async def __aenter__(self):
return SimpleNamespace(
execute=lambda *a, **k: SimpleNamespace(scalar_one_or_none=lambda: None),
add=lambda *a, **k: None,
flush=lambda *a, **k: None,
)
async def __aexit__(self, *a):
return None
async def _fake_upsert(session, **kw):
return SimpleNamespace(id=1)
@pytest.mark.asyncio
async def test_avg_goals_no_data_returns_zero():
"""无历史数据时场均进球为 0(不抛异常)。"""
@@ -68,7 +87,9 @@ async def test_predict_baseline_no_llm():
match_status = "scheduled"
with patch("src.llm.baseline._avg_goals", fake_avg), \
patch("src.llm.baseline.AsyncSessionLocal") as SLC:
patch("src.llm.baseline.AsyncSessionLocal") as SLC, \
patch("src.db.unit_of_work.get_uow", _FakeUoW), \
patch("src.llm.baseline._upsert_prediction", _fake_upsert):
class FakeSession:
async def get(self, cls, mid):
return FakeMatch()
@@ -81,18 +102,20 @@ async def test_predict_baseline_no_llm():
result = await predict_baseline(1)
assert result["provider"] == "baseline"
assert result["model"] == "baseline"
assert result["mode"] == "baseline"
assert result["latency_ms"] == 0
assert result["prompt_tokens"] == 0
assert result["completion_tokens"] == 0
assert result.provider == "baseline"
assert result.model == "baseline"
assert result.mode == "baseline"
assert result.latency_ms == 0
assert result.prompt_tokens == 0
assert result.completion_tokens == 0
# 2.4 → round = 2, 1.6 → round = 2 → 平局 X
assert result["pred_home_goals"] == 2.0
assert result["pred_away_goals"] == 2.0
assert result["pred_1x2"] == "X"
assert result["subjective_confidence"] == 0.5
assert "非投注建议" in result["reasoning"]
assert result.pred_home_goals == 2.0
assert result.pred_away_goals == 2.0
assert result.pred_1x2 == "X"
assert result.subjective_confidence == 0.5
assert "非投注建议" in result.reasoning
# P3-2:服务层落库,回填真实 prediction_id
assert result.prediction_id == 1
# 确认未调用任何 LLM 相关模块
assert "home_10" in captured and "away_20" in captured
@@ -112,7 +135,9 @@ async def test_predict_baseline_clamps_to_range():
match_status = "scheduled"
with patch("src.llm.baseline._avg_goals", fake_avg), \
patch("src.llm.baseline.AsyncSessionLocal") as SLC:
patch("src.llm.baseline.AsyncSessionLocal") as SLC, \
patch("src.db.unit_of_work.get_uow", _FakeUoW), \
patch("src.llm.baseline._upsert_prediction", _fake_upsert):
class FakeSession:
async def get(self, cls, mid):
return FakeMatch()
@@ -125,6 +150,8 @@ async def test_predict_baseline_clamps_to_range():
result = await predict_baseline(2)
assert result["pred_home_goals"] == 10.0 # clamped
assert result["pred_away_goals"] == 0.0 # clamped
assert result["pred_1x2"] == "1" # 10:0 主胜
assert result.pred_home_goals == 10.0 # clamped
assert result.pred_away_goals == 0.0 # clamped
assert result.pred_1x2 == "1" # 10:0 主胜
# P3-2:服务层落库,回填真实 prediction_id
assert result.prediction_id == 1
+301
View File
@@ -0,0 +1,301 @@
"""D2 工程债回归测试: 统一预测结果类型。
背景: predict_match 三条路径返回类型不一 single 返回 PredictResult,
multi 返回字段重复定义的 MultiPredictResult dataclass,baseline 返回裸 dict
后果: (1) 预测路由 PredictOut 映射被迫写 isinstance(result, dict) 双分支;
(2) backtest baseline 模式直接 AttributeError(dict 没有 .prediction_id,
潜伏 bug);(3) 字段清单在两处 dataclass 重复维护,加字段必漏一处
统一方案: 扩展 PredictResult(可选字段)承载全部模式;
MultiPredictResult 变为其别名(保留 orchestrator 签名标记,兼容 re-export);
baseline 返回 PredictResult;路由单一字段映射
本测试守护四件事:
1. predict_baseline 返回 PredictResult(属性访问)
2. MultiPredictResult PredictResult 兼容(orchestrator 构造调用的
全字段 kwargs 可直接构造别名)
3. 预测路由不再有 isinstance(result, dict) 分支(源码守卫,仿 R4 范式)
4. _persist_baseline 用属性访问构造 upsert values(baseline 落库语义不变)
"""
from __future__ import annotations
from pathlib import Path
from types import SimpleNamespace
import pytest
from src.llm.baseline import predict_baseline
from src.llm.predict import PredictResult
from src.llm.agents import MultiPredictResult
ROUTE_PATH = Path(__file__).resolve().parents[1] / "src" / "api" / "routes" / "predict.py"
# ============================================================
# 1. baseline 返回 PredictResult
# ============================================================
@pytest.mark.asyncio
async def test_predict_baseline_returns_predict_result():
"""基线预测返回 PredictResult 实例,mode=baseline,token/延迟为 0。"""
from unittest.mock import patch
async def fake_avg(db, *, team_id, side, league_id, before):
return 2.4 if side == "home" else 1.6
class FakeMatch:
id = 1
home_team_id = 10
away_team_id = 20
league_id = 1
match_status = "scheduled"
class FakeSession:
async def get(self, cls, mid):
return FakeMatch()
class FakeCM:
async def __aenter__(self):
return FakeSession()
async def __aexit__(self, *a):
return None
# P3-2:baseline 在服务层落库(get_uow + _upsert_prediction),需 mock 掉。
class FakeUoW:
async def __aenter__(self):
return _make_session()
async def __aexit__(self, *a):
return None
captured = {}
async def fake_upsert(session, **kw):
captured.update(kw)
return SimpleNamespace(id=77)
# baseline.py 内部 from-import get_uow / _upsert_prediction,需 patch 真实来源模块。
with patch("src.llm.baseline._avg_goals", fake_avg), \
patch("src.llm.baseline.AsyncSessionLocal") as SLC, \
patch("src.db.unit_of_work.get_uow", FakeUoW), \
patch("src.llm.baseline._upsert_prediction", fake_upsert):
SLC.return_value = FakeCM()
result = await predict_baseline(1)
# P3-2:验证服务层落库被调用且属性映射正确
assert captured["match_id"] == 1
assert captured["provider_name"] == "baseline"
assert captured["run_type"] == "live"
assert captured["values"]["pred_home_goals"] == 2.0
assert isinstance(result, PredictResult)
assert result.mode == "baseline"
assert result.provider == "baseline"
assert result.model == "baseline"
assert result.prompt_version == "baseline_v1"
assert result.pred_home_goals == 2.0
assert result.pred_away_goals == 2.0
assert result.pred_1x2 == "X"
assert result.subjective_confidence == 0.5
assert result.prompt_tokens == 0
assert result.completion_tokens == 0
assert result.latency_ms == 0
assert result.status == "success"
assert "非投注建议" in (result.reasoning or "")
# baseline 不构建 LLM 上下文,但字段必须存在且可安全序列化
assert result.context == ""
# 原始统计快照保留在 raw 中
assert result.raw is not None
assert "home_avg" in result.raw
# ============================================================
# 2. MultiPredictResult 与扩展后的 PredictResult 兼容
# ============================================================
def test_multi_predict_result_is_predict_result_alias():
"""multi 结果不再是重复定义的 dataclass,而是扩展 PredictResult 的别名。"""
assert MultiPredictResult is PredictResult
def test_multi_result_constructor_kwargs_still_supported():
"""orchestrator 现有构造调用的全部字段 kwargs 必须仍可构造(别名完整性)。"""
# 与 orchestrator.predict_match_multi 的 return MultiPredictResult(...) 逐一对应
result = MultiPredictResult(
prediction_id=1,
provider="openai",
model="gpt-x",
prompt_version="multi_v1",
mode="multi",
pred_home_goals=2.0,
pred_away_goals=1.0,
alt_pred_home_goals=None,
alt_pred_away_goals=None,
pred_1x2="1",
subjective_confidence=0.7,
reasoning="r",
status="success",
agent_outputs=[{"agent": "form"}],
agent_weights={"form": 0.2},
context="ctx",
latency_ms=100,
prompt_tokens=10,
completion_tokens=5,
raw={"final": True},
)
assert result.mode == "multi"
assert result.agent_outputs == [{"agent": "form"}]
assert result.agent_weights == {"form": 0.2}
assert result.prompt_tokens == 10
assert result.completion_tokens == 5
# ============================================================
# 3. 路由去 dict 分支(源码守卫)
# ============================================================
def test_predict_route_has_no_dict_branch():
"""PredictOut 映射必须统一走属性访问,禁止 isinstance(result, dict) 回潮。"""
src = ROUTE_PATH.read_text(encoding="utf-8")
assert "isinstance(result, dict)" not in src
assert ".get(\"pred_home_goals\")" not in src
# ============================================================
# 4. P3-2:baseline 服务层落库属性映射(落库已从路由移到 baseline.py)
# ============================================================
class _FakeResult:
"""支持 .scalar_one_or_none() 的最小假结果集。"""
def __init__(self, items):
self._items = list(items)
def scalars(self):
return self
def all(self):
return self._items
def scalar_one_or_none(self):
return self._items[0] if self._items else None
class _FakeUoW:
"""替代 get_uow 的最小上下文管理器(session.execute 是 async 的)。"""
def __init__(self):
self.session = _make_session()
async def __aenter__(self):
return self.session
async def __aexit__(self, *a):
return None
def __call__(self):
return self
def _make_session(existing=None):
"""构造带 async execute / add / flush 的假 session。"""
sess = SimpleNamespace()
async def execute(*a, **k):
return _FakeResult(existing or [])
sess.execute = execute
sess.add = lambda *a, **k: None
async def flush(*a, **k):
return None
sess.flush = flush
return sess
@pytest.mark.asyncio
async def test_baseline_service_persists_with_correct_attributes(monkeypatch):
"""P3-2:baseline 在服务层(predict_baseline)落库,属性映射与路由旧版一致。"""
captured = {}
async def fake_upsert(session, **kwargs):
captured.update(kwargs)
return SimpleNamespace(id=77)
class FakeMatch:
id = 1
home_team_id = 10
away_team_id = 20
league_id = 1
match_status = "scheduled"
class FakeSession:
async def get(self, cls, mid):
return FakeMatch()
class FakeSLC:
async def __aenter__(self):
return FakeSession()
async def __aexit__(self, *a):
return None
async def fake_avg(db, *, team_id, side, league_id, before):
return 2.0 if side == "home" else 1.0
monkeypatch.setattr("src.llm.baseline._avg_goals", fake_avg)
monkeypatch.setattr("src.llm.baseline.AsyncSessionLocal", FakeSLC)
monkeypatch.setattr("src.db.unit_of_work.get_uow", lambda: _FakeUoW())
# baseline.py 模块级 import _upsert_prediction(第 15 行),需 patch baseline 模块属性
monkeypatch.setattr("src.llm.baseline._upsert_prediction", fake_upsert)
result = await predict_baseline(1)
# 落库被调用且属性映射正确
assert captured, f"predict_baseline 应调用 _upsert_prediction 落库,但 captured 为空(result.prediction_id={result.prediction_id!r})"
assert captured["match_id"] == 1
assert captured["provider_name"] == "baseline"
assert captured["model"] == "baseline"
assert captured["mode"] == "baseline"
assert captured["run_type"] == "live"
v = captured["values"]
assert v["prompt_version"] == "baseline_v1"
assert v["pred_home_goals"] == 2.0
assert v["pred_away_goals"] == 1.0
assert v["pred_1x2"] == "1"
assert v["subjective_confidence"] == 0.5
assert v["prompt_tokens"] == 0
assert v["completion_tokens"] == 0
assert v["latency_ms"] == 0
assert v["raw_response"] == {"home_avg": 2.0, "away_avg": 1.0}
assert v["status"] == "success"
# 回填真实 prediction_id(服务层落库后取得)
assert result.prediction_id == 77
assert result.pred_1x2 == "1"
assert captured["match_id"] == 1
assert captured["provider_name"] == "baseline"
assert captured["model"] == "baseline"
assert captured["mode"] == "baseline"
assert captured["run_type"] == "live"
v = captured["values"]
assert v["prompt_version"] == "baseline_v1"
assert v["pred_home_goals"] == 2.0
assert v["pred_away_goals"] == 1.0
assert v["pred_1x2"] == "1"
assert v["subjective_confidence"] == 0.5
assert v["prompt_tokens"] == 0
assert v["completion_tokens"] == 0
assert v["latency_ms"] == 0
assert v["raw_response"] == {"home_avg": 2.0, "away_avg": 1.0}
assert v["status"] == "success"
# 回填真实 prediction_id(服务层落库后取得)
assert result.prediction_id == 77
+259
View File
@@ -0,0 +1,259 @@
"""D1 工程债回归测试: events 成功路径必须写 Bronze 层(RawEvent + DataLineage)。
背景: stats 回填管线早已有 RawEvent/DataLineage 写入, events 管线(比赛
主数据的唯一入口)成功插入/更新后既不留原始载荷,也不留血缘 数据溯源
链条在最关键的一环断掉本测试守护:
1. 插入新比赛 RawEvent(幂等键=source_event_id 或合成键) + Lineage
(target_table="matches", transform_name="events_ingest")
2. 变更更新(如补比分/状态) 同样写血缘
3. 无变化跳过 不写(避免 lineage 刷屏)
4. RawEvent 幂等: source_record_id 已存在则跳过
5. 基础设施写入失败 warning,不拖垮采集主流程
范式: db(按查询实体分发预置数据 + 记录 add,flush 分配自增 id)
+ monkeypatch 抓取函数,不依赖真实数据库
"""
from __future__ import annotations
from datetime import date, datetime, timezone
import pytest
import src.data.bzzoiro as bz
from src.db.models import DataLineage, League, Match, RawEvent, Team
def _event(eid=1001, status="finished", home="Arsenal", away="Chelsea", hs=2, as_=1):
"""构造一条最小合法的 bzzoiro /events/ 原始载荷。"""
raw = {
"event_date": "2026-09-20 15:00:00",
"status": status,
"home_team": home,
"away_team": away,
"home_score": hs,
"away_score": as_,
}
if eid is not None:
raw["id"] = eid
return raw
class _FakeResult:
"""支持 .scalars().all() / .scalar_one_or_none() 的最小假结果集。"""
def __init__(self, items):
self._items = list(items)
def scalars(self):
return self
def __iter__(self):
return iter(self._items)
def all(self):
return self._items
def scalar_one_or_none(self):
return self._items[0] if self._items else None
def scalar(self):
return None
class _FakeDB:
"""按查询实体分发预置数据;记录 add();flush 为无 id 对象分配自增主键。"""
def __init__(self, matches=(), teams=(), leagues=(), raw_events=()):
self.added = []
self._by_entity = {
Match: list(matches),
Team: list(teams),
League: list(leagues),
RawEvent: list(raw_events),
}
self._next_id = 0
def add(self, obj):
self.added.append(obj)
async def execute(self, stmt):
entities = set()
for d in (stmt.column_descriptions or []):
entities.add(d.get("entity") or d.get("type"))
for entity, items in self._by_entity.items():
if entity in entities:
return _FakeResult(items)
return _FakeResult([])
async def flush(self):
for obj in self.added:
if getattr(obj, "id", None) is None:
self._next_id += 1
obj.id = self._next_id
@pytest.fixture(autouse=True)
def _no_request_interval(monkeypatch):
monkeypatch.setattr(bz, "REQUEST_INTERVAL", 0)
def _patch_fetch(monkeypatch, events):
async def _fetch(league_code, **kwargs):
return list(events)
monkeypatch.setattr(bz, "fetch_bzzoiro_events", _fetch)
def _matches(db):
return [o for o in db.added if isinstance(o, Match)]
def _raw_events(db):
return [o for o in db.added if isinstance(o, RawEvent)]
def _lineages(db):
return [o for o in db.added if isinstance(o, DataLineage)]
# ============================================================
# 1. 插入新比赛 → RawEvent + DataLineage
# ============================================================
class TestEventsBronzeOnInsert:
async def test_insert_writes_raw_event_and_lineage(self, monkeypatch):
_patch_fetch(monkeypatch, [_event()])
db = _FakeDB()
result = await bz.BzzoiroSource().ingest(db, leagues=["E0"])
assert result["total_inserted"] == 1
raws = _raw_events(db)
assert len(raws) == 1
raw = raws[0]
assert raw.source_system == "bzzoiro"
assert raw.source_record_id == "1001" # 有上游 id 时直接用
assert raw.ingest_batch_id.startswith("bzzoiro-events-E0-")
assert raw.raw_payload["id"] == 1001 # 原始载荷完整保留
lineages = _lineages(db)
assert len(lineages) == 1
lin = lineages[0]
assert lin.source_system == "bzzoiro"
assert lin.source_record_id == "1001"
assert lin.target_table == "matches"
assert lin.target_id == _matches(db)[0].id
assert lin.transform_name == "events_ingest"
# RawEvent 与 Lineage 同批次,便于按批追溯
assert lin.batch_id == raw.ingest_batch_id
async def test_missing_source_id_uses_synthetic_stable_key(self, monkeypatch):
"""上游 id 缺失时,用 (league:home:away:date) 合成稳定幂等键。"""
_patch_fetch(monkeypatch, [_event(eid=None)])
db = _FakeDB()
result = await bz.BzzoiroSource().ingest(db, leagues=["E0"])
assert result["total_inserted"] == 1
raws = _raw_events(db)
assert len(raws) == 1
# 期望键基于 normalize 后的队名与天级日期 —— 与 _match_key 同口径,
# 不依赖 DB 自增 id,跨批次可复现
nm = bz.normalize_bzzoiro(_event(eid=None), "E0")
expected = f"E0:{nm.home_team}:{nm.away_team}:{nm.date.date().isoformat()}"
assert raws[0].source_record_id == expected
async def test_existing_raw_event_is_skipped(self, monkeypatch):
"""RawEvent 幂等: 同 source_record_id 已存在则不再新增,但血缘照写。"""
existing = RawEvent(
source_system="bzzoiro",
source_record_id="1001",
raw_payload={"old": True},
)
_patch_fetch(monkeypatch, [_event()])
db = _FakeDB(raw_events=[existing])
await bz.BzzoiroSource().ingest(db, leagues=["E0"])
new_raws = [r for r in _raw_events(db) if r is not existing]
assert new_raws == []
assert len(_lineages(db)) == 1 # 血缘仍然记录本次采集
# ============================================================
# 2. 变更更新 → 写血缘;无变化 → 不写
# ============================================================
class TestEventsBronzeOnUpdate:
def _existing_match(self, **overrides):
m = Match(
league_id=1,
home_team_id=2, # 与本轮 Team 创建后 fake 自增 id 对齐(league=1, home=2, away=3)
away_team_id=3,
match_date=datetime(2026, 9, 20, 15, 0, tzinfo=timezone.utc),
match_date_date=date(2026, 9, 20),
match_status="scheduled",
source_event_id=1001,
)
m.id = 42
for k, v in overrides.items():
setattr(m, k, v)
return m
async def test_changed_update_writes_lineage(self, monkeypatch):
# 已有比赛处于 scheduled 且无比分;新载荷为 finished 2:1 → 触发变更更新
db = _FakeDB(matches=[self._existing_match()])
_patch_fetch(monkeypatch, [_event()])
result = await bz.BzzoiroSource().ingest(db, leagues=["E0"])
assert result["total_inserted"] == 0
assert result["leagues"]["E0"]["updated"] == 1
lineages = _lineages(db)
assert len(lineages) == 1
assert lineages[0].target_id == 42
assert lineages[0].target_table == "matches"
assert lineages[0].transform_name == "events_ingest"
async def test_unchanged_match_writes_nothing(self, monkeypatch):
# 已有比赛与新载荷完全一致 → 无变化,不应产生 RawEvent/Lineage
existing = self._existing_match(
match_status="finished",
home_goals=2,
away_goals=1,
)
db = _FakeDB(matches=[existing])
_patch_fetch(monkeypatch, [_event()])
result = await bz.BzzoiroSource().ingest(db, leagues=["E0"])
assert result["total_inserted"] == 0
assert result["leagues"]["E0"]["updated"] == 0
assert _raw_events(db) == []
assert _lineages(db) == []
# ============================================================
# 3. 基础设施写入失败: 尽力而为,不拖垮主流程
# ============================================================
class TestEventsBronzeIsBestEffort:
async def test_bronze_write_failure_does_not_break_ingest(self, monkeypatch):
async def _boom(*args, **kwargs):
raise RuntimeError("infra down")
monkeypatch.setattr(bz, "_write_raw_event", _boom)
monkeypatch.setattr(bz, "_write_lineage", _boom)
_patch_fetch(monkeypatch, [_event()])
db = _FakeDB()
# 不应抛异常:Bronze 写不进去只记 warning
result = await bz.BzzoiroSource().ingest(db, leagues=["E0"])
assert result["total_inserted"] == 1
assert len(_matches(db)) == 1
+163
View File
@@ -0,0 +1,163 @@
"""公开只读 API 回归(P1-2 / P1-3):context 与 leagues 匿名可访问。
背景:
- 公开站 MatchDetailSection 展开详情会请求 /matches/{id}/context,
此前该端点挂 require_admin,未登录 401 fetchMatchContext
catch 吞掉 近况/交锋静默为空;
- 公开站联赛筛选需要 /leagues,此前同样 require_admin,前端写死五大联赛
守卫(双保险):
1. 功能层:最小 FastAPI app + dependency_overrides 注入 fake session,
匿名请求(无任何凭据) 200;不存在的 match 404
(不启动完整 app lifespan,遵循 test_api_critical.py 的既定约束)
2. 鉴权层:检查路由依赖声明,require_admin 不得出现在
/leagues /matches/{match_id}/contextdev 环境 require_admin
未配置鉴权时 fail-open,功能层测不出加回了 require_admin的回归,
必须靠本层声明检查;并用 ingest 路由证明检查器本身有判别力
(变异保护:若有人给公开端点加回 require_admin,此处变红)
"""
from __future__ import annotations
from datetime import datetime, timezone
from fastapi import FastAPI
from fastapi.testclient import TestClient
from src.api.deps import require_admin
from src.api.routes.ingest import router as ingest_router
from src.api.routes.matches import router as matches_router
from src.db.base import get_db_read
from src.db.models import League, Match
# ── fake DB(对齐 routes/matches.py 的实际查询面) ──────────────────
class _FakeResult:
def __init__(self, items):
self._items = list(items)
def scalars(self):
return self
def all(self):
return list(self._items)
def scalar_one_or_none(self):
return self._items[0] if self._items else None
class _FakeDB:
"""match_context 的查询次序:① match 主查询(scalar_one_or_none)
home_recent away_recent h2h( scalars().all())
League 查询按实体识别直接返回列表(对应 /leagues)"""
def __init__(self, match=None, match_lists=(), leagues=()):
self._match = match
self._match_lists = list(match_lists)
self._leagues = list(leagues)
self._calls = 0
async def execute(self, stmt):
entity = stmt.column_descriptions[0]["entity"]
if entity is League:
return _FakeResult(self._leagues)
if self._calls == 0:
self._calls += 1
return _FakeResult([self._match] if self._match is not None else [])
idx = self._calls - 1
self._calls += 1
return _FakeResult(self._match_lists[idx] if idx < len(self._match_lists) else [])
def _client(fake_db: _FakeDB) -> TestClient:
app = FastAPI()
app.include_router(matches_router)
app.dependency_overrides[get_db_read] = lambda: fake_db
# 不用 with:不触发 lifespan,无真实 DB 引擎连接
return TestClient(app)
def _league(lid: int, code: str) -> League:
return League(id=lid, code=code, name=f"League {code}", country=f"Country {code}")
def _match(mid: int) -> Match:
return Match(
id=mid,
match_date=datetime(2026, 9, 20, 15, 0, tzinfo=timezone.utc),
home_goals=2,
away_goals=1,
)
# ── 功能层:匿名可访问(P1-2 / P1-3) ──────────────────────────────
def test_leagues_anonymous_200_and_shape():
"""/leagues 匿名 200;仅暴露 id/code/name/country 四字段。"""
db = _FakeDB(leagues=[_league(1, "E0"), _league(2, "SP1")])
r = _client(db).get("/api/v1/leagues")
assert r.status_code == 200
body = r.json()
assert body == [
{"id": 1, "code": "E0", "name": "League E0", "country": "Country E0"},
{"id": 2, "code": "SP1", "name": "League SP1", "country": "Country SP1"},
]
# 不暴露敏感配置字段
assert all(set(item.keys()) == {"id", "code", "name", "country"} for item in body)
def test_context_anonymous_200_empty_data():
"""/context 匿名 200;无数据时三个列表为空(前端空态),结构不变。"""
db = _FakeDB(match=_match(42), match_lists=[[], [], []])
r = _client(db).get("/api/v1/matches/42/context")
assert r.status_code == 200
assert r.json() == {"home_recent": [], "away_recent": [], "h2h": []}
def test_context_anonymous_200_row_shape():
"""/context 行结构与既有前端契约一致(5 字段)。"""
db = _FakeDB(match=_match(42), match_lists=[[_match(1)], [], [_match(2), _match(3)]])
r = _client(db).get("/api/v1/matches/42/context")
assert r.status_code == 200
body = r.json()
assert len(body["home_recent"]) == 1
assert len(body["h2h"]) == 2
assert set(body["h2h"][0].keys()) == {
"match_date", "home_team", "away_team", "home_goals", "away_goals",
}
def test_context_not_found_404():
"""/context 匿名访问不存在的 match → 404(而非 401/503)。"""
db = _FakeDB(match=None)
r = _client(db).get("/api/v1/matches/999/context")
assert r.status_code == 404
# ── 鉴权层:路由依赖声明检查(防 require_admin 回潜) ────────────────
def _admin_paths(router) -> set[str]:
paths: set[str] = set()
for route in router.routes:
for dep in route.dependant.dependencies:
if dep.call is require_admin:
paths.add(route.path)
break
return paths
def test_public_routes_have_no_admin_dependency():
"""/leagues 与 /context 不得挂 require_admin;matches 路由全部公开只读。"""
admin_paths = _admin_paths(matches_router)
assert "/api/v1/leagues" not in admin_paths
assert "/api/v1/matches/{match_id}/context" not in admin_paths
assert admin_paths == set(), f"matches 路由应全部公开只读,仍有 {admin_paths}"
def test_guard_detector_has_discrimination_power():
"""变异保护:检查器必须能在 ingest 路由上发现 require_admin,
否则上一条 admin 依赖断言恒真毫无判别力"""
assert "/api/v1/ingest/bzzoiro" in _admin_paths(ingest_router), (
"ingest 路由应仍存在 require_admin 保护;若本断言失败,"
"说明公开只读守卫的检查器已失效,请修复检查逻辑"
)
+3 -3
View File
@@ -175,7 +175,7 @@ class TestBzzoiroLineage:
return bad
def test_normalized_matches_carries_raw(self):
src = _read("data/bzzoiro.py")
src = _read("data/bzzoiro_events.py")
# 规范化结果必须与原始 event 成对保存
assert "normalized_matches.append((nm, raw))" in src, (
"normalized_matches 未携带 (nm, raw) 元组 —— raw 变量泄漏会回归 (P0-3)"
@@ -193,7 +193,7 @@ class TestBzzoiroLineage:
它不是 `raw.get(` 同一行,但同样正确非法写法(回归)是直接
`existing_match.source_event_id = orphan_var`
"""
src = _read("data/bzzoiro.py")
src = _read("data/bzzoiro_events.py")
seg = self._consume_loop_body(src)
bad = self._bad_assignments(seg)
assert len(bad) == 0, (
@@ -206,7 +206,7 @@ class TestBzzoiroLineage:
下游 `_backfill_stats` 里合法地在 ORM 对象上访问 `m.source_event_id`
(与配对 raw 无关) seg 越界,test_no_orphan_raw_use 会误报
"""
src = _read("data/bzzoiro.py")
src = _read("data/bzzoiro_events.py")
seg = self._consume_loop_body(src)
assert "m.source_event_id" not in seg, (
"循环体截取越界,扫到了下游 stats 管线 —— 会误报 P0-3"
+5 -2
View File
@@ -125,7 +125,7 @@ class _FakeDb:
async def test_r2_standings_actually_upserts(monkeypatch):
"""行为测试: 喂一份积分榜 payload,断言真的构造了 Standing 且计数 > 0。"""
import src.data.bzzoiro as bz
from src.db.models import League, Standing, Team
from src.db.models import DataLineage, League, RawEvent, Standing, Team
payload = {
"season": {"start_date": "2025-08-01", "end_date": "2026-05-31"},
@@ -166,7 +166,10 @@ async def test_r2_standings_actually_upserts(monkeypatch):
standings = [o for o in db.added if isinstance(o, Standing)]
assert len(standings) == 2, "应真的构造 Standing 行"
assert all(isinstance(o, (Standing, Team)) for o in db.added)
# standings 采集接线 Bronze 后(RawEvent + DataLineage),add 的对象类型白名单随之放宽
assert all(
isinstance(o, (Standing, Team, RawEvent, DataLineage)) for o in db.added
)
first = standings[0]
assert first.league_id == 42
+288
View File
@@ -0,0 +1,288 @@
"""standings 成功路径 Bronze 层回归测试(RawEvent + DataLineage)。
背景: events/stats 管线成功后均已补写 Bronze ,唯独 standings 采集
成功后既不留原始载荷,也不留血缘 三条管线的溯源链条在积分榜一环
缺失本测试守护( test_events_bronze.py 对称):
1. 联赛成功 upsert RawEvent(幂等键=standings:{league}:{season})
+ Lineage(target_table="standings", transform_name="standings_ingest")
2. 更新已有快照(非插入)同样写 Bronze 积分榜是快照,刷新即采集
3. RawEvent 幂等: source_record_id 已存在则跳过,血缘照写
4. 基础设施写入失败 warning,不拖垮采集主流程
5. 抓取失败路径继续走 _safe_write_ingest_failure,且不写 Bronze
范式: db(按查询实体分发预置数据 + 记录 add,flush 分配自增 id)
+ monkeypatch 抓取函数,不依赖真实数据库
"""
from __future__ import annotations
import pytest
import src.data.bzzoiro as bz
from src.db.models import DataLineage, IngestFailure, League, RawEvent, Standing, Team
def _payload():
"""构造一份最小合法的 bzzoiro /leagues/{id}/standings/ 原始载荷。"""
return {
"season": {"start_date": "2025-08-01", "end_date": "2026-05-31"},
"standings": [
{
"position": 1, "team_name": "Arsenal FC",
"played": 10, "won": 8, "drawn": 1, "lost": 1,
"gf": 22, "ga": 8, "gd": 14, "pts": 25,
"zone": {"key": "champions_league", "label": "Champions League"},
},
{
"position": 2, "team_name": "Chelsea FC",
"played": 10, "won": 6, "drawn": 2, "lost": 2,
"gf": 18, "ga": 12, "gd": 6, "pts": 20,
},
],
}
def _patch_fetch(monkeypatch, payload):
async def _fetch(league_code, season=None):
return payload
monkeypatch.setattr(bz, "fetch_bzzoiro_standings", _fetch)
class _FakeResult:
"""支持 .scalars().all() / .scalar_one_or_none() 的最小假结果集。"""
def __init__(self, items):
self._items = list(items)
def scalars(self):
return self
def __iter__(self):
return iter(self._items)
def all(self):
return self._items
def scalar_one_or_none(self):
return self._items[0] if self._items else None
class _FakeDB:
"""按查询实体分发预置数据;记录 add();flush 为无 id 对象分配自增主键。"""
def __init__(self, leagues=(), teams=(), standings=(), raw_events=()):
self.added = []
self._by_entity = {
League: list(leagues),
Team: list(teams),
Standing: list(standings),
RawEvent: list(raw_events),
}
self._next_id = 0
def add(self, obj):
self.added.append(obj)
async def execute(self, stmt):
entities = set()
for d in (stmt.column_descriptions or []):
entities.add(d.get("entity") or d.get("type"))
for entity, items in self._by_entity.items():
if entity in entities:
return _FakeResult(self._filter(entity, items, stmt))
return _FakeResult([])
@staticmethod
def _filter(entity, items, stmt):
"""RawEvent 查询按 source_record_id 过滤 —— 幂等测试需区分不同键。"""
if entity is RawEvent:
try:
params = stmt.compile().params
except Exception:
return items
rid = next((v for k, v in params.items() if "source_record_id" in k), None)
if rid is not None:
return [i for i in items if i.source_record_id == rid]
return items
async def flush(self):
for obj in self.added:
if getattr(obj, "id", None) is None:
self._next_id += 1
obj.id = self._next_id
def _preset_league():
lg = League(code="EPL", name="Premier League", country="England")
lg.id = 42
return lg
def _raw_events(db):
return [o for o in db.added if isinstance(o, RawEvent)]
def _lineages(db):
return [o for o in db.added if isinstance(o, DataLineage)]
# ============================================================
# 1. 成功 upsert → RawEvent + DataLineage
# ============================================================
class TestStandingsBronzeOnUpsert:
async def test_upsert_writes_raw_event_and_lineage(self, monkeypatch):
_patch_fetch(monkeypatch, _payload())
db = _FakeDB(leagues=[_preset_league()])
result = await bz.ingest_bzzoiro_standings(db, leagues=["EPL"])
assert result["errors"] == []
assert result["total_upserted"] == 2
raws = _raw_events(db)
assert len(raws) == 1
raw = raws[0]
assert raw.source_system == "bzzoiro"
# 幂等键: 联赛 + 实际入库的赛季标签(由载荷日期推导,与 Standing.season 同口径)
assert raw.source_record_id == "standings:EPL:2025-2026"
assert raw.ingest_batch_id.startswith("bzzoiro-standings-EPL-")
# 整份原始载荷完整保留
assert raw.raw_payload["standings"][0]["team_name"] == "Arsenal FC"
lineages = _lineages(db)
assert len(lineages) == 1
lin = lineages[0]
assert lin.source_system == "bzzoiro"
assert lin.source_record_id == "standings:EPL:2025-2026"
assert lin.target_table == "standings"
assert lin.target_id == 42 # 联赛 id
assert lin.transform_name == "standings_ingest"
assert lin.transform_detail == {
"league": "EPL", "season": "2025-2026", "rows_upserted": 2,
}
# RawEvent 与 Lineage 同批次,便于按批追溯
assert lin.batch_id == raw.ingest_batch_id
async def test_updated_snapshot_also_writes_bronze(self, monkeypatch):
"""已有快照就地更新(非插入)同样是成功采集,必须留 Bronze 记录。"""
payload = _payload()
payload["standings"] = payload["standings"][:1] # 单队,便于命中同一行
_patch_fetch(monkeypatch, payload)
team = Team(name="Arsenal FC", name_zh="阿森纳")
team.id = 7
existing = Standing(league_id=42, season="2025-2026", team_id=7, position=9)
existing.points = 1
db = _FakeDB(leagues=[_preset_league()], teams=[team], standings=[existing])
result = await bz.ingest_bzzoiro_standings(db, leagues=["EPL"])
assert result["total_upserted"] == 1
assert result["leagues"]["EPL"]["teams_created"] == 0
# 快照刷新也要留痕: RawEvent(幂等) + 血缘
assert len(_raw_events(db)) == 1
lineages = _lineages(db)
assert len(lineages) == 1
assert lineages[0].transform_name == "standings_ingest"
assert lineages[0].transform_detail["rows_upserted"] == 1
async def test_empty_upsert_writes_no_bronze(self, monkeypatch):
"""载荷有行但全部队名为空 → 没有任何 upsert,不应产生 RawEvent/Lineage。"""
payload = {"standings": [{"position": 1, "team_name": ""}]}
_patch_fetch(monkeypatch, payload)
db = _FakeDB(leagues=[_preset_league()])
result = await bz.ingest_bzzoiro_standings(db, leagues=["EPL"])
assert result["total_upserted"] == 0
assert _raw_events(db) == []
assert _lineages(db) == []
# ============================================================
# 2. RawEvent 幂等: 同 source_record_id 跳过
# ============================================================
class TestStandingsRawEventIdempotent:
async def test_existing_raw_event_is_skipped(self, monkeypatch):
existing = RawEvent(
source_system="bzzoiro",
source_record_id="standings:EPL:2025-2026",
raw_payload={"old": True},
)
_patch_fetch(monkeypatch, _payload())
db = _FakeDB(leagues=[_preset_league()], raw_events=[existing])
await bz.ingest_bzzoiro_standings(db, leagues=["EPL"])
new_raws = [r for r in _raw_events(db) if r is not existing]
assert new_raws == []
assert len(_lineages(db)) == 1 # 血缘仍然记录本次采集
async def test_different_season_writes_new_raw_event(self, monkeypatch):
"""幂等键含赛季: 同联赛不同赛季各留一条 RawEvent。"""
payload = _payload()
payload["season"] = {"start_date": "2024-08-01", "end_date": "2025-05-31"}
existing = RawEvent(
source_system="bzzoiro",
source_record_id="standings:EPL:2025-2026",
raw_payload={"old": True},
)
_patch_fetch(monkeypatch, payload)
db = _FakeDB(leagues=[_preset_league()], raw_events=[existing])
await bz.ingest_bzzoiro_standings(db, leagues=["EPL"])
new_raws = [r for r in _raw_events(db) if r is not existing]
assert len(new_raws) == 1
assert new_raws[0].source_record_id == "standings:EPL:2024-2025"
# ============================================================
# 3. 基础设施写入失败: 尽力而为,不拖垮主流程
# ============================================================
class TestStandingsBronzeIsBestEffort:
async def test_bronze_write_failure_does_not_break_ingest(self, monkeypatch):
async def _boom(*args, **kwargs):
raise RuntimeError("infra down")
monkeypatch.setattr(bz, "_write_raw_event", _boom)
monkeypatch.setattr(bz, "_write_lineage", _boom)
_patch_fetch(monkeypatch, _payload())
db = _FakeDB(leagues=[_preset_league()])
# 不应抛异常:Bronze 写不进去只记 warning
result = await bz.ingest_bzzoiro_standings(db, leagues=["EPL"])
assert result["total_upserted"] == 2
assert [o for o in db.added if isinstance(o, Standing)]
# ============================================================
# 4. 抓取失败: 继续写死信,且不写 Bronze
# ============================================================
class TestStandingsFailurePathKeepsDeadLetter:
async def test_fetch_failure_writes_deadletter_and_no_bronze(self, monkeypatch):
async def _boom(league_code, season=None):
raise RuntimeError("upstream 500")
monkeypatch.setattr(bz, "fetch_bzzoiro_standings", _boom)
db = _FakeDB()
result = await bz.ingest_bzzoiro_standings(db, leagues=["SP1"], season="2025-2026")
assert result["errors"]
failures = [o for o in db.added if isinstance(o, IngestFailure)]
assert len(failures) == 1
assert failures[0].entity_type == "standings"
assert failures[0].error_type == "fetch_error"
# 失败路径绝不写 Bronze(没有任何成功 upsert)
assert _raw_events(db) == []
assert _lineages(db) == []