Compare commits
6
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c7be14c518 | ||
|
|
983363dab7 | ||
|
|
6bdb1f8ae6 | ||
|
|
80616cf459 | ||
|
|
a24017eb69 | ||
|
|
3056a95ef4 |
@@ -40,7 +40,6 @@ LLM_TIMEOUT=60
|
||||
|
||||
# ---- 数据源 ----
|
||||
BZZOIRO_KEY=
|
||||
API_FOOTBALL_KEY=
|
||||
|
||||
# ---- CORS ----
|
||||
CORS_ORIGINS=http://localhost:5173,http://localhost:3000
|
||||
|
||||
@@ -12,28 +12,30 @@
|
||||
│ REST API
|
||||
┌──────────────────────▼──────────────────────────────┐
|
||||
│ FastAPI │
|
||||
│ ├── /api/v1/matches 比赛查询 │
|
||||
│ ├── /api/v1/matches 比赛查询(公开只读) │
|
||||
│ ├── /api/v1/predict LLM 预测 (单/多 Agent) │
|
||||
│ ├── /api/v1/ingest/* 数据采集 │
|
||||
│ ├── /api/v1/eval/* 评估回填 │
|
||||
│ └── /api/v1/backtest 回测 │
|
||||
│ ├── /api/v1/ingest/* 数据采集(需管理员) │
|
||||
│ ├── /api/v1/eval/* 评估回填(需管理员) │
|
||||
│ └── /api/v1/backtest 回测(需管理员) │
|
||||
└──────────┬─────────────────────────────┬────────────┘
|
||||
│ │
|
||||
┌──────────▼──────────┐ ┌─────────────▼────────────┐
|
||||
│ PostgreSQL │ │ LLM (OpenAI-compatible) │
|
||||
│ 6 张表 │ │ OpenAI / Deepseek / │
|
||||
│ 13 张表 │ │ OpenAI / Deepseek / │
|
||||
│ leagues/teams/ │ │ Ollama / 任意网关 │
|
||||
│ matches/match_ │ └──────────────────────────┘
|
||||
│ stats/predictions/ │
|
||||
│ injuries │
|
||||
│ stats/standings/ │
|
||||
│ predictions/ │
|
||||
│ app_settings/ │
|
||||
│ schedules + │
|
||||
│ raw_events 等 4 张 │
|
||||
│ 数据治理表 │
|
||||
└─────────────────────┘
|
||||
▲
|
||||
│ 采集
|
||||
┌──────────┴─────────────────────────────────────────┐
|
||||
│ 数据源 (DataSource 协议 + 注册表) │
|
||||
│ ├── bzzoiro 比分 / 统计 / xG │
|
||||
│ ├── understat xG 回填 │
|
||||
│ └── injuries 伤停数据 (api-football) │
|
||||
│ └── bzzoiro 比分 / 赛程 / 统计 / 积分榜 │
|
||||
└────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
@@ -55,7 +57,7 @@ API Route → Application Service → Repository → UnitOfWork → DB
|
||||
比赛数据 → 切片 ─┬─→ A 近期状态专家 ─┐
|
||||
├─→ B 攻防数据专家 ─┤
|
||||
├─→ C 主客因素专家 ─┼─→ 终裁专家 ─→ 最终预测
|
||||
├─→ D 阵容完整专家 ─┤
|
||||
├─→ D 联赛排名专家 ─┤
|
||||
└─→ E 历史交锋专家 ─┘
|
||||
```
|
||||
|
||||
@@ -66,8 +68,7 @@ API Route → Application Service → Repository → UnitOfWork → DB
|
||||
|
||||
### 数据正确性保障
|
||||
|
||||
- **Cutoff 机制**: 回测时只使用 `cutoff_at` 之前已采集的数据
|
||||
- **Injury 防泄漏**: 伤停查询强制 `retrieved_at <= cutoff`
|
||||
- **Cutoff 机制**: 回测时只使用 `cutoff_at` 之前已采集的数据(近况/交锋/统计/积分榜切片统一生效)
|
||||
- **LLM 输出校验**: Pydantic 严格校验 + 语义一致性检查
|
||||
- **数据库约束**: CHECK 约束作为最后一道防线
|
||||
|
||||
@@ -130,6 +131,21 @@ cd frontend && npm install && npm run dev
|
||||
|
||||
后端运行在 `http://localhost:8000`,前端在 `http://localhost:5173`。
|
||||
|
||||
## 生产上线检查清单
|
||||
|
||||
公网部署前逐项确认(第 1–4 项由启动校验强制,不满足拒绝启动;详见 [docs/06-deployment.md](docs/06-deployment.md#生产上线检查清单)):
|
||||
|
||||
- [ ] `APP_ENV=production`(安全校验 / Cookie `Secure` / 管理端点 fail-closed 的总开关)
|
||||
- [ ] `SECRET_KEY` 强随机:`openssl rand -base64 32`,禁止弱值
|
||||
- [ ] `ADMIN_PASSWORD` 或 `ADMIN_API_KEY` 至少配置其一
|
||||
- [ ] 数据库强密码,禁止 `football:football` 等示例弱密码
|
||||
- [ ] HTTPS(反代终结 TLS;production 下会话 Cookie 自动 `Secure`)
|
||||
- [ ] 反代后设 `TRUST_PROXY_HEADERS=True`,仅可信反代可达 API,并配置 `X-Forwarded-For` / `X-Real-IP`
|
||||
- [ ] 限流前置到 Nginx `limit_req`;应用内限流与 KeyRing 仅单进程有效,多 worker 会放大配额
|
||||
- [ ] uvicorn 单 worker(默认);需扩容先网关统一限流再起多实例
|
||||
- [ ] 启动后验证 `/health` 与 `/health/ready` 均 200
|
||||
- [ ] 数据库迁移已内置:compose/Dockerfile 启动即执行 `alembic upgrade head`
|
||||
|
||||
## 安全与限流
|
||||
|
||||
- `/api/v1/predict`: 内存滑动窗口限流(10 次/分钟/IP),多 worker 时每进程独立计数
|
||||
@@ -141,20 +157,29 @@ cd frontend && npm install && npm run dev
|
||||
|
||||
## API 概览
|
||||
|
||||
**公开只读**(无需登录;`predict` 带内存限流):
|
||||
|
||||
| 方法 | 路径 | 说明 |
|
||||
|---|---|---|
|
||||
| GET | `/api/v1/matches` | 比赛查询(筛选/分页) |
|
||||
| GET | `/api/v1/leagues` | 联赛列表 |
|
||||
| POST | `/api/v1/predict` | LLM 预测 (`mode=single`/`multi`) |
|
||||
| GET | `/api/v1/predictions` | 预测历史 |
|
||||
| POST | `/api/v1/ingest/bzzoiro` | 采集比分/统计 |
|
||||
| POST | `/api/v1/ingest/understat` | 回填 xG |
|
||||
| POST | `/api/v1/ingest/injuries` | 采集伤停 |
|
||||
| GET | `/api/v1/leagues` | 联赛列表(仅 id/code/name/country) |
|
||||
| GET | `/api/v1/matches` | 比赛查询(筛选/游标分页) |
|
||||
| GET | `/api/v1/matches/{id}` | 比赛详情(含统计与最近预测) |
|
||||
| GET | `/api/v1/matches/{id}/context` | 比赛上下文(双方近况 + 历史交锋) |
|
||||
| GET | `/api/v1/standings` | 联赛积分榜 |
|
||||
| POST | `/api/v1/predict` | LLM 预测 (`mode=single`/`multi`/`baseline`) |
|
||||
| GET | `/health`、`/health/ready` | 存活 / 就绪检查(含 DB) |
|
||||
|
||||
**需管理员**(Cookie 会话或 `X-API-Key`):
|
||||
|
||||
| 方法 | 路径 | 说明 |
|
||||
|---|---|---|
|
||||
| POST | `/api/v1/ingest/bzzoiro` | 采集赛果/赛程/统计/积分榜 |
|
||||
| GET | `/api/v1/predictions` | 预测历史(列表) |
|
||||
| GET | `/api/v1/predictions/{id}` | 单条预测详情 |
|
||||
| POST | `/api/v1/eval/settle` | 回填实际结果 |
|
||||
| GET | `/api/v1/eval/summary` | 准确率汇总 |
|
||||
| POST | `/api/v1/backtest` | 历史回测 |
|
||||
| GET | `/health` | 存活检查 |
|
||||
| GET | `/health/ready` | 就绪检查(含 DB) |
|
||||
| `/api/v1/admin/**` | 配置/采集状态/日志/定时任务/死信等 | 管理后台(router 级鉴权) |
|
||||
|
||||
## 项目结构
|
||||
|
||||
@@ -163,36 +188,46 @@ Profeto/
|
||||
├── src/
|
||||
│ ├── api/ # FastAPI 路由层
|
||||
│ │ ├── app.py # 应用工厂 + lifespan
|
||||
│ │ ├── deps.py # 依赖注入:鉴权 / 限流
|
||||
│ │ ├── schemas.py # Pydantic 请求/响应模型
|
||||
│ │ └── routes/
|
||||
│ │ ├── matches.py # 比赛查询
|
||||
│ │ ├── predict.py # 预测入口
|
||||
│ │ ├── ingest.py # 数据采集
|
||||
│ │ ├── eval.py # 评估回填
|
||||
│ │ └── backtest.py # 回测
|
||||
│ │ ├── matches.py # 比赛查询(公开只读)
|
||||
│ │ ├── predict.py # 预测入口 + 预测历史
|
||||
│ │ ├── ingest.py # 数据采集(需管理员)
|
||||
│ │ ├── eval.py # 评估回填(需管理员)
|
||||
│ │ ├── backtest.py # 回测(需管理员)
|
||||
│ │ ├── auth.py # 登录/登出/改密
|
||||
│ │ ├── admin_settings.py # /admin/** 配置/日志/数据质量(router 级鉴权)
|
||||
│ │ └── schedules.py # 定时任务 + 死信重试(router 级鉴权)
|
||||
│ ├── core/ # 基础设施
|
||||
│ │ ├── config.py # pydantic-settings 配置
|
||||
│ │ ├── crypto.py # 加密/哈希
|
||||
│ │ ├── http_client.py # 共享 httpx 客户端
|
||||
│ │ └── retry.py # 重试工具(指数退避)
|
||||
│ │ ├── log_buffer.py # 内存日志缓冲(admin 日志页)
|
||||
│ │ ├── runtime_config.py # DB 配置覆盖(.env → app_settings)
|
||||
│ │ ├── scheduler.py # 进程内 cron 调度器
|
||||
│ │ └── security_check.py # 启动安全校验
|
||||
│ ├── data/ # 数据层
|
||||
│ │ ├── sources.py # DataSource 协议 + 注册表
|
||||
│ │ ├── bzzoiro.py # bzzoiro 数据源(events/standings/stats)
|
||||
│ │ ├── normalize.py # 数据规范化契约
|
||||
│ │ ├── bzzoiro.py # bzzoiro 数据源
|
||||
│ │ ├── understat.py # understat xG 数据源
|
||||
│ │ ├── injuries.py # 伤停数据
|
||||
│ │ ├── config.py # 联赛映射常量
|
||||
│ │ └── team_names.py # 队名归一化
|
||||
│ │ ├── key_ring.py # API Key 轮换环(429 冷却)
|
||||
│ │ ├── team_names.py # 队名归一化
|
||||
│ │ └── team_names_zh.py # 队名中文映射
|
||||
│ ├── db/ # 数据库
|
||||
│ │ ├── base.py # SQLAlchemy async engine
|
||||
│ │ ├── models.py # ORM 模型 (6 表)
|
||||
│ │ ├── models.py # ORM 模型 (12 表)
|
||||
│ │ ├── unit_of_work.py # UnitOfWork 事务封装
|
||||
│ │ └── repositories.py # Repository 数据访问
|
||||
│ └── llm/ # LLM 预测核心
|
||||
│ ├── predict.py # 预测服务 (缓存 + 单/多模式)
|
||||
│ ├── predict.py # 预测服务 (缓存 + 单/多/基线模式)
|
||||
│ ├── context_builder.py # 数据切片 + 上下文拼接
|
||||
│ ├── baseline.py # 基线预测(均值模型)
|
||||
│ ├── eval.py # 评估统计
|
||||
│ ├── backtest.py # 回测框架
|
||||
│ ├── provider.py # 多提供商 LLM 抽象
|
||||
│ ├── utils.py # LLM 工具函数
|
||||
│ ├── validation.py # LLM 输出校验
|
||||
│ ├── agents/
|
||||
│ │ ├── base.py # Agent 基础设施 + 解析
|
||||
@@ -200,6 +235,11 @@ Profeto/
|
||||
│ └── prompts/ # Prompt 模板
|
||||
├── alembic/ # 数据库迁移
|
||||
├── frontend/ # React 前端
|
||||
│ └── src/
|
||||
│ ├── pages/ # 公开站(赛程 Matches + 积分榜 Standings)
|
||||
│ ├── admin/ # 管理后台(布局/页面/数据访问层 dal.ts)
|
||||
│ ├── components/ # 共享组件
|
||||
│ └── lib/http.ts # 唯一 HTTP 实现(带凭据/超时/错误处理)
|
||||
├── docs/ # 详细文档
|
||||
├── tests/ # 单元测试
|
||||
├── docker-compose.yml
|
||||
@@ -234,7 +274,6 @@ Profeto/
|
||||
| `LLM_SPECIALIST_MODEL` | 专家模型 (空=回落 LLM_MODEL) | |
|
||||
| `LLM_AGGREGATOR_MODEL` | 终裁模型 (空=回落 LLM_MODEL) | |
|
||||
| `BZZOIRO_KEY` | bzzoiro API Key | *(必填)* |
|
||||
| `API_FOOTBALL_KEY` | api-football Key (伤停) | |
|
||||
| `CORS_ORIGINS` | 允许的跨域来源 | `http://localhost:5173` |
|
||||
|
||||
## 测试
|
||||
|
||||
@@ -0,0 +1,46 @@
|
||||
"""新增 ingest_jobs 表
|
||||
|
||||
Revision ID: 0019_ingest_jobs
|
||||
Revises: 0018_match_checks
|
||||
Create Date: 2026-09-21
|
||||
|
||||
采集任务状态跟踪:POST /ingest/bzzoiro 创建 pending job 后台执行,
|
||||
解决 fire-and-forget 不可观测问题(admin 可查询任务级状态与结果摘要)。
|
||||
"""
|
||||
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.dialects.postgresql import JSONB
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = '0019_ingest_jobs'
|
||||
down_revision: Union[str, None] = '0018_match_checks'
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
'ingest_jobs',
|
||||
sa.Column('id', sa.String(36), primary_key=True),
|
||||
sa.Column('task', sa.String(20), nullable=False),
|
||||
sa.Column('params', JSONB, nullable=False),
|
||||
sa.Column('status', sa.String(20), nullable=False, server_default='pending'),
|
||||
sa.Column('result', JSONB),
|
||||
sa.Column('error', sa.Text()),
|
||||
sa.Column('created_at', sa.DateTime(timezone=True)),
|
||||
sa.Column('started_at', sa.DateTime(timezone=True)),
|
||||
sa.Column('finished_at', sa.DateTime(timezone=True)),
|
||||
sa.CheckConstraint(
|
||||
"status IN ('pending','running','success','failed')",
|
||||
name='ck_ingest_jobs_status',
|
||||
),
|
||||
)
|
||||
op.create_index('ix_ingest_jobs_created', 'ingest_jobs', ['created_at'])
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index('ix_ingest_jobs_created', table_name='ingest_jobs')
|
||||
op.drop_table('ingest_jobs')
|
||||
+35
-27
@@ -13,16 +13,15 @@
|
||||
│ │
|
||||
│ 数据查询 预测编排 采集(手动/cron 触发) │
|
||||
│ ┌──────┐ ┌────────────┐ ┌───────────────────┐ │
|
||||
│ │matches│ │ orchestrator│ │ bzzoiro (赛果) │ │
|
||||
│ │matches│ │ orchestrator│ │ bzzoiro (唯一源) │ │
|
||||
│ │leagues│ │ ┌─ 5 专家并行(便宜模型) │ │
|
||||
│ └──┬───┘ │ │ h2h / form / stats / │ │
|
||||
│ │ │ │ home_away / injuries │ │
|
||||
│ │ │ │ home_away / standings │ │
|
||||
│ │ │ └─ aggregator 终裁(强模型) │ │
|
||||
│ │ └────────────┘ └───────────────────┘ │
|
||||
│ │ │ └ understat (xG) │
|
||||
│ ┌──┴──────────────┴──┐ └ injuries (伤停) │
|
||||
│ │ PostgreSQL (6 张表) │ httpx → 外部 API │
|
||||
│ └────────────────────┘ │
|
||||
│ │ └────────────┘ │ events / standings │ │
|
||||
│ ┌──┴──────────────┴──┐ │ /stats 三条管线 │ │
|
||||
│ │ PostgreSQL (13 张表)│ └───────────────────┘ │
|
||||
│ └────────────────────┘ httpx → 外部 API │
|
||||
└─────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
@@ -31,7 +30,7 @@
|
||||
1. `POST /predict {match_id}` → orchestrator
|
||||
2. `load_match_header`: 查比赛 + 双方 + 联赛(一次 eager load)
|
||||
3. **5 个专家 agent 并行**(`asyncio.gather`),每个:
|
||||
- 各自的数据切片函数查库(近况/交锋/积分榜 SQL 聚合/伤停/xG)
|
||||
- 各自的数据切片函数查库(近况/交锋/积分榜聚合/射门控球/xG)
|
||||
- 切片无数据 → **跳过 LLM**,直接 `no_data` stub(省 token、防幻觉)
|
||||
- 有数据 → 专属 prompt(专家模型,便宜快)→ 结构化 JSON 报告(`home_edge` 方向性评分 + 证据)
|
||||
4. **终裁 agent**:5 份报告 + 比赛信息 → 权衡采信度(`agent_weights`)→ 最终预测 JSON
|
||||
@@ -44,7 +43,7 @@
|
||||
|---|---|
|
||||
| **多专家并行而非单次大 prompt** | 每维度独立迭代 prompt;报告可归因(哪个维度分析错了);总延迟 ≈ 2 次串行调用 |
|
||||
| **专家/终裁模型分档** | 专家用便宜模型快速分析,终裁用强模型汇总决策,成本与质量平衡(`LLM_SPECIALIST_MODEL` / `LLM_AGGREGATOR_MODEL`) |
|
||||
| **no_data 门控** | 无数据维度(如伤停未接入)不调 LLM,终裁知道维度缺失,不编造 |
|
||||
| **no_data 门控** | 无数据维度(如积分榜未采集)不调 LLM,终裁知道维度缺失,不编造 |
|
||||
| **fail-open** | 单个专家失败只标记 `status=error`,其余照常;研究场景可用性优先 |
|
||||
| **`match_date_date` 天级去重** | 不同源时间精度不同,秒级匹配会产生重复行;天级 + 数据库唯一约束 |
|
||||
| **积分榜 SQL 聚合 + season 过滤** | `UNION ALL` 主客双视角 + `GROUP BY` 在库内算,只算当前赛季(修复过跨赛季 bug) |
|
||||
@@ -57,33 +56,38 @@
|
||||
Profeto/
|
||||
├── src/
|
||||
│ ├── api/
|
||||
│ │ ├── app.py # FastAPI 工厂(lifespan 仅验证 DB 连接,不建表)
|
||||
│ │ ├── deps.py # 依赖:管理接口鉴权(X-API-Key)
|
||||
│ │ ├── app.py # FastAPI 工厂(lifespan:迁移校验/定时任务/生产限流提醒)
|
||||
│ │ ├── deps.py # 依赖:管理接口鉴权(Cookie/X-API-Key)+ 限流
|
||||
│ │ ├── schemas.py # Pydantic v2 请求/响应
|
||||
│ │ └── routes/
|
||||
│ │ ├── matches.py # 联赛/比赛查询(游标分页)
|
||||
│ │ ├── predict.py # 预测 + 预测历史
|
||||
│ │ ├── ingest.py # 采集触发(自管 session,需鉴权)
|
||||
│ │ ├── eval.py # 赛后回填 + 准确率汇总
|
||||
│ │ └── backtest.py # 历史回测(需鉴权)
|
||||
│ │ ├── matches.py # 联赛/比赛/上下文/积分榜(公开只读)
|
||||
│ │ ├── predict.py # 预测(限流)+ 预测历史(需鉴权)
|
||||
│ │ ├── ingest.py # 采集触发(需鉴权)
|
||||
│ │ ├── eval.py # 赛后回填 + 准确率汇总(需鉴权)
|
||||
│ │ ├── backtest.py # 历史回测(需鉴权)
|
||||
│ │ ├── auth.py # 登录/登出/改密
|
||||
│ │ ├── admin_settings.py # /admin/** 配置/日志/数据质量(router 级鉴权)
|
||||
│ │ └── schedules.py # 定时任务 + 死信重试(router 级鉴权)
|
||||
│ ├── db/
|
||||
│ │ ├── base.py # async engine + get_db/get_db_read
|
||||
│ │ ├── models.py # 6 张表 ORM
|
||||
│ │ ├── models.py # 13 张表 ORM
|
||||
│ │ ├── repositories.py # 仓储层
|
||||
│ │ └── unit_of_work.py # 事务边界
|
||||
│ ├── data/
|
||||
│ │ ├── bzzoiro.py # 赛果采集 + 幂等入库
|
||||
│ │ ├── understat.py # xG 回填
|
||||
│ │ ├── injuries.py # 伤停采集(带文件缓存)
|
||||
│ │ ├── bzzoiro.py # 唯一数据源:events/standings/stats 三管线 + Bronze 层
|
||||
│ │ ├── normalize.py # NormalizedMatch 清洗契约
|
||||
│ │ ├── team_names.py # 队名归一映射
|
||||
│ │ ├── team_names_zh.py # 队名中文名映射
|
||||
│ │ ├── key_ring.py # 多 key 轮换(429 冷却,进程内)
|
||||
│ │ ├── sources.py # 数据源注册表
|
||||
│ │ └── config.py # 联赛代码映射
|
||||
│ ├── llm/
|
||||
│ │ ├── provider.py # OpenAI-compatible 抽象(共享连接池/JSON 兜底解析)
|
||||
│ │ ├── context_builder.py # 数据切片(h2h/form/stats/home_away/injuries)+ 单 agent 拼接
|
||||
│ │ ├── context_builder.py # 数据切片(h2h/form/stats/home_away/standings)+ 单 agent 拼接
|
||||
│ │ ├── predict.py # 预测入口(mode 分派 + 缓存)
|
||||
│ │ ├── baseline.py # 基线预测(均值模型,mode=baseline)
|
||||
│ │ ├── eval.py # 准确率统计
|
||||
│ │ ├── utils.py # LLM 工具函数
|
||||
│ │ ├── validation.py # LLM 输出严格校验(Pydantic)
|
||||
│ │ ├── backtest.py # 回测执行
|
||||
│ │ ├── agents/
|
||||
@@ -91,14 +95,18 @@ Profeto/
|
||||
│ │ │ └── orchestrator.py # 并行专家 → 终裁 → 存库
|
||||
│ │ └── prompts/
|
||||
│ │ ├── match_prediction_v1/v2.md # 单 agent 模板
|
||||
│ │ └── agents/{h2h,form,stats,home_away,injuries,aggregator}_v1.md
|
||||
│ │ └── agents/{form,stats,home_away,standings,h2h,aggregator}_v1.md
|
||||
│ └── core/
|
||||
│ ├── config.py # pydantic-settings
|
||||
│ ├── crypto.py # 加密/哈希
|
||||
│ ├── http_client.py # 共享 httpx 客户端
|
||||
│ └── retry.py # 重试工具
|
||||
├── alembic/versions/ # 0001~0006(0001 建表 → 0006 漂移清理)
|
||||
├── frontend/src/pages/Matches.tsx # 单页(预测面板 + 专家报告折叠区 + 游标分页)
|
||||
├── tests/ # 核心 + agent 测试
|
||||
│ ├── log_buffer.py # 内存日志缓冲(admin 日志页)
|
||||
│ ├── runtime_config.py # DB 配置覆盖(app_settings)
|
||||
│ ├── scheduler.py # 进程内 cron 调度器
|
||||
│ └── security_check.py # 启动安全校验
|
||||
├── alembic/versions/ # 0001~0018(建表 → Bronze 层 → 单一数据源 → 基线模式等)
|
||||
├── frontend/src/ # pages/(公开站) + admin/(管理后台) + lib/http.ts(唯一 HTTP 实现)
|
||||
├── tests/ # 核心 + agent 测试(250+ 项,自包含)
|
||||
├── docker-compose.yml # api + postgres 两容器
|
||||
└── docs/ # 本文档
|
||||
```
|
||||
@@ -113,5 +121,5 @@ Profeto/
|
||||
| HTTP | httpx(共享连接池)/ urllib(bzzoiro 同步限速) |
|
||||
| LLM | OpenAI-compatible 接口(openai/deepseek/ollama 等任一) |
|
||||
| 前端 | Vite + React 18 + TypeScript + Tailwind |
|
||||
| 测试 | pytest + pytest-asyncio(33 项,自包含) |
|
||||
| 测试 | pytest + pytest-asyncio(250+ 项,自包含) |
|
||||
| 部署 | Docker Compose(api + postgres) |
|
||||
|
||||
+13
-4
@@ -66,6 +66,9 @@ curl -X POST http://localhost:8000/api/v1/ingest/bzzoiro \
|
||||
-d '{"leagues":["E0"],"date_from":"2026-08-01","date_to":"2026-09-08"}'
|
||||
```
|
||||
|
||||
> 注:采集/评估端点需管理员凭据。本地开发环境(未配置鉴权、非 production)默认放行;
|
||||
> 生产环境需先 `POST /auth/login` 取 Cookie,或带 `X-API-Key` 头。
|
||||
|
||||
数据量大时**直接拉整赛季**(约 380 场,含近几个赛季更好,近况/交锋/积分榜都需要历史):
|
||||
|
||||
```bash
|
||||
@@ -74,14 +77,20 @@ curl -X POST http://localhost:8000/api/v1/ingest/bzzoiro \
|
||||
-d '{"leagues":["E0"],"date_from":"2025-08-01","date_to":"2026-09-08"}'
|
||||
```
|
||||
|
||||
### 回填 xG(可选,让攻防数据 agent 有数据)
|
||||
### 回填积分榜与统计(让攻防/排名专家有数据)
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8000/api/v1/ingest/understat \
|
||||
curl -X POST http://localhost:8000/api/v1/ingest/bzzoiro \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"league":"E0","season":2025}'
|
||||
-d '{"task":"standings"}'
|
||||
|
||||
curl -X POST http://localhost:8000/api/v1/ingest/bzzoiro \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"task":"stats","leagues":["E0"],"limit":300}'
|
||||
```
|
||||
|
||||
`task=stats` 只补空字段(xG/射门/控球等),不创建比赛。
|
||||
|
||||
### 查比赛
|
||||
|
||||
浏览器打开 http://localhost:5173 ,选"英超 / 未开赛";
|
||||
@@ -126,4 +135,4 @@ curl http://localhost:8000/api/v1/eval/summary
|
||||
| predict 返回 502 | 看 uvicorn 日志的 LLM error;确认 `LLM_BASE_URL`/`LLM_API_KEY`;`response_format` 不兼容的网关会报错(改用支持 json mode 的模型) |
|
||||
| 采集 0 场 | bzzoiro Key 失效或联赛代码写错;先 `GET /api/v1/leagues` 看库里有没有联赛 |
|
||||
| 专家报告全是 no_data | 历史数据不够 —— 近况需要每队近 5 场、积分榜需要本赛季已完赛比赛,多拉几周数据 |
|
||||
| xg agent 报无 xG 数据 | 先跑 understat 回填;注意 understat 只有五大联赛 |
|
||||
| stats 专家报无 xG/统计 | 先跑 `task=stats` 回填(bzzoiro 统计管线,只补空字段) |
|
||||
|
||||
+91
-27
@@ -4,18 +4,30 @@ Base URL: `http://localhost:8000` · 交互式文档: `/docs`(Swagger)与 `/redo
|
||||
|
||||
所有数据端点返回 JSON。错误统一为 `{"detail": "<message>"}` + 对应 HTTP 状态码。
|
||||
|
||||
## 鉴权模型
|
||||
|
||||
| 级别 | 端点 | 说明 |
|
||||
|---|---|---|
|
||||
| **公开只读** | `GET /leagues`、`GET /matches`、`GET /matches/{id}`、`GET /matches/{id}/context`、`GET /standings`、`GET /health*` | 无需任何凭据;公开站直接调用 |
|
||||
| **公开 + 限流** | `POST /predict` | 内存滑动窗口限流(10 次/分钟/IP) |
|
||||
| **需管理员** | `GET /predictions*`、`POST /ingest/bzzoiro`、`/eval/*`、`POST /backtest`、`/admin/**` | Cookie 会话(`POST /auth/login` 颁发)或 `X-API-Key` 头 |
|
||||
|
||||
管理端点在生产环境未配置鉴权时 fail-closed(503),不会静默放行。
|
||||
|
||||
---
|
||||
|
||||
## 数据查询
|
||||
## 数据查询(公开只读)
|
||||
|
||||
### `GET /api/v1/leagues`
|
||||
|
||||
列出已入库联赛。
|
||||
列出已入库联赛(P1-3: 公开站联赛筛选动态加载来源)。
|
||||
|
||||
```json
|
||||
[{"id": 1, "code": "E0", "name": "Premier League", "country": "England"}]
|
||||
```
|
||||
|
||||
仅返回 `id/code/name/country` 四个展示字段,不含任何配置或密钥信息。
|
||||
|
||||
### `GET /api/v1/matches`
|
||||
|
||||
比赛列表,游标分页。
|
||||
@@ -47,7 +59,44 @@ Base URL: `http://localhost:8000` · 交互式文档: `/docs`(Swagger)与 `/redo
|
||||
|
||||
### `GET /api/v1/matches/{id}`
|
||||
|
||||
单场比赛详情,字段同上。
|
||||
单场比赛详情,字段同上,另含 `stats`(统计)与 `recent_predictions`(最近 5 条预测摘要)。
|
||||
|
||||
### `GET /api/v1/matches/{id}/context`
|
||||
|
||||
比赛上下文(公开只读,P1-2: 公开站详情页「近况/交锋」数据来源;不触发 LLM):
|
||||
|
||||
- `home_recent`: 主队最近 5 场已完赛
|
||||
- `away_recent`: 客队最近 5 场已完赛
|
||||
- `h2h`: 双方最近 5 次交手
|
||||
|
||||
```json
|
||||
{
|
||||
"home_recent": [
|
||||
{"match_date": "2026-09-12T14:00:00+00:00", "home_team": "阿森纳",
|
||||
"away_team": "切尔西", "home_goals": 2, "away_goals": 1}
|
||||
],
|
||||
"away_recent": [],
|
||||
"h2h": []
|
||||
}
|
||||
```
|
||||
|
||||
数据不足时对应列表为空(前端展示空态)。比赛不存在返回 404。
|
||||
|
||||
### `GET /api/v1/standings`
|
||||
|
||||
联赛积分榜(公开只读)。参数:`league`(联赛代码,空 = 全部)、`season`(空 = 各联赛最新赛季)。
|
||||
|
||||
```json
|
||||
{"leagues": [
|
||||
{"league_code": "E0", "league_name": "Premier League", "season": "2026-2027",
|
||||
"retrieved_at": "2026-09-20T08:00:00+00:00",
|
||||
"rows": [{"position": 1, "team": "阿森纳", "team_en": "Arsenal",
|
||||
"played": 5, "won": 4, "drawn": 1, "lost": 0,
|
||||
"goals_for": 11, "goals_against": 3, "goal_diff": 8,
|
||||
"points": 13, "xg_for": 9.8, "xg_against": 3.9,
|
||||
"form": "WWWDW", "zone": "UEFA Champions League"}]}
|
||||
]}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
@@ -96,13 +145,13 @@ Base URL: `http://localhost:8000` · 交互式文档: `/docs`(Swagger)与 `/redo
|
||||
"exp_home_goals": null, "exp_away_goals": null, "probable_score": null,
|
||||
"model": "gpt-4o-mini", "latency_ms": 2100,
|
||||
"prompt_tokens": 380, "completion_tokens": 120},
|
||||
{"agent": "injuries", "status": "no_data", "data_sufficiency": "none",
|
||||
{"agent": "standings", "status": "no_data", "data_sufficiency": "none",
|
||||
"analysis": "该维度无数据,跳过分析。", "home_edge": null, "subjective_confidence": null,
|
||||
"key_evidence": [], "exp_home_goals": null, "exp_away_goals": null,
|
||||
"probable_score": null, "model": "", "latency_ms": null,
|
||||
"prompt_tokens": null, "completion_tokens": null}
|
||||
],
|
||||
"agent_weights": {"form": 0.9, "stats": 0.8, "home_away": 0.7, "injuries": 0.0, "h2h": 0.8},
|
||||
"agent_weights": {"form": 0.9, "stats": 0.8, "home_away": 0.7, "standings": 0.8, "h2h": 0.8},
|
||||
"context": "[5 份报告的 JSON 串]",
|
||||
"latency_ms": 9800
|
||||
}
|
||||
@@ -112,11 +161,11 @@ Base URL: `http://localhost:8000` · 交互式文档: `/docs`(Swagger)与 `/redo
|
||||
|
||||
错误:404 比赛不存在;502 LLM 调用失败(终裁失败时整体失败,专家失败不会)。
|
||||
|
||||
### `GET /api/v1/predictions?match_id=&limit=`
|
||||
### `GET /api/v1/predictions?match_id=&limit=`(需管理员)
|
||||
|
||||
预测历史(倒序),含 `settled` 与实际比分回填状态。
|
||||
|
||||
### `GET /api/v1/predictions/{id}`
|
||||
### `GET /api/v1/predictions/{id}`(需管理员)
|
||||
|
||||
单条预测详情(含完整 `agent_outputs`)。
|
||||
|
||||
@@ -124,40 +173,52 @@ Base URL: `http://localhost:8000` · 交互式文档: `/docs`(Swagger)与 `/redo
|
||||
|
||||
## 数据采集
|
||||
|
||||
### `POST /api/v1/ingest/bzzoiro`
|
||||
### `POST /api/v1/ingest/bzzoiro`(需管理员)
|
||||
|
||||
从 bzzoiro 采集赛果/赛程并入库(幂等,重复跑安全)。
|
||||
从 bzzoiro(唯一数据源)采集数据并入库(幂等,重复跑安全)。任务在后台异步执行,请求立即返回。
|
||||
|
||||
```json
|
||||
{"leagues": ["E0", "SP1"], "date_from": "2025-08-01", "date_to": "2026-09-08", "status": "finished"}
|
||||
{"task": "all", "leagues": ["E0", "SP1"], "date_from": "2025-08-01", "date_to": "2026-09-08", "status": "finished"}
|
||||
```
|
||||
|
||||
- `status` 还可传 `scheduled` 拉未来赛程
|
||||
- 响应含每联赛 `inserted`/`updated`/`errors` 统计
|
||||
| 字段 | 默认 | 说明 |
|
||||
|---|---|---|
|
||||
| `task` | `events` | 采集任务:`events`(比赛)/ `standings`(积分榜)/ `stats`(统计回填,含 xG)/ `all` |
|
||||
| `leagues` | 全部已知联赛 | 联赛代码列表,如 `["E0", "SP1"]` |
|
||||
| `date_from` / `date_to` | 空 | 日期范围(`YYYY-MM-DD`) |
|
||||
| `status` | 空(两者都采) | `finished`(已完赛)/ `scheduled`(未来赛程) |
|
||||
| `season` | 当前赛季 | standings 赛季,如 `"2026-2027"` |
|
||||
| `limit` | 100 | stats 回填单次最大比赛数(1–500) |
|
||||
|
||||
### `POST /api/v1/ingest/understat`
|
||||
- 响应:`{"ok": true, "job_id": "<uuid>", "message": "……"}`,`job_id` 用于查询任务状态
|
||||
- `task=stats` 只补空字段、不创建比赛(xG/射门/控球等统计回填)
|
||||
|
||||
回填 xG(只补空字段,不创建比赛):
|
||||
### `GET /api/v1/admin/ingest/jobs/{job_id}`(需管理员)
|
||||
|
||||
查询一次采集任务的状态(`ingest_jobs` 表,任务级可观测性):
|
||||
|
||||
```json
|
||||
{"league": "E0", "season": 2025}
|
||||
{"id": "…", "task": "standings", "params": {"leagues": ["E0"]},
|
||||
"status": "success", "result": {"total_upserted": 20, "errors": []},
|
||||
"error": null, "created_at": "…", "started_at": "…", "finished_at": "…"}
|
||||
```
|
||||
|
||||
`season=2025` 表示 2025-2026 赛季。仅支持五大联赛。
|
||||
- `status` 取值:`pending`(已创建未开始)/ `running` / `success` / `failed`
|
||||
- 管理端采集页提交后凭 `job_id` 轮询本端点直至终态
|
||||
- 与 `ingest_failures` 死信独立:死信记录单条管线抓取失败(行级),job 记录整次任务结果
|
||||
|
||||
### `POST /api/v1/ingest/injuries`
|
||||
### `GET /api/v1/admin/ingest/jobs?limit=20&status=`(需管理员)
|
||||
|
||||
采集伤停(需 `API_FOOTBALL_KEY`,当前只返回计数,尚未接入 context):
|
||||
列出最近采集任务(最新在前),可按 `status` 过滤,`limit` 1–100。
|
||||
|
||||
```json
|
||||
{"date": "2026-09-10"}
|
||||
```
|
||||
> 历史版本曾有独立的 understat(xG)与 injuries(伤停)采集端点,
|
||||
> 已随数据源收敛为 bzzoiro 唯一来源而移除。
|
||||
|
||||
---
|
||||
|
||||
## 评估
|
||||
|
||||
### `POST /api/v1/eval/settle`
|
||||
### `POST /api/v1/eval/settle`(需管理员)
|
||||
|
||||
赛后回填实际比分:
|
||||
|
||||
@@ -165,7 +226,7 @@ Base URL: `http://localhost:8000` · 交互式文档: `/docs`(Swagger)与 `/redo
|
||||
{"prediction_id": 7, "home_goals": 2, "away_goals": 1}
|
||||
```
|
||||
|
||||
### `GET /api/v1/eval/summary`
|
||||
### `GET /api/v1/eval/summary`(需管理员)
|
||||
|
||||
按 `provider × model` 聚合已结算预测:
|
||||
|
||||
@@ -183,7 +244,10 @@ Base URL: `http://localhost:8000` · 交互式文档: `/docs`(Swagger)与 `/redo
|
||||
|
||||
## 基础
|
||||
|
||||
| 端点 | 说明 |
|
||||
|---|---|
|
||||
| `GET /health` | 存活检查 |
|
||||
| `GET /docs` | Swagger UI |
|
||||
| 端点 | 权限 | 说明 |
|
||||
|---|---|---|
|
||||
| `GET /health` | 公开 | 存活检查 |
|
||||
| `GET /docs`、`GET /redoc` | 公开 | Swagger UI / ReDoc |
|
||||
| `POST /auth/login`、`POST /auth/logout`、`GET /auth/me` | 公开 | 管理员 Cookie 会话登录/登出/当前用户 |
|
||||
| `POST /auth/change-password` | 需管理员 | 修改管理员密码 |
|
||||
| `POST /backtest` | 需管理员 | 历史回测(对已完赛比赛批量预测并评估) |
|
||||
|
||||
+7
-7
@@ -10,7 +10,7 @@ Profeto 的核心预测路径是 **5 个领域专家 Agent 并行分析 + 1 个
|
||||
| `form` 近期状态 | 分析比分与关键事件,判断近期走势 | 两队近 N 场赛果(含 xG) | `home_edge` + 走势判断 |
|
||||
| `stats` 攻防数据 | 评估进球、射门与控球,量化攻防强度 | 近 N 场进球/射门/控球/xG 统计 | `home_edge` + 攻防强度 |
|
||||
| `home_away` 主客因素 | 对比主场与客场表现,评估地理优势影响 | 主队主场战绩 + 客队客场战绩 | `home_edge` + 地理优势 |
|
||||
| `injuries` 阵容完整性 | 汇总伤停与停赛名单,评估战力缺失程度 | 伤停数据(当前无源 → no_data 门控) | `home_edge` 或 `no_data` |
|
||||
| `standings` 联赛排名 | 结合积分榜排名、积分与分区(欧冠/欧联/降级),评估双方竞争位置 | 两队当前赛季积分榜行(排名/积分/分区/近期战绩) | `home_edge` 或 `no_data` |
|
||||
| `h2h` 历史交锋 | 分析过去数年以及近期的交手数据,提取交手规律 | 近 N 次交锋(含主客方向 + 总计统计) | `home_edge` + 交手规律 |
|
||||
| `aggregator` 终裁 | 权衡 5 份报告 → 最终结论 | 5 份结构化报告 + 比赛头信息 | 最终预测 + 各报告采信度 |
|
||||
|
||||
@@ -25,7 +25,7 @@ POST /predict {match_id, mode: "multi"}
|
||||
│ ├─ form agent ─┐
|
||||
│ ├─ stats agent │ 每个 agent 拿到专属数据切片
|
||||
│ ├─ home_away agent │ → no_data 门控 → 调 LLM → 输出 JSON 报告
|
||||
│ ├─ injuries agent │ (无数据 → 跳过 LLM,返回 stub)
|
||||
│ ├─ standings agent │ (无数据 → 跳过 LLM,返回 stub)
|
||||
│ └─ h2h agent ─┘
|
||||
│
|
||||
├─ aggregator agent(5 份报告 + 比赛头 → 最终 JSON)
|
||||
@@ -39,12 +39,12 @@ POST /predict {match_id, mode: "multi"}
|
||||
5 个专家通过 `asyncio.gather` 并发,总延迟 ≈ `max(专家延迟) + 终裁延迟` ≈ 2 次串行 LLM 调用。
|
||||
|
||||
### 2. no_data 门控(省 token、防幻觉)
|
||||
数据切片为空时(如伤停数据源未接入),**跳过 LLM 调用**,直接返回:
|
||||
数据切片为空时(如该场比赛的积分榜尚未采集),**跳过 LLM 调用**,直接返回:
|
||||
```json
|
||||
{"agent": "injuries", "status": "no_data", "data_sufficiency": "none",
|
||||
{"agent": "standings", "status": "no_data", "data_sufficiency": "none",
|
||||
"analysis": "该维度无数据,跳过分析。"}
|
||||
```
|
||||
终裁 Agent 会看到这个 `no_data` 状态,不会编造伤停分析。
|
||||
终裁 Agent 会看到这个 `no_data` 状态,不会编造积分榜分析。
|
||||
|
||||
### 3. fail-open(单专家失败不阻断)
|
||||
单个专家 LLM 调用失败 → 其报告标记 `status: error`,其余 4 份 + 终裁照常执行。
|
||||
@@ -89,7 +89,7 @@ POST /predict {match_id, mode: "multi"}
|
||||
"1x2": "1",
|
||||
"subjective_confidence": 0.68,
|
||||
"reasoning": "综合 stats 报告的攻防强度与 form 报告的三连胜势头……",
|
||||
"agent_weights": {"form": 0.9, "stats": 0.8, "home_away": 0.7, "injuries": 0.0, "h2h": 0.8}
|
||||
"agent_weights": {"form": 0.9, "stats": 0.8, "home_away": 0.7, "standings": 0.8, "h2h": 0.8}
|
||||
}
|
||||
```
|
||||
|
||||
@@ -120,7 +120,7 @@ agents/
|
||||
├── form_v1.md # 近期状态专家
|
||||
├── stats_v1.md # 攻防数据专家
|
||||
├── home_away_v1.md # 主客因素专家
|
||||
├── injuries_v1.md # 阵容完整性专家
|
||||
├── standings_v1.md # 联赛排名专家
|
||||
├── h2h_v1.md # 历史交锋专家
|
||||
└── aggregator_v1.md # 终裁
|
||||
```
|
||||
|
||||
+27
-14
@@ -4,9 +4,10 @@
|
||||
|
||||
| 数据源 | 用途 | 必需 Key | 说明 |
|
||||
|---|---|---|---|
|
||||
| bzzoiro | 赛果/赛程(主源) | `BZZOIRO_KEY` | 五大联赛历史 + 实时 |
|
||||
| understat | xG 回填 | 无(公开) | 仅五大联赛,补 `match_stats.xg` |
|
||||
| api-football | 伤停 | `API_FOOTBALL_KEY` | 当前只采集计数,未接入 context |
|
||||
| bzzoiro(唯一) | 赛果/赛程/积分榜/统计(xG、射门、控球等) | `BZZOIRO_KEY` | 五大联赛 + 欧战,历史 + 实时 |
|
||||
|
||||
> 历史版本曾有 understat(xG 回填)与 api-football(伤停)两个辅助源,
|
||||
> 现已移除:数据源收敛为 bzzoiro 唯一来源,统计与积分榜均由 bzzoiro 管线采集。
|
||||
|
||||
### bzzoiro
|
||||
|
||||
@@ -28,12 +29,6 @@
|
||||
> - 黄牌: `home_yellow_cards` / `away_yellow_cards`
|
||||
> - 红牌: `home_red_cards` / `away_red_cards`
|
||||
|
||||
### understat
|
||||
|
||||
- 端点:`/getLeagueData/{league}/{season}`,返回 JS 包裹的 JSON(需正则提取)
|
||||
- 只回填 xG(`match_stats.home_xg`/`away_xg`),**不创建新比赛**
|
||||
- 通过"天级日期 + 队名归一"匹配已有比赛
|
||||
|
||||
## 数据清洗契约
|
||||
|
||||
所有数据源统一清洗为 `NormalizedMatch`(`src/data/normalize.py`),字段:
|
||||
@@ -70,7 +65,7 @@
|
||||
|
||||
## 数据库 Schema
|
||||
|
||||
6 张表:
|
||||
12 张业务/配置表 + 1 张任务状态表(`ingest_jobs`):核心业务表 5 张见下方 DDL,其余见后文表格。
|
||||
|
||||
```sql
|
||||
-- 联赛
|
||||
@@ -144,12 +139,26 @@ CREATE TABLE predictions (
|
||||
reasoning TEXT,
|
||||
raw_response JSONB, -- LLM 完整原始响应
|
||||
agent_outputs JSONB, -- multi 模式: 5 份专家报告
|
||||
agent_weights JSONB, -- multi 模式: 终裁给出的各专家权重
|
||||
created_at TIMESTAMPTZ,
|
||||
actual_home_goals INT, actual_away_goals INT, -- 赛后回填
|
||||
settled BOOLEAN DEFAULT FALSE
|
||||
);
|
||||
```
|
||||
|
||||
其余 8 张表(DDL 略,详见 `src/db/models.py` 与 alembic 迁移):
|
||||
|
||||
| 表 | 状态 | 用途 |
|
||||
|---|---|---|
|
||||
| `standings` | 已启用 | 联赛积分榜快照,按 `(league_id, season, team_id)` upsert,同联赛同赛季只保留最新快照;含排名/战绩/进失球/积分/分区(zone) |
|
||||
| `app_settings` | 已启用 | 后台运行时设置(如数据源 API Key),读取时优先于 `.env` 默认值 |
|
||||
| `schedules` | 已启用 | 定时采集任务配置(task/cron/leagues/enabled),供内置调度器执行 |
|
||||
| `ingest_jobs` | 已启用 | 采集任务状态:POST ingest 创建 pending,后台流转 running→success/failed;result 存统计摘要,供 admin 轮询(任务级,与死信互补) |
|
||||
| `raw_events` | 预留未启用 | Bronze 层原始事件存档;规划中用于重放与审计 |
|
||||
| `ingest_failures` | 已启用 | 采集失败死信:bzzoiro 三条管线(events/standings/stats)抓取失败时写入,admin 后台可查看与重试 |
|
||||
| `data_quality_checks` | 预留未启用 | 数据质量检查结果;规划中定时检查比赛/统计/积分榜完整性 |
|
||||
| `data_lineage` | 预留未启用 | ETL 血缘追踪;规划中记录源记录到目标表的映射 |
|
||||
|
||||
### 关键设计点
|
||||
|
||||
1. **`match_date_date`(天级日期)**: 用于天级去重。bzzoiro 返回的时间带时分秒,精确匹配不可靠,故拆出 `DATE` 列做唯一键。
|
||||
@@ -170,7 +179,9 @@ CREATE TABLE predictions (
|
||||
- 状态:只允许单向升级(`scheduled` → `finished`),防止完赛行被覆盖成赛程
|
||||
- stats:只补空(`home_xg` 已有值时不覆盖)
|
||||
|
||||
`ingest_understat` 只回填 xG(也只补空),不创建比赛。
|
||||
`task=stats` 只回填统计(xG/射门/控球等,也只补空),不创建比赛。
|
||||
|
||||
`task=standings` 按 `(league_id, season, team_id)` upsert 积分榜快照,同一联赛同一赛季只保留最新一份。
|
||||
|
||||
## 采集建议
|
||||
|
||||
@@ -183,9 +194,11 @@ curl -X POST /api/v1/ingest/bzzoiro \
|
||||
curl -X POST /api/v1/ingest/bzzoiro \
|
||||
-d '{"leagues":["E0"],"date_from":"2026-09-01","date_to":"2026-09-08"}'
|
||||
|
||||
# 3. xG 回填(可选,提升 xg agent 质量)
|
||||
curl -X POST /api/v1/ingest/understat -d '{"league":"E0","season":2025}'
|
||||
curl -X POST /api/v1/ingest/understat -d '{"league":"E0","season":2026}'
|
||||
# 3. 积分榜 + 统计回填(xG/射门/控球,提升 stats/standings 专家质量)
|
||||
curl -X POST /api/v1/ingest/bzzoiro -d '{"task":"standings"}'
|
||||
curl -X POST /api/v1/ingest/bzzoiro -d '{"task":"stats","leagues":["E0"],"limit":300}'
|
||||
|
||||
# 注:采集端点需管理员凭据(Cookie 会话或 X-API-Key 头),下同
|
||||
```
|
||||
|
||||
建议用外部 cron(如系统 crontab)定时触发,不引入 worker/redis。
|
||||
|
||||
+17
-2
@@ -33,6 +33,22 @@ curl http://localhost:8000/health
|
||||
> **注意**: `api` 服务启动时会先执行 `alembic upgrade head` 迁移数据库,再启动 uvicorn。
|
||||
> 容器内数据库连接自动使用 `postgres` 服务名(通过 compose `environment` 覆盖 `.env` 中的 `DB_HOST`)。
|
||||
|
||||
## 生产上线检查清单
|
||||
|
||||
公网上线前逐项勾选。第 1–4 项由 `src/core/security_check.py` 在 `APP_ENV=production`
|
||||
启动时**强制校验**,不满足直接拒绝启动(开发环境仅告警);管理鉴权另有请求期 fail-closed(503)。
|
||||
|
||||
- [ ] **1. `APP_ENV=production`** — 安全校验、Cookie `Secure`、管理端点 fail-closed 均以它为总开关
|
||||
- [ ] **2. `SECRET_KEY` 强随机** — 用 `openssl rand -base64 32` 生成;禁止弱值/短值(弱值黑名单会拒绝启动,含把生成指令原样粘进去的情况)
|
||||
- [ ] **3. 管理鉴权至少其一** — `ADMIN_PASSWORD` 或 `ADMIN_API_KEY`(后台改过密码后以数据库哈希优先);两者皆空时管理接口 503
|
||||
- [ ] **4. 数据库强密码** — 禁止 `football:football` 等示例弱密码(启动校验会拒绝);compose 的 `POSTGRES_PASSWORD` 必填,缺失时容器拒绝启动
|
||||
- [ ] **5. HTTPS** — 由反代(Nginx/Caddy)终结 TLS;`APP_ENV=production` 下会话 Cookie 自动 `Secure`(且 HttpOnly + SameSite=Lax)
|
||||
- [ ] **6. 反代信任头** — `TRUST_PROXY_HEADERS=True`,且**仅可信反代可达 API**;反代需设置 `X-Forwarded-For`(`$proxy_add_x_forwarded_for`)与 `X-Real-IP`,否则限流/日志按反代 IP 计数
|
||||
- [ ] **7. 限流前置到网关** — 推荐 Nginx `limit_req`(配置见[安全与限流](#安全与限流));应用内限流与 KeyRing 为**单进程内存实现**,多 worker 各自独立计数会把实际配额放大 N 倍(启动时会打印一次性告警)
|
||||
- [ ] **8. uvicorn 单 worker** — compose/Dockerfile 默认单 worker,保持即可;需横向扩容时先在网关统一限流,再起多实例(每实例仍单 worker)
|
||||
- [ ] **9. 启动后健康检查** — `curl /health` 返回 200(存活);`curl /health/ready` 返回 200(就绪,校验数据库连通,不可达时 503)
|
||||
- [ ] **10. 数据库迁移** — compose/Dockerfile 启动命令已内置 `alembic upgrade head && uvicorn …`,升级镜像重启即自动迁移,无需手动执行
|
||||
|
||||
## 本地开发部署
|
||||
|
||||
```bash
|
||||
@@ -81,8 +97,7 @@ cd frontend && npm install && npm run dev
|
||||
| `LLM_TIMEOUT` | ❌ | `60` | 单次调用超时(秒) |
|
||||
| `LLM_SPECIALIST_MODEL` | ❌ | — | 专家模型(回落 `LLM_MODEL`) |
|
||||
| `LLM_AGGREGATOR_MODEL` | ❌ | — | 终裁模型(回落 `LLM_MODEL`) |
|
||||
| `BZZOIRO_KEY` | ✅ | — | bzzoiro 数据源 Key |
|
||||
| `API_FOOTBALL_KEY` | ❌ | — | 伤停数据源 Key |
|
||||
| `BZZOIRO_KEY` | ✅ | — | bzzoiro 数据源 Key(唯一数据源) |
|
||||
| `CORS_ORIGINS` | ❌ | `http://localhost:5173,...` | 允许的跨域来源 |
|
||||
| `SECRET_KEY` | ❌ | — | 加密主密钥(生产环境必填) |
|
||||
| `ADMIN_PASSWORD` | ❌ | — | 管理后台密码(留空=不启用) |
|
||||
|
||||
+13
-8
@@ -83,16 +83,16 @@ Profeto/
|
||||
│ │ └── app.py # FastAPI 工厂
|
||||
│ ├── db/
|
||||
│ │ ├── base.py # SQLAlchemy async engine + session
|
||||
│ │ ├── models.py # 6 张表 ORM
|
||||
│ │ ├── models.py # 12 张表 ORM
|
||||
│ │ ├── repositories.py # 仓储层(查询封装)
|
||||
│ │ └── unit_of_work.py # 事务边界
|
||||
│ ├── data/
|
||||
│ │ ├── bzzoiro.py # bzzoiro 采集 + 入库
|
||||
│ │ ├── understat.py # understat xG 回填
|
||||
│ │ ├── injuries.py # 伤停采集
|
||||
│ │ ├── bzzoiro.py # bzzoiro 采集 + 入库(唯一数据源)
|
||||
│ │ ├── normalize.py # 数据清洗契约
|
||||
│ │ ├── team_names.py # 队名归一化映射
|
||||
│ │ ├── team_names_zh.py # 队名中文名映射
|
||||
│ │ ├── sources.py # 数据源注册表
|
||||
│ │ ├── key_ring.py # 数据源 Key 读取(DB 设置优先于 env)
|
||||
│ │ └── config.py # 联赛映射常量
|
||||
│ ├── llm/
|
||||
│ │ ├── provider.py # LLM 提供商抽象(OpenAI-compatible)
|
||||
@@ -110,13 +110,17 @@ Profeto/
|
||||
│ │ ├── form_v1.md
|
||||
│ │ ├── stats_v1.md
|
||||
│ │ ├── home_away_v1.md
|
||||
│ │ ├── injuries_v1.md
|
||||
│ │ ├── standings_v1.md
|
||||
│ │ ├── h2h_v1.md
|
||||
│ │ └── aggregator_v1.md
|
||||
│ └── core/
|
||||
│ ├── config.py # pydantic-settings 配置
|
||||
│ ├── http_client.py # 共享 httpx 客户端
|
||||
│ └── retry.py # 重试工具
|
||||
│ ├── crypto.py # 对称加密(Fernet)与密码哈希
|
||||
│ ├── log_buffer.py # 内存日志缓冲(admin「系统日志」页)
|
||||
│ ├── runtime_config.py # 运行时配置(数据库优先,回落 .env)
|
||||
│ ├── scheduler.py # 定时任务调度器(cron 触发采集)
|
||||
│ └── security_check.py # 生产启动安全校验(缺配置拒绝启动)
|
||||
├── frontend/ # React 单页前端
|
||||
├── alembic/ # 数据库迁移
|
||||
│ └── versions/
|
||||
@@ -125,7 +129,8 @@ Profeto/
|
||||
│ ├── 0003_injuries.py
|
||||
│ ├── 0004_snapshot_and_constraints.py
|
||||
│ ├── 0005_prediction_status_and_stats_provenance.py
|
||||
│ └── 0006_schema_model_drift_cleanup.py
|
||||
│ ├── 0006_schema_model_drift_cleanup.py
|
||||
│ └── …… (共 19 个迁移,最新 0019_ingest_jobs)
|
||||
├── tests/ # 测试
|
||||
│ ├── test_core.py # 核心逻辑测试
|
||||
│ └── test_agents.py # 多 agent 测试
|
||||
@@ -194,7 +199,7 @@ cp src/llm/prompts/agents/h2h_v1.md src/llm/prompts/agents/h2h_v2.md
|
||||
|
||||
### 3. 新增数据源
|
||||
|
||||
1. 在 `src/data/` 写采集模块(参考 `understat.py`)
|
||||
1. 在 `src/data/` 写采集模块(参考 `bzzoiro.py`)
|
||||
2. 在 `normalize.py` 加清洗函数
|
||||
3. 在 `context_builder.py` 加切片函数
|
||||
4. 在 `api/routes/ingest.py` 加端点
|
||||
|
||||
@@ -1,3 +1,9 @@
|
||||
> ⚠️ **历史文档(已过时)**:伤停(injuries)数据源与 api-football 集成已移除,
|
||||
> 数据源收敛为 bzzoiro 唯一来源。本文仅作历史决策记录保留,
|
||||
> 现状请见 [05-data.md](05-data.md) 与 [01-architecture.md](01-architecture.md)。
|
||||
|
||||
---
|
||||
|
||||
"""检查 injuries 数据源 api-football 的响应结构。"""
|
||||
API_FOOTBALL_INJURY_RESPONSE_EXAMPLE = """
|
||||
{
|
||||
|
||||
@@ -21,6 +21,7 @@ import type {
|
||||
LLMAgentConfig,
|
||||
LogEntry,
|
||||
IngestSourceStatus,
|
||||
IngestJob,
|
||||
MatchDetailOut,
|
||||
MatchContextOut,
|
||||
AdminStats,
|
||||
@@ -71,6 +72,11 @@ export async function triggerCollection(req: CollectionRequest): Promise<any> {
|
||||
return api.post(`${API_BASE}/ingest/bzzoiro`, body)
|
||||
}
|
||||
|
||||
// 查询采集任务状态(ingest_jobs;提交采集返回 job_id 后轮询用)
|
||||
export async function fetchIngestJob(jobId: string): Promise<IngestJob> {
|
||||
return api.get<IngestJob>(`${API_BASE}/admin/ingest/jobs/${jobId}`)
|
||||
}
|
||||
|
||||
// ── 预测管理 ────────────────────────────────────────────────────
|
||||
|
||||
export async function triggerPrediction(req: { match_id: number; mode?: string }): Promise<any> {
|
||||
|
||||
@@ -11,9 +11,9 @@
|
||||
*/
|
||||
|
||||
import { useEffect, useState, useCallback, useRef } from 'react'
|
||||
import { triggerCollection, fetchLeagues, fetchIngestStatus } from '../dal'
|
||||
import { triggerCollection, fetchLeagues, fetchIngestStatus, fetchIngestJob } from '../dal'
|
||||
import type { IngestSourceStatus } from '../types'
|
||||
import type { CollectionRequest, League } from '../types'
|
||||
import type { CollectionRequest, League, IngestJob } from '../types'
|
||||
import { Card, CardBody, CardHeader, Badge, SectionHeader, Alert, Spinner } from '../components'
|
||||
|
||||
const TASKS = [
|
||||
@@ -25,6 +25,32 @@ const TASKS = [
|
||||
|
||||
type TaskStatus = 'idle' | 'running' | 'done' | 'error'
|
||||
|
||||
const JOB_STATUS_LABEL: Record<IngestJob['status'], string> = {
|
||||
pending: '等待中',
|
||||
running: '执行中',
|
||||
success: '已完成',
|
||||
failed: '失败',
|
||||
}
|
||||
|
||||
/** job.result 摘要 → 多行文本(错误只显示条数,明细见「采集失败」页与系统日志) */
|
||||
function summarizeResult(result: Record<string, any> | null): string[] {
|
||||
if (!result) return []
|
||||
return Object.entries(result).map(([k, v]) => {
|
||||
if (v && typeof v === 'object') {
|
||||
const bits: string[] = []
|
||||
for (const [kk, vv] of Object.entries(v)) {
|
||||
if (kk === 'errors') {
|
||||
if (Array.isArray(vv) && vv.length > 0) bits.push(`错误 ${vv.length} 条`)
|
||||
} else if (vv !== null && vv !== undefined) {
|
||||
bits.push(`${kk} ${vv}`)
|
||||
}
|
||||
}
|
||||
return `${k}: ${bits.length ? bits.join(' · ') : '无变更'}`
|
||||
}
|
||||
return `${k}: ${String(v)}`
|
||||
})
|
||||
}
|
||||
|
||||
export default function CollectionPage() {
|
||||
const [leagues, setLeagues] = useState<League[]>([])
|
||||
const [task, setTask] = useState<string>('events')
|
||||
@@ -55,7 +81,10 @@ export default function CollectionPage() {
|
||||
const [taskStatus, setTaskStatus] = useState<TaskStatus>('idle')
|
||||
const [taskStartedAt, setTaskStartedAt] = useState<number | null>(null)
|
||||
const pollRef = useRef<ReturnType<typeof setInterval> | null>(null)
|
||||
const jobPollRef = useRef<ReturnType<typeof setInterval> | null>(null)
|
||||
const [ingestSnap, setIngestSnap] = useState<IngestSourceStatus | null>(null)
|
||||
// 采集任务状态(ingest_jobs):提交后有 job_id 即轮询,替代「30 秒盲等」
|
||||
const [job, setJob] = useState<IngestJob | null>(null)
|
||||
|
||||
const loadLeagues = useCallback(async () => {
|
||||
const lg = await fetchLeagues()
|
||||
@@ -80,7 +109,27 @@ export default function CollectionPage() {
|
||||
if (pollRef.current) { clearInterval(pollRef.current); pollRef.current = null }
|
||||
}, [])
|
||||
|
||||
useEffect(() => () => stopPolling(), [stopPolling])
|
||||
// job 轮询:3 秒一次,终态(success/failed)自动停止;10 分钟兜底防泄漏
|
||||
const stopJobPolling = useCallback(() => {
|
||||
if (jobPollRef.current) { clearInterval(jobPollRef.current); jobPollRef.current = null }
|
||||
}, [])
|
||||
|
||||
const startJobPolling = useCallback((jobId: string) => {
|
||||
stopJobPolling()
|
||||
let ticks = 0
|
||||
jobPollRef.current = setInterval(async () => {
|
||||
ticks += 1
|
||||
if (ticks > 200) { stopJobPolling(); setTaskStatus('done'); return }
|
||||
try {
|
||||
const j = await fetchIngestJob(jobId)
|
||||
setJob(j)
|
||||
if (j.status === 'success') { stopJobPolling(); setTaskStatus('done') }
|
||||
else if (j.status === 'failed') { stopJobPolling(); setTaskStatus('error') }
|
||||
} catch { /* 网络抖动忽略,下个周期重试 */ }
|
||||
}, 3_000)
|
||||
}, [stopJobPolling])
|
||||
|
||||
useEffect(() => () => { stopPolling(); stopJobPolling() }, [stopPolling, stopJobPolling])
|
||||
|
||||
const isEventsTask = task === 'events' || task === 'all'
|
||||
|
||||
@@ -88,6 +137,7 @@ export default function CollectionPage() {
|
||||
e.preventDefault()
|
||||
setError(null)
|
||||
setResult(null)
|
||||
setJob(null)
|
||||
setLoading(true)
|
||||
setTaskStatus('running')
|
||||
setTaskStartedAt(Date.now())
|
||||
@@ -103,22 +153,32 @@ export default function CollectionPage() {
|
||||
date_from: isEventsTask ? dateFrom || undefined : undefined,
|
||||
date_to: isEventsTask ? dateTo || undefined : undefined,
|
||||
}
|
||||
await triggerCollection(body)
|
||||
const resp = await triggerCollection(body)
|
||||
const jobId: string | undefined = resp?.job_id
|
||||
if (jobId) {
|
||||
// 有 job_id:轮询任务状态直至终态(3 秒/次)
|
||||
setResult({
|
||||
title: '采集任务已启动',
|
||||
detail: `任务 ID: ${jobId}。正在跟踪执行进度,完成后展示结果摘要。`,
|
||||
})
|
||||
startJobPolling(jobId)
|
||||
} else {
|
||||
// 兜底:后端未返回 job_id(旧版本),退回「30 秒盲等 + 系统日志」
|
||||
setResult({
|
||||
title: '采集任务已启动',
|
||||
detail: '正在后台执行(上游限速时可能需要数分钟)。完成结果与错误请到「系统日志」页查看(支持自动刷新)。',
|
||||
})
|
||||
// 启动轮询,跟踪状态
|
||||
startPolling()
|
||||
// 30 秒后自动停止轮询并标记完成
|
||||
setTimeout(() => {
|
||||
setTaskStatus('done')
|
||||
stopPolling()
|
||||
}, 30_000)
|
||||
}
|
||||
} catch (err: unknown) {
|
||||
setTaskStatus('error')
|
||||
setError(err instanceof Error ? err.message : '采集触发失败')
|
||||
stopPolling()
|
||||
stopJobPolling()
|
||||
} finally {
|
||||
setLoading(false)
|
||||
}
|
||||
@@ -278,7 +338,11 @@ export default function CollectionPage() {
|
||||
<div className="space-y-3">
|
||||
<div className="flex items-center gap-2 text-xs text-ink-700">
|
||||
<Spinner />
|
||||
<span>任务执行中,已运行 {elapsed}s…</span>
|
||||
<span>
|
||||
{job
|
||||
? `${JOB_STATUS_LABEL[job.status]}(轮询中),已运行 ${elapsed}s…`
|
||||
: `任务执行中,已运行 ${elapsed}s…`}
|
||||
</span>
|
||||
</div>
|
||||
<p className="text-2xs text-ink-400">
|
||||
后台异步执行,关闭页面不影响结果。可稍后查看「系统日志」确认完成。
|
||||
@@ -289,15 +353,37 @@ export default function CollectionPage() {
|
||||
<div className="space-y-2">
|
||||
<div className="flex items-center gap-2 text-xs text-emerald-700">
|
||||
<span className="inline-block h-2 w-2 rounded-full bg-emerald-500" />
|
||||
<span>任务已提交,后台执行中(可能尚未完成)</span>
|
||||
<span>
|
||||
{job
|
||||
? `采集完成(${JOB_STATUS_LABEL[job.status]})`
|
||||
: '任务已提交,后台执行中(可能尚未完成)'}
|
||||
</span>
|
||||
</div>
|
||||
{job?.result && summarizeResult(job.result).length > 0 && (
|
||||
<div className="rounded-md bg-ink-50 px-3 py-2">
|
||||
{summarizeResult(job.result).map((line, i) => (
|
||||
<p key={i} className="font-mono text-2xs leading-relaxed text-ink-600">{line}</p>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
{!job && (
|
||||
<p className="text-2xs text-ink-400">
|
||||
采集耗时取决于数据量。请到「系统日志」页查看最终结果。
|
||||
</p>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
{taskStatus === 'error' && (
|
||||
<p className="text-xs text-press">任务触发失败,请检查配置或网络。</p>
|
||||
<div className="space-y-2">
|
||||
<p className="text-xs text-press">
|
||||
{job?.status === 'failed' ? '采集任务失败' : '任务触发失败,请检查配置或网络。'}
|
||||
</p>
|
||||
{job?.error && (
|
||||
<p className="break-all rounded-md bg-red-50 px-3 py-2 font-mono text-2xs text-press">
|
||||
{job.error}
|
||||
</p>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
{ingestSnap?.last_success_at && (
|
||||
<div className="mt-3 border-t border-ink-100 pt-3">
|
||||
|
||||
@@ -103,6 +103,19 @@ export interface CollectionRequest {
|
||||
date_to?: string
|
||||
}
|
||||
|
||||
// 采集任务状态(ingest_jobs,提交采集后轮询)
|
||||
export interface IngestJob {
|
||||
id: string
|
||||
task: 'events' | 'standings' | 'stats' | 'all'
|
||||
params: Record<string, unknown>
|
||||
status: 'pending' | 'running' | 'success' | 'failed'
|
||||
result: Record<string, any> | null
|
||||
error: string | null
|
||||
created_at: string | null
|
||||
started_at: string | null
|
||||
finished_at: string | null
|
||||
}
|
||||
|
||||
// ── 评估 & 回测 ─────────────────────────────────────────────────
|
||||
|
||||
export interface EvalCalibrationBucket {
|
||||
|
||||
@@ -15,13 +15,16 @@ import { fetchMatchDetail, fetchMatchContext } from '../admin/dal'
|
||||
import type { MatchDetailOut, MatchContextOut } from '../admin/types'
|
||||
import { useMatchesList } from './matches/hooks/useMatchesList'
|
||||
import { useMatchPredict } from './matches/hooks/useMatchPredict'
|
||||
import { useLeagues } from './matches/hooks/useLeagues'
|
||||
import { PredictModal } from './matches/components/MatchPredictPanel'
|
||||
import { MatchRow } from './matches/components/MatchDetailSection'
|
||||
import { Spinner, SkeletonRows, Switch, formatDateHeader, groupByDate, withinNext3Days } from './matches/ui'
|
||||
import { LEAGUES, type Match } from './matches/types'
|
||||
import { type Match } from './matches/types'
|
||||
|
||||
export default function Matches() {
|
||||
const [error, setError] = useState<string | null>(null) // 列表与预测共用(拆分前即如此)
|
||||
// P1-3: 联赛列表优先请求 /api/v1/leagues,失败/空回退本地五大联赛常量
|
||||
const leagues = useLeagues()
|
||||
const {
|
||||
league, setLeague,
|
||||
status, setStatus,
|
||||
@@ -58,7 +61,7 @@ export default function Matches() {
|
||||
}, [])
|
||||
const scrollToTop = () => window.scrollTo({ top: 0, behavior: 'smooth' })
|
||||
|
||||
const leagueName = LEAGUES.find(l => l.code === league)?.name ?? league
|
||||
const leagueName = leagues.find(l => l.code === league)?.name ?? league
|
||||
|
||||
/** 未开赛默认仅展示未来 3 天;其余状态展示全部。showAllUpcoming=true 时展开全部。 */
|
||||
const isScheduledView = status === 'scheduled'
|
||||
@@ -91,7 +94,7 @@ export default function Matches() {
|
||||
<div className="space-y-5">
|
||||
{/* ── 联赛版面切换 ── */}
|
||||
<nav className="flex items-center gap-6 overflow-x-auto border-b border-ink-900" aria-label="联赛">
|
||||
{LEAGUES.map(l => (
|
||||
{leagues.map(l => (
|
||||
<button
|
||||
key={l.code}
|
||||
onClick={() => setLeague(l.code)}
|
||||
|
||||
@@ -0,0 +1,38 @@
|
||||
/**
|
||||
* 公开站联赛列表(P1-3):优先请求 GET /api/v1/leagues,
|
||||
* 失败或返回空数组则回退本地五大联赛常量(LEAGUES)。
|
||||
*
|
||||
* 显示名规则:常量里已有的 code 沿用中文标签(保持现有 UI 语言不变),
|
||||
* 新增联赛用 API 返回的 name;排序按常量顺序优先、新联赛按 API 返回序追加。
|
||||
* API 仅返回 {id, code, name, country},无敏感配置字段。
|
||||
*/
|
||||
import { useEffect, useState } from 'react'
|
||||
import { fetchLeagues } from '../../../admin/dal'
|
||||
import { LEAGUES } from '../types'
|
||||
|
||||
export function useLeagues(): { code: string; name: string }[] {
|
||||
const [leagues, setLeagues] = useState(LEAGUES)
|
||||
|
||||
useEffect(() => {
|
||||
let alive = true
|
||||
;(async () => {
|
||||
// dal.fetchLeagues 已兜底:网络/权限异常时返回 []
|
||||
const rows = await fetchLeagues()
|
||||
if (!alive || rows.length === 0) return
|
||||
const zhName = new Map(LEAGUES.map(l => [l.code, l.name] as const))
|
||||
const rank = new Map(LEAGUES.map((l, i) => [l.code, i] as const))
|
||||
const merged = rows
|
||||
.slice()
|
||||
.sort(
|
||||
(a, b) => (rank.get(a.code) ?? LEAGUES.length) - (rank.get(b.code) ?? LEAGUES.length),
|
||||
)
|
||||
.map(l => ({ code: l.code, name: zhName.get(l.code) ?? l.name }))
|
||||
setLeagues(merged)
|
||||
})()
|
||||
return () => {
|
||||
alive = false
|
||||
}
|
||||
}, [])
|
||||
|
||||
return leagues
|
||||
}
|
||||
@@ -10,14 +10,17 @@ from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from sqlalchemy import select
|
||||
|
||||
from src.api.deps import require_admin
|
||||
from src.api.schemas import IngestBzzoiroRequest
|
||||
from src.data.config import BZZOIRO_LEAGUE_IDS
|
||||
from src.data.bzzoiro import ingest_bzzoiro_event_stats, ingest_bzzoiro_standings
|
||||
from src.data.sources import get_source
|
||||
from src.db.models import IngestJob
|
||||
from src.db.unit_of_work import get_uow
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -30,6 +33,29 @@ _background_tasks: set[asyncio.Task] = set()
|
||||
VALID_TASKS = {"events", "standings", "stats", "all"}
|
||||
|
||||
|
||||
def _utcnow() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
async def _update_job(job_id: str, **fields) -> None:
|
||||
"""更新采集任务状态(尽力而为:失败只记日志,绝不拖垮采集主流程)。
|
||||
|
||||
job 记录是任务级可观测性,IngestFailure 死信是行级失败记录,
|
||||
两者独立工作 —— 本函数抛错不应影响采集结果,故整体吞异常。
|
||||
"""
|
||||
try:
|
||||
async with get_uow() as session:
|
||||
stmt = select(IngestJob).where(IngestJob.id == job_id)
|
||||
job = (await session.execute(stmt)).scalar_one_or_none()
|
||||
if job is None:
|
||||
logger.warning("ingest job %s 不存在,跳过状态更新(%s)", job_id, fields)
|
||||
return
|
||||
for k, v in fields.items():
|
||||
setattr(job, k, v)
|
||||
except Exception:
|
||||
logger.warning("ingest job %s 状态更新失败(不影响采集): %s", job_id, fields, exc_info=True)
|
||||
|
||||
|
||||
def _spawn(coro) -> None:
|
||||
"""启动后台采集任务;异常已在任务内记录到系统日志。"""
|
||||
task = asyncio.create_task(coro)
|
||||
@@ -39,20 +65,50 @@ def _spawn(coro) -> None:
|
||||
|
||||
@router.post("/ingest/bzzoiro", dependencies=[Depends(require_admin)])
|
||||
async def ingest_bzzoiro_route(req: IngestBzzoiroRequest):
|
||||
"""触发 bzzoiro 采集(events / standings / stats / all)。"""
|
||||
"""触发 bzzoiro 采集(events / standings / stats / all)。
|
||||
|
||||
启动后台任务前先创建 ingest_jobs 记录(pending),响应返回 job_id,
|
||||
供 admin 通过 GET /api/v1/admin/ingest/jobs/{job_id} 轮询状态。
|
||||
"""
|
||||
if req.task not in VALID_TASKS:
|
||||
raise HTTPException(status_code=422, detail=f"未知任务类型: {req.task}(可选: {', '.join(sorted(VALID_TASKS))})")
|
||||
leagues = req.leagues or list(BZZOIRO_LEAGUE_IDS.keys())
|
||||
task_label = {"events": "比赛数据", "standings": "积分榜", "stats": "统计回填", "all": "全量(比赛+积分榜+统计)"}[req.task]
|
||||
_spawn(_run_bzzoiro(req.task, leagues, req))
|
||||
|
||||
# 任务级可观测性:先落 pending 记录,后台 _run_bzzoiro 接管状态流转
|
||||
job = IngestJob(
|
||||
task=req.task,
|
||||
params={
|
||||
"leagues": leagues,
|
||||
"date_from": req.date_from,
|
||||
"date_to": req.date_to,
|
||||
"status": req.status,
|
||||
"limit": req.limit,
|
||||
"season": req.season,
|
||||
},
|
||||
status="pending",
|
||||
)
|
||||
async with get_uow() as session:
|
||||
session.add(job)
|
||||
|
||||
_spawn(_run_bzzoiro(job.id, req.task, leagues, req))
|
||||
return {
|
||||
"ok": True,
|
||||
"message": f"采集任务已启动(后台执行,任务: {task_label}),请在「系统日志」查看进度与结果",
|
||||
"job_id": job.id,
|
||||
"message": f"采集任务已启动(后台执行,任务: {task_label}),可用 job_id 查询状态或在「系统日志」查看进度",
|
||||
}
|
||||
|
||||
|
||||
async def _run_bzzoiro(task: str, leagues: list[str], req: IngestBzzoiroRequest) -> None:
|
||||
"""后台执行 bzzoiro 采集:上游限速时单次可能耗时数分钟,必须脱离请求生命周期。"""
|
||||
async def _run_bzzoiro(job_id: str, task: str, leagues: list[str], req: IngestBzzoiroRequest) -> None:
|
||||
"""后台执行 bzzoiro 采集:上游限速时单次可能耗时数分钟,必须脱离请求生命周期。
|
||||
|
||||
同时维护 ingest_jobs 状态(pending → running → success/failed):
|
||||
- result 存各子任务统计摘要(errors 截断到前 10 条)
|
||||
- 状态更新经 _update_job 尽力而为,失败不影响采集本身
|
||||
- 单条管线抓取失败的行级记录仍由 bzzoiro 死信逻辑(IngestFailure)负责
|
||||
"""
|
||||
await _update_job(job_id, status="running", started_at=_utcnow())
|
||||
summary: dict = {}
|
||||
try:
|
||||
if task in ("events", "all"):
|
||||
statuses = [req.status] if req.status else ["finished", "scheduled"]
|
||||
@@ -79,6 +135,7 @@ async def _run_bzzoiro(task: str, leagues: list[str], req: IngestBzzoiroRequest)
|
||||
)
|
||||
if merged["errors"]:
|
||||
logger.warning("bzzoiro 比赛采集错误 %d 条: %s", len(merged["errors"]), merged["errors"][:3])
|
||||
summary["events"] = {**merged, "errors": merged["errors"][:10]}
|
||||
|
||||
if task in ("standings", "all"):
|
||||
async with get_uow() as session:
|
||||
@@ -87,6 +144,9 @@ async def _run_bzzoiro(task: str, leagues: list[str], req: IngestBzzoiroRequest)
|
||||
logger.warning("bzzoiro 积分榜采集部分失败: %s", r["errors"][:3])
|
||||
else:
|
||||
logger.info("bzzoiro 积分榜采集完成: upsert %d 条", r["total_upserted"])
|
||||
summary["standings"] = {k: r.get(k) for k in ("total_upserted", "errors") if k in r}
|
||||
if isinstance(summary["standings"].get("errors"), list):
|
||||
summary["standings"]["errors"] = summary["standings"]["errors"][:10]
|
||||
|
||||
if task in ("stats", "all"):
|
||||
async with get_uow() as session:
|
||||
@@ -95,5 +155,12 @@ async def _run_bzzoiro(task: str, leagues: list[str], req: IngestBzzoiroRequest)
|
||||
)
|
||||
if r["errors"]:
|
||||
logger.warning("bzzoiro 统计回填错误 %d 条: %s", len(r["errors"]), r["errors"][:3])
|
||||
except Exception:
|
||||
summary["stats"] = {k: r.get(k) for k in ("total_inserted", "total_updated", "errors") if k in r}
|
||||
if isinstance(summary["stats"].get("errors"), list):
|
||||
summary["stats"]["errors"] = summary["stats"]["errors"][:10]
|
||||
except Exception as exc:
|
||||
logger.exception("bzzoiro 采集任务失败(task=%s)", task)
|
||||
await _update_job(job_id, status="failed", finished_at=_utcnow(), error=repr(exc))
|
||||
return
|
||||
|
||||
await _update_job(job_id, status="success", finished_at=_utcnow(), result=summary)
|
||||
|
||||
@@ -7,7 +7,6 @@ from fastapi import APIRouter, Depends, HTTPException, Query
|
||||
from sqlalchemy import func, or_, select
|
||||
from sqlalchemy.orm import selectinload
|
||||
|
||||
from src.api.deps import require_admin
|
||||
from src.api.schemas import MatchListOut, MatchOut, PredictionOut
|
||||
from src.db.base import AsyncSession, get_db_read
|
||||
from src.db.models import League, Match, Prediction, Standing
|
||||
@@ -32,8 +31,9 @@ def _stats_dict(stats) -> dict | None:
|
||||
}
|
||||
|
||||
|
||||
@router.get("/leagues", response_model=list[dict], dependencies=[Depends(require_admin)])
|
||||
@router.get("/leagues", response_model=list[dict])
|
||||
async def list_leagues(db: AsyncSession = Depends(get_db_read)):
|
||||
"""联赛列表(公开只读,P1-3: 公开站联赛筛选需要;仅返回展示字段)。"""
|
||||
stmt = select(League).order_by(League.name)
|
||||
result = await db.execute(stmt)
|
||||
leagues = result.scalars().all()
|
||||
@@ -190,9 +190,9 @@ async def get_match(match_id: int, db: AsyncSession = Depends(get_db_read)):
|
||||
)
|
||||
|
||||
|
||||
@router.get("/matches/{match_id}/context", dependencies=[Depends(require_admin)])
|
||||
@router.get("/matches/{match_id}/context")
|
||||
async def match_context(match_id: int, db: AsyncSession = Depends(get_db_read)):
|
||||
"""比赛上下文(只读,不触发 LLM):双方近况 + 历史交锋。
|
||||
"""比赛上下文(公开只读,P1-2: 公开站详情页需要;不触发 LLM):双方近况 + 历史交锋。
|
||||
|
||||
全部基于现有数据聚合:
|
||||
- recent_home / recent客队:该队最近 5 场已完赛(进球/结果)
|
||||
|
||||
@@ -182,3 +182,49 @@ async def retry_ingest_failure(failure_id: int, db: AsyncSession = Depends(get_d
|
||||
|
||||
# 触发重试(简化版:仅标记状态,实际重试逻辑由调度器处理)
|
||||
return {"ok": True, "message": f"已标记重试 (第 {failure.retry_count} 次)"}
|
||||
|
||||
|
||||
# ── 采集任务状态(ingest_jobs) ──────────────────────────────────
|
||||
|
||||
|
||||
def _serialize_job(j) -> dict:
|
||||
"""ingest_jobs 行 → 响应 dict(时间统一 isoformat)。"""
|
||||
return {
|
||||
"id": j.id,
|
||||
"task": j.task,
|
||||
"params": j.params,
|
||||
"status": j.status,
|
||||
"result": j.result,
|
||||
"error": j.error,
|
||||
"created_at": j.created_at.isoformat() if j.created_at else None,
|
||||
"started_at": j.started_at.isoformat() if j.started_at else None,
|
||||
"finished_at": j.finished_at.isoformat() if j.finished_at else None,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/ingest/jobs")
|
||||
async def list_ingest_jobs(
|
||||
limit: int = 20,
|
||||
status: str | None = None,
|
||||
db: AsyncSession = Depends(get_db_read),
|
||||
):
|
||||
"""列出采集任务状态(最新在前),可按 status 过滤。"""
|
||||
from src.db.models import IngestJob
|
||||
|
||||
stmt = select(IngestJob).order_by(IngestJob.created_at.desc()).limit(max(1, min(limit, 100)))
|
||||
if status:
|
||||
stmt = stmt.where(IngestJob.status == status)
|
||||
rows = (await db.execute(stmt)).scalars().all()
|
||||
return [_serialize_job(j) for j in rows]
|
||||
|
||||
|
||||
@router.get("/ingest/jobs/{job_id}")
|
||||
async def get_ingest_job(job_id: str, db: AsyncSession = Depends(get_db_read)):
|
||||
"""查询单次采集任务的状态与结果(供前端提交后轮询)。"""
|
||||
from src.db.models import IngestJob
|
||||
|
||||
stmt = select(IngestJob).where(IngestJob.id == job_id)
|
||||
job = (await db.execute(stmt)).scalar_one_or_none()
|
||||
if job is None:
|
||||
raise HTTPException(404, "采集任务不存在")
|
||||
return _serialize_job(job)
|
||||
|
||||
+35
-2
@@ -1,9 +1,12 @@
|
||||
"""ORM 模型: leagues / teams / matches / match_stats / standings / predictions。
|
||||
"""ORM 模型(13 张表)。
|
||||
|
||||
数据源统一为 bzzoiro(单一数据源),伤停(injuries)与 Understat 已移除。
|
||||
业务表: leagues / teams / matches / match_stats / standings / predictions
|
||||
治理与配置表: raw_events / ingest_failures / data_quality_checks / data_lineage /
|
||||
app_settings / schedules / ingest_jobs(完整说明见 docs/05-data.md)。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
from datetime import date, datetime, timezone
|
||||
|
||||
from sqlalchemy import (
|
||||
@@ -404,3 +407,33 @@ class DataLineage(Base):
|
||||
Index("ix_lineage_target", "target_table", "target_id"),
|
||||
Index("ix_lineage_batch", "batch_id"),
|
||||
)
|
||||
|
||||
|
||||
class IngestJob(Base):
|
||||
"""采集任务状态跟踪:解决 POST /ingest/bzzoiro fire-and-forget 不可观测问题。
|
||||
|
||||
生命周期: pending(路由创建) → running(后台开始) → success / failed(终态)。
|
||||
result 存各任务统计摘要(如 inserted/updated/errors 截断);error 存失败原因。
|
||||
与 IngestFailure 死信相互独立:死信记录单条管线抓取失败(行级),
|
||||
job 记录整次任务执行结果(任务级),两者可同时存在。
|
||||
job 状态更新是「尽力而为」:更新失败只记日志,不影响采集主流程。
|
||||
"""
|
||||
__tablename__ = "ingest_jobs"
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=lambda: str(uuid.uuid4()))
|
||||
task: Mapped[str] = mapped_column(String(20), nullable=False) # events / standings / stats / all
|
||||
params: Mapped[dict] = mapped_column(JSONB, nullable=False, default=dict) # leagues/日期等请求参数
|
||||
status: Mapped[str] = mapped_column(String(20), nullable=False, default="pending")
|
||||
result: Mapped[dict | None] = mapped_column(JSONB) # 成功时的统计摘要
|
||||
error: Mapped[str | None] = mapped_column(Text) # 失败原因(repr)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_utcnow)
|
||||
started_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
|
||||
__table_args__ = (
|
||||
CheckConstraint(
|
||||
"status IN ('pending','running','success','failed')",
|
||||
name="ck_ingest_jobs_status",
|
||||
),
|
||||
Index("ix_ingest_jobs_created", "created_at"),
|
||||
)
|
||||
|
||||
@@ -0,0 +1,313 @@
|
||||
"""采集任务状态(ingest_jobs)测试。
|
||||
|
||||
背景: POST /ingest/bzzoiro 此前 fire-and-forget —— 触发后只能翻系统日志,
|
||||
无法程序化查询「这次采集跑到哪了/成没成」。本次改造:
|
||||
1. 路由创建 pending job → 响应返回 job_id
|
||||
2. 后台 _run_bzzoiro 维护 running → success / failed + result/error
|
||||
3. admin 端点 /admin/ingest/jobs/{job_id} 与列表可查询
|
||||
|
||||
守护点:
|
||||
- job 更新是「尽力而为」: _update_job 自身失败被吞掉,不影响采集主流程
|
||||
- job(任务级)与 IngestFailure 死信(行级)相互独立,可同时存在
|
||||
(死信路径由 test_ingest_deadletter.py 守护,本文件不动 bzzoiro 内部)
|
||||
|
||||
范式: 假 UoW(记录 add / 返回预设 job)+ monkeypatch source,不依赖真实数据库
|
||||
(与 test_ingest_deadletter.py / test_public_readonly_api.py 相同)。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from contextlib import asynccontextmanager
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from fastapi import FastAPI
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
import src.api.routes.ingest as ingest
|
||||
import src.api.routes.schedules as schedules
|
||||
from src.api.deps import require_admin
|
||||
from src.api.schemas import IngestBzzoiroRequest
|
||||
from src.db.base import get_db_read
|
||||
from src.db.models import IngestJob
|
||||
|
||||
|
||||
# ── 假基础设施 ──────────────────────────────────────────────────
|
||||
|
||||
|
||||
class _FakeResult:
|
||||
"""支持 .scalars().all() / .scalar_one_or_none() / .scalar() 的最小假结果集。"""
|
||||
|
||||
def __init__(self, items):
|
||||
self._items = items
|
||||
|
||||
def scalars(self):
|
||||
return self
|
||||
|
||||
def all(self):
|
||||
return self._items
|
||||
|
||||
def scalar_one_or_none(self):
|
||||
return self._items[0] if self._items else None
|
||||
|
||||
def scalar(self):
|
||||
return None
|
||||
|
||||
|
||||
class _FakeSession:
|
||||
"""记录 add();execute 按预设队列依次返回(与真实 UoW 的单会话用法对齐)。"""
|
||||
|
||||
def __init__(self, results=None):
|
||||
self.added = []
|
||||
self._results = list(results or [])
|
||||
|
||||
def add(self, obj):
|
||||
self.added.append(obj)
|
||||
|
||||
async def execute(self, stmt):
|
||||
if self._results:
|
||||
return self._results.pop(0)
|
||||
return _FakeResult([])
|
||||
|
||||
|
||||
def _patch_uow(monkeypatch, session) -> None:
|
||||
"""把 ingest 模块的 get_uow 指向假会话。"""
|
||||
|
||||
@asynccontextmanager
|
||||
async def _fake():
|
||||
yield session
|
||||
|
||||
monkeypatch.setattr(ingest, "get_uow", _fake)
|
||||
|
||||
|
||||
def _ingest_app() -> TestClient:
|
||||
app = FastAPI()
|
||||
app.include_router(ingest.router)
|
||||
app.dependency_overrides[require_admin] = lambda: None
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
def _admin_app(fake_db) -> TestClient:
|
||||
app = FastAPI()
|
||||
app.include_router(schedules.router)
|
||||
app.dependency_overrides[require_admin] = lambda: None
|
||||
app.dependency_overrides[get_db_read] = lambda: fake_db
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
def _job(jid: str = "job-1", status: str = "success", **kw) -> IngestJob:
|
||||
return IngestJob(
|
||||
id=jid,
|
||||
task=kw.pop("task", "standings"),
|
||||
params=kw.pop("params", {"leagues": ["E0"]}),
|
||||
status=status,
|
||||
result=kw.pop("result", {"total_upserted": 20}),
|
||||
error=kw.pop("error", None),
|
||||
created_at=kw.pop("created_at", datetime.now(timezone.utc)),
|
||||
**kw,
|
||||
)
|
||||
|
||||
|
||||
# ── 1. 路由:创建 job + 返回 job_id ─────────────────────────────
|
||||
|
||||
|
||||
class TestRouteCreatesJob:
|
||||
def test_post_returns_job_id_and_persists_pending_job(self, monkeypatch):
|
||||
session = _FakeSession()
|
||||
_patch_uow(monkeypatch, session)
|
||||
|
||||
spawned: list = []
|
||||
monkeypatch.setattr(ingest, "_spawn", lambda coro: spawned.append(coro))
|
||||
|
||||
client = _ingest_app()
|
||||
resp = client.post(
|
||||
"/api/v1/ingest/bzzoiro",
|
||||
json={"task": "standings", "leagues": ["E0", "SP1"], "season": "2026-2027"},
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
body = resp.json()
|
||||
assert body["ok"] is True
|
||||
|
||||
jobs = [o for o in session.added if isinstance(o, IngestJob)]
|
||||
assert len(jobs) == 1
|
||||
job = jobs[0]
|
||||
assert body["job_id"] == job.id
|
||||
assert job.status == "pending"
|
||||
assert job.task == "standings"
|
||||
# params 记录的是「实际将执行」的参数(含默认联赛展开)
|
||||
assert job.params["leagues"] == ["E0", "SP1"]
|
||||
assert job.params["season"] == "2026-2027"
|
||||
|
||||
# 后台协程被捕获但未执行;显式关闭避免 un-awaited 告警
|
||||
assert len(spawned) == 1
|
||||
spawned[0].close()
|
||||
|
||||
def test_post_invalid_task_422_and_no_job(self, monkeypatch):
|
||||
session = _FakeSession()
|
||||
_patch_uow(monkeypatch, session)
|
||||
monkeypatch.setattr(ingest, "_spawn", lambda coro: coro.close())
|
||||
|
||||
client = _ingest_app()
|
||||
resp = client.post("/api/v1/ingest/bzzoiro", json={"task": "bogus"})
|
||||
assert resp.status_code == 422
|
||||
assert not [o for o in session.added if isinstance(o, IngestJob)]
|
||||
|
||||
|
||||
# ── 2. 后台执行:状态流转 ────────────────────────────────────────
|
||||
|
||||
|
||||
class TestRunBzzoiroJobLifecycle:
|
||||
async def test_success_flow_updates_job_running_then_success(self, monkeypatch):
|
||||
job = _job(jid="job-ok", status="pending", task="events")
|
||||
session = _FakeSession(results=[_FakeResult([job])] * 10)
|
||||
_patch_uow(monkeypatch, session)
|
||||
|
||||
class _FakeSource:
|
||||
async def ingest(self, session, **kw):
|
||||
return {"inserted": 3, "updated": 1, "total_inserted": 3,
|
||||
"total_updated": 1, "errors": ["e1", "e2"]}
|
||||
|
||||
monkeypatch.setattr(ingest, "get_source", lambda name: _FakeSource())
|
||||
|
||||
req = IngestBzzoiroRequest(task="events", leagues=["E0"], status="finished")
|
||||
await ingest._run_bzzoiro("job-ok", "events", ["E0"], req)
|
||||
|
||||
assert job.status == "success"
|
||||
assert job.started_at is not None
|
||||
assert job.finished_at is not None
|
||||
assert job.error is None
|
||||
# result 含 events 子任务摘要,errors 截断到前 10 条
|
||||
assert job.result["events"]["total_inserted"] == 3
|
||||
assert job.result["events"]["errors"] == ["e1", "e2"]
|
||||
|
||||
async def test_failure_flow_marks_failed_with_error(self, monkeypatch):
|
||||
job = _job(jid="job-bad", status="running", task="standings", result=None)
|
||||
session = _FakeSession(results=[_FakeResult([job])] * 10)
|
||||
_patch_uow(monkeypatch, session)
|
||||
|
||||
async def _boom(*args, **kwargs):
|
||||
raise RuntimeError("network down")
|
||||
|
||||
monkeypatch.setattr(ingest, "ingest_bzzoiro_standings", _boom)
|
||||
|
||||
req = IngestBzzoiroRequest(task="standings", leagues=["E0"])
|
||||
await ingest._run_bzzoiro("job-bad", "standings", ["E0"], req)
|
||||
|
||||
assert job.status == "failed"
|
||||
assert "network down" in job.error
|
||||
assert job.finished_at is not None
|
||||
assert job.result is None
|
||||
|
||||
async def test_all_task_collects_per_task_summaries(self, monkeypatch):
|
||||
job = _job(jid="job-all", status="pending", task="all")
|
||||
session = _FakeSession(results=[_FakeResult([job])] * 20)
|
||||
_patch_uow(monkeypatch, session)
|
||||
|
||||
class _FakeSource:
|
||||
async def ingest(self, session, **kw):
|
||||
return {"inserted": 1, "updated": 0, "total_inserted": 1,
|
||||
"total_updated": 0, "errors": []}
|
||||
|
||||
async def _standings(*args, **kwargs):
|
||||
return {"total_upserted": 20, "errors": []}
|
||||
|
||||
async def _stats(*args, **kwargs):
|
||||
return {"total_inserted": 5, "total_updated": 2, "errors": []}
|
||||
|
||||
monkeypatch.setattr(ingest, "get_source", lambda name: _FakeSource())
|
||||
monkeypatch.setattr(ingest, "ingest_bzzoiro_standings", _standings)
|
||||
monkeypatch.setattr(ingest, "ingest_bzzoiro_event_stats", _stats)
|
||||
|
||||
req = IngestBzzoiroRequest(task="all", leagues=["E0"])
|
||||
await ingest._run_bzzoiro("job-all", "all", ["E0"], req)
|
||||
|
||||
assert job.status == "success"
|
||||
assert set(job.result.keys()) == {"events", "standings", "stats"}
|
||||
assert job.result["standings"]["total_upserted"] == 20
|
||||
assert job.result["stats"]["total_inserted"] == 5
|
||||
|
||||
async def test_update_job_failure_does_not_break_ingest(self, monkeypatch):
|
||||
"""_update_job 抛错必须被吞掉:job 可观测性失败 ≠ 采集失败。"""
|
||||
session = _FakeSession()
|
||||
_patch_uow(monkeypatch, session)
|
||||
|
||||
class _FakeSource:
|
||||
async def ingest(self, session, **kw):
|
||||
return {"inserted": 1, "updated": 0, "total_inserted": 1,
|
||||
"total_updated": 0, "errors": []}
|
||||
|
||||
monkeypatch.setattr(ingest, "get_source", lambda name: _FakeSource())
|
||||
|
||||
# 让 execute 抛错(_update_job 内部会捕获)
|
||||
async def _broken_execute(stmt):
|
||||
raise RuntimeError("db gone")
|
||||
|
||||
session.execute = _broken_execute
|
||||
|
||||
req = IngestBzzoiroRequest(task="events", leagues=["E0"], status="finished")
|
||||
# 不抛异常即通过;采集逻辑本身照常跑完
|
||||
await ingest._run_bzzoiro("job-x", "events", ["E0"], req)
|
||||
|
||||
async def test_job_and_deadletter_are_independent_layers(self, monkeypatch):
|
||||
"""任务级(job)与行级(死信)互不干扰:source 内部返回 errors 时,
|
||||
job 仍为 success(部分失败不算任务失败),死信由 bzzoiro 层另行记录。"""
|
||||
job = _job(jid="job-part", status="pending", task="events")
|
||||
session = _FakeSession(results=[_FakeResult([job])] * 10)
|
||||
_patch_uow(monkeypatch, session)
|
||||
|
||||
class _FakeSource:
|
||||
async def ingest(self, session, **kw):
|
||||
# 模拟 bzzoiro 管线:单条失败已写死信,汇总 errors 非空但返回正常
|
||||
return {"inserted": 9, "updated": 0, "total_inserted": 9,
|
||||
"total_updated": 0, "errors": ["league F1 fetch failed"]}
|
||||
|
||||
monkeypatch.setattr(ingest, "get_source", lambda name: _FakeSource())
|
||||
|
||||
req = IngestBzzoiroRequest(task="events", leagues=["E0"], status="finished")
|
||||
await ingest._run_bzzoiro("job-part", "events", ["E0"], req)
|
||||
|
||||
assert job.status == "success"
|
||||
assert job.result["events"]["errors"] == ["league F1 fetch failed"]
|
||||
|
||||
|
||||
# ── 3. admin 查询端点 ──────────────────────────────────────────
|
||||
|
||||
|
||||
class TestAdminJobEndpoints:
|
||||
def test_get_job_detail_200(self, monkeypatch):
|
||||
job = _job(jid="abc-123", status="running")
|
||||
client = _admin_app(_FakeSession(results=[_FakeResult([job])]))
|
||||
|
||||
resp = client.get("/api/v1/admin/ingest/jobs/abc-123")
|
||||
assert resp.status_code == 200
|
||||
body = resp.json()
|
||||
assert body["id"] == "abc-123"
|
||||
assert body["status"] == "running"
|
||||
assert body["task"] == "standings"
|
||||
assert body["params"] == {"leagues": ["E0"]}
|
||||
assert body["result"] == {"total_upserted": 20}
|
||||
assert body["error"] is None
|
||||
assert body["created_at"] is not None
|
||||
|
||||
def test_get_job_detail_404(self):
|
||||
client = _admin_app(_FakeSession(results=[_FakeResult([])]))
|
||||
resp = client.get("/api/v1/admin/ingest/jobs/missing")
|
||||
assert resp.status_code == 404
|
||||
|
||||
def test_list_jobs_returns_serialized_rows(self):
|
||||
j1 = _job(jid="j1", status="success")
|
||||
j2 = _job(jid="j2", status="failed", task="events",
|
||||
result=None, error="RuntimeError('x')")
|
||||
client = _admin_app(_FakeSession(results=[_FakeResult([j1, j2])]))
|
||||
|
||||
resp = client.get("/api/v1/admin/ingest/jobs?limit=10")
|
||||
assert resp.status_code == 200
|
||||
rows = resp.json()
|
||||
assert [r["id"] for r in rows] == ["j1", "j2"]
|
||||
assert rows[1]["status"] == "failed"
|
||||
assert rows[1]["error"] == "RuntimeError('x')"
|
||||
|
||||
def test_job_routes_live_under_admin_router_with_require_admin(self):
|
||||
"""结构守护:job 端点必须挂在 /api/v1/admin 路由(路由级 require_admin)。"""
|
||||
assert any(dep.dependency is require_admin for dep in schedules.router.dependencies)
|
||||
paths = {getattr(r, "path", "") for r in schedules.router.routes}
|
||||
assert "/api/v1/admin/ingest/jobs" in paths
|
||||
assert "/api/v1/admin/ingest/jobs/{job_id}" in paths
|
||||
@@ -0,0 +1,163 @@
|
||||
"""公开只读 API 回归(P1-2 / P1-3):context 与 leagues 匿名可访问。
|
||||
|
||||
背景:
|
||||
- 公开站 MatchDetailSection 展开详情会请求 /matches/{id}/context,
|
||||
此前该端点挂 require_admin,未登录 401 被 fetchMatchContext 的
|
||||
catch 吞掉 → 近况/交锋静默为空;
|
||||
- 公开站联赛筛选需要 /leagues,此前同样 require_admin,前端写死五大联赛。
|
||||
|
||||
守卫(双保险):
|
||||
1. 功能层:最小 FastAPI app + dependency_overrides 注入 fake session,
|
||||
匿名请求(无任何凭据)→ 200;不存在的 match → 404。
|
||||
(不启动完整 app lifespan,遵循 test_api_critical.py 的既定约束)
|
||||
2. 鉴权层:检查路由依赖声明,require_admin 不得出现在
|
||||
/leagues 与 /matches/{match_id}/context。dev 环境 require_admin
|
||||
未配置鉴权时 fail-open,功能层测不出「加回了 require_admin」的回归,
|
||||
必须靠本层声明检查;并用 ingest 路由证明检查器本身有判别力
|
||||
(变异保护:若有人给公开端点加回 require_admin,此处变红)。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from fastapi import FastAPI
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from src.api.deps import require_admin
|
||||
from src.api.routes.ingest import router as ingest_router
|
||||
from src.api.routes.matches import router as matches_router
|
||||
from src.db.base import get_db_read
|
||||
from src.db.models import League, Match
|
||||
|
||||
|
||||
# ── fake DB(对齐 routes/matches.py 的实际查询面) ──────────────────
|
||||
|
||||
class _FakeResult:
|
||||
def __init__(self, items):
|
||||
self._items = list(items)
|
||||
|
||||
def scalars(self):
|
||||
return self
|
||||
|
||||
def all(self):
|
||||
return list(self._items)
|
||||
|
||||
def scalar_one_or_none(self):
|
||||
return self._items[0] if self._items else None
|
||||
|
||||
|
||||
class _FakeDB:
|
||||
"""match_context 的查询次序:① match 主查询(scalar_one_or_none)
|
||||
② home_recent ③ away_recent ④ h2h(均 scalars().all())。
|
||||
League 查询按实体识别直接返回列表(对应 /leagues)。"""
|
||||
|
||||
def __init__(self, match=None, match_lists=(), leagues=()):
|
||||
self._match = match
|
||||
self._match_lists = list(match_lists)
|
||||
self._leagues = list(leagues)
|
||||
self._calls = 0
|
||||
|
||||
async def execute(self, stmt):
|
||||
entity = stmt.column_descriptions[0]["entity"]
|
||||
if entity is League:
|
||||
return _FakeResult(self._leagues)
|
||||
if self._calls == 0:
|
||||
self._calls += 1
|
||||
return _FakeResult([self._match] if self._match is not None else [])
|
||||
idx = self._calls - 1
|
||||
self._calls += 1
|
||||
return _FakeResult(self._match_lists[idx] if idx < len(self._match_lists) else [])
|
||||
|
||||
|
||||
def _client(fake_db: _FakeDB) -> TestClient:
|
||||
app = FastAPI()
|
||||
app.include_router(matches_router)
|
||||
app.dependency_overrides[get_db_read] = lambda: fake_db
|
||||
# 不用 with:不触发 lifespan,无真实 DB 引擎连接
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
def _league(lid: int, code: str) -> League:
|
||||
return League(id=lid, code=code, name=f"League {code}", country=f"Country {code}")
|
||||
|
||||
|
||||
def _match(mid: int) -> Match:
|
||||
return Match(
|
||||
id=mid,
|
||||
match_date=datetime(2026, 9, 20, 15, 0, tzinfo=timezone.utc),
|
||||
home_goals=2,
|
||||
away_goals=1,
|
||||
)
|
||||
|
||||
|
||||
# ── 功能层:匿名可访问(P1-2 / P1-3) ──────────────────────────────
|
||||
|
||||
def test_leagues_anonymous_200_and_shape():
|
||||
"""/leagues 匿名 200;仅暴露 id/code/name/country 四字段。"""
|
||||
db = _FakeDB(leagues=[_league(1, "E0"), _league(2, "SP1")])
|
||||
r = _client(db).get("/api/v1/leagues")
|
||||
assert r.status_code == 200
|
||||
body = r.json()
|
||||
assert body == [
|
||||
{"id": 1, "code": "E0", "name": "League E0", "country": "Country E0"},
|
||||
{"id": 2, "code": "SP1", "name": "League SP1", "country": "Country SP1"},
|
||||
]
|
||||
# 不暴露敏感配置字段
|
||||
assert all(set(item.keys()) == {"id", "code", "name", "country"} for item in body)
|
||||
|
||||
|
||||
def test_context_anonymous_200_empty_data():
|
||||
"""/context 匿名 200;无数据时三个列表为空(前端空态),结构不变。"""
|
||||
db = _FakeDB(match=_match(42), match_lists=[[], [], []])
|
||||
r = _client(db).get("/api/v1/matches/42/context")
|
||||
assert r.status_code == 200
|
||||
assert r.json() == {"home_recent": [], "away_recent": [], "h2h": []}
|
||||
|
||||
|
||||
def test_context_anonymous_200_row_shape():
|
||||
"""/context 行结构与既有前端契约一致(5 字段)。"""
|
||||
db = _FakeDB(match=_match(42), match_lists=[[_match(1)], [], [_match(2), _match(3)]])
|
||||
r = _client(db).get("/api/v1/matches/42/context")
|
||||
assert r.status_code == 200
|
||||
body = r.json()
|
||||
assert len(body["home_recent"]) == 1
|
||||
assert len(body["h2h"]) == 2
|
||||
assert set(body["h2h"][0].keys()) == {
|
||||
"match_date", "home_team", "away_team", "home_goals", "away_goals",
|
||||
}
|
||||
|
||||
|
||||
def test_context_not_found_404():
|
||||
"""/context 匿名访问不存在的 match → 404(而非 401/503)。"""
|
||||
db = _FakeDB(match=None)
|
||||
r = _client(db).get("/api/v1/matches/999/context")
|
||||
assert r.status_code == 404
|
||||
|
||||
|
||||
# ── 鉴权层:路由依赖声明检查(防 require_admin 回潜) ────────────────
|
||||
|
||||
def _admin_paths(router) -> set[str]:
|
||||
paths: set[str] = set()
|
||||
for route in router.routes:
|
||||
for dep in route.dependant.dependencies:
|
||||
if dep.call is require_admin:
|
||||
paths.add(route.path)
|
||||
break
|
||||
return paths
|
||||
|
||||
|
||||
def test_public_routes_have_no_admin_dependency():
|
||||
"""/leagues 与 /context 不得挂 require_admin;matches 路由全部公开只读。"""
|
||||
admin_paths = _admin_paths(matches_router)
|
||||
assert "/api/v1/leagues" not in admin_paths
|
||||
assert "/api/v1/matches/{match_id}/context" not in admin_paths
|
||||
assert admin_paths == set(), f"matches 路由应全部公开只读,仍有 {admin_paths}"
|
||||
|
||||
|
||||
def test_guard_detector_has_discrimination_power():
|
||||
"""变异保护:检查器必须能在 ingest 路由上发现 require_admin,
|
||||
否则上一条「无 admin 依赖」断言恒真、毫无判别力。"""
|
||||
assert "/api/v1/ingest/bzzoiro" in _admin_paths(ingest_router), (
|
||||
"ingest 路由应仍存在 require_admin 保护;若本断言失败,"
|
||||
"说明公开只读守卫的检查器已失效,请修复检查逻辑"
|
||||
)
|
||||
Reference in New Issue
Block a user