fix: 数据库与数据管线 6 个 P1 + 5 个 P2 审查问题修复
P1-1 [context_builder] build_context 共享 session,切片函数传 db 参数,
回测 20 场并发连接需求从 100+ 降至每场 1 个
P1-2 [bzzoiro] 预加载改为按 raw_events 日期范围 ±30 天按需加载
P1-3 [understat] 批量查询球队 + 比赛,从 1140 次往返降到 3 次
P1-4 [injuries] 批量幂等检查 + 分批 flush,IntegrityError 逐条回退
P1-5 [predict] 删除 threading.Lock,dict 操作原子无需同步锁
P1-6 [models] 添加 (match_id, provider, model) 唯一约束 + 迁移
P2-1 [unit_of_work] get_uow 返回类型改为 AsyncIterator[AsyncSession]
P2-2 [normalize] _parse_date 失败时记录 warning 避免静默丢数据
P2-3 [repositories] find_by_teams_and_date 改用 match_date_date 等值匹配
P2-4 [migration] 幽灵列 cutoff_at 已在 0006 迁移删除(已有)
P2-5 [migration] injuries 约束命名对齐 ORM,UniqueConstraint → 唯一索引
This commit is contained in:
+19
-5
@@ -195,11 +195,25 @@ class BzzoiroSource:
|
||||
teams = (await db.execute(stmt)).scalars().all()
|
||||
team_name_to_id = {t.name: t.id for t in teams}
|
||||
|
||||
# 预加载已有比赛(完整对象)
|
||||
stmt = select(Match).where(Match.league_id == league.id)
|
||||
for m in (await db.execute(stmt)).scalars():
|
||||
key = _match_key(m.home_team_id, m.away_team_id, m.match_date_date)
|
||||
existing_matches[key] = m
|
||||
# P1-2: 按需加载,只加载 raw_events 涉及日期范围的比赛(加 30 天缓冲)
|
||||
# 避免加载联赛全部历史比赛到内存(多赛季采集时内存溢出)
|
||||
if normalized_matches:
|
||||
from datetime import timedelta
|
||||
dates = [nm.date for nm in normalized_matches if nm.date is not None]
|
||||
if dates:
|
||||
min_dt = min(dates) - timedelta(days=30)
|
||||
max_dt = max(dates) + timedelta(days=30)
|
||||
stmt = (
|
||||
select(Match)
|
||||
.where(Match.league_id == league.id)
|
||||
.where(Match.match_date >= min_dt)
|
||||
.where(Match.match_date <= max_dt)
|
||||
)
|
||||
existing_matches = {
|
||||
_match_key(m.home_team_id, m.away_team_id, m.match_date_date): m
|
||||
for m in (await db.execute(stmt)).scalars()
|
||||
}
|
||||
# else: existing_matches 保持空 dict(全量新比赛)
|
||||
|
||||
for nm, raw in normalized_matches:
|
||||
# 球队: 内存查找 + 按需创建
|
||||
|
||||
Reference in New Issue
Block a user