shangfangjian
03727bda00
bzzoiro_events / bzzoiro_standings / bzzoiro_stats 直接 import src.data.pipeline_write(_write_raw_event/_write_lineage/_safe_write_ingest_failure), 删除 bzzoiro.py 门面中的 pipeline_write 转发胶水。 保留 fetch_*/_fetch_json_async/REQUEST_INTERVAL 经 bz. 门面调用(测试 monkeypatch 入口); 测试 best-effort 改为 patch 管线模块自身命名空间(from-import 绑定语义)。 函数语义与「失败不拖垮主流程」不变;source_record_id/transform_name 约定不变。 全量测试 270 通过。
67 lines
2.7 KiB
Python
67 lines
2.7 KiB
Python
"""Bzzoiro 数据源:抓取 + 入库(单一数据源)—— 聚合门面。
|
|
|
|
实现按管线拆分(单文件 → 多模块),本模块只做再导出,保持不变量:
|
|
1. sources._load_sources() 仍从本模块导入 BzzoiroSource(注册表入口不变);
|
|
2. 测试与脚本对 `bz.<名称>` 的 monkeypatch 语义不变 —— 抓取函数 / REQUEST_INTERVAL
|
|
仍经本门面解析可替换;Bronze 写入助手(_write_raw_event/_write_lineage/
|
|
_safe_write_ingest_failure)已改为管线模块直接 import pipeline_write,
|
|
测试需 patch `src.data.pipeline_write.*` 源模块。
|
|
|
|
三条管线(各自模块):
|
|
1. events — 比赛日程/比分(/events/),含 source_event_id 血缘 → bzzoiro_events.py
|
|
2. standings— 联赛积分榜快照(/leagues/{id}/standings/) → bzzoiro_standings.py
|
|
3. stats — 已完赛比赛详细统计回填(/events/{id}/stats/) → bzzoiro_stats.py
|
|
|
|
共享基础:HTTP 抓取(多 key 轮换)与字段转换 → bzzoiro_common.py;
|
|
Bronze 基础设施(RawEvent/IngestFailure/DataLineage)→ pipeline_write.py
|
|
(各管线模块直接 import pipeline_write,不再经本门面转发)。
|
|
|
|
D4(工程债): Team/League/Match 的查找/创建经 Repository 层(src/db/repositories.py),
|
|
各管线不直接控制事务(commit/rollback 由调用方 UnitOfWork 控制,只 flush)。
|
|
Standing/RawEvent/Lineage 等管线内私有读写仍不强行 Repository 化。
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
# ── 配置常量(原文件即从 config 再导出,维持 bz.REQUEST_INTERVAL 等引用) ──
|
|
from src.data.config import ( # noqa: F401
|
|
BZZOIRO_LEAGUE_IDS,
|
|
LEAGUE_COUNTRIES,
|
|
LEAGUE_NAMES,
|
|
REQUEST_INTERVAL,
|
|
)
|
|
from src.data.key_ring import _mask # noqa: F401 (R1 测试引用 bz._mask)
|
|
from src.data.normalize import normalize_bzzoiro # noqa: F401
|
|
|
|
# ── 共享原语:HTTP 抓取 + 宽松字段转换 ──
|
|
# NOTE: 管线模块同时从 bzzoiro_common 直接 import _fetch_json_async 等(经本处也转发)。
|
|
from src.data.bzzoiro_common import ( # noqa: F401
|
|
_fetch_json_async,
|
|
_match_key,
|
|
_to_date,
|
|
_to_float_or_none,
|
|
_to_int_or_none,
|
|
)
|
|
|
|
# ── events 管线:BzzoiroSource(注册表入口)+ 抓取/入库 ──
|
|
from src.data.bzzoiro_events import ( # noqa: F401
|
|
BzzoiroSource,
|
|
_events_record_id,
|
|
_write_events_bronze,
|
|
fetch_bzzoiro_events,
|
|
)
|
|
|
|
# ── standings 管线 ──
|
|
from src.data.bzzoiro_standings import ( # noqa: F401
|
|
_season_label_from_dates,
|
|
_write_standings_bronze,
|
|
fetch_bzzoiro_standings,
|
|
ingest_bzzoiro_standings,
|
|
)
|
|
|
|
# ── stats 回填管线 ──
|
|
from src.data.bzzoiro_stats import ( # noqa: F401
|
|
_pick,
|
|
_stats_from_payload,
|
|
ingest_bzzoiro_event_stats,
|
|
)
|