feat: 数据管线架构优化 — Bronze 层 + 死信队列 + 令牌桶 + 血缘追踪
P0: injuries 缓存原子写(tempfile+os.replace) + TTL 分级 + asyncio.Lock 并发安全 P1: 引入 Bronze 层 RawEvent 表(原始事件存档,不可变) P1: IngestFailure 死信表(失败持久化,支持重试恢复) P1: understat 改为 xG 覆盖更新模式 + xG 来源追踪字段 P1: 令牌桶限流器(TokenBucket)替换固定 sleep P1: DataQualityCheck 数据质量监控表 P1: DataLineage 血缘追踪表 P2: 回测并发控制 Semaphore(3) 新增文件: - src/data/rate_limiter.py (令牌桶限流器) - alembic/versions/0008 (4 张新表迁移) - alembic/versions/0009 (MatchStats xG 字段迁移)
This commit is contained in:
@@ -0,0 +1,35 @@
|
||||
"""MatchStats 新增 xG 追踪字段
|
||||
|
||||
Revision ID: 0009_match_stats_xg_fields
|
||||
Revises: 0008_raw_event_and_ingest_failure
|
||||
Create Date: 2026-09-17
|
||||
|
||||
架构审查报告 P1-4 实施:
|
||||
understat 允许纠正旧 xG 值。新增字段追踪 xG 具体来源和更新时间,
|
||||
实现全量覆盖模式:当 understat 数据更新时覆盖旧值而非跳过。
|
||||
"""
|
||||
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = '0009_match_stats_xg_fields'
|
||||
down_revision: Union[str, None] = '0008_raw_event_and_ingest_failure'
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column("match_stats", sa.Column("xg_source", sa.String(30), nullable=True))
|
||||
op.add_column("match_stats", sa.Column("xg_updated_at", sa.DateTime(timezone=True), nullable=True))
|
||||
op.add_column("match_stats", sa.Column("xg_source_record_id", sa.String(100), nullable=True))
|
||||
op.create_index("ix_match_stats_xg_source", "match_stats", ["xg_source", "xg_updated_at"])
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index("ix_match_stats_xg_source", table_name="match_stats")
|
||||
op.drop_column("match_stats", "xg_source_record_id")
|
||||
op.drop_column("match_stats", "xg_updated_at")
|
||||
op.drop_column("match_stats", "xg_source")
|
||||
Reference in New Issue
Block a user