ETL Pipeline Documentation

1. L1A (Raw Data Ingestion)

Status: ✅ Supports Incremental Update

This script ingests raw JSON files from output_arena/ into database/L1A/L1A.sqlite.

Usage

# Standard Run (Incremental)
# Only processes new files that are not yet in the database.
python ETL/L1A.py

# Force Refresh
# Reprocesses ALL files, overwriting existing records.
python ETL/L1A.py --force

L1B demoparser2 -> L1B.sqlite

L2 L1A.sqlite (+L1b.sqlite) -> L2.sqlite

L3 Deep Dive

509 B Raw Blame History

ETL Pipeline Documentation

1. L1A (Raw Data Ingestion)

Usage

509 B

Raw Blame History