2026-01-25 01:06:16 +08:00
|
|
|
# ETL Pipeline Documentation
|
|
|
|
|
|
|
|
|
|
## 1. L1A (Raw Data Ingestion)
|
|
|
|
|
**Status**: ✅ Supports Incremental Update
|
|
|
|
|
|
|
|
|
|
This script ingests raw JSON files from `output_arena/` into `database/L1A/L1A.sqlite`.
|
|
|
|
|
|
|
|
|
|
### Usage
|
|
|
|
|
```bash
|
|
|
|
|
# Standard Run (Incremental)
|
|
|
|
|
# Only processes new files that are not yet in the database.
|
|
|
|
|
python ETL/L1A.py
|
|
|
|
|
|
|
|
|
|
# Force Refresh
|
|
|
|
|
# Reprocesses ALL files, overwriting existing records.
|
|
|
|
|
python ETL/L1A.py --force
|
|
|
|
|
```
|
2026-01-24 01:20:36 +08:00
|
|
|
|
|
|
|
|
L1B demoparser2 -> L1B.sqlite
|
|
|
|
|
|
|
|
|
|
L2 L1A.sqlite (+L1b.sqlite) -> L2.sqlite
|
|
|
|
|
|
2026-01-25 01:06:16 +08:00
|
|
|
L3 Deep Dive
|
