Rust Streaming ETL Pipeline
A production-grade streaming ingestion and micro-batch processing pipeline built in Rust. Events are ingested via an HTTP API, buffered in a bounded async channel, transformed in micro-batches using Polars, and loaded into PostgreSQL — with AES-256-GCM encryption at rest, Prometheus metrics, structured JSON logging, idempotent delivery guarantees, a dead-letter queue, and a live web dashboard.
Data EngineeringETLREST APIserdesqlxtokioRust
