varveio
-
swath
Parallel, resumable listing for very large S3 buckets. Learns the key distribution as it runs, streams JSONL or TSV, and writes crash-resumable Parquet you can query directly with DuckDB.
Parallel, resumable listing for very large S3 buckets. Learns the key distribution as it runs, streams JSONL or TSV, and writes crash-resumable Parquet you can query directly with DuckDB.