Lars dove into data pipelines, and emerged bearing arrows and wishing for a lot fewer copies. What is there to think about regarding data pipelines, what is interesting about them? Which tools are out there, and why might you want to use them? Why all this talk about making fewer copies of data? What does Lars' current ideal pipeline look like, and where does Elixir fit in? Links Matt Topol Apache Arrow Large language models Vector search BigQuery sed AWK jq Replacing Hadoop with bash - "Command-line Tools can be 235x Faster than your Hadoop Cluster" Hadoop MapReduce Unix pipes Directed acyclic graph tee - to "materialize inbetween states" Apache Beam Apache Spark Apache Flink Apache Pulsar Airbyte - shoves data between systems using connectors Cronjob Fivetran - Airbyte competitor Apache Airflow ETL - Extract, transform, load Designing data-intensive applications Stream processing Ephemerality Data lake Data warehouse The people's front of Judea DBT - SQL-SQL batch-work-thingy SQL with Jinja templates Snowflake - data warehouse thing Scala Broadway Oban - "robust job processing for Elixir" Dashbit pandas - Python data library APL Arrow flight GRPC DataFusion - query execution engine Polars - "DataFrames in Rust" Explorer - built on top of Polars Voltron data The Composable Codex Pyarrow - Arrow bindings for Python Quotes I've been reading a lot about data pipelines What's so special about data pipelines? There's a lot of special tooling There's a lot of bad, bad tooling Less than optimal tooling Converging on something biggerlk He got me eventually All of your steps in one bucket What tools do you associate with data? I inherited a data pipeline BashReduce Iterate on the L and the T The modern data stack And then you demand more work No unnecessary copies Barely a copy Reconnecting with my Python roots
