2023 · intermediate
The Wonderful World of Incremental Time-Series ETL
ETL is a simple process until it isn't. When you're dealing with billions of rows, how can you choose the synchronization strategy that best fits your needs? In this talk, we dive down the rabbit hole of incremental time-series ETL. We consider the trade-offs with each strategy and how to choose the right one for you. Are your priorities perfect accuracy? Bandwidth? Run-time? Or something in between? To put these strategies to the test, we will build several time-series ETL streams using the open source framework Meerschaum. We'll dive into a new Meerschaum Compose project and write several plugins to ingest outside data sources. Lastly, we'll wrap things up with a secure Docker image for easy deployment that Cloud Ops will love. From beginners to experts, there's something here for everyone!