Podcasts/Patrick Wheeler and Jason Gauci/Programming Throwdown
185: Workflow Orchestrators
Programming Throwdown•byPatrick Wheeler and Jason Gauci

185: Workflow Orchestrators

0:000:00

Intro topic: Asymmetric Returns


News/Links:

Book of the Show


Patreon Plug https://www.patreon.com/programmingthrowdown?ty=h


Tool of the Show


Topic: Workflow Orchestrators

  • Why
    • Batch jobs (embarrassingly parallel)
    • Long-running tasks (e.g. transcoding video)
    • Checkpointing/resuming
  • How
    • Message Queues
    • Containerization
    • Worker Pools & Autoscaling
    • History & Backfill
  • Steps to run workflows:
    • Containerize the workflow definition and send to the cloud
    • Containerize all the individual tasks
    • Submit job(s)
  • Examples
    • Airflow
      • Legacy but dominant
    • Dagster
      • Great UX for python developers
    • Temporal: https://temporal.io/
      • The new hotness
    • Ray
      • Low-level but very powerful
    • Kubeflow
      • Designed for ML workflows, integrated dashboard

★ Support this podcast on Patreon ★
🏢 Patrick Wheeler and Jason Gauci•186 Episodes•0 Subscribers•0 Total Listens

Episode Playlist

25 Episodes
185: Workflow Orchestrators
Nov 4, 2025•—

185: Workflow Orchestrators

184: Asynchronous Programming
Sep 23, 2025•—

184: Asynchronous Programming

183: Landing a Software Job in 2025
Jul 31, 2025•—

183: Landing a Software Job in 2025

182: AI Assisted Coding
Jun 30, 2025•—

182: AI Assisted Coding

181: Memory Management
May 12, 2025•—

181: Memory Management

180: Reinforcement Learning
Mar 17, 2025•—

180: Reinforcement Learning

179: Project Planning
Feb 3, 2025•—

179: Project Planning

178: Working from Home
Dec 3, 2024•—

178: Working from Home

177: Vector Databases
Nov 4, 2024•—

177: Vector Databases

176: MLOps at SwampUp
Sep 24, 2024•—

176: MLOps at SwampUp

175: Resume Writing
Aug 16, 2024•—

175: Resume Writing

174: Devops
Jun 10, 2024•—

174: Devops

173: Mocking and Unit Tests
Apr 29, 2024•—

173: Mocking and Unit Tests

172: Transformers and Large Language Models
Mar 11, 2024•—

172: Transformers and Large Language Models

171: Compilers and Interpreters
Feb 12, 2024•—

171: Compilers and Interpreters

170: 2023 Holiday Special Live
Dec 24, 2023•—

170: 2023 Holiday Special Live

169: HyperLogLog
Nov 27, 2023•—

169: HyperLogLog

168: Godot
Nov 20, 2023•—

168: Godot

167: Desktop User Interfaces
Oct 23, 2023•—

167: Desktop User Interfaces

166: Speedy Database Queries with Lukas Fittl
Oct 16, 2023•—

166: Speedy Database Queries with Lukas Fittl

165: Differential Equations
Sep 25, 2023•—

165: Differential Equations

164: Choosing a Database For Your Project With Kris Zyp
Sep 11, 2023•—

164: Choosing a Database For Your Project With Kris Zyp

163: Recursion
Aug 14, 2023•—

163: Recursion

162: Interactive Fiction
Jul 24, 2023•—

162: Interactive Fiction

161: Leveraging Generative AI Models with Hagay Lupesko
Jul 10, 2023•—

161: Leveraging Generative AI Models with Hagay Lupesko

Może ci się też spodobać