--

Hi Thomas, that's a great question!

Indeed, now every dbt entity (i.e. model, seed, test, or snapshot) will run on their own k8s pod, through KubernetesPodOperator.

However, from my own personal experience, I haven't noticed any significant increase (no more than 5%) in the execution time. The projects I currently maintain consist of around ~1K models each, in case that matters to your specific use-case.

--

--

Giorgos Myrianthous
Giorgos Myrianthous

Written by Giorgos Myrianthous

I strive to build data-intensive systems that are not only functional, but also scalable, cost effective and maintainable over the long term.

No responses yet