Is this the right approach for everyone? I actually don’t know. Airflow came to market prior to the rise of Docker and Kubernetes, but at this point I have a hard time imagining wanting to run a huge Airflow installation without the infrastructure they provide.
Running jobs with heterogenous dependencies as a part of a single DAG feels like it shouldn’t be that hard, but as soon as you have two requirements.txt files things can get bad quickly.
One engineering team developed a solution involving Docker and Kubernetes.
Reference: https://medium.com/bluecore-engineering/were-all-using-airflow-wrong-and-how-to-fix-it-a56f14cb0753
Comentarios