Apache Airflow

What do you like best?

I like the ease of use, how easily I can make DAGs and the scheduler is just amazing. I have a lot of python scripts that I needed to run manually and daily. Airflow just made it seamless.

What do you dislike?

The initial setup is a bit overwhelming to me.

Recommendations to others considering the product:

Go for it. Replace your cron jobs with this, you'll be fine.

What problems are you solving with the product? What benefits have you realized?

We have an ElasticSearch Index and a Redis database.

We have build an app to move data from elasticsearch index to redis database. We had to do this daily or weekly, but by deploying the container again and again.

What we did with Airflow is we made this excercise into a task and setup an Airflow scheduler for this. Now it runs automatically based on our schedule.