It is not a secret that with workflow orchestration tools like Airflow, testing and iterating a pipeline is notoriously hard, especially when cloud components come to play. In this article, we will explore an easy way to set up a complete mock infrastructure setup using docker compose, localstack, and the…

Some of the most in-demand and hot certifications nowadays are those that the known cloud providers offer. These are ways to validate your knowledge, NOT your experience, in the given space and serve as means to increase your opportunities getting hired. In this article I am going to focus on…

After spending some time with Spark, I noticed some quirks while developing — some major, some minor. I hope this article helps shed some light on some difficulties and confusing aspects that a lot of Spark users have encountered over time. Compression Spark makes connecting to data sources a no-brain operation…