Version 2.8.0 Gives You Early Access to Zookeeper-Less Kafka — Introduction Apache Kafka 2.8.0 is finally out and you can now have early-access to KIP-500 that removes the Apache Zookeeper dependency. Instead, Kafka now relies on an internal Raft quorum that can be activated through Kafka Raft metadata mode. The new feature simplifies cluster administration and infrastructure management and marks a…

Save time when converting large Spark DataFrames to Pandas — Converting a PySpark DataFrame to Pandas is quite trivial thanks to toPandas()method however, this is probably one of the most costly operations that must be used sparingly, especially when dealing with fairly large volume of data. Why is it so costly? Pandas DataFrames are stored in-memory which means that the operations over them are faster…

Sorting pandas DataFrames using one or more columns — When inspecting our data, we may sometimes want or even have to sort it based on one or multiple columns. This simple process can help us investigate a specific use-case, explore edge cases and so on. In today’s tutorial we will explain in detail how to sort pandas DataFrames either…

Computing the mean in pandas using Group By expressions — Introduction When working with pandas DataFrames, we usually need to compute certain metrics for specific groups. The typical way to achieve this is through the traditional group-by expression followed by the relevant calculation we need to compute. In today’s short tutorial we will be showcasing how to perform Group-By operations over…

Discussing the difference between positional and keyword arguments and how to use *args and **kwargs in Python — Introduction In Python, the arguments provided in a function call are passed by assignment (i.e. by object reference). This means that whenever a function gets called, every argument will become a variable pointing to the value that was specified. Obviously, the scope of these variables will be limited to the function…

Authoring Apache Airflow DAGs and Tasks with Python decorators — Introduction The most common way for writing pipelines in Airflows is by using the DAG context managers to automatically assign new operators to that DAG. As of Airflow 2, you can now use decorators in order to author Airflow DAGs and Tasks. In today’s tutorial, we will introduce these decorators and…

Aggregating column values in pandas DataFrames using Group By expressions — Introduction One of the most commonly used applications of GROUP BY clause it to let users compute the sum of specific groups of values within a given set of data. In today’s short tutorial we will be showcasing a few different approaches when it comes to computing the sum after a…

Fixing the SIGTERM signal in Apache Airflow tasks — Introduction While I have been recently working on migrating DAGs from Airflow 1 (v1.10.15) to Airflow 2 (v2.2.5) I’ve spent a lot of time trying to figure out one error that I was getting for some of the DAGs that wasn’t informative at all. WARNING airflow.exceptions.AirflowException: Task received SIGTERM signal INFO …

Computing the NYSE Market bank holidays in Python using Pandas — Introduction Over the last few days, I’ve been trying to find a proper way in order to calculate the NYSE market bank holidays for any given year. Even though I came across a few different open-source packages, most of them were unreliable, containing a few bugs and wouldn’t let me implement…