Version 2.8.0 Gives You Early Access to Zookeeper-Less Kafka — Introduction Apache Kafka 2.8.0 is finally out and you can now have early-access to KIP-500 that removes the Apache Zookeeper dependency. Instead, Kafka now relies on an internal Raft quorum that can be activated through Kafka Raft metadata mode. …

Save time when converting large Spark DataFrames to Pandas — Converting a PySpark DataFrame to Pandas is quite trivial thanks to toPandas()method however, this is probably one of the most costly operations that must be used sparingly, especially when dealing with fairly large volume of data. Why is it so costly? Pandas DataFrames are stored in-memory which means that the operations over them are faster…

Who is an Analytics Engineer and what are they supposed to do — Traditionally, data teams were formed by Data Engineers and Data Analysts. The Data Engineers are responsible for building up the infrastructure to support data operations. These would include the configuration of databases and the implementation of ETL processes that are used to ingest data from external sources into a destination…

Here’s why you should avoid representing paths as strings and use Pathlib instead — Working with filesystems is one of the most trivial tasks in programming. Surprisingly, many of us still get it wrong since we tend to represent file paths as strings. …

Medium alternatives to consider after the recent changes in Medium Partner Program — The recent changes to the Medium Partner Program have had a negative impact on the earnings of many technical writes on the platform. As a consequence, many authors have already begun seeking for alternative options that could eventually help them monetize their content. While technical writing is slowly dying on…

Exploring batch and real-time design paradigms for data processing — Extract, Transform, Load (ETL) and Extract, Load, Transform (ELT) are two fundamental concepts in the context of data processing, used to describe data ingestion and transformation design paradigms. While these terms are often used interchangeably, they refer to slightly different concepts and are applicable to different use cases that also…

Here are the most important feature updates that will make your life easier and save you time — Apache Airflow 2.7.0 is finally out and we are all excited to see all these notable features being shipped in this latest release. The new version consists of 40 new features, 53 bug fixes, 49 improvements and 15 documentation updates. The main focus of this release has been security but…

Automated rendering of dbt projects on Apache Airflow — data build tool (dbt) is undeniably one of the rapidly emerging technologies in the field of data and analytics engineering. dbt serves as a data transformation tool, playing a crucial role in the T-stage of ELT pipelines. It empowers teams to effectively deploy analytics code while adhering to best practices…

Fixing the error reported in Python as a result of Pillow upgrade — Recently, an error has started showing up for Python developers utilizing Python Imaging Library (PIL), either directly or indirectly (for example, when interacting with OpenCV library). More specifically, the error that many users see is the following: Traceback (most recent call last): File "<module1>", line 19, in <module> AttributeError: module…