Version 2.8.0 Gives You Early Access to Zookeeper-Less Kafka — Introduction Apache Kafka 2.8.0 is finally out and you can now have early-access to KIP-500 that removes the Apache Zookeeper dependency. Instead, Kafka now relies on an internal Raft quorum that can be activated through Kafka Raft metadata mode. The new feature simplifies cluster administration and infrastructure management and marks a…

Save time when converting large Spark DataFrames to Pandas — Converting a PySpark DataFrame to Pandas is quite trivial thanks to toPandas()method however, this is probably one of the most costly operations that must be used sparingly, especially when dealing with fairly large volume of data. Why is it so costly? Pandas DataFrames are stored in-memory which means that the operations over them are faster…

Here’s why you should avoid representing paths as strings and use Pathlib instead — Working with filesystems is one of the most trivial tasks in programming. Surprisingly, many of us still get it wrong since we tend to represent file paths as strings. This is fundamentally wrong and one of the most common anti-patterns that for sure you have probably already seen in many…

Medium alternatives to consider after the recent changes in Medium Partner Program — The recent changes to the Medium Partner Program have had a negative impact on the earnings of many technical writes on the platform. As a consequence, many authors have already begun seeking for alternative options that could eventually help them monetize their content. While technical writing is slowly dying on…

Exploring batch and real-time design paradigms for data processing — Extract, Transform, Load (ETL) and Extract, Load, Transform (ELT) are two fundamental concepts in the context of data processing, used to describe data ingestion and transformation design paradigms. While these terms are often used interchangeably, they refer to slightly different concepts and are applicable to different use cases that also…

Here are the most important feature updates that will make your life easier and save you time — Apache Airflow 2.7.0 is finally out and we are all excited to see all these notable features being shipped in this latest release. The new version consists of 40 new features, 53 bug fixes, 49 improvements and 15 documentation updates. The main focus of this release has been security but…

Automated rendering of dbt projects on Apache Airflow — data build tool (dbt) is undeniably one of the rapidly emerging technologies in the field of data and analytics engineering. dbt serves as a data transformation tool, playing a crucial role in the T-stage of ELT pipelines. It empowers teams to effectively deploy analytics code while adhering to best practices…

Fixing the error reported in Python as a result of Pillow upgrade — Recently, an error has started showing up for Python developers utilizing Python Imaging Library (PIL), either directly or indirectly (for example, when interacting with OpenCV library). More specifically, the error that many users see is the following: Traceback (most recent call last): File "<module1>", line 19, in <module> AttributeError: module…

Useful prompts to speed up development and improve efficiency of SQL queries — ChatGPT has changed the way developers write, test and maintain code. While not every single task can be automated (in an accurate way) with LLMs, it’s still important to make sure that you can consult models such as ChatGPT when they can probably do the dirty work for you. Writing…