Version 2.8.0 Gives You Early Access to Zookeeper-Less Kafka — Introduction Apache Kafka 2.8.0 is finally out and you can now have early-access to KIP-500 that removes the Apache Zookeeper dependency. Instead, Kafka now relies on an internal Raft quorum that can be activated through Kafka Raft metadata mode. …

Save time when converting large Spark DataFrames to Pandas — Converting a PySpark DataFrame to Pandas is quite trivial thanks to toPandas()method however, this is probably one of the most costly operations that must be used sparingly, especially when dealing with fairly large volume of data. Why is it so costly? Pandas DataFrames are stored in-memory which means that the operations over them are faster…

Compare tables and extract their differences with standard SQL — Comparing tables in BigQuery is a crucial task when testing the results of data pipelines and queries prior to productionizing them. The ability to compare tables allows for the detection of any changes or discrepancies in the data, ensuring that the data remains accurate and consistent. In this article we…

A complete cheatsheet for selecting specific models when running dbt commands — When working on dbt projects you need to ensure that the CLI commands used to run or test models, seeds and snapshots encompass only the resource (or a subset) of interest. In other words, you need to be able to target specific models, tests, seeds or snapshots in order to…

Fixing AttributeError introduced in recent cryptography release in Python — Recently, many Python users have started reporting AttributeError: module ‘lib’ has no attribute ‘OpenSSL_add_all_algorithms’ when attempting to install new or update existing modules via pip. File "/home/patrick/.local/lib/python3.8/site-packages/OpenSSL/__init__.py", line 8, in <module> from OpenSSL import crypto, SSL File "/home/patrick/.local/lib/python3.8/site-packages/OpenSSL/crypto.py", line 3268, in <module> _lib.OpenSSL_add_all_algorithms() …

Configuring Application Default Credentials and fixing oauth2client.client.ApplicationDefaultCredentialsError — Welcome to our tutorial on configuring Application Default Credentials for Google Cloud and Python. In this article, we will cover how to properly set GOOGLE_APPLICATION_CREDENTIALS in Python. In order to be able to programmatically interact with Google Cloud Platform services, such as Google BigQuery, you first need to properly authenticate…

A comparison between ETL and ELT in the context of Data Engineering — ETL (Extract-Transform-Load) and ELT (Extract-Load-Transform) are two terms commonly used in the realm of Data Engineering and more specifically in the context of data ingestion and transformation. While these terms are often used interchangeably, they refer to slightly different concepts and have different implications for the design of a data…

Checking if one ore more columns exist in a pandas DataFrame — pandas is one of the most popular packages of the Python ecosystem that lets developers analyse, process and transform data quickly and effectively. Some of the most common tasks when working with pandas DataFrames involve the check of whether a particular column already exists. In this article, we will be…

Taking advantage of all the options available when writing pandas DataFrames into CSV files — One very common task when working with pandas involves the export of a DataFrame into a CSV file. In this article, we will be demonstrating how to write a pandas DataFrame into CSV files. Furthermore, we will also be showcasing how to use the various options of the pandas API…