A comparison between ETL and ELT in the context of Data Engineering — ETL (Extract-Transform-Load) and ELT (Extract-Load-Transform) are two terms commonly used in the realm of Data Engineering and more specifically in the context of data ingestion and transformation. While these terms are often used interchangeably, they refer to slightly different concepts and have different implications for the design of a data…

Understanding the purpose of requirements.txt, setup.py and setup.cfg in Python when developing and distributing packages — Introduction Managing dependencies in Python projects could be quite challenging, especially for people new to the language. When developing a new Python package, the chances are you will also need to utilise some other packages that will eventually help you write less code (in less time) so that you don’t have…

Version 2.8.0 Gives You Early Access to Zookeeper-Less Kafka — Introduction Apache Kafka 2.8.0 is finally out and you can now have early-access to KIP-500 that removes the Apache Zookeeper dependency. Instead, Kafka now relies on an internal Raft quorum that can be activated through Kafka Raft metadata mode. The new feature simplifies cluster administration and infrastructure management and marks a…

Save time when converting large Spark DataFrames to Pandas — Converting a PySpark DataFrame to Pandas is quite trivial thanks to toPandas()method however, this is probably one of the most costly operations that must be used sparingly, especially when dealing with fairly large volume of data. Why is it so costly? Pandas DataFrames are stored in-memory which means that the operations over them are faster…

Fixing ImportError when importing Python packages that rely on urllib3 and OpenSSL — A growing number of Python developers have been complaining about an ImportError being reported when attempting to import packages that depend on urrlib3 and OpenSSL, like openai Python package. More specifically, the error indicates that the latest versions of urllib3 only support OpenSSL versions, 1.1.1+: ImportError: urllib3 v2.0 only supports…

Programmatically list all datasets and tables using BigQuery API and Python — BigQuery is the managed Data Warehousing service on Google Cloud Platform that lets users store, manage and query data. A big part of Data and/or Analytics Engineering is the automation of certain tasks, including some interactions with BigQuery. Such automations usually require us to make use of the BigQuery API…

How to provide a connection profile when running dbt commands — data build tool (dbt) is one of the hottest (relatively) recent addition to modern data stacks. dbt offers both, a CLI tool (dbt-core) as well as a cloud tool (dbt Cloud), which is a paid service where you can host and run your dbt projects. When using the command-line interface…

Hiding __pycache__ metadata from VS Code IDE — I recently switched from PyCharm to Visual Studio Code for my Python projects. However, I encountered an issue where VS Code displays __pycache__ files, which can clutter the File Explorer. This can be particularly frustrating for those working on large projects with multiple modules and folders given that cache files…

Managing Python project dependencies in pyproject.toml files — Dependency management in Python is tricky, and sometimes frustrating work. Newcomers, are usually tempted to install any dependency (i.e. package) they may find useful, even in a single virtual enviroment. Therefore, this approach increases the chances of having conflicting package dependencies and ending up in the so-called dependency hell.