Version 2.8.0 Gives You Early Access to Zookeeper-Less Kafka — Introduction Apache Kafka 2.8.0 is finally out and you can now have early-access to KIP-500 that removes the Apache Zookeeper dependency. Instead, Kafka now relies on an internal Raft quorum that can be activated through Kafka Raft metadata mode. The new feature simplifies cluster administration and infrastructure management and marks a…

Save time when converting large Spark DataFrames to Pandas — Converting a PySpark DataFrame to Pandas is quite trivial thanks to toPandas()method however, this is probably one of the most costly operations that must be used sparingly, especially when dealing with fairly large volume of data. Why is it so costly? Pandas DataFrames are stored in-memory which means that the operations over them are faster…

Understanding Common Table Expression (CTE) in SQL — Writing clean, readable and efficient SQL queries is an important aspect of any engineering or analytical process across a team. Such queries be maintained effectively and scale well when the time comes. One SQL construct that can be easily adopted by both developers and analysts in order to make this…

Best practices and things to avoid when running SQL on Google Cloud BigQuery — BigQuery is the managed Data Warehouse service on Google Cloud Platform, and like most services and technologies it comes with a set of principles that one needs to have in mind whilst using it. In the next few sections we will outline a set of best practices to avoid common…

Search operators that you can use with Gmail — Google searching is definitely one of the most underrated skills and if you think that by just typing a bunch of words in the search box you are gonna get the most optimal results, then you need to re-consider. In one of my articles a while back, I discussed about…

Incorporating null values whilst grouping by in pandas columns — Performing aggregations with the use of group by clauses is probably something you come across in your day-to-day work. Pandas in Python is no exception to this since this is an operation you will definitely see in many different places of a repository utilising the library. However, pandas’ default behaviour…

Highlighted syntax for code blocks is now available on Medium platform — While I’ve been working on the latest piece I was about to publish on Medium, I came across a brand new feature I was expecting for so long — and I guess many technical writes on the platform! You guessed it right — Syntax highlighting on code blocks is now…

Enforcing meaningful structure to dbt data models — Data models are abstract constructs that represent a particular object within an application domain. Such models could be the users, products, and orders of a particular organisation. In modern organisations where data-driven decisions are supposed to be made, it is quite important to to be able to manage data models…

Installing data build tool for your specific data warehouse — data build tool (dbt) is undoubtedly one of the most powerful tools in modern data stacks as it allows teams and organisations manage and transform data models in a scalable, efficient and effective way. dbt will handle all data model inter-dependencies and provide you with everything you need to in…