User Notification Throttling System: Design and Implementation with Confluent KafkaDesigning an effective user notification throttling system is crucial for applications that need to manage large-scale user notifications…Jun 15, 2023Jun 15, 2023
End-to-End ETL: How to Streamline Data Transfer from Freshdesk to Snowflake with Mage and build…In this article let us see how we can use Mage and perform ETL (extract, transform, load) to transfer data from Freshdesk, a customer…May 5, 2023May 5, 2023
Building a Streamlit app on the data residing in datalakehouse using Deltalake and DuckDbIn today’s data-driven world, businesses are generating and collecting massive amounts of data from various sources. However, managing and…Apr 28, 2023Apr 28, 2023
In this article lets see how DuckDB is a game changer in analytical world and how it helps data…What is DuckDB ?Feb 3, 2023Feb 3, 2023
Generate Sample Data using FakerIn this article lets see how we can generate sample data in CSV or Json format that will be needed for our quick testing of Data Pipeline…Dec 13, 2022Dec 13, 2022
Moving data from BigQuery to On-premise HiveIn this article let’s discuss the solution for the use case that we received from our customer, Can you help us move some 200+ tables from…Dec 13, 2022Dec 13, 2022
Published inDev GeniusReal-time Streaming Analytics with Databricks Delta Live TableIn this story let us understand how useful the streaming analytics is with a real world example. Let us see how can we detect COVID cases…Nov 7, 20221Nov 7, 20221
How to create BigQuery tables on top of external data?In this article let’s see how we can create external Big query table on top of the files present in the google storage bucket using python…Nov 2, 2022Nov 2, 2022
SFTP File WatcherIn this article let’s discuss about a use-case where the need is to trigger the ingestion pipeline for the data files that land in the SFTP…Nov 2, 2022Nov 2, 2022
Delta Lake Streaming InternalsIn this article, we are going to see how to use Delta lake as a streaming source & also as a sink along with the internals of delta…Jul 9, 2021Jul 9, 2021