Disaster Strikes!
All hell broke loose when a developer deleted a table in production in BigQuery and we had to fight the fire early in the morning. The table was getting continuously populated through streaming data from pubsub. Since pubsub … Continue Reading ››
CSV or comma separated files are a lifeline for many organizations and managers as we receive these files from our clients, dump data from databases like Oracle or MySQl in the delimited formats and load them to various tools to process them. Here CSV … Continue Reading ››
Many organizations have massive amount of data in their data warehouse that keeps growing every year. Business use of this data has also increased with analysis and getting actionable insights from data becoming more relevant. However, analyzing this huge … Continue Reading ››
Distributed message processing
As part of big data analytics project, you need to ingest huge amount of data in real time from external systems into your big data analytics system. You might have distributed your processing using tools like Apache Kafka but how to distribute data ingestion? That is, getting data into your big data system? … Continue Reading ››
{{unknown}}
By Ganapathi Devappa,
Many stream processing frameworks have come up in the last few years like Apache Storm, Apache Flink and Apache Kafka. In this blog post, I am … Continue Reading ››
{{unknown}}
Apache hadoop is the most popular tool for big data processing. Hadoop is most suitable for batch and offline processing but what if real time big data processing is required? Then we have to look at … Continue Reading ››
By Ganapathi Devappa,
It is a great thing that time never stops or slows down (even if it did, nobody can notice it!). Mapreduce paradigm of hadoop has been around for a few years that there has to be some improved version that can do more. … Continue Reading ››
By Ganapathi Devappa
When I am discussing with clients about big data solutions, they say the relational databases cannot handle their data and ask me if they can try Cassandra. It is just that they have heard about Cassandra as … Continue Reading ››