4 posts tagged with "deep-dive"

Every cab I have ever ridden has been complaining about how hard it is to make ends meet as a driver. The public is generally quick to blame unfair competition from the likes of Uber. However, additional forces are also to blame.

Going through more than 10 years worth of NYC taxi data, I analyse how the antiquated meter system impacts the livelihood of NYC cabbies by drawing an analogy with stock options trading. Interestingly, this approach allows us to show that drivers have progressively been worse-off, independently of competition from Uber.

In order to do so, we have loaded a dataset into our database QuestDB. This dataset includes over 1.6 billion taxi rides, 700 million FHV rides (Uber, Lyft etc), and 10 years of weather and gas prices data.

How does QuestDB get the kind of performance it does, and how are we continuing to squeeze another 50-60% out of it? We are constantly learning more about the fundamental concepts of memory performance, and this is one example of how what we at first thought would be worse for performance ended up bringing us a rather substantial boost in overall memory performance.

We will walk you through how some of our initial thoughts on storage and memory-mapping evolved to bring us better performance overall.

If you like QuestDB, please do give us a star on GitHub

In the world of databases, benchmarking performance has always been the hottest topic. Who is faster for data ingestion and queries? About a month ago we announced a new release with SIMD aggregations on HackerNews and Reddit. Fast. But were those results numerically accurate?

Inter-thread messaging is a fundamental part of any asynchronous system. It is the component responsible for transportation of data between threads. Messaging forms the infrastructure, the scaffolding of multi-threaded application and just like real-world transport infrastructure we want it to be inexpensive, fast, reliable and clean.

For QuestDB we wrote our own messaging system and this post is about how it works and how fast it is.