Use case: Market data
Industry: Fintech
Deployment: QuestDB Cloud
Invezo is a fast growing startup that aims to become the gold standard for digital assets analytics. Traditional companies in equity markets are assessed and valued based on established frameworks; their coverage from equity research companies is plentiful and all of this information is aggregated on platforms such as the Bloomberg Terminal. It's a different situation for crypto assets, which is a brand new asset class. There is a lack of tools to analyze market price behavior and underlying valuations. Invezo pulls on-chain and off-chain metrics to understand the behavior of crypto assets to a much deeper level, while offering superior charting capabilities and also providing an API.
Emmett Miller, co-founder and CEO, Invezo, tells us about his journey building his company using QuestDB Cloud.
All of the data in the Invezo platform is time series. We track all of our metrics over time. There are plenty of metrics to track: cryptocurrency token price, GitHub commits, reddit subscribers, on-chain wallet transactions. This data is then exposed to the end user of the platform in real-time. The latency needs to be as low as possible to guarantee a good user experience. We want the dashboards to refresh live in milliseconds rather than seconds. This may seem a trivial thing to achieve but it is not: users zooming in and out of our dynamic charts are performing heavy operations for the underlying database to compute. New data also flows in real-time, requiring aggregations on the fly. Performant time-series databases are crucial to cope with the ingestion rate: we currently pull market data for hundreds of cryptocurrencies from more than nine different exchanges. We also need very fast querying capabilities to power our live dashboards and analytics.
Example of live dashboards and analytics in real time. Each refresh is a SQL query for QuestDB.
We collect data from crypto exchanges APIs and build our main application in Python. We use the official QuestDB Python client library to send the data via the InfluxDB Line Protocol ("ILP") over TCP. Each query is computed programmatically via PGWire using SQL and feeds real-time data, aggregations and downsampled data to plot out charts.
QuestDB has built time-series specific syntax to make queries less verbose. For example, SAMPLE BY automatically downsamples the data for a given interval. An example of SQL query to compute the number of twitter mentions over time is the following:
Example of live dashboards and analytics in real time. Each refresh is a SQL query for QuestDB.
We started with AWS RD but the database was too slow for our use case as queries would take up to 6 minutes to come back. We then looked into MongoDB, which recently released time-series capabilities. Here again, slow ingestion and queries made our application unusable (see a comparison with QuestDB toward the end of the case study). In addition, we had a crack at InfluxDB but their query language "Flux" was a show stopper because it is unintuitive.
Here is a comparison of the same query between MongoDB for time series and QuestDB:
MongoDB query execution time: 5.4 seconds
QuestDB query execution time: 0.1 seconds
We saw the light at the end of the tunnel when we found out about QuestDB. We liked it for its superior ingestion rates and the ability to query data on the fly. On top of the superior performance we are impressed by how vibrant the community is; we asked a few questions on the QuestDB Slack and got a response immediately. We love SQL and wanted to build the queries for our application with a familiar language.
SAMPLE BY queries, which are on the fly
We initially deployed QuestDB open source via the AWS marketplace. With QuestDB open source, we do not have the luxury of web console authentication, although we knew that a workaround with setting NGINX as a proxy was possible. We also did not have the bandwidth nor internal resources to properly monitor and manage the instances. It made sense to us that the instances should be overseen by the QuestDB creators, who know the product inside out.
By offloading the management and monitoring responsibilities to the QuestDB team, we gained precious hours to focus on our business. We also benefit from authentication for the web console, and scheduled snapshots without any downtime at the click of a button. The UI is intuitive and the team is shipping new features every week. I am excited about the upcoming cold storage integration with AWS S3, which should free a lot of space on our EBS volume.
“Our customers value a low-latency API, so speed is extremely important to us. With QuestDB, our ingestion rate is 5x faster and query execution time went from minutes to milliseconds”
Emmett Miller, Co-founder, Invezo