Direct cost reduction (¼ of the machines)
Script to read from one side & ingest in the other
Machines never overtaxed
Queries are >300x faster
Proactive customer support
Imported 600 million data points in a few minutes
“TOGGLE is a SaaS company building state-of-the-art AI technology to help investors turn Big Data into investment insights.”
In this case study, Toggle’s CTO, Armenak, summarises the migration experience and goes through the improvements they saw.
Toggle uses AI & Machine Learning to help investors extract insights on their portfolio & investments. The system distills billions of data points into alerts like “Analyst expectations are turning negative for AAPL, historically this led to stock’s outperformance.” As you can imagine, this sort of system requires a tremendous amount of timeseries data — prices, fundamentals, sentiment, etc. All of this data is stored as series and needs to be easily accessible for analysis by our models. It is critical that every step in the process is optimized.
We utilized many databases, including Mongo, Cassandra, and TimescaleDB. After much testing, we settled on InfluxDB, as it had the best performance. That said, as we were growing, performance started to degrade and it became expensive to run. After a short time, we had a small cluster of 4 x m4.2xlarge machines, and memory on all 4 was often at least 80%, hitting 100% a few times per week. Modeling out our future infrastructure spend based on this baseline, we knew InfluxDB wasn’t a viable option as we scaled.
When evaluating a new solution, we knew we had to answer the following questions:
Of all the possible solutions evaluated, QuestDB was the only that met all of our criteria.
A side by side comparison of QuestDB vs InfluxDB
Direct cost reduction (¼ of the machines) and performance improvements means that we can do much more for less.
The actual data migration was easy with a script to read from one side & ingest in the other. We imported over 600 million data points in a few minutes.
The QuestDB team assisted us in all steps along the way. They were proactive in supporting our changeover, helping to debug issues as they arose and optimize our deployment as we moved things into production.