Direct cost reduction (¼ of the machines)
Script to read from one side & ingest in the other
Machines never overtaxed
Queries are >300x faster
Proactive customer support
Imported 600 million data points in a few minutes
“TOGGLE is a SaaS company building state-of-the-art AI technology to help investors turn Big Data into investment insights.”
In this case study, Toggle’s CTO, Armenak, summarises the migration experience and goes through the improvements they saw.
Toggle uses AI & Machine Learning to help investors extract insights on their portfolio & investments. The system distills billions of data points into alerts like “Analyst expectations are turning negative for AAPL; historically, this led to stock’s outperformance.” As you can imagine, this sort of system requires a tremendous amount of timeseries data — prices, fundamentals, sentiment, etc. All of this data is stored as a series and needs to be easily accessible for analysis by our models. It is critical that every step in the process is optimized.
Toggle utilized many databases, including Mongo, Cassandra, and TimescaleDB. After much testing, they settled on InfluxDB, as it had the best performance. As the company was growing, performance started to degrade, and it became expensive to run. They had a small cluster of 4 x m4.2xlarge machines after a short time, and memory on all 4 was often at least 80%, hitting 100% a few times per week. Modeling out the future infrastructure spend based on this baseline, InfluxDB wasn’t a viable option as the company scaled.
When evaluating a new solution, Toggle knew that they had to answer the following questions:
Of all the possible solutions evaluated, QuestDB was the only one that met all of our criteria.
The actual data migration was easy with a script to read from one side & ingest in the other. Toggle imported over 600 million data points in a few minutes.
Armenak, Toggle's CTO: The QuestDB team assisted us in all steps along the way. They were proactive in supporting our changeover, helping to debug issues as they arose, and optimize our deployment as we moved things into production.