logoCase study

Toggle migration to QuestDB

Toggle switched from InfluxDB to QuestDB, leading to massive cost reduction and performance improvements.

An illustration representing data as a connected mesh network

Dollar iconDirect cost reduction (¼ of the machines)

Workflow iconScript to read from one side & ingest in the other

Leaf iconMachines never overtaxed

Gauge iconQueries are >300x faster

Voice iconProactive customer support

Time iconImported 600 million data points in a few minutes

TOGGLE is a SaaS company building state-of-the-art AI technology to help investors turn Big Data into investment insights.

In this case study, we summarize Toggle’s migration experience from InfluxDB to QuestDB and the improvements the migration brought.

Description of Toggle use case with QuestDB

Toggle uses AI & Machine Learning to help investors extract insights on their portfolio & investments. The system distills billions of data points into alerts like “Analyst expectations are turning negative for AAPL; historically, this led to outperformance of the stock.” As you can imagine, this sort of system requires a tremendous amount of timeseries data — prices, fundamentals, sentiment, etc. All of this data is stored as a series and needs to be easily accessible for analysis by our models. It is critical that every step in the process is optimized.

Improvements over InfluxDB

Toggle utilized many databases, including Mongo, Cassandra, and TimescaleDB. After much testing, they settled on InfluxDB, as it had the best performance. As the company was growing, performance started to degrade, and it became expensive to run. They had a small cluster of 4 x m4.2xlarge machines, and memory on all 4 was often at least 80%, hitting 100% a few times per week. Modeling out the future infrastructure spend based on this baseline, InfluxDB wasn’t a viable option as the company scaled.

The process to migrate data from InfluxDB to QuestDB

When evaluating a new solution, Toggle knew that they had to answer the following questions:

  • Can we move the data seamlessly and promptly?
  • Can we query a sample of our data with at least the response times of InfluxDB?
  • Can we ingest new data seamlessly?
  • Can we create time series on the fly?
  • Can we maintain the performance that we have today after we’ve imported all of our data?

Of all the possible solutions evaluated, QuestDB was the only one that met all of our criteria.

A side by side comparison of QuestDB vs. InfluxDB

  • InfluxDB on a cluster of 4 machines (m4.2xlarge with 128GiB of RAM) was averaging a response time of over 5 seconds.
  • After a few weeks with QuestDB in production (still with a single machine), the performance averaged 15 milliseconds, i.e., more than 300x faster queries.
    Chart showing the average transaction duration for QuestDB on a given day
  • The virtual machine’s stats indicated that the servers were never overtaxed (user: 17%, system: 4%).
  • Direct cost reduction (¼ of the machines) and performance improvements meant that Toggle could do much more for less.

The actual data migration was easy with a script to read from one side & ingest in the other. Toggle imported over 600 million data points in a few minutes.

The QuestDB team assisted us in all steps along the way. They were proactive in supporting our changeover, helping to debug issues as they arose, and optimize our deployment as we moved things into production.

Armenak, Toggle's CTO