Toggle.global logoCase study

Migrating to QuestDB

Toggle switched from InfluxDB to QuestDB, leading to massive cost reduction and performance improvements.

Toggle.global background

Dollar iconDirect cost reduction (¼ of the machines)

Workflow iconScript to read from one side & ingest in the other

Leaf iconMachines never overtaxed

Gauge iconQueries are >300x faster

Voice iconProactive customer support

Time iconImported 600 million data points in a few minutes

TOGGLE is a SaaS company building state-of-the-art AI technology to help investors turn Big Data into investment insights.

In this case study, Toggle’s CTO, Armenak, summarises the migration experience and goes through the improvements they saw.

Description of Toggle use case with QuestDB

Toggle uses AI & Machine Learning to help investors extract insights on their portfolio & investments. The system distills billions of data points into alerts like “Analyst expectations are turning negative for AAPL, historically this led to stock’s outperformance.” As you can imagine, this sort of system requires a tremendous amount of timeseries data — prices, fundamentals, sentiment, etc. All of this data is stored as series and needs to be easily accessible for analysis by our models. It is critical that every step in the process is optimized.

Improvements versus InfluxDB

We utilized many databases, including Mongo, Cassandra, and TimescaleDB. After much testing, we settled on InfluxDB, as it had the best performance. That said, as we were growing, performance started to degrade and it became expensive to run. After a short time, we had a small cluster of 4 x m4.2xlarge machines, and memory on all 4 was often at least 80%, hitting 100% a few times per week. Modeling out our future infrastructure spend based on this baseline, we knew InfluxDB wasn’t a viable option as we scaled.

Process to migrate data from InfluxDB to QuestDB

When evaluating a new solution, we knew we had to answer the following questions:

  • Can we move the data seamlessly and in a timely manner?
  • Can we query a sample of our data in at least the same response times of influx?
  • Can we ingest new data seamlessly?
  • Can we create time series on the fly?
  • Can we maintain the performance that we have today after we’ve imported all of our data?

Of all the possible solutions evaluated, QuestDB was the only that met all of our criteria.

A side by side comparison of QuestDB vs InfluxDB

  • InfluxDB on our cluster of 4 x m4.2xlarge with 128GiB of RAM was averaging a response time of over 5sChart showing the average transaction duration for InfluxDB over 2 days
  • The first day of QuestDB in production on a single m4.2xlarge virtual machine, saw an average response time of 19ms
  • After a few weeks with QuestDB in production (still with a single machine), the performance averaged 15msChart showing the average transaction duration for QuestDB over 2 days
  • When looking at the virtual machine’s statistics, it never seems to be overtaxed (User: 17%, system: 4%)

Direct cost reduction (¼ of the machines) and performance improvements means that we can do much more for less.

The actual data migration was easy with a script to read from one side & ingest in the other. We imported over 600 million data points in a few minutes.

Customer support experience from QuestDB’s team during the process

The QuestDB team assisted us in all steps along the way. They were proactive in supporting our changeover, helping to debug issues as they arose and optimize our deployment as we moved things into production.