Date ingestion over InfluxDB Line Protocol makes use of 4 logical thread pools:
- I/O worker pool -
line.tcp.io.worker.count, threads responsible for handling incoming TCP connections and parsing received InfluxDB Line Protocol messages
- writer pool -
line.tcp.writer.worker.count, threads responsible for non-WAL table writes
- WAL apply pool -
wal.apply.worker.count, threads responsible for applying data written to Write-Ahead Log (WAL) to WAL tables
- shared pool -
shared.worker.count, threads responsible for handling out-of-order data
Depending on the number of concurrent TCP connections
io worker pool size
might need to be adjusted. The ideal ratio is
1:1 - a thread per connection.
In less busy environments it is possible for single
io worker thread to handle
multiple connections simultaneously. We recommend starting with conservative
ratio, measure and increase the ratio up to
1:1. More threads than connections
will be wasting server CPU.
Another consideration applicable to non-WAL tables only is the number of tables
writer pool should be tuned to increase concurrency.
writer threads can also handle multiple non-WAL tables concurrently.
ratio is the maximum required ratio between
writer threads and tables. If
1:1 ratio is not an option, avoid writing to all tables from each connection.
Instead, group connections and tables. For example, if there are 10 non-WAL
tables, 8 TCP connections and
writer pool size is set to 2, 4 TCP connections
may be used to write into tables 1-5, while 4 connections may write into tables
Note that the last recommendation does not apply to WAL tables. QuestDB supports
concurrent writes to these tables, which removes the need for the
Instead, WAL tables use
wal apply pool. This pool seldom requires tuning as
applying WAL data to the end table is much more efficient than non-WAL writes.
Sending updates for multiple tables from a single TCP connection might be
inefficient. Consider using multiple connections to improve performance. If a
single connection is unavoidable, keep
writer pool size set to 1 for optimal
CPU resource utilization.
When ingesting data out of order (O3)
shared pool accelerates O3 tasks. It is
also responsible for SQL execution.
shared pool size should be set to use the
remaining available CPU cores.