QuestDB implements the InfluxDB line protocol to ingest data. This enables you to use QuestDB as a drop-in replacement for InfluxDB and others implementing the protocol. QuestDB can listen for line protocol packets both over TCP and UDP.
This page describes InfluxDB line protocol APIs. More details on the protocol, including including practical hints for understanding and working with the message format can be found in the InfluxDB line protocol guide.
|Name of the table where QuestDB will write data.|
|Array of string key-value pairs separated by commas that represent the reading's associated metadata|
|Array of key-value pairs separated by commas that represent readings. The keys are string, values can be string, numeric, boolean or timestamp|
|UNIX timestamp. By default in nanoseconds. Can be changed in the configuration|
- When the
table_namedoes not correspond to an existing table, QuestDB will create the table on the fly using the name provided. Column types will be automatically recognized and assigned based on the data.
timestampcolumn is automatically created as designated timestamp with the partition strategy set to
NONE. If you would like to define a partition strategy, you should CREATE the table beforehand.
- When the timestamp is empty, QuestDB will use the server timestamp.
Let's assume the following data:
The line protocol syntax for that table is:
InfluxDB line protocol makes it possible to send data under different shapes. Each new entry may contain certain metadata tags or readings, and others not. QuestDB can support on-the-fly data structure changes with minimal overhead. Whilst the example just above highlights structured data, it is possible for InfluxDB line protocol users to send data as follows:
This would result in the following table:
Whilst we offer this function for flexibility, we recommend that users try to minimise structural changes to maintain operational simplicity.
The TCP receiver can handle both single and multi-row write requests. It is fully multi-threaded and customizable. It can work from the common worker pool or out of dedicated threads. A load balancing mechanism dynamically assigns work between the threads.
By default, QuestDB listens to line protocol packets over TCP on
If you are running QuestDB with Docker, you will need to publish the port
-p 9009:9009. This port can be customized.
Although the original protocol does not support it, we have added authentication over TCP. This works by using an elliptic curve P-256 JSON Web Token (JWT) to sign a server challenge. Details of authentication over ILP can be found in the authentication documentation
A load balancing job reassigns work between threads in order to relieve the busiest threads and maintain high ingestion speed. It can be triggered in two ways.
- After a certain number of updates per table
- After a certain amount of time has passed
Once either is met, QuestDB will calculate a load ratio as the number of writes by the busiest thread divided by the number of writes in the least busy thread. If this ratio is above the threshold, the table with the least writes in the busiest worker thread will be reassigned to the least busy worker thread.
Uncommitted rows are committed either:
line.tcp.maintenance.job.hysterisis.in.msmilliseconds have passed
- once reaching
The TCP receiver configuration can be completely customized using configuration keys. You can use this to configure the tread pool, buffer and queue sizes, receiver IP address and port, load balancing etc.
Please check or Develop section:
The UDP receiver can handle both single and multi row write requests. It is currently single-threaded, and performs both network IO and write jobs out of one thread. The UDP worker thread can work either on its own thread or use the common thread pool. It supports both multicast and unicast.
By default, QuestDB listens for
multicast line protocol packets over UDP on
188.8.131.52:9009. If you are running QuestDB with Docker, you will need to
publish the port
-p 9009:9009 and publish multicast packets with
TTL of at least 2. This port can be customized, and you can also configure
QuestDB to listen for
Uncommitted rows are committed either:
- after receiving a number of continuous messages equal to
- when messages are no longer being received
The UDP receiver configuration can be completely customized using configuration keys. You can use this to configure the IP address and port the receiver binds to, commit rates, buffer size, whether it should run on a separate thread etc.
Find an example of how to use this in the InfluxDB sender library section.