QuestDB is able to ingest data over Influx Line Protocol. Existing systems already publishing line protocol need not change at all. Although QuestDB uses relational model, line protocol parser will automatically create tables and add missing columns.
For the time being QuestDB can listen for UDP packets, either multicast or unicast. TCP and HTTP support for line protocol is on our road map. Find an example of how to use this here
QuestDB listens for multicast on
22.214.171.124:9009. The address change or switch to unicast can be done via configuration. If you run QuestDB via Docker start container as
run --rm -it -p 9000:9000 -p 8812:8812 -p 9009:9009 --net=host questdb/questdband publish multicast packets with TTL of at least 2.
Influx Line Protocol follows this syntax
table_name,tagset valueset timestamp
|table_name||Name of the table where QuestDB will write data.|
|tagset||Array of string key-value pairs separated by commas that represent the reading's associated metadata|
|values||Array of key-value pairs separated by commas that represent the readings. The keys are string, values can be numeric or boolean|
|timestamp||UNIX timestamp. By default in microseconds. Can be changed in the configuration (see below)|
table_namedoes not correspond to an existing table, QuestDB will create the table on the fly using the name provided. Column types will be automatically recognized and assigned based on the data.
Let's assume the following data:
Line protocol to insert this data in the
readings table would look like this:
readings,city=London,make=Omron temperature=23.5,humidity=0.343 1465839830100400000 readings,city=Bristol,make=Honeywell temperature=23.2,humidity=0.443 1465839830100600000 readings,city=London,make=Omron temperature=23.6,humidity=0.348 1465839830100700000
Note there are only 2 spaces in each line. First between the
values. Second between
Dealing with irregularly-structured data
QuestDB can support on-the-fly data structure changes with minimal overhead. Should users decide to send varying quantities of readings or metadata tags for different entries, QuestDB will adapt on the fly.
Influx line protocol makes it possible to send data under different shapes. Each new entry may contain certain metadata tags or readings, and others not. Whilst the example just above highlights structured data, it is possible for Influx line protocol users to send data as follows.
readings,city=London temperature=23.2 1465839830100400000 readings,city=London temperature=23.6 1465839830100700000 readings,make=Honeywell temperature=23.2,humidity=0.443 1465839830100800000
Note that on the third line,
- a new
tagis added: "make"
- a new
fieldis added: "humidity"
After writing two entries, the data would look like this:
The third entry would result in the following table:
Adding columns on the fly is no issue for QuestDB. New columns will be created in the affected partitions, and only populated if they contain values. Whilst we offer this function for flexibility, we recommend that users try to minimise structural changes to maintain operational simplicity.
The following configuration parameters can be added to the configuration file to customise the interaction using
Influx line protocol. The configuration file is found under
|line.udp.join||"126.96.36.199"||Multicast address receiver joins. This values is ignored when receiver is in "unicast" mode|
|line.udp.bind.to||"0.0.0.0:9009"||IP address of the network interface to bind listener to and port. By default UDP receiver listens on all network interfaces|
|line.udp.commit.rate||1000000||For packet bursts the number of continuously received messages after which receiver will force commit. Receiver will commit irrespective of this parameter when there are no messages.|
|line.udp.msg.buffer.size||2048||Buffer used to receive single message. This value should be roughly equal to your MTU size.|
|line.udp.msg.count||10000||Only for Linux. On Linix QuestDB will use recvmmsg(). This is the max number of messages to receive at once.|
|line.udp.receive.buffer.size||8388608||UDP socket buffer size. Larger size of the buffer will help reduce message loss during bursts.|
|line.udp.enabled||true||Flag to enable or disable UDP receiver|
|line.udp.own.thread||false||When "true" UDP receiver will use its own thread and busy spin that for performance reasons. "false" makes receiver use worker threads that do everything else in QuestDB.|
|line.udp.own.thread.affinity||-1||-1 does not set thread affinity. OS will schedule thread and it will be liable to run on random cores and jump between the. 0 or higher pins thread to give core. This propertly is only valid when UDP receiver uses own thread.|
|line.udp.unicast||false||When true, UDP will me unicast. Otherwise multicast|
|line.udp.timestamp||"n"||Input timestamp resolution. Possible values are "n", "u", "ms", "s" and "h".|
|line.udp.commit.mode||"nosync"||Commit durability. Available values are "nosync", "sync" and "async"|