InfluxDB Line Protocol Overview

QuestDB implements the InfluxDB Line Protocol to ingest data.

The InfluxDB Line Protocol is for data ingestion only.

For building queries, see the Query & SQL Overview.

Each ILP client library also has its own language-specific documentation set.

This supporting document thus provides an overview to aid in client selection and initial configuration:

  1. Client libraries
  2. Basic ILP usage
  3. TCP vs HTTP ILP transport options
  4. Configuration
  5. Authentication

Client libraries#

The quickest way to get started is to select your library of choice.

From there, its documentation will carry you through to implementation.

Client libraries are available for several languages:

Client LanguageClient Docs
.NETSee GitHub README
C & C++See GitHub README
Rustdocs.rs
JavaQuestDB docs
Pythonreadthedocs.io
NodeJSJSDocs
Gopackage.go.dev

If you'd like more context on ILP overall, please continue reading.

Usage#

This section provides examples of data ingestion via InfluxDB Line Protocol.

They use existing client libraries or raw TCP socket connections when a client library is not available.

The following examples send a few rows of input using insecure basic auth.

Swap in with your own credentials.

Python client library docs and repo.

See more examples, including ingesting data from Pandas dataframes.

python3 -m pip install questdb
from questdb.ingress import Sender, IngressError, TimestampNanos
import sys
import datetime
def example(host: str = 'localhost', port: int = 9009):
try:
with Sender(host, port) as sender:
# Record with provided designated timestamp (using the 'at' param)
# Notice the designated timestamp is expected in Nanoseconds,
# but timestamps in other columns are expected in Microseconds.
# The API provides convenient functions
sender.row(
'trades',
symbols={
'pair': 'USDGBP',
'type': 'buy'},
columns={
'traded_price': 0.83,
'limit_price': 0.84,
'qty': 100,
'traded_ts': datetime.datetime(
2022, 8, 6, 7, 35, 23, 189062,
tzinfo=datetime.timezone.utc)},
at=TimestampNanos.now())
# You can call `sender.row` multiple times inside the same `with`
# block. The client will buffer the rows and send them in batches.
# We recommend flushing periodically, for example every few seconds.
# If you don't flush explicitly, the client will flush automatically
# once the buffer is reaches 63KiB and just before the connection
# is closed.
sender.flush()
except IngressError as e:
sys.stderr.write(f'Got error: {e}\n')
if __name__ == '__main__':
example()

HTTP and TCP overview#

The InfluxDB Line Protocol within QuestDB supports both HTTP and TCP receivers.

Both transports offer a similar performance profile, however HTTP enables additional benefits like precision timestamp parameters, chunking and multi-part requests, error feedback, and a health check endpoint. The HTTP endpoint also works as a drop-in replacement in the context of existing InfluxDB clients.

Both both transports also offer similar features:

  • ingestion only, there is no query capability
  • accepts plain text input in the InfluxDB Line Protocol format
  • batching: TCP batching is implicit, HTTP batching is explicit
  • supports automatic table and column creation
  • multi-threaded, non-blocking
  • supports authentication
  • encryption requires an optional external reverse-proxy

If going over TCP, ILP will use a dedicated port 9009 (default).

If going over HTTP, ILP will use shared HTTP port 9000 (default) if the following is set in server.conf:

line.http.enabled=true

While HTTP is available, please note that:

  1. QuestDB clients are still being updated for full HTTP support. This means that each client may be in a different state of support for HTTP-based methods and may not have HTTP-based examples. If your client-of-choice does not present enough information, please join us in our community Slack and we can help you.
  2. There is a transactionality caveat.

Configuration#

The TCP and HTTP receiver configuration can be completely customized using QuestDB configuration keys for ILP.

Configure the thread pools, buffer and queue sizes, receiver IP address and port, load balancing, and more.

For more guidance in how to tune QuestDB, see capacity planning.

Authentication#

note

Using QuestDB Enterprise?

Skip to advanced security features instead, which provides holistic security out-of-the-box.

InfluxDB Line Protocol supports authentication.

A similar pattern is used across all client libraries.

However, the underlying TCP or HTTP transport method supports different auth methods:

This document will break down and demonstrate the configuration keys and core configuration options.

Once a client has been selected and configured, resume from your language client documentation.

Configuration strings#

Configuration strings combine a set of key/value pairs.

Assembling a string connects an ILP client to a QuestDB ILP server.

The standard configuration string pattern is:

schema::key1=value1;key2=value2;key3=value3;

It is made up of the following parts:

  • Schema: One of the specified schemas in the base values section below
  • Key=Value: Each key-value pair sets a specific parameter for the client
  • Terminating semicolon: A semicolon must follow the last key-value pair

Basic Examples#

Configuration strings can be utilized directly or with a builder pattern.

For client specific documentation, see your chosen client library.

// Direct usage in Rust
let mut sender = Sender::connect("http::addr=localhost;user=joe;pass=bloggs;")?;
// Builder usage in Rust
let mut sender = SenderBuilder::from_conf("http::addr=localhost;user=joe;")?
.pass(read_secret()?)
.build()?;
# From env var, in Python
with Sender.fromEnv() as sender:
sender.row(...)

Note in these examples that:

  • Clients enforce full parameter validation upon completion of connection
  • Once specified in the initial string, parameters cannot be overridden

Client parameters#

Below is a list of common parameters that ILP clients will accept.

These params facilitate connection to QuestDB's ILP server and define client-specific behaviors.

Some are shared across all clients, while some are client specific.

See the Usage section for write examples that use these schemas.

Base parameters#

  • schema: Specifies the transport method, with support for: http, https, tcp & tcps
  • addr: The address and port of the QuestDB server.
  • user: Username for authentication (HTTP Basic or TCP).

Sensitive parameters#

Warning! Please remember to treat the following as sensitive.

Exposing these tokens could expose your database.

  • pass: Password for HTTP Basic authentication.
  • token, token_x, token_y: Authentication tokens and their parts for TCP authentication.
    • token_x & token_y only needed for C/C++/Rust & Python clients.

Auto-flushing behavior#

  • auto_flush: Enable or disable automatic flushing (on/off).

    • Default is “on” for clients that support auto-flushing (All except C, C++ & Rust).
  • auto_flush_rows: Auto-flushing is triggered above this row count.

    • Defaults to 600 if unspecified
    • If set explicitly implies “auto_flush=on”.
  • auto_flush_bytes Auto-flushing is triggered above this buffer size.

    • Python only.
    • Errors if “auto_flush_rows” is set.
    • Disabled by default.

Network configuration#

Optional.

  • bind_interface: Specify the local network interface for outbound connections.
    • Not to be confused with the QuestDB port in the addr param.

TLS configuration#

  • tls_verify: Toggle verification of TLS certificates. Default is on.
  • tls_roots: Specify the source of bundled TLS certificates.
    • The defaults and possible param values are client-specific.
      • In Rust and Python this might be “webpki”, “os-certs” or a path to a “pem” file.
      • In Java this might be a path to a “jks” trust store.
      • tls_roots_password Password to a configured tls_roots if any.
        • Passwords are sensitive! Manage appropriately.
  • tls_ca: Path to single certificate authourity, not supported on all clients.
    • Java for instance would apply tls_roots=/path/to/Java/key/store

TCP Auth via JSON Web Keys#

InfluxDB Line Protocol auth over TCP works via an elliptic curve P-256 JSON Web Token (JWT) to sign a server challenge.

Prerequisites#

Use jose to generate cryptographic keys.

We also recommend jq to parse the JSON output from the keys generated by jose

brew install jose
brew install jq

Auth file creation#

First, create an authentication file using the following template:

testUser1 ec-p-256-sha256 fLKYEaoEb9lrn3nkwLDA-M_xnuFOdSt9y0Z7_vWSHLU Dt5tbS1dEDMSYfym3fgMv0B99szno-dFc1rYF9t0aac
# [key/user id] [key type] {keyX keyY}

Only elliptic curve (P-256) are supported (key type ec-p-256-sha256).

Generate an authentication file using the jose utility:

jose jwk gen -i '{"alg":"ES256", "kid": "testUser1"}' -o /var/lib/questdb/conf/full_auth.json
KID=$(cat /var/lib/questdb/conf/full_auth.json | jq -r '.kid')
X=$(cat /var/lib/questdb/conf/full_auth.json | jq -r '.x')
Y=$(cat /var/lib/questdb/conf/full_auth.json | jq -r '.y')
echo "$KID ec-p-256-sha256 $X $Y" | tee /var/lib/questdb/conf/auth.txt

Reference the new file in your server.conf configuration:

/path/to/server.conf
line.tcp.auth.db.path=conf/auth.txt

Create a client key#

For the server configuration above, the corresponding JSON Web Key must be stored on the client side.

When sending a fully-composed JWK, it will have the following keys:

{
"kty": "EC",
"d": "5UjEMuA0Pj5pjK8a-fa24dyIf-Es5mYny3oE_Wmus48",
"crv": "P-256",
"kid": "testUser1",
"x": "fLKYEaoEb9lrn3nkwLDA-M_xnuFOdSt9y0Z7_vWSHLU",
"y": "Dt5tbS1dEDMSYfym3fgMv0B99szno-dFc1rYF9t0aac"
}

For this kind of key, the d property is used to generate the the secret key.

The x and y parameters are used to generate the public key (values that we retrieve in the server authentication file).

Configure client library#

The server will now expect the client to send its key id (terminated with \n) straight after connect(). The server will respond with a challenge (printable characters terminated with \n). The client needs to sign the challenge and respond to the server with the base64 encoded signature (terminated with \n). If all is good the client can then continue, if not the server will disconnect and log the failure.

const { Socket } = require("net")
const { Crypto } = require("node-webcrypto-ossl")
const crypto = new Crypto()
const PORT = 9009
const HOST = "localhost"
const PRIVATE_KEY = "5UjEMuA0Pj5pjK8a-fa24dyIf-Es5mYny3oE_Wmus48"
const PUBLIC_KEY = {
x: "fLKYEaoEb9lrn3nkwLDA-M_xnuFOdSt9y0Z7_vWSHLU",
y: "Dt5tbS1dEDMSYfym3fgMv0B99szno-dFc1rYF9t0aac",
}
const JWK = {
...PUBLIC_KEY,
kid: "testUser1",
kty: "EC",
d: PRIVATE_KEY,
crv: "P-256",
}
const client = new Socket()
async function write(data) {
return new Promise((resolve) => {
client.write(data, () => {
resolve()
})
})
}
async function authenticate(challenge) {
// Check for trailing \n which ends the challenge
if (challenge.slice(-1).readInt8() === 10) {
const apiKey = await crypto.subtle.importKey(
"jwk",
JWK,
{ name: "ECDSA", namedCurve: "P-256" },
true,
["sign"],
)
const signature = await crypto.subtle.sign(
{ name: "ECDSA", hash: "SHA-256" },
apiKey,
challenge.slice(0, challenge.length - 1),
)
await write(`${Buffer.from(signature).toString("base64")}\n`)
return true
}
return false
}
async function sendData() {
const rows = [
`test,location=us temperature=22.4 ${Date.now() * 1e6}`,
`test,location=us temperature=21.4 ${Date.now() * 1e6}`,
]
for (row of rows) {
await write(`${row}\n`)
}
}
async function run() {
let authenticated = false
let data
client.on("data", async function (raw) {
data = !data ? raw : Buffer.concat([data, raw])
if (!authenticated) {
authenticated = await authenticate(data)
await sendData()
setTimeout(() => {
client.destroy()
}, 0)
}
})
client.on("ready", async function () {
await write(`${JWK.kid}\n`)
})
client.connect(PORT, HOST)
}
run()

Something missing? Page not helpful? Please suggest an edit on GitHub.