InfluxDB Line Protocol Overview
QuestDB implements the InfluxDB Line Protocol to ingest data.
The InfluxDB Line Protocol is for data ingestion only.
For building queries, see the Query & SQL Overview.
Each ILP client library also has its own language-specific documentation set.
This supporting document thus provides an overview to aid in client selection and initial configuration:
#
Client librariesThe quickest way to get started is to select your library of choice.
From there, its documentation will carry you through to implementation.
Client libraries are available for several languages:
Client Language | Client Docs |
---|---|
.NET | See GitHub README |
C & C++ | See GitHub README |
Rust | docs.rs |
Java | QuestDB docs |
Python | readthedocs.io |
NodeJS | JSDocs |
Go | package.go.dev |
If you'd like more context on ILP overall, please continue reading.
#
UsageThis section provides examples of data ingestion via InfluxDB Line Protocol.
They use existing client libraries or raw TCP socket connections when a client library is not available.
The following examples send a few rows of input using insecure basic auth.
Swap in with your own credentials.
- Python
- Go
- Java
- NodeJS
- C#
- C
- C++
- Rust
- Ruby
- PHP
Java client library docs and Maven artifact.
Maven
<dependency>
<groupId>org.questdb</groupId>
<artifactId>questdb</artifactId>
<version>6.5</version>
</dependency>
Gradle
compile group: 'org.questdb', name: 'questdb', version: '6.5'
NodeJS client library repo.
#
HTTP and TCP overviewThe InfluxDB Line Protocol within QuestDB supports both HTTP and TCP receivers.
Both transports offer a similar performance profile, however HTTP enables additional benefits like precision timestamp parameters, chunking and multi-part requests, error feedback, and a health check endpoint. The HTTP endpoint also works as a drop-in replacement in the context of existing InfluxDB clients.
Both both transports also offer similar features:
- ingestion only, there is no query capability
- accepts plain text input in the InfluxDB Line Protocol format
- batching: TCP batching is implicit, HTTP batching is explicit
- supports automatic table and column creation
- multi-threaded, non-blocking
- supports authentication
- encryption requires an optional external reverse-proxy
If going over TCP, ILP will use a dedicated port 9009
(default).
If going over HTTP, ILP will use shared HTTP port 9000
(default) if the
following is set in server.conf
:
While HTTP is available, please note that:
- QuestDB clients are still being updated for full HTTP support. This means that each client may be in a different state of support for HTTP-based methods and may not have HTTP-based examples. If your client-of-choice does not present enough information, please join us in our community Slack and we can help you.
- There is a transactionality caveat.
#
ConfigurationThe TCP and HTTP receiver configuration can be completely customized using QuestDB configuration keys for ILP.
Configure the thread pools, buffer and queue sizes, receiver IP address and port, load balancing, and more.
For more guidance in how to tune QuestDB, see capacity planning.
#
Authenticationnote
Using QuestDB Enterprise?
Skip to advanced security features instead, which provides holistic security out-of-the-box.
InfluxDB Line Protocol supports authentication.
A similar pattern is used across all client libraries.
However, the underlying TCP or HTTP transport method supports different auth methods:
- HTTP(S) supports basic auth
- TCP supports JSON Web Keys (JWK)
This document will break down and demonstrate the configuration keys and core configuration options.
Once a client has been selected and configured, resume from your language client documentation.
#
Configuration stringsConfiguration strings combine a set of key/value pairs.
Assembling a string connects an ILP client to a QuestDB ILP server.
The standard configuration string pattern is:
It is made up of the following parts:
- Schema: One of the specified schemas in the base values section below
- Key=Value: Each key-value pair sets a specific parameter for the client
- Terminating semicolon: A semicolon must follow the last key-value pair
#
Basic ExamplesConfiguration strings can be utilized directly or with a builder pattern.
For client specific documentation, see your chosen client library.
Note in these examples that:
- Clients enforce full parameter validation upon completion of connection
- Once specified in the initial string, parameters cannot be overridden
#
Client parametersBelow is a list of common parameters that ILP clients will accept.
These params facilitate connection to QuestDB's ILP server and define client-specific behaviors.
Some are shared across all clients, while some are client specific.
See the Usage section for write examples that use these schemas.
#
Base parameters- schema: Specifies the transport method, with support for:
http
,https
,tcp
&tcps
- addr: The address and port of the QuestDB server.
- user: Username for authentication (HTTP Basic or TCP).
#
Sensitive parametersWarning! Please remember to treat the following as sensitive.
Exposing these tokens could expose your database.
- pass: Password for HTTP Basic authentication.
- token, token_x, token_y: Authentication tokens and their parts for
TCP authentication.
- token_x & token_y only needed for C/C++/Rust & Python clients.
#
Auto-flushing behaviorauto_flush: Enable or disable automatic flushing (
on
/off
).- Default is “on” for clients that support auto-flushing (All except C, C++ & Rust).
auto_flush_rows: Auto-flushing is triggered above this row count.
- Defaults to
600
if unspecified - If set explicitly implies “auto_flush=on”.
- Defaults to
auto_flush_bytes Auto-flushing is triggered above this buffer size.
- Python only.
- Errors if “auto_flush_rows” is set.
- Disabled by default.
#
Network configurationOptional.
- bind_interface: Specify the local network interface for outbound
connections.
- Not to be confused with the QuestDB port in the
addr
param.
- Not to be confused with the QuestDB port in the
#
TLS configuration- tls_verify: Toggle verification of TLS certificates. Default is
on
. - tls_roots: Specify the source of bundled TLS certificates.
- The defaults and possible param values are client-specific.
- In Rust and Python this might be “webpki”, “os-certs” or a path to a “pem” file.
- In Java this might be a path to a “jks” trust store.
- tls_roots_password Password to a configured tls_roots if any.
- Passwords are sensitive! Manage appropriately.
- The defaults and possible param values are client-specific.
- tls_ca: Path to single certificate authourity, not supported on all
clients.
- Java for instance would apply
tls_roots=/path/to/Java/key/store
- Java for instance would apply
#
TCP Auth via JSON Web KeysInfluxDB Line Protocol auth over TCP works via an elliptic curve P-256 JSON Web Token (JWT) to sign a server challenge.
#
PrerequisitesUse jose to generate cryptographic keys.
We also recommend jq to parse the
JSON output from the keys generated by jose
- macOS
- Debian
- Ubuntu
#
Auth file creationFirst, create an authentication file using the following template:
Only elliptic curve (P-256) are supported (key type ec-p-256-sha256
).
Generate an authentication file using the jose
utility:
Reference the new file in your server.conf configuration:
#
Create a client keyFor the server configuration above, the corresponding JSON Web Key must be stored on the client side.
When sending a fully-composed JWK, it will have the following keys:
For this kind of key, the d
property is used to generate the the secret key.
The x
and y
parameters are used to generate the public key (values that we
retrieve in the server authentication file).
#
Configure client libraryThe server will now expect the client to send its key id (terminated with \n
)
straight after connect()
. The server will respond with a challenge (printable
characters terminated with \n
). The client needs to sign the challenge and
respond to the server with the base64
encoded signature (terminated with
\n
). If all is good the client can then continue, if not the server will
disconnect and log the failure.
- NodeJS
- Go
- Python