QuestDB is written in Java and can be used as any other Java library. Moreover, it is a single JAR with no additional dependencies.
To include QuestDB in your project, use the following:
This section provides example codes to write data to WAL and non-WAL tables. See Write Ahead Log for details about the differences between WAL and non-WAL tables.
The following writers are available for data ingestion:
WalWriterfor WAL tables
TableWriterfor non-WAL tables
TableWriterAPIfor both WAL and non-WAL tables as it is an interface for
Writing data using
WalWriter facilitates table writes to WAL tables. To successfully create
an instance of
WalWriter, the table must already exist.
Writing data using
Non-WAL tables do not allow concurrent writes via multiple interfaces. To successfully create an instance, the table must:
- Already exist
- Have no other open writers against it as the
TableWriterconstructor will attempt to obtain an exclusive cross-process lock on the table.
Writing data using
TableWriterAPI allows writing to both WAL and non-WAL tables by returning the
Writer based on the table configurations. The table must already
#Configure Cairo engine
CairoEngine is a resource manager for the embedded QuestDB. Its main function is
to facilitate concurrent access to pools of
TableReader and suitable writer
A typical application will need only one instance of
instance will start when the application starts and shuts down when the
application closes. You will need to close
CairoEngine gracefully when the
QuestDB provides a default configuration which only requires the
data directory to be specified. For a more advanced usage, the whole
CairoConfiguration interface can be overridden.
#Create an instance of SqlExecutionContext
Execution context is a conduit for passing SQL execution artifacts to the execution site. This instance is not thread-safe and it must not be shared between threads.
The second argument is the number of threads that will be helping to execute SQL statements. Unless you are building another QuestDB server, this value should always be 1.
#New SqlCompiler instance and blank table
Before we start writing data using a writer, the target table has to exist.
There are several ways to create a new table and we recommend using
As you will be able to see below, the table field types and indexes must match the code that is populating the table.
#A new writer instance
engine to create an instance of the writer. This will enable reusing
this writer instance later, when we use the same method of creating table writer
TableWriter - A non-WAL table uses
TableWriter, which will hold an exclusive
lock on table
abc until it is closed and
testing will be used as the lock
reason. This lock is both intra- and inter-process. If you have two Java
applications accessing the same table only one will succeed at one time.
WalWriter - A WAL table uses
WalWriter to enable concurrent data ingestion,
data modification, and schema changes, as the table is not locked.
TableWriterAPI - Both WAL and Non-WAL tables can use
TableWriterAPI. It is
an interface implemented by both writers.
#Create a new row
Although this operation semantically looks like a new object creation, the row instance is actually being re-used under the hood. A Timestamp is necessary to determine a partition for the new row. Its value has to be either increment or stay the same as the last row. When the table is not partitioned and does not have a designated timestamp column, the timestamp value can be omitted.
There are put* methods for every supported data type. Columns are updated by an index as opposed to by name.
Column update order is not important and updates can be sparse. All unset columns will default to NULL values.
#Append a row
Following method call:
Appended rows are not visible to readers until they are committed. An unneeded row can also be canceled if required.
A pending row is automatically cancelled when
writer.newRow() is called.
Consider the following scenario:
newRow() call would cancel all the updates to the row since the last
To make changes visible to readers, writer has to commit.
this job. Unlike traditional SQL databases, the size of the transaction does not
matter. You can commit anything between 1 and 1 trillion rows. We also spent
considerable effort to ensure
commit() is lightweight. You can drip one row at
a time in applications that require such behaviour.
#Writing columns in blocks
QuestDB supports writing blocks of columnar data at once via the use of the
TableBlockWriter instance is obtained from a
TableWriter and can then be used to write in memory frames of columnar data. A
frame of columnar data is just a piece of contiguous memory with each column
value stored in it one after another. The
TableBlockWriter will allow any
number of such frames of columnar data to be written with an invocation of the
appendPageFrameColumn method, before the block is either committed or
cancelled (rolled back). Use of the
TableBlockWriter requires that all columns
have the same number of rows written to them and within each column the frames
need to be added in append order.
PageFrame instance can optionally be used as a convenient interface to hold
the columnar frames and a
PageFrameCursor instance can be used as an interface
to provide a sequence of frames to be committed. Many of QuestDB's
RecordCursorFactory implementations provide a
We provide a single API for executing all kinds of SQL queries. The example
below focuses on
SELECT and how to fetch data from a cursor.
The steps to setup CairoEngine, execution context and SqlCompiler are the same as those we explained in writing data section. We will skip them here and focus on fetching data.
You can think of
RecordCursorFactory as PreparedStatement. This is the entity
that holds SQL execution plan with all of the execution artefacts. Factories are
designed to be reused and we strongly encourage caching them. You also need to
make sure that you close factories explicitly when you no longer need them.
Failing to do so can cause memory and/or other resources leak.
This instance allows iterating over the dataset produced by SQL. Cursors are relatively short-lived and do not imply fetching all the data. Note that you have to close a cursor as soon as enough data is fetched ; the closing process can happen at any time.
Cursors are not thread safe and cannot be shared between threads.
This is cursor's data access API. Record instance is obtained from the cursor outside of the fetch loop.
Record does not hold the data. Instead, it is an API to pull data when data is needed. Record instance remains the same while cursor goes over the data, making caching of records pointless.
#InfluxDB sender library
We have a Java ILP client library to allow fast data ingestion.