In QuestDB 7.0, we added a new approach to ingest data using a write-ahead log (WAL). This page introduces the properties of a WAL-enabled table (WAL table) and compares it to a non-WAL table. It also contains a summary of key components, relevant functions, as well as related SQL keywords.
A WAL table must be partitioned. It permits the following concurrent transactions:
- Data ingestion through different interfaces
- Data modifications
- Table schema changes
The following keywords enable WAL tables:
WAL table creation via
Converting an existing table to a WAL table or vice versa via
SET TYPEfollowing a database restart.
Server-wide configuration via
cairo.wal.enabled.defaultis set to
CREATE TABLESQL keyword generates WAL tables without
BYPASS WALkeyword still works as expected.
Parallel threads to apply WAL data to the table storage can be configured, see WAL table configuration for more details.
The following table highlights the main difference between a WAL and a non-WAL table:
|WAL table||Non-WAL table|
|Concurrent data ingestion via multiple interfaces||InfluxDB Line Protocol locks the table for ingestion; concurrent data ingestion via other interfaces is not allowed - |
|Unconstrained concurrent DDLs and DMLs||Concurrent DDLs and DMLs for InfluxDB Line Protocol interface only|
|Asynchronous operations - in rare situations there may be slight delays in data visibility||Synchronous operations - no-wait commits|
|Improved data freshness for ||No change|
|Some impacts on existing operations||No change|
We are working hard to reduce the below limitations.
For a WAL table, the following existing operations may have different behaviors from a non-WAL table:
- No row count returned
- No support for
A WAL table uses the following components to manage concurrent commit requests:
WAL: acts as a dedicated API for each ingestion interface. When data is ingested via multiple interfaces, dedicated
WALsensure that the table is not locked by one interface only.
Sequencer: centrally manages transactions, providing a single source of truth. The sequencer generates unique
txnnumbers as transaction identifiers and keeps a log that tracks their allocation, preventing duplicates. This log is called
TransactionLogand is stored in a meta file called
_txnlog. See root directory for more information.
WAL apply job: collects the commit requests based on the unique
txnnumbers and sends them to the
TableWriterto be committed.
TableWriter: updates the database and resolves any out-of-order data writes.
The following table metadata functions are useful for checking WAL table settings: