Write-Ahead Log (WAL)
In QuestDB 7.0, we added a new approach to ingest data using a write-ahead log (WAL). This page introduces the properties of a WAL-enabled table (WAL table) and compares it to a non-WAL table. It also contains a summary of key components, relevant functions, as well as related SQL keywords.
A WAL table must be partitioned. It permits the following concurrent transactions:
- Data ingestion through different interfaces
- Data modifications
- Table schema changes
The following configurations and keywords enable and create WAL tables:
WAL table creation is enabled by the following methods:
Table-wide configuration via
WALgenerates a WAL table.
BYPASS WALgenerates a non-WAL table.
Server-wide configuration via
cairo.wal.enabled.defaultis set to
CREATE TABLESQL keyword generates WAL tables without
BYPASS WALkeyword still works as expected.
Parallel threads to apply WAL data to the table storage can be configured, see WAL table configuration for more details.
The following table highlights the main difference between a WAL and a non-WAL table:
|WAL table||Non-WAL table|
|Concurrent data ingestion via multiple interfaces||ILP locks the table for ingestion; concurrent data ingestion via other interfaces is not allowed - |
|Unconstrained concurrent DDLs and DMLs||Concurrent DDLs and DMLs for ILP interface only|
|Asynchronous operations - in rare situations there may be slight delays in data visibility||Synchronous operations - no-wait commits|
|Improved data freshness for ||No change|
|Some impacts on existing operations||No change|
We are working hard to reduce the below limitations.
For a WAL table, the following existing operations may have different behaviors from a non-WAL table:
- No row count returned
- No support for
ADD COLUMNcan only add 1 column per statement
Non-structural operations may fail silently. These are partition-level and configuration operations:
A WAL table uses the following components to manage concurrent commit requests:
WAL: acts as a dedicated API for each ingestion interface. When data is ingested via multiple interfaces, dedicated
WALsensure that the table is not locked by one interface only.
Sequencer: centrally manages transactions, providing a single source of truth. The sequencer generates unique
txnnumbers as transaction identifiers and keeps a log that tracks their allocation, preventing duplicates. This log is called
TransactionLogand is stored in a meta file called
_txnlog. See root directory for more information.
WAL apply job: collects the commit requests based on the unique
txnnumbers and sends them to the
TableWriterto be committed.
TableWriter: updates the database and resolves any out-of-order data writes.
#Checking WAL configurations
The following table metadata functions are useful for checking WAL table settings:
tables()returns general table metadata, including whether a table is a WAL table or not.
wal_tables()returns WAL-table status.
- ALTER TABLE RESUME WAL restarts suspended transactions.