Core Storage Engine
DonaDB
DonaDB is TruthLinked Labs' embedded LSM storage engine for validator state, chain indexing, historical reads, and durable block execution. It combines WAL-backed writes, in-memory memtables, immutable SST files, snapshots, point-in-time values, range scans, and filesystem checkpoints.
01 Overview
DonaDB is designed for blockchain state, not generic web app persistence. Every write can be attached to a block height. Each key has a latest value for normal reads and historical values for replay, verification, indexers, and recovery flows. The database stores byte keys and byte values directly; application layers decide how account records, blocks, cells, receipts, or index entries are encoded.
get() returns the current live value for a key inside a domain.
get_at() returns the newest value at or before a given block height.
DomainId separates account state, block state, indexes, cells, and other storage families.
Startup loads the latest valid snapshot, then replays durable WAL entries.
02 Install
Add DonaDB to a Rust crate that participates in the TruthLinked stack:
[dependencies] donadb = "0.1.1"
For workspace development, use the local crate path:
[dependencies]
donadb = { path = "../donadb" }
The primary imports are:
use donadb::{DonaDB, DonaDbConfig, DomainId, BlockHeight};
03 Open A Database
Open DonaDB with a durable data directory. The directory becomes the database root for the WAL, snapshot, and SST state.
use donadb::{DonaDB, DonaDbConfig};
use std::path::PathBuf;
let db = DonaDB::open(DonaDbConfig {
data_dir: PathBuf::from("./chain-state"),
..DonaDbConfig::default()
})?;
For shared access across services or threads, wrap the database handle in Arc. Cloning the
handle is cheap because mutable storage state is shared internally.
use std::sync::Arc; let db = Arc::new(DonaDB::open(DonaDbConfig::default())?); let worker_db = Arc::clone(&db);
04 Storage Model
| Concept | Meaning |
|---|---|
| DomainId | A u32 namespace. Use separate domains for logically independent storage families. |
| BlockHeight | A u64 height attached to state mutations for historical reads. |
| Head value | The latest live value for a domain/key pair. |
| Version value | A historical value keyed by domain, user key, and block height. |
| Tombstone | A deletion marker. Deletes hide current values while preserving historical lookup semantics. |
| SST | An immutable sorted-string-table file used after memtables are flushed. |
05 Write State
Single Write
Use set() when one key is changing at one block height.
const ACCOUNTS: u32 = 1; let height: u64 = 100; db.set(ACCOUNTS, b"alice", b"balance:1000", height); db.finalize_block(height)?;
Block-Scoped Batch
Use begin_batch() when a block, transaction, or state transition touches multiple keys.
The batch commits through the same WAL-backed path.
let height = 101; let mut batch = db.begin_batch(height, b"block-101"); batch.put(ACCOUNTS, b"alice", b"balance:900"); batch.put(ACCOUNTS, b"bob", b"balance:100"); batch.put(2, b"tx:101:0", b"alice->bob:100"); batch.commit()?; db.finalize_block(height)?;
Delete
Use del() when the current value should disappear while older heights remain queryable.
db.del(ACCOUNTS, b"alice-temp-lock", 102); db.finalize_block(102)?;
06 Read State
Latest Value
if let Some(value) = db.get(ACCOUNTS, b"alice")? {
println!("alice latest = {:?}", value);
}
Historical Value
get_at() returns the newest historical value whose height is less than or equal to the
requested height. This is the important primitive for replay, audit, and deterministic chain queries.
let before_transfer = db.get_at(ACCOUNTS, b"alice", 100)?; let after_transfer = db.get_at(ACCOUNTS, b"alice", 101)?;
07 Scan State
DonaDB supports ordered scans over live head keys inside a domain.
| Method | Use |
|---|---|
| scan(domain, start, end) | Range scan live keys in [start, end). |
| scan_prefix_domain(domain, prefix) | Read all live keys matching a prefix in one domain. |
| scan_all(domain) | Read every live key in a domain. |
let accounts = db.scan_prefix_domain(ACCOUNTS, b"acct:")?;
for (key, value) in accounts {
println!("{:?} -> {:?}", key, value);
}
Use domains and key prefixes deliberately. Good key design keeps scans narrow and makes indexes predictable.
08 Durability
DonaDB writes immediately into memory and sends durable operations to a dedicated WAL writer. Calling
finalize_block() waits for submitted writes to reach disk. On restart, DonaDB loads the
latest valid snapshot and then replays WAL entries, so the database can recover committed state after
process death.
Write-ahead log used as the durable append path for state changes.
In-memory write layer used for fast latest reads before SST flush.
Compact representation used to speed recovery and rotate durable state.
Immutable sorted files searched after active memory layers.
09 Checkpoints
checkpoint() creates a standalone filesystem copy of the database state. The destination can
be opened as a DonaDB database using DonaDbConfig.data_dir.
use std::path::PathBuf;
db.finalize_block(150)?;
db.checkpoint("./checkpoint-150")?;
let checkpoint_db = DonaDB::open(DonaDbConfig {
data_dir: PathBuf::from("./checkpoint-150"),
..DonaDbConfig::default()
})?;
let alice = checkpoint_db.get(ACCOUNTS, b"alice")?;
Checkpoints are the storage primitive to use for validator backup, replay staging, indexer handoff, and safe state transfer.
10 Metrics
metrics() returns a point-in-time storage snapshot useful for operators and health pages.
The fields describe memory layers, disk layers, WAL growth, compaction state, and read amplification.
let m = db.metrics();
println!("active entries: {}", m.active_entries);
println!("wal bytes: {}", m.wal_file_bytes);
println!("snapshot bytes: {}", m.snapshot_file_bytes);
println!("sst files: {}/{}/{}", m.sst_l0_files, m.sst_l1_files, m.sst_l2_files);
println!("read amplification estimate: {}", m.estimated_read_amplification);
| Metric | Meaning |
|---|---|
| active_entries | Entries currently in the active memory layer. |
| flushing_entries | Entries in a memory layer being flushed to SST. |
| compacting_entries | Entries visible while snapshot compaction is active. |
| wal_file_bytes | Current durable WAL file size. |
| snapshot_file_bytes | Current snapshot file size. |
| sst_l0_files / sst_l1_files / sst_l2_files | Current SST files by level. |
| estimated_read_amplification | Memory layers plus SST file count used as a simple read-cost signal. |
11 API Reference
| API | Purpose |
|---|---|
| DonaDB::open(config) | Open or create a database from a durable directory. |
| DonaDB::open_wal(path) | Open using an explicit WAL file path. |
| DonaDB::open_wal_with_shards(path, shard_count) | Open with explicit WAL path and memory shard count. |
| begin_batch(height, entropy) | Create a block-scoped write batch. |
| WriteBatch::put(domain, key, value) | Add a put operation to a batch. |
| WriteBatch::del(domain, key) | Add a delete operation to a batch. |
| WriteBatch::commit() | Apply a bound batch through DonaDB. |
| set(domain, key, value, height) | Write one latest and historical value. |
| del(domain, key, height) | Delete the current value with a tombstone. |
| get(domain, key) | Read the latest live value. |
| get_at(domain, key, height) | Read the newest historical value at or before a height. |
| scan(domain, start, end) | Range scan live keys in a domain. |
| scan_prefix_domain(domain, prefix) | Prefix scan live keys in a domain. |
| scan_all(domain) | Scan every live key in a domain. |
| sync() | Flush queued WAL work and wait for disk sync. |
| finalize_block(height) | Durably finalize submitted writes for a block. |
| checkpoint(destination) | Create an independent filesystem checkpoint. |
| metrics() | Return storage health and amplification counters. |
| sst_stats() | Return SST file counts for L0, L1, and L2. |