Core Storage Engine

DonaDB

DonaDB is TruthLinked Labs' embedded LSM storage engine for validator state, chain indexing, historical reads, and durable block execution. It combines WAL-backed writes, in-memory memtables, immutable SST files, snapshots, point-in-time values, range scans, and filesystem checkpoints.

donadb 0.1.1 WAL-backed Block-height history SST compaction Validator storage

01 Overview

DonaDB is designed for blockchain state, not generic web app persistence. Every write can be attached to a block height. Each key has a latest value for normal reads and historical values for replay, verification, indexers, and recovery flows. The database stores byte keys and byte values directly; application layers decide how account records, blocks, cells, receipts, or index entries are encoded.

Latest state

get() returns the current live value for a key inside a domain.

Historical state

get_at() returns the newest value at or before a given block height.

Domain separation

DomainId separates account state, block state, indexes, cells, and other storage families.

Crash recovery

Startup loads the latest valid snapshot, then replays durable WAL entries.

02 Install

Add DonaDB to a Rust crate that participates in the TruthLinked stack:

[dependencies]
donadb = "0.1.1"

For workspace development, use the local crate path:

[dependencies]
donadb = { path = "../donadb" }

The primary imports are:

use donadb::{DonaDB, DonaDbConfig, DomainId, BlockHeight};

03 Open A Database

Open DonaDB with a durable data directory. The directory becomes the database root for the WAL, snapshot, and SST state.

use donadb::{DonaDB, DonaDbConfig};
use std::path::PathBuf;

let db = DonaDB::open(DonaDbConfig {
    data_dir: PathBuf::from("./chain-state"),
    ..DonaDbConfig::default()
})?;

For shared access across services or threads, wrap the database handle in Arc. Cloning the handle is cheap because mutable storage state is shared internally.

use std::sync::Arc;

let db = Arc::new(DonaDB::open(DonaDbConfig::default())?);
let worker_db = Arc::clone(&db);

04 Storage Model

block-scoped writes
->
active memtable
WAL writer
->
durable append log
memtable flush
->
immutable SST files
SST pressure
->
L0 / L1 / L2 compaction
restart
->
snapshot + WAL replay
ConceptMeaning
DomainIdA u32 namespace. Use separate domains for logically independent storage families.
BlockHeightA u64 height attached to state mutations for historical reads.
Head valueThe latest live value for a domain/key pair.
Version valueA historical value keyed by domain, user key, and block height.
TombstoneA deletion marker. Deletes hide current values while preserving historical lookup semantics.
SSTAn immutable sorted-string-table file used after memtables are flushed.

05 Write State

Single Write

Use set() when one key is changing at one block height.

const ACCOUNTS: u32 = 1;
let height: u64 = 100;

db.set(ACCOUNTS, b"alice", b"balance:1000", height);
db.finalize_block(height)?;

Block-Scoped Batch

Use begin_batch() when a block, transaction, or state transition touches multiple keys. The batch commits through the same WAL-backed path.

let height = 101;
let mut batch = db.begin_batch(height, b"block-101");

batch.put(ACCOUNTS, b"alice", b"balance:900");
batch.put(ACCOUNTS, b"bob", b"balance:100");
batch.put(2, b"tx:101:0", b"alice->bob:100");

batch.commit()?;
db.finalize_block(height)?;

Delete

Use del() when the current value should disappear while older heights remain queryable.

db.del(ACCOUNTS, b"alice-temp-lock", 102);
db.finalize_block(102)?;

06 Read State

Latest Value

if let Some(value) = db.get(ACCOUNTS, b"alice")? {
    println!("alice latest = {:?}", value);
}

Historical Value

get_at() returns the newest historical value whose height is less than or equal to the requested height. This is the important primitive for replay, audit, and deterministic chain queries.

let before_transfer = db.get_at(ACCOUNTS, b"alice", 100)?;
let after_transfer = db.get_at(ACCOUNTS, b"alice", 101)?;

07 Scan State

DonaDB supports ordered scans over live head keys inside a domain.

MethodUse
scan(domain, start, end)Range scan live keys in [start, end).
scan_prefix_domain(domain, prefix)Read all live keys matching a prefix in one domain.
scan_all(domain)Read every live key in a domain.
let accounts = db.scan_prefix_domain(ACCOUNTS, b"acct:")?;

for (key, value) in accounts {
    println!("{:?} -> {:?}", key, value);
}

Use domains and key prefixes deliberately. Good key design keeps scans narrow and makes indexes predictable.

08 Durability

DonaDB writes immediately into memory and sends durable operations to a dedicated WAL writer. Calling finalize_block() waits for submitted writes to reach disk. On restart, DonaDB loads the latest valid snapshot and then replays WAL entries, so the database can recover committed state after process death.

WAL

Write-ahead log used as the durable append path for state changes.

Memtable

In-memory write layer used for fast latest reads before SST flush.

Snapshot

Compact representation used to speed recovery and rotate durable state.

SST

Immutable sorted files searched after active memory layers.

09 Checkpoints

checkpoint() creates a standalone filesystem copy of the database state. The destination can be opened as a DonaDB database using DonaDbConfig.data_dir.

use std::path::PathBuf;

db.finalize_block(150)?;
db.checkpoint("./checkpoint-150")?;

let checkpoint_db = DonaDB::open(DonaDbConfig {
    data_dir: PathBuf::from("./checkpoint-150"),
    ..DonaDbConfig::default()
})?;

let alice = checkpoint_db.get(ACCOUNTS, b"alice")?;

Checkpoints are the storage primitive to use for validator backup, replay staging, indexer handoff, and safe state transfer.

10 Metrics

metrics() returns a point-in-time storage snapshot useful for operators and health pages. The fields describe memory layers, disk layers, WAL growth, compaction state, and read amplification.

let m = db.metrics();

println!("active entries: {}", m.active_entries);
println!("wal bytes: {}", m.wal_file_bytes);
println!("snapshot bytes: {}", m.snapshot_file_bytes);
println!("sst files: {}/{}/{}", m.sst_l0_files, m.sst_l1_files, m.sst_l2_files);
println!("read amplification estimate: {}", m.estimated_read_amplification);
MetricMeaning
active_entriesEntries currently in the active memory layer.
flushing_entriesEntries in a memory layer being flushed to SST.
compacting_entriesEntries visible while snapshot compaction is active.
wal_file_bytesCurrent durable WAL file size.
snapshot_file_bytesCurrent snapshot file size.
sst_l0_files / sst_l1_files / sst_l2_filesCurrent SST files by level.
estimated_read_amplificationMemory layers plus SST file count used as a simple read-cost signal.

11 API Reference

APIPurpose
DonaDB::open(config)Open or create a database from a durable directory.
DonaDB::open_wal(path)Open using an explicit WAL file path.
DonaDB::open_wal_with_shards(path, shard_count)Open with explicit WAL path and memory shard count.
begin_batch(height, entropy)Create a block-scoped write batch.
WriteBatch::put(domain, key, value)Add a put operation to a batch.
WriteBatch::del(domain, key)Add a delete operation to a batch.
WriteBatch::commit()Apply a bound batch through DonaDB.
set(domain, key, value, height)Write one latest and historical value.
del(domain, key, height)Delete the current value with a tombstone.
get(domain, key)Read the latest live value.
get_at(domain, key, height)Read the newest historical value at or before a height.
scan(domain, start, end)Range scan live keys in a domain.
scan_prefix_domain(domain, prefix)Prefix scan live keys in a domain.
scan_all(domain)Scan every live key in a domain.
sync()Flush queued WAL work and wait for disk sync.
finalize_block(height)Durably finalize submitted writes for a block.
checkpoint(destination)Create an independent filesystem checkpoint.
metrics()Return storage health and amplification counters.
sst_stats()Return SST file counts for L0, L1, and L2.