CAD-SYS // ULTSQL-KERNEL v1.0.26
COORD: X: 0000 | Y: 0000
JITTER: 0.08ms
View on pub.dev GitHub Repository
VERSION 1.0.26 100% PURE DART ZERO C/C++ DEPENDENCIES

ULTSQL Documentation

Complete technical guide for ULTSQL: the converged multi-model database engine combining Relational SQL, NoSQL Dotted JSON, Redis-style KV caching, and AI Vector RAG Search on a unified slotted-page core.

1. System Architecture & Overview #

ULTSQL is an autonomous, converged database engine implemented in 100% pure Dart. Unlike traditional systems that require running separate database daemons for relational queries (PostgreSQL/SQLite), document storage (MongoDB), high-speed key-value caching (Redis), and vector embeddings (Milvus/Pinecone), ULTSQL unifies all four operational models into a single engine binary and single storage container.

💡 Pure Dart Architecture
ULTSQL compiles ahead-of-time (AOT) to standalone native binaries or runs in Flutter, Dart VM, Node.js, and web browsers with WebAssembly. It requires zero native C/C++ libraries, zero dynamic link libraries (.so/.dylib/.dll), and zero C-compiler toolchains.

Storage is structured around a slotted-page architecture with an ARIES-compliant Write-Ahead Log (WAL), CRC32 integrity checksums, bottom-up B+ Tree construction, and graph-based HNSW vector indexing.

2. Installation & SDKs #

Install ULTSQL in your language environment or install the standalone CLI globally.

# Add to Dart / Flutter pubspec.yaml
dart pub add ultsql
# Or Flutter:
flutter pub add ultsql

3. Quickstart Guide #

Initialize an embedded database instance, create a table with primary key constraints, insert records, and query with high-speed interpreted execution.

import 'package:ultsql/ultsql.dart';

void main() async {
  // 1. Open persistent disk database (or ':memory:' for RAM)
  final db = Database('./my_database');
  await db.init();
  final it = Interpreter(db);

  // 2. Create relational schema
  await it.executeScript('''
    CREATE TABLE users (
      id INT PRIMARY KEY,
      name TEXT,
      email TEXT,
      created_at TEXT
    );
  ''');

  // 3. Insert records
  await it.executeScript('''
    INSERT INTO users VALUES (1, 'Alice Smith', 'alice@ultsql.io', '2026-10-01');
    INSERT INTO users VALUES (2, 'Bob Jones', 'bob@ultsql.io', '2026-10-02');
  ''');

  // 4. Query with standard SQL
  final res = await it.executeScript("SELECT name, email FROM users WHERE id = 1;");
  for (final row in res.rows) {
    print('User: ${row[0]} <${row[1]}>');
  }

  // 5. Clean shutdown (flushes cache and closes WAL)
  await db.close();
}

4. Storage Modes #

ULTSQL offers four switchable persistence profiles depending on performance and durability requirements:

Storage Mode Initialization Throughput (Measured) Crash Safety Recommended Use Case
⚡ In-Memory Database(':memory:') ~140k–200k rows/s Ephemeral (RAM) Automated unit tests, fast ephemeral caches, session states
💾 Durable Disk Database('./path') ~670k–718k rows/s ACID WAL + CRC32 Production transactional OLTP, embedded applications, microservices
🔄 Hybrid Snapshot Database('./path', syncWal: false) ~1.2M rows/s Periodic Snapshots Time-series ingestion, sensory IoT streams, high-volume event logs
⚡ Public Batch API insertBatchRecordsSync() ~2.08M rows/s Slotted-Page Direct Packing ETL bulk loading, initial data migrations, parquet conversions

5. SQL & Dialect Reference #

ULTSQL supports an ANSI-SQL 92/99 dialect with modern PostgreSQL-style extensions including JSON operators, vector types, and procedural scripting.

Data Types

SQL Type Dart Representation Storage Footprint Description
INT / INTEGER / BIGINTDbInt (int)8 bytes64-bit signed integer
DOUBLE / FLOAT / REALDbDouble (double)8 bytes64-bit IEEE floating point
TEXT / VARCHARDbText (String)Variable (UTF-8)Arbitrary-length text string
BOOLEAN / BOOLDbBool (bool)1 byteTrue or false boolean
BLOBDbBlob (Uint8List)VariableRaw binary byte array
JSONDbJson (Map/List)Variable (Structured)Dotted schema-less JSON object
VECTORDbVector (List<double>)4 bytes/dimensionFixed-dimension float32 embedding vector
UUIDDbUuid (String)16 bytes packedStandard RFC 4122 UUID
DATETIME / TIMESTAMPDbDateTime (DateTime)8 bytes (epoch ms)ISO 8601 UTC timestamp
DECIMAL(p, s)DbDecimalVariableExact fixed-precision decimal arithmetic

DDL (Data Definition Language)

SQL DDL
-- Create table with constraints
CREATE TABLE orders (
  order_id INT PRIMARY KEY,
  customer_id INT NOT NULL,
  amount DOUBLE DEFAULT 0.0,
  status TEXT,
  metadata JSON,
  embedding VECTOR,
  placed_at DATETIME
);

-- Modify schema
ALTER TABLE orders ADD COLUMN discount DOUBLE;
DROP TABLE IF EXISTS old_orders;

-- Inspect database metadata
SELECT * FROM information_schema.tables;
SELECT * FROM information_schema.columns WHERE table_name = 'orders';

DML & Query Capabilities

SQL DML
-- Multi-row batch insert
INSERT INTO orders VALUES 
  (101, 1, 149.99, 'completed', '{"tier": "gold"}', '[0.1, 0.2, 0.3]', '2026-10-01T12:00:00Z'),
  (102, 2, 89.50, 'pending', '{"tier": "silver"}', '[0.4, 0.5, 0.6]', '2026-10-02T14:30:00Z');

-- UPSERT / REPLACE
REPLACE INTO orders VALUES (101, 1, 139.99, 'refunded', '{"tier": "gold"}', '[0.1, 0.2, 0.3]', '2026-10-01T12:00:00Z');

-- Aggregations with GROUP BY and HAVING
SELECT 
  status, 
  COUNT(*) AS total_orders, 
  AVG(amount) AS avg_amount,
  SUM(amount) AS gross_revenue
FROM orders
WHERE amount > 50.0
GROUP BY status
HAVING COUNT(*) >= 1
ORDER BY gross_revenue DESC
LIMIT 10 OFFSET 0;

-- Query Planner Execution Plan
EXPLAIN SELECT * FROM orders WHERE order_id = 101;

6. Prepared Statements & Batch API #

Prepared statements compile the SQL query string into an execution plan once, allowing parameterized parameters to be executed repeatedly without re-parsing overhead.

Dart API
final stmt = db.prepare('INSERT INTO metrics VALUES (?, ?, ?);');

// 1. Single parameterized execution (sub-5µs point write)
stmt.executeSync([DbInt(101), DbText('cpu_load'), DbDouble(0.42)]);

// 2. High-speed synchronized batch insertion (~670k rows/sec)
final batch = List<List<DbValue>>.generate(50000, (i) {
  return [DbInt(i), DbText('metric_$i'), DbDouble(i * 0.1)];
});
stmt.executeBatchSync(batch);

// 3. Query parameterized prepared statement
final queryStmt = db.prepare('SELECT val FROM metrics WHERE tag = ?;');
final result = queryStmt.executeSync([DbText('cpu_load')]);
print('Metric value: ${result.rows.first.first}');

7. B+ Tree Indexes & Bottom-Up Construction #

ULTSQL builds standard B+ Tree indexes on disk with bottom-up bulk index construction. Instead of performing N random top-down tree insertions, creating an index on existing tables sorts leaf keys sequentially and constructs upper index layers in a single cache-coherent pass.

SQL Indexing
-- Standard B+ Tree Index on single column
CREATE INDEX idx_orders_customer ON orders(customer_id);

-- Compound / Multi-column Index
CREATE INDEX idx_orders_status_amount ON orders(status, amount);

-- Unique Constraint Index
CREATE UNIQUE INDEX idx_users_email ON users(email);

-- Drop Index
DROP INDEX idx_orders_customer;

8. ACID Transactions & MVCC #

Transactions in ULTSQL guarantee full ACID semantics:

  • Atomicity: Statements executed inside BEGIN...COMMIT either all commit to disk or are completely discarded on ROLLBACK.
  • Consistency: Schema constraints, unique keys, and primary key requirements are validated before slot assignment.
  • Isolation: Multi-Version Concurrency Control (MVCC) ensures read operations never block write operations and writes never block reads.
  • Durability: Changes are recorded sequentially in the Write-Ahead Log before acknowledging commit.
SQL Transaction
BEGIN TRANSACTION;
UPDATE accounts SET balance = balance - 100.0 WHERE account_id = 1;
UPDATE accounts SET balance = balance + 100.0 WHERE account_id = 2;
-- In case of failure: ROLLBACK;
COMMIT;

9. ARIES WAL, Checkpoints & Crash Recovery #

ULTSQL utilizes an ARIES-style Write-Ahead Log (`wal.log`) fortified with CRC32 checksums on every log entry. If the host machine loses power, crashes, or terminates abruptly, the engine automatically engages recovery upon startup during db.init():

  1. Analysis Phase: Identifies unwritten dirty pages in the slotted-page cache and locates active uncommitted transactions.
  2. Redo Phase: Replays committed page updates from the log sequentially to recover database state.
  3. Undo Phase: Rolls back operations from transactions that were active during the crash.
Pragma Tuning
-- Checkpoint active WAL into main database table pages
SET engine_option wal_checkpoint = true;

-- Enable automated autovacuum to reclaim fragmented slots
SET engine_option enable_autovacuum = true;

10. Converged NoSQL Document Store #

Store and query schema-less JSON documents using MongoDB-compatible collection APIs directly inside ULTSQL with zero configuration:

Dart NoSQL API
final users = db.collection('customers');

// 1. Insert single document (auto-generates 24-char unique _id)
final doc = await users.insertOne({
  'name': 'Sarah Connor',
  'email': 'sarah@resistance.org',
  'profile': {
    'score': 95000,
    'tags': ['vip', 'verified'],
    'location': {'city': 'Los Angeles', 'country': 'USA'}
  }
});

// 2. High-speed batch ingestion (~92k docs/sec)
await users.insertMany([
  {'name': 'John Connor', 'profile': {'score': 88000, 'tags': ['member']}},
  {'name': 'Kyle Reese', 'profile': {'score': 72000, 'tags': ['veteran']}},
]);

// 3. Fast point-read by _id (direct B+ Tree index lookup: ~4µs)
final found = await users.findOne({'_id': doc.id});
print('Found user: ${found?.getByPath('name')}');

// 4. Deep nested dotted-path queries with pagination & sorting
final cursor = users.find({
  'profile.score': {r'$gte': 80000},
  'profile.location.country': 'USA',
})
.sort({'profile.score': -1})
.limit(10);

final vips = await cursor.toList();

// 5. In-place atomic mutations ($set, $inc, $unset, $push)
await users.updateOne(
  filter: {'_id': doc.id},
  update: {
    r'$set': {'profile.location.city': 'San Francisco'},
    r'$inc': {'profile.score': 500},
  },
);

// 6. Delete document
await users.deleteOne({'name': 'Kyle Reese'});

Supported MongoDB Filter Operators

Filter Operator Meaning Example Syntax
$eq / $neEquality / Inequality{'role': {'$eq': 'admin'}}
$gt / $gteGreater than / Greater or equal{'profile.score': {'$gte': 90000}}
$lt / $lteLess than / Less or equal{'age': {'$lt': 30}}
$in / $ninArray membership{'role': {'$in': ['admin', 'dev']}}
$existsKey existence check{'profile.location': {'$exists': true}}
$regexRegular expression match{'email': {'$regex': r'@domain\.io$'}}

11. Redis-Style Embedded Key-Value Engine #

ULTSQL embeds an ultra-high-throughput Key-Value caching engine with in-memory hot lookups (876,000+ ops/sec), atomic WAL persistence, TTL expiration, and atomic counters:

Dart KV API
// 1. Set key with TTL expiration
await db.kv.set('session:user_101', 'token_xyz123', ttl: const Duration(hours: 2));

// 2. High-speed hot in-memory get (< 1.5 µs latency)
final token = await db.kv.get('session:user_101');

// 3. Batch set in single WAL transaction (~342k ops/sec)
await db.kv.mset({
  'config:timeout': 30,
  'config:max_conns': 100,
  'config:endpoint': 'https://api.ultsql.io',
});

// 4. Multi-key read
final configs = await db.kv.mget(['config:timeout', 'config:max_conns']);

// 5. Atomic increment and decrement counters
final views = await db.kv.incr('stats:page_views');      // 1
final views5 = await db.kv.incr('stats:page_views', 5);   // 6
final active = await db.kv.decr('stats:page_views', 2);   // 4

// 6. Pattern wildcard key scanning
final keys = await db.kv.keys(pattern: 'config:*');

// 7. Delete key
await db.kv.delete('session:user_101');

12. Cross-Model SQL ↔ NoSQL Bridge #

Query NoSQL document collections directly from SQL using the collection('name') function and dotted navigation operators (-> and ->>), or join relational SQL tables with schema-less collections in the same query:

Cross-Model SQL
-- Query document collection using relational SQL with dotted navigation
SELECT 
  _id,
  doc->>'serial' AS dev_serial,
  (doc->'telemetry'->>'battery')::INT AS battery
FROM collection('devices')
WHERE (doc->'telemetry'->>'battery')::INT > 80;

-- Join a relational table with a NoSQL document collection
SELECT 
  o.order_id,
  o.amount,
  u.doc->>'name' AS customer_name,
  u.doc->>'email' AS customer_email
FROM orders o
JOIN collection('customers') u ON o.customer_id = u._id
WHERE o.amount > 100.0;

14. Security, Encryption & Envelopes #

ULTSQL provides enterprise-grade data security at rest and in transit:

  • AES-256-GCM Authenticated Encryption: Encrypt database files, pages, and WAL records with military-grade authenticated ciphers.
  • Dual Authenticated Envelopes: Choose between speed-optimized and security-hardened cryptographic envelope modes.
  • Tamper Traps: Any unauthorized bit modification on disk triggers active tamper trap detection, raising DatabaseIntegrityException.
  • Zeroization: Cryptographic keys in memory are automatically zeroized upon database closure.
Encrypted Storage
// Open encrypted database with secret passphrase
final db = Database(
  './secure_db',
  passphrase: 'super_secret_master_key_99',
);
await db.init();

15. PostgreSQL Wire Protocol Server #

ULTSQL can run as a standalone or embedded network database daemon implementing the PostgreSQL Wire Protocol v3.0 (port 5432). Connect using existing tools like psql, DBeaver, TablePlus, or native drivers across all languages:

Dart Server Daemon
import 'package:ultsql/ultsql.dart';

void main() async {
  final db = Database('./production_db');
  await db.init();

  // Start PGWire TCP server on port 5432
  final server = PgWireServer(db, port: 5432);
  await server.start();
  print('ULTSQL PostgreSQL Wire Server listening on port 5432...');
}
Connect with psql CLI
# Connect using standard PostgreSQL CLI
psql -h localhost -p 5432 -U postgres -d ultsql

# Output:
# Welcome to psql 16.2.
# Connected to: ULTSQL Engine v1.0.26 (Pure Dart)
# ultsql=> SELECT 1 + 1;

16. Embedded REST Server #

Need HTTP access from web applications or serverless lambdas? Start the built-in REST server:

Dart REST Daemon
final rest = RestServer(db, port: 8080);
await rest.start();
print('REST API active on http://localhost:8080');
Method Route Payload Description
POST/api/query{"sql": "SELECT * FROM users"}Executes SQL query and returns JSON rows
GET/api/healthNoneReturns engine health, uptime, and page cache stats
GET/api/metricsNoneReturns Prometheus-compatible telemetry metrics

17. CLI & REPL Commands #

The ultsql CLI provides an interactive REPL with syntax highlighting, autocomplete, dot-commands, and headless CI/CD batch execution:

Terminal Commands
# 1. Launch interactive REPL
ultsql ./app.db

# 2. Interactive dot-commands inside REPL:
.tables                          # List all tables and row counts
.schema users                    # Display CREATE TABLE DDL
.collections                     # List all NoSQL document collections
.kv list config:*                # Scan KV keys matching pattern
.mode json                       # Switch output mode (box, json, csv, markdown)
.quit                            # Exit REPL

# 3. Headless CI/CD execution with JSON output piped to jq
ultsql ./app.db -m json -c "SELECT * FROM users;" | jq .

# 4. Ingest via MongoDB syntax directly in CLI
db.users.insertOne({"name": "Diana", "role": "admin"})

18. Native Multi-Language SDK Bindings #

Native client SDKs connect seamlessly to embedded or PGWire ULTSQL instances across all major languages:

import { Client } from 'pg';

const client = new Client({
  host: 'localhost',
  port: 5432,
  user: 'postgres',
  database: 'ultsql',
});

await client.connect();
const res = await client.query('SELECT * FROM users WHERE id = $1', [1]);
console.log(res.rows);
await client.end();

19. Engine Configuration & Pragmas #

Fine-tune runtime engine parameters via SQL pragmas or Dart constructors:

Option Default Values Description
page_size40961024–65536Storage page size in bytes
cache_capacity10000100–1000000Maximum pages retained in hot LRU cache
wal_synchronousNORMALOFF, NORMAL, FULLWAL fsync flush behavior on commit
enable_autovacuumtruetrue, falseAutomatic background slot compaction
auto_create_indexesfalsetrue, falseAutonomous index creation based on query telemetry

20. Errors, Troubleshooting & FAQ #

Frequently Encountered Exceptions

Exception Class Cause Resolution
DatabaseLockException Another process holds an exclusive lock on the database file. Ensure only one process opens the database in read-write mode, or use PGWire/REST server for multi-process concurrency.
DatabaseIntegrityException CRC32 checksum mismatch or disk tampering detected. Check disk hardware integrity; restore from WAL or backup snapshot.
TableNotFoundException The referenced table or collection does not exist in the catalog. Verify spelling or run CREATE TABLE / db.collection('name') before querying.

Frequently Asked Questions

Q: Does ULTSQL use SQLite underneath?
No. ULTSQL contains zero lines of SQLite code and zero native C libraries. The B+ Tree, slotted-page layout, ARIES WAL, query parser, JIT compiler, and HNSW vector index are written 100% in pure Dart.
Q: Can I use ULTSQL on iOS, Android, macOS, Linux, and Windows?
Yes! Because ULTSQL is pure Dart, it compiles natively to ARM64 and x86_64 across iOS, Android, macOS, Linux, and Windows without platform channel bridging or FFI configuration.
ESC
✓ Copied