Querying Data Without a Database: TONL's SQL-Like Query API

TONL ships a SQL-like query API with sub-0.1ms indexed lookups, letting you filter, aggregate, and select rows from a TONL file directly — no database server, no ORM, just a file and a query. It also delivers 32–50% fewer tokens than JSON and runs with zero runtime dependencies, making it practical for test fixtures, edge analytics, and LLM pre-filtering alike.

What Is TONL's Query API?

Most serialization formats stop at encoding and decoding. TONL goes further: tonl.dev documents a built-in query layer that lets you express filter conditions, field projections, and aggregations against an in-memory index built from any TONL document. The syntax is intentionally SQL-adjacent — developers familiar with SELECT … WHERE … or MongoDB-style query objects pick it up in minutes.

Crucially, the entire stack ships with zero runtime dependencies. There is no query engine to install separately, no connection pool to configure, and no daemon to keep alive. You load the tonl npm package, point it at a file or an in-memory buffer, and start querying.

This is a meaningful architectural difference from SQLite (requires native bindings), DuckDB (heavier binary), or even lowdb (no query language beyond JavaScript array methods). For scripts, test harnesses, and edge functions where spinning up a database is overkill, TONL's query API fills a genuine gap. See also the architecture deep-dive for how TONL's internal document model makes indexing efficient.

How Fast Are TONL Queries?

The official TONL documentation reports indexed lookups completing in under 0.1 milliseconds. That figure is for in-process, in-memory indexed access — the kind of access pattern that replaces a db.findMany({ where: { status: "active" } }) call in a test helper or a pre-processing step.

The index is built from the parsed document structure and held in memory for the lifetime of the query session. For read-heavy workloads on a static or slowly-changing file, you build the index once and re-use it across many queries. For write-heavy workloads TONL also ships CRUD with change-tracking and rollback, so the index stays consistent after mutations.

A Realistic Query Example: Filter and Aggregate

The following snippet illustrates the query pattern. Treat it as representative; consult tonl.dev for the exact current API, which is at v2.5.2 at time of writing.

import { loadTonl } from "tonl";

// Load a TONL file — could be a test fixture, a local export, or an in-memory buffer
const doc = await loadTonl("./orders.tonl");

// Filter: orders placed in the last 30 days with status "shipped"
const recentShipped = doc.query({
  from: "orders",
  where: { status: "shipped", days_ago: { lte: 30 } },
  select: ["id", "customer", "total", "shipped_at"],
});

// Aggregate: total revenue and order count grouped by region
const regionSummary = doc.aggregate({
  from: "orders",
  where: { status: "shipped" },
  groupBy: "region",
  compute: { revenue: { sum: "total" }, count: { count: "*" } },
});

console.log(recentShipped);   // plain JS array — no cursor, no streaming overhead
console.log(regionSummary);   // [{ region: "EU", revenue: 84200, count: 312 }, ...]

The result is a plain JavaScript array — no cursor to drain, no connection to close. If you need to send the filtered slice to an LLM, you can serialize it back to TONL (saving 32–50% tokens versus JSON) or keep it as a JS object for further processing.

Building an Index: One Call, Persistent for the Session

Indexed performance requires an explicit index-build step. Again, consult tonl.dev for the authoritative API; the pattern below is illustrative:

import { loadTonl } from "tonl";

const doc = await loadTonl("./events.tonl");

// Build an index on the "events" collection, keyed by "timestamp" and "user_id"
doc.buildIndex("events", ["timestamp", "user_id"]);

// Subsequent queries on these fields hit the index — sub-0.1ms per lookup
const userEvents = doc.query({
  from: "events",
  where: { user_id: "u_8821", timestamp: { gte: "2026-05-01" } },
});

The index lives in the Node.js process heap. There is no on-disk index file to manage, no WAL to rotate, and no lock contention across processes. This makes it particularly clean for serverless functions and edge runtimes where persistent storage is either unavailable or expensive.

Three Use Cases Where TONL Queries Beat a Database

1. Test Fixtures Without a Test Database

Integration tests often need realistic, queryable data — but bootstrapping a real database for every CI run is slow and brittle. A TONL fixture file checked into the repo gives you the same query expressiveness with zero infrastructure. Load it in beforeAll, query it in each test, and tear it down with garbage collection. No Docker container, no migration runner.

TONL's 2,300+ passing tests and stable API surface mean the query behavior itself is well-specified — you can rely on it as the ground truth in your test harness.

2. Pre-Filtering Data Before It Hits an LLM

Sending an entire dataset to an LLM is expensive and often counter-productive — the model's attention dilutes across irrelevant rows. A better pattern is to filter down to the relevant slice first, then serialize only that slice into the prompt.

TONL makes this a two-step pipeline: query the document to get the relevant rows, then let TONL serialize the result at 32–50% fewer tokens than JSON. The TOON vs TONL comparison covers the serialization trade-offs in detail, but for LLM pre-filtering TONL is the natural choice because the query and the serialization live in the same package.

This matters at scale. TONL's streaming mode handles 50GB+ files in under 100MB of memory, so pre-filtering is feasible even on large local exports without loading the entire file into RAM.

3. Local and Edge Analytics

Product analytics at the edge — think Vercel Edge Functions or Cloudflare Workers — can't reach a Postgres instance without significant latency and cost. A TONL file stored in KV or on-disk lets you run aggregation queries (sum, count, groupBy) in the same process that handles the request. Response times stay in single-digit milliseconds; there is no cold-start penalty for a database connection pool.

You can also pipe TONL data through the JSON tools on this site to inspect or transform it before deploying.

TONL Query API vs Spinning Up a Database

The table below is a practical decision guide, not a comprehensive benchmark. For high-write, multi-process, or persistence-critical workloads, a real database is the right tool. For the use cases above, TONL's query API removes meaningful friction.

Schema Validation on the Same Document

A practical complaint about file-based data is that nothing enforces schema consistency — a hand-edited fixture can silently introduce a null where a u32 was expected. TONL addresses this with built-in schema validation and automatic TypeScript type generation, documented at tonl.dev.

Type hints like u32, str, and bool add roughly 20 tokens to a document header but enable validation at load time and generate TypeScript interfaces automatically. Even with type hints, a typed TONL document is approximately 32% smaller than equivalent JSON. That means your test fixtures are both safer and smaller than their JSON equivalents.

The introduction to TONL covers the full feature surface including compression layers (Dictionary, Delta, RLE, Bit Packing) that push savings toward 60% for the right data shapes.

When TONL Queries Are Not the Right Tool

It is worth being direct about the limits. TONL's query API is single-process and in-memory. If you need concurrent writes from multiple processes, referential integrity across collections, or sub-second queries over billions of rows, a real database is the correct choice. The decision is not "TONL or nothing" — it is "does this workload actually need the guarantees a database provides?"

For production application data where multiple services write concurrently, use Postgres. For analytical workloads over hundreds of millions of rows, use DuckDB or a columnar store. For everything in between — test data, edge pre-processing, local analytics, LLM pipeline glue — TONL's query API removes real infrastructure overhead without meaningful trade-offs.

See the TONL architecture post for a deeper look at how the format's columnar-friendly layout makes aggregation especially efficient, and the TOON vs TONL comparison if you are deciding between the two lighter-weight formats for a given project.

Getting Started

Install the package, load a file, and run a query:

npm install tonl

If you have existing JSON data, the free converter on this site converts JSON to TONL in the browser — no data leaves your machine. The JSON tools page also has utilities for inspecting and transforming structured data before you load it into a TONL document.

For the full query syntax, schema validation reference, and streaming API, the tonl.dev documentation and GitHub repository are the authoritative sources.

Frequently Asked Questions