Time-Aware Data in TONL: @now, @today and Temporal Queries

TONL temporal queries let you filter time-stamped datasets using built-in primitives like @now and @today — no date math, no external libraries. Combined with TONL's SQL-like query API (indexed lookups under 0.1ms, 32–50% fewer tokens than JSON, zero runtime dependencies), they make time-aware data retrieval concise, correct, and LLM-friendly.

What Is TONL and Why Does Its Query API Matter?

TONL (Token-Optimized Notation Language) is a text-first, LLM-friendly serialization format. According to tonl.dev, the current release (v2.5.2) delivers 32–50% fewer tokens than JSON and up to roughly 60% with optional compression layers. Even adding type hints — u32, str, bool — which unlock schema validation and auto TypeScript generation, keeps the payload approximately 32% smaller than the JSON equivalent.

What sets TONL apart from simpler compact formats is its built-in query API. Rather than parsing a document and filtering it in application code, you express the predicate directly in the TONL query language. The runtime resolves it using indexed lookups that stay under 0.1ms, without any external database or query engine. The library ships with 2,300+ tests passing and zero runtime dependencies, making it suitable for both edge deployments and server-side pipelines.

For a broader comparison of TONL versus the simpler TOON format, see TOON vs TONL, or read the introduction to TONL for a feature overview.

What Are Temporal Primitives in TONL?

TONL ships two temporal primitives, documented at tonl.dev and the tonl-dev/tonl GitHub repository:

• @today — resolves to midnight of the current calendar day. Use for date-level comparisons: "created today," "due today," "expires on or before today."
• @now — resolves to the current timestamp with sub-second precision. Use for time-level comparisons: "last updated within the past hour," "sessions active right now."

Both primitives are evaluated at query execution time by TONL's temporal evaluator. They integrate directly with the standard query predicate syntax, so you write a filter expression the same way you would compare against any literal value — the runtime substitutes the resolved timestamp transparently.

This matters for LLM pipelines because time-aware filtering often happens on the application side before data reaches the prompt. Every row you can exclude before serialization is a row whose tokens you never pay. TONL lets you push that filter into the data layer rather than the application layer.

Temporal Query Example: @today vs Manual Date Math

Consider a dataset of support tickets, each with a created_at timestamp. You want only the tickets opened today.

Here is the raw TONL dataset:

tickets[5]{id:u32, subject:str, status:str, created_at:str}:
  1, "Login broken",       open,    2026-06-03T08:14:00Z
  2, "Export fails",       open,    2026-06-03T09:47:00Z
  3, "Slow dashboard",     pending, 2026-06-02T16:30:00Z
  4, "Password reset bug", closed,  2026-06-01T11:05:00Z
  5, "Missing invoice",    open,    2026-06-03T10:22:00Z

With TONL temporal queries you filter directly in the query expression (conceptual syntax per tonl.dev):

// TONL temporal query — filter in the data layer, no date math in app code
SELECT id, subject, status
FROM tickets
WHERE created_at >= @today

Result: records 1, 2, and 5. Only those three rows are serialized and sent to the LLM.

Now contrast that with the manual approach you would use without temporal primitives:

// Manual date math in JavaScript — application-layer filtering
const todayStart = new Date();
todayStart.setHours(0, 0, 0, 0);
const todayISO = todayStart.toISOString();

const todayTickets = tickets.filter(
  (t) => t.created_at >= todayISO
);

// Then serialize the filtered array to your prompt format

Both approaches produce the same filtered rows. The difference is where the logic lives: TONL keeps it in a single declarative expression evaluated by the data layer; the manual approach scatters date-handling code across the application. In a team setting, the declarative form is auditable, testable, and far less likely to introduce timezone bugs.

Manual Date Math vs TONL Temporal Primitives: A Comparison

The table below compares the two approaches across the dimensions that matter in production.

How Temporal Queries Fit Into TONL's Broader Feature Set

Temporal primitives are one component of TONL's query layer. The same query API also handles standard field predicates, aggregation (count, sum, average, group-by), and schema-validated projections. Because TONL uses indexed lookups, all of these operations remain under 0.1ms per query regardless of whether the predicate is temporal or structural.

The streaming engine is complementary: TONL can process files larger than 50GB in under 100MB of memory. In a time-series pipeline — logs, telemetry, financial ticks — you can stream the full dataset through a temporal filter and emit only the matching window to the prompt. This combination of streaming and temporal filtering is what makes TONL practical for large-scale LLM data pipelines rather than just small in-memory datasets.

For a deep dive into how TONL is structured internally, see the architecture of TONL. For production-use patterns with large datasets, see optimizing RAG pipelines with TOON.

When Should You Use TONL Temporal Queries?

Temporal queries are most valuable when one or more of these conditions holds:

• Your dataset is time-stamped and you need to show the LLM only a recent window — today's events, the last 24 hours, the current week.
• The full dataset is too large to fit in context and time-based scoping is the natural way to reduce it.
• Your pipeline runs on a schedule (hourly, daily) and the "today" boundary shifts automatically without code changes.
• You want to avoid shipping date utilities as runtime dependencies to edge or serverless environments.

Temporal queries are less valuable when the dataset is small and fully fits in context, or when the time boundary is complex enough (business-day calendars, fiscal quarters, multi-timezone adjustments) that the TONL primitives do not cover the full logic and you need custom code regardless.

If you are evaluating TONL's token efficiency before committing to the format, use the free converter at json2toon.co to paste a sample dataset and compare token counts directly.

Frequently Asked Questions