Structured log search — Splunk, Graylog, Papertrail, Sumo Logic

Why this page exists

The four hosted log stores in this library used to expose exactly one way to search: query(searchString), where the caller wrote the search string by hand. That contradicts the library's stated principle — the public API never requires you to write a query language yourself — and it puts every escaping decision on the caller, in four languages that escape differently.

Each store now has a compiler: a findAll-shaped spec in, the exact search text out, with every caller-supplied value escaped or refused. query() stays where it is, as the deliberate escape hatch for what the compilers will not emit.

Store	Language	Compiler module	Store methods
Splunk	SPL (piped)	`src/nosql/splunk/spl.ts`	`compileSearch()`, `search()`
Graylog	Lucene filter + Views JSON	`src/nosql/graylog/lucene.ts`	`compileSearch()`, `search()`, `compileAggregation()`, `aggregate()`
Papertrail	word-boolean expression	`src/nosql/papertrail/search-expression.ts`	`compileSearch()`, `search()`
Sumo Logic	Sumo search (piped)	`src/nosql/sumologic/sumo-query.ts`	`compileSearch()`, `search()`

Every compiler is compile-only: it holds no connection and performs no I/O, so compileSearch() works without connecting and is the thing to reach for in a test or a log line. This mirrors N1qlCompiler (src/nosql/couchbase/n1ql.ts), which the stores use the same way.

They share one thing, and it is the right one: the ORM's predicate surface. Every compiler folds WhereOptions through normalizeWhere() from src/graph/types.ts into a GraphPredicate tree, exactly as the N1QL, CQL and graph dialects do, and then renders that tree in its own language. Re-parsing WhereOptions per store would be four drifting copies of the ~80 Op.* operators.

Beyond that, src/nosql/shared/log-search.ts holds about a hundred lines: the LogSearchCapabilityError type, integer validation for the values that are written into a search as bare digits, Date → epoch conversion, and sort-term normalization. There is deliberately no shared escaper and no shared search spec.

A shared spec would be a lie, because the same concept lives in a different place in each language:

	Splunk	Graylog	Papertrail	Sumo Logic
Filter	search command	the whole language	the whole language	scope + `where`
Time range	`earliest=`/`latest=` in the language	REST parameter (3 different endpoints)	`min_time`/`max_time` parameters	property of the search job
Limit	`\| head` command	`limit` parameter	`limit` parameter	`\| limit` operator
Ordering	`\| sort` command	single `sort=field:dir` parameter	does not exist (always newest-first)	`\| sort by` operator
Aggregation	`\| stats`	a separate JSON API (Views/pivots)	does not exist	`\| count by` etc.

So "limit" is a language construct in two of them and an HTTP parameter in the other two; "sort" is absent from one entirely; "aggregate" is a pipeline stage in two, a different API in one, and nothing at all in the fourth. Forcing one spec over that would make three-quarters of its fields conditionally meaningless, and the failure would be silent — a limit that a store quietly ignores is worse than one it refuses.

One tree, four renderers, and three possible results: exact text, a refusal
of a value, or a refusal of a feature. There is no fourth "best effort" path.

One question, four stores

"Requests that failed with a 5xx on any web* host in the last 15 minutes,
first 100." The filter is the same WhereOptions each time; everything around
it is where the four stores differ. These are the actual compiled outputs:

import { Op } from 'ts-prorm-orm';
import { SplunkStore } from 'ts-prorm-orm/nosql';

const where = { status: { [Op.gte]: 500 }, host: { [Op.startsWith]: 'web' } };

new SplunkStore({ token }).compileSearch({
  index: 'web',
  earliest: '-15m',
  where,
  pipeline: [{ command: 'head', count: 100 }],
});
// { spl: 'search index="web" earliest=-15m (status>=500 AND host="web*") | head 100',
//   earliest: '-15m' }

import { Op } from 'ts-prorm-orm';
import { GraylogStore } from 'ts-prorm-orm/nosql';

const where = { status: { [Op.gte]: 500 }, host: { [Op.startsWith]: 'web' } };

new GraylogStore({ baseURL }).compileSearch({
  where,
  timeRange: { type: 'relative', range: 900 },
  limit: 100,
});
// { endpoint: '/api/search/universal/relative',
//   query: '(status:>=500 AND host:web*)',
//   params: { query: '(status:>=500 AND host:web*)', range: '900', limit: '100' } }

import { Op } from 'ts-prorm-orm';
import { SumoLogicStore } from 'ts-prorm-orm/nosql';

const where = { status: { [Op.gte]: 500 }, host: { [Op.startsWith]: 'web' } };

new SumoLogicStore({}).compileSearch({
  sourceCategory: 'prod/web',
  from: '-15m',
  pipeline: [
    { operator: 'json', field: '_raw', paths: ['status', 'host'] },
    { operator: 'where', where },
    { operator: 'limit', count: 100 },
  ],
});
// { query: '_sourceCategory="prod/web" | json field=_raw "status", "host"
//           | where (status >= 500 and host matches /^web/) | limit 100',
//   timeZone: 'UTC', from: '-15m' }

import { Op } from 'ts-prorm-orm';
import { PapertrailStore } from 'ts-prorm-orm/nosql';

// `status` is not something Papertrail indexes, so the same where is refused:
new PapertrailStore({ token }).compileSearch({ where: { status: { [Op.gte]: 500 } } });
// LogSearchCapabilityError: "filter on "status"" is not supported by the papertrail
// search language. Papertrail indexes syslog metadata and message text, not structured
// fields. The only filters it has are program, host, severity, facility. …

// The nearest honest question uses its metadata and severity instead:
new PapertrailStore({ token }).compileSearch({
  where: { host: { [Op.startsWith]: 'web' }, severity: { [Op.in]: ['err', 'crit'] } },
  minTime: new Date(Date.now() - 15 * 60 * 1000),
  limit: 100,
});
// { query: '(host:web* (severity:err OR severity:crit))',
//   params: { q: '…same…', min_time: '1790459719', limit: '100' } }

Note where the 15 minutes went: into the SPL text, into a Graylog URL
parameter, onto the Sumo search job, and into a Papertrail min_time.
Op.startsWith became a quoted wildcard, a bare wildcard, an anchored regex,
and a trailing *.

The honest grouping is two and two: SPL and Sumo are piped languages where a query is a scope plus a chain of stages, so their specs have pipeline arrays and look alike in shape. Graylog and Papertrail are filter expressions where everything else is a request parameter, so their specs are flat and their compiled output is { query, params }. Even within each pair the details do not transfer — SPL's stats is not Sumo's count by, and Graylog's Lucene is nothing like Papertrail's word matching — so the shapes rhyme and the code does not.

Escaping is the whole safety story

All four searches travel to the server as text. None of them has wire-level parameter binding. There is no $1 to hide behind, so escaping correctness is the only thing between a caller's value and an altered search. This module follows the discipline the SPARQL dialect established (src/graph/sparql/terms.ts): a small, audited set of functions is allowed to turn a JavaScript value into query text, and nothing else concatenates one.

Two rules run through all four compilers:

Field names are validated, never rewritten. safeIdent-style rewriting is right for a Cypher label, where the label is the name. It is wrong here: user-id and user_id are two different fields, and quietly searching the wrong one returns a plausible answer nobody checks.

A value that cannot be escaped is refused, not mangled. Where a language defines no escape for a character — Papertrail's ", Sumo's ", SPL's * in the search command — the compiler raises a ValidationError naming the character, the reason, and the alternative construct that does work.

The wildcard problem, per language

An unescaped * in a value does not raise an error. It silently widens the search and returns plausible rows. That makes it the most important case in this whole module, and it is handled differently in each language because the languages differ:

Language	Wildcards	Literal `*` in a value	Where a deliberate wildcard comes from
SPL `search`	`` only, live inside quotes too*	refused — SPL has no escape for it	`Op.startsWith`/`endsWith`/`substring`/`like` append it after escaping
SPL `where` (eval)	none; `like()` uses SQL `%`/`_`	passes through, quoted	`Op.like` → `like()`; literal operators → `match()` over an escaped regex
Graylog Lucene	`*` and `?`	escaped (`\*`, `\?`) in phrases and in bare terms	appended after `luceneEscapeTerm()`
Papertrail	trailing `*` only	refused — no escape exists	`Op.startsWith` only; `endsWith`/`substring` are refused
Sumo scope	`*` and `?`, unquoted terms only	refused — quoting is not a reliable defence	appended after `sumoBareToken()` validation
Sumo `where`	none in the exact form	passes through, quoted	`matches /regex/` over a regex-escaped value

The recurring trick, borrowed from the N1QL compiler, is that literal-valued operators (startsWith, endsWith, substring) do not compile to a pattern with the value pasted into it. Where the language has an exact alternative — SPL's match(), Sumo's matches /…/ — they compile to that instead, over a regex-escaped value, so there is no wildcard in the output at all. Only when the language has nothing better (Graylog, Papertrail, Sumo's scope) is a wildcard emitted, and then only after the value has been escaped or validated, so the wildcard the compiler adds is provably the only live one.

Refusals

[!WARNING]
Refusals are the feature, not a gap to work around. Catching
LogSearchCapabilityError and retrying through query() with the same
value pasted in reintroduces exactly the escaping problem the compiler refused.

Two real refusals, each naming what to do instead:

import { SplunkStore, SumoLogicStore } from 'ts-prorm-orm/nosql';

new SplunkStore({ token }).compileSearch({ index: 'web', where: { user: 'a*' } });
// ValidationError: Splunk: value for "user" contains "*", which is a wildcard in the
// search command even inside double quotes, and SPL has no escape that makes it literal.
// Refusing rather than silently widening the search: "a*". If you meant a wildcard, use
// Op.startsWith / Op.endsWith / Op.substring / Op.like. If you meant a literal asterisk,
// compare it in a `{ command: "where" }` stage, where "=" is exact.

new SumoLogicStore({}).compileSearch({ sourceCategory: 'prod/web', where: { status: 500 } });
// LogSearchCapabilityError: "scope filter on "status"" is not supported by the sumologic
// search language. Only the built-in metadata fields (_sourceCategory, _source, …) exist
// before the first pipe; a parsed field does not. … Put it in a `{ operator: "where" }`
// stage instead.

When a language cannot express something, the compiler throws LogSearchCapabilityError (src/nosql/shared/log-search.ts), which names the feature and the store — the pattern set by GraphCapabilityError and N1qlCapabilityError. Emitting a best-effort approximation would be worse here than anywhere else in the repo: a search the server rejects is annoying, while a search the server accepts that means something else returns rows and is never questioned.

Splunk — SPL

import { SplunkStore } from 'ts-prorm-orm/nosql';

const store = new SplunkStore({ token: process.env.SPLUNK_TOKEN });
const compiled = store.compileSearch({
  index: 'main',
  sourcetype: 'access_combined',
  earliest: '-24h',
  where: { status: { [Op.gte]: 500 } },
  pipeline: [
    { command: 'eval', assignments: { ms: { op: '*', left: { field: 'seconds' }, right: 1000 } } },
    { command: 'where', where: { ms: { [Op.gt]: 250 } } },
    { command: 'stats', functions: [{ fn: 'count', as: 'errors' }], by: 'host' },
    { command: 'sort', by: '-errors' },
    { command: 'head', count: 10 },
    { command: 'table', fields: ['host', 'errors'] },
  ],
});

compiled.spl;
// search index="main" sourcetype="access_combined" earliest=-24h status>=500
//   | eval ms=(seconds * 1000) | where ms > 250 | stats count AS errors BY host
//   | sort -errors | head 10 | table host, errors

The two predicate languages

SPL has two places you can write a comparison and they do not mean the same thing. This is the single most important thing to know about the Splunk compiler:

	`search` command (`SplSearchSpec.where`)	`where` command (`{ command: 'where' }`)
`field=value`	value is a wildcard pattern	exact comparison
case	field values compare case-insensitively	case-sensitive
wildcards	`*` only, no single-character wildcard	none; `like()` uses `%` and `_`
regex	not available (refused, by name)	`match(field, "re")`
absent field	`NOT field=*`	`isnull(field)`

Both renderers walk the same normalized tree, so Op.startsWith works in both — it just compiles to path="/api*" in one and match(path, "^/api") in the other.

Covered

Search command: index/sourcetype/source/host constraints (single or ORed lists), bare quoted terms, the full boolean predicate (AND/OR/NOT/grouping), field comparisons (=, !=, >, >=, <, <=, IN), wildcard patterns, field=* existence tests, and earliest/latest time modifiers validated against SPL's grammar.

Pipeline: stats (with BY, aliases, and percentile functions such as perc95), eval (a typed expression AST: field refs, literals, arithmetic and . concatenation, whitelisted functions, if over a WhereOptions condition), where, sort (with an optional leading count), head (count or condition), tail, table, fields (+/-), dedup (count, sortby, keepevents, keepempty), rename.

Not covered, and why

timechart, chart, transaction, join, lookup, rex, spath, eventstats, streamstats, tstats, subsearches and macros. Each is a feature in its own right rather than a variation on the ones above — timechart needs a span/bin vocabulary, join and subsearches embed a second search whose escaping is a separate problem, rex is a regex-extraction DSL. They stay behind query().

Field names are restricted to [A-Za-z_][A-Za-z0-9_.]*. Real Splunk data does contain k8s-pod and foo{}.bar, which need single-quoting in eval and a rename in search — different rules on either side — so they are refused rather than handled correctly in one context and silently wrongly in the other. Wildcard renames (foo* AS bar*) are refused by the same rule.

Op.iLike in the search command compiles identically to Op.like, because search-command field values already compare case-insensitively. In eval context it lowers both sides.

Graylog — Lucene, plus the Views API

import { GraylogStore } from 'ts-prorm-orm/nosql';

const store = new GraylogStore({ baseURL: 'https://graylog.example.com' });
const compiled = store.compileSearch({
  where: { status: { [Op.gte]: 500 }, source: 'web01' },
  timeRange: { type: 'relative', range: 900 },
  limit: 50,
  order: '-timestamp',
  streamId: '000000000000000000000001',
});
// endpoint: '/api/search/universal/relative'
// query:    '(status:>=500 AND source:"web01")'
// params:   { query, range: '900', limit: '50', sort: 'timestamp:desc', filter: 'streams:…' }

await store.aggregate({
  where: { status: { [Op.gte]: 500 } },
  timeRange: { type: 'relative', range: 900 },
  groupBy: ['source', { field: 'status', limit: 5 }],
  series: [{ fn: 'count' }, { fn: 'percentile', field: 'took_ms', percentile: 95 }],
});

Two APIs, because Graylog has two

Graylog's query language expresses a filter and nothing else — no ordering, no limit, no time range, no aggregation. So the compiler has two halves that do not resemble each other:

compileSearch() → { endpoint, query, params } for the universal-search API. The endpoint is chosen by the time range, because Graylog splits the same search across /relative, /absolute and /keyword URLs that take different parameters.

compileViewsSearch() → the POST /api/views/search request body: pivots, row and column groups, series, sorting and limits, all as JSON. aggregate() runs it (create, then execute — two calls, because that is what the API is).

Values in the Views request are JSON fields, so JSON.stringify handles the quoting and there is nothing to escape; the one exception is the embedded query_string, which goes through the same query compiler as (1). Field names are still validated, because a pivot field is an Elasticsearch field reference and a wrong one aggregates the wrong column rather than failing.

Covered

field:value and field:"phrase", _exists_ for presence, prefix comparisons (field:>10), ranges (field:[lo TO hi]), field:(a OR b) lists, wildcards (* and ?), regex literals (field:/re/), the full boolean structure, bare terms against the default field; plus limit/offset/sort/stream filter as parameters and three time-range kinds. On the Views side: values and time row/column groupings, and the count, sum, avg, min, max, stddev, variance, card, latest, sumofsquares and percentile series, with an optional messages search type alongside the pivot.

Not covered, and why

Op.iLike is refused. In Elasticsearch, case folding is a property of the field's analyzer and is fixed at index time; there is no query-side case-insensitive operator. Lowering the pattern would match on an analyzed text field, miss on a keyword field, and never say which happened.

Multi-field ordering on the universal-search API is refused: that endpoint takes a single sort=field:dir. The Views API takes a list.

Fuzzy (~) and proximity searches, saved-search and dashboard management, and the decorator/highlight machinery are out of scope.

Escaping

luceneQuotePhrase() escapes \, ", * and ? inside "…". The last two are escaped defensively: a phrase is normally not wildcard-expanded, but Graylog versions differ on analyze_wildcard, the parser unescapes \* back to a literal, and the cost of being wrong is a silently wider search. luceneEscapeTerm() escapes every operator character and whitespace for the bare terms that wildcard queries must be. Regex patterns have only their / delimiter escaped — doubling the backslashes would turn \d into "a backslash then a d".

Papertrail — a word-boolean expression

import { PapertrailStore } from 'ts-prorm-orm/nosql';

const store = new PapertrailStore({ token: process.env.PAPERTRAIL_TOKEN });
const compiled = store.compileSearch({
  where: { program: 'nginx', severity: { [Op.in]: ['error', 'crit'] } },
  terms: 'connection refused',
  minTime: new Date('2024-01-01T00:00:00Z'),
  limit: 250,
});
// query:  '"connection refused" (program:nginx (severity:error OR severity:crit))'
// params: { q, min_time: '1704067200', limit: '250' }

Covered

Bare terms and quoted phrases, implicit AND, OR, grouping, -term exclusion, a trailing * prefix match, and the four metadata filters Papertrail indexes: program, host, severity, facility. Everything else — the time window, the page size, the pagination cursors, the system/group scope — is a request parameter, because the language has no way to say any of it.

Not covered — and this is most of the ORM's operator surface

Papertrail is the shallowest of the four and the compiler says so out loud. Each of these throws LogSearchCapabilityError naming the feature and the reason:

Refused	Reason
a filter on any other field	Papertrail indexes syslog metadata and message text, not structured fields. `user_id:42` would compile happily and search for those words.
`gt`/`gte`/`lt`/`lte`/`between`	the search has no comparison operators at all; time is the only ordered dimension, and it is a parameter
`endsWith`/`substring`	matching is word-boundary based with a trailing wildcard only — there is no leading-wildcard form
`like`/`iLike`	no LIKE-style pattern operator exists; the trailing wildcard is the whole of its pattern matching
`regexp`	the index is word-based and exposes no regex matching
`isNull`/`isNotNull`	every event carries all four metadata fields, so absence has no meaning
`NOT` over a group	Papertrail negates one term at a time with `-`; `NOT (a OR b)` has no faithful spelling. Push it down yourself: `-a -b`.

Escaping

Papertrail documents no escape character, so this compiler validates rather than escapes. A " is refused (there is nothing to turn it into), a * is refused in every literal position, metadata values must be bare tokens matching [A-Za-z0-9_.@/:+-]+, and free-text terms are quoted as phrases whenever they are not plain tokens — which also neutralizes a leading - and the words OR/AND/NOT. severity and facility values are checked against the syslog vocabularies, because those are closed sets and a typo there means "match nothing" rather than an error.

Sumo Logic — scope plus pipeline

import { SumoLogicStore } from 'ts-prorm-orm/nosql';

const store = new SumoLogicStore({ /* accessId, accessKey */ });
const compiled = store.compileSearch({
  sourceCategory: 'prod/app',
  terms: 'error',
  from: '-15m',
  pipeline: [
    { operator: 'json', field: '_raw', paths: 'status', as: 'status' },
    { operator: 'where', where: { status: { [Op.gte]: 500 } } },
    { operator: 'timeslice', interval: '5m' },
    { operator: 'aggregate', functions: { fn: 'count', as: 'errors' }, by: '_timeslice' },
    { operator: 'sort', by: '-errors' },
    { operator: 'limit', count: 10 },
  ],
});
// query: _sourceCategory="prod/app" "error" | json field=_raw "status" as status
//        | where status >= 500 | timeslice 5m | count as errors by _timeslice
//        | sort by errors desc | limit 10
// from:  '-15m'   (a property of the search job, not of the query)

Scope and `where` are two different dialects

The scope — everything before the first | — can only match the built-in metadata fields (_sourceCategory, _source, _sourceHost, _sourceName, _collector, _index, _view) and free-text keywords. A parsed field does not exist yet at that point. Writing status=500 there is not an error: Sumo reads it as a keyword search for the literal text status=500 and returns whatever lines contain it. The compiler therefore refuses a non-metadata field in the scope and names the where stage instead.

The two contexts also differ in spelling, and the compiler follows the language rather than picking one:

	scope	`where` clause
booleans	`AND` / `OR` / `NOT`, uppercase, AND implicit	`and` / `or` / `!`, lowercase
not-equal	no `!=`; spelled `NOT field=value`	`<>`
wildcards	`*`/`?` on unquoted terms	none — `matches` with a glob or an RE2 regex
comparisons	none	`=`, `<>`, `<`, `<=`, `>`, `>=`, `in`, `not in`, `isNull`

Covered

Scope metadata constraints (single or ORed lists), keyword terms, and the scope predicate; then the pipeline: where, parse (anchor and parse regex, with field= and nodrop), json (auto or explicit paths, with as and nodrop), timeslice, aggregations (count, count_distinct, sum, avg, min, max, stddev, variance, pct, first, last, each with as and by), sort by, limit, fields (include/exclude) and dedup (consecutive, count, by). The search-job window (from/to/timeZone/byReceiptTime) is returned beside the query rather than inside it, because that is where Sumo puts it.

Not covered, and why

lookup, join, transpose, outlier, save, transaction and top — each is a feature with its own argument grammar rather than a variation on the above, and several (lookup, save) touch resources outside the search. They stay behind query().

An unscoped search is refused. * would technically work, but an unscoped Sumo search reads every message in the retention window — an expensive accident, and never what a forgotten argument meant.

Escaping

Sumo documents no escape character either, so sumoQuote() refuses a " rather than emitting a literal the value could break out of. In the scope, * and ? are refused inside literals: quoting is documented as suppressing wildcard expansion, but the failure mode of being wrong is a silently wider search, so a deliberate wildcard has to come from Op.startsWith/Op.like, which validate the value as a bare token first. In a where clause the literal operators compile to matches /^regex$/ over a regex-escaped value, which is exact — that is also how Op.like keeps SQL's _ semantics, which the scope's glob cannot express.

Testing

tests/nosql/splunk-spl.test.ts, graylog-lucene.test.ts, papertrail-search.test.ts and sumologic-query.test.ts assert the exact compiled search for every construct, with toBe rather than toContain — a "contains the value" assertion passes just as happily against a compiler that widened the search or dropped a negation. Every value-carrying position is then fed hostile input — a double quote, a backslash, an asterisk, a pipe, the language's own boolean operators, a newline, a NUL — and the result is asserted to be either correctly escaped or refused with a named error. Field names, sort fields, output aliases, rename targets, stream ids, time modifiers, intervals and limits are all covered, since each of those is a position where a value reaches the search unquoted.

SplunkStore · GraylogStore · PapertrailStore · SumoLogicStore

CQL queries and Couchbase (N1QL) — the same compile-only pattern for Cassandra and Couchbase

Documentation index — including the Op.* operator surface these compilers render

Structured log search — Splunk, Graylog, Papertrail, Sumo Logic

Why this page exists#

How much these four share, and why it is so little#

One question, four stores#

Escaping is the whole safety story#

The wildcard problem, per language#

Refusals#

Splunk — SPL#

The two predicate languages#

Covered#

Not covered, and why#

Graylog — Lucene, plus the Views API#

Two APIs, because Graylog has two#

Covered#

Not covered, and why#

Escaping#

Papertrail — a word-boolean expression#

Covered#

Not covered — and this is most of the ORM's operator surface#

Escaping#

Sumo Logic — scope plus pipeline#

Scope and where are two different dialects#

Covered#

Not covered, and why#

Escaping#

Testing#

Related reading#

Why this page exists

How much these four share, and why it is so little

One question, four stores

Escaping is the whole safety story

The wildcard problem, per language

Refusals

Splunk — SPL

The two predicate languages

Covered

Not covered, and why

Graylog — Lucene, plus the Views API

Two APIs, because Graylog has two

Covered

Not covered, and why

Escaping

Papertrail — a word-boolean expression

Covered

Not covered — and this is most of the ORM's operator surface

Escaping

Sumo Logic — scope plus pipeline

Scope and `where` are two different dialects

Covered

Not covered, and why

Escaping

Testing

Related reading