A lexer that is one pattern and no allocations

Hand-written lexers are mostly a loop, a switch, and a lot of careful index arithmetic. Regex-based ones usually cost an allocation per token and a pattern compilation at startup. CTRE gives a third option with neither.

The shape is one alternation, one capture per token kind:

constexpr auto kToken =
    ctll::fixed_string{R"(([A-Za-z_]\w*)|(\d+)|([+*/=()\-])|(\s+))"};

Whichever capture engaged tells you what you matched, so the pattern is also the token table.

for (auto m : ctre::tokenize<kToken>(src)) {
    if (m.get<1>())      { sink(token{kind::ident,  m.get<1>().to_view()}); }
    else if (m.get<2>()) { sink(token{kind::number, m.get<2>().to_view()}); }
    else if (m.get<3>()) { sink(token{kind::op,     m.get<3>().to_view()}); }
    else if (m.get<4>()) { sink(token{kind::space,  m.get<4>().to_view()}); }
}

Running it over total = price * 3 + tax_rate gives the expected stream, with whitespace dropped by the sink rather than by the pattern:

ident   total      op      =        ident   price
op      *          number  3        ident   tax_rate

Why tokenize rather than search_all

Because of the behaviour from the previous episode. Feed the lexer ok then #bad and it produces [ok] [then] and stops. The # matches no branch, so the scan ends there.

That is the correct failure. A lexer that skipped the # would hand the parser a token stream that looks valid and describes a program the author did not write. Stopping at the first thing you cannot explain is how a lexer reports a lexical error, and tokenize does it by construction rather than by an extra check.

It also runs at compile time

Nothing about the lexer is run-time-only, so counting tokens in a constant expression works:

constexpr int counted = [] {
    int n = 0;
    lex("a = 1 + 22"sv, [&n](token t) { if (t.k != kind::space) { ++n; } });
    return n;
}();
static_assert(counted == 5);

Which is more useful than it first appears. A domain-specific string embedded in your source, a format specifier, a query template, can be lexed during compilation and rejected then, rather than being parsed on every call at run time.

The escaped dash

One detail in the pattern that cost me two compiles. The operator class is written [+*/=()\-], with the dash escaped. PCRE treats a dash at the start or the end of a character class as a literal; CTRE accepts neither position and rejects the pattern at compile time, naming the offset:

[-+*/=()]   -> problem_at_position<23>
[+*/=()-]   -> problem_at_position<30>
[+*/=()\-]  -> compiles

Which is a fair trade, and the final episode looks at what that error message is made of and what C++26 could do about how it reads.

Sources: CTRE · lexer source in the examples repo