// tag
#tokenize
2 posts tagged #tokenize.
A lexer that is one pattern and no allocations
An alternation where each branch is a capture group doubles as a token table: whichever capture engaged names the token kind. Built on tokenize, unmatched input stops the scan instead of being skipped, which is what a lexer wants. The whole thing runs at compile time as readily as at run time.
Iterating matches, and a manual that has fallen behind
search_all, split and tokenize cover scanning, separating and lexing, and the difference between the first and the last one decides which you want. The documentation still teaches ctre::range, which the compiler now reports as deprecated, so this episode is written against the header.