XMLTokenizer.h File Reference

Standalone XML tokenizer for syntax highlighting and source-location-aware editing. Unlike XMLParser::Parse, this tokenizer: More...

#include <cstddef>
#include <string_view>
#include "donner/base/xml/XMLTokenType.h"

Include dependency graph for XMLTokenizer.h:

Classes
class	donner::xml::detail::XMLTokenizerImpl
	Internal tokenizer state, factored out of the template so the header doesn't carry lambdas across translation units (which would cause the goto_into_protected_scope issue with goto + lambdas in the same scope). More...

Namespaces
namespace	donner
	Top-level Donner namespace, which is split into different sub-namespaces such as donner::svg and donner::css.
namespace	donner::xml
	XML parsing and document model support, top-level objects are donner::xml::XMLParser and donner::xml::XMLDocument.

Functions
template<typename TokenSink>
void	donner::xml::Tokenize (std::string_view source, TokenSink &&sink)
	Tokenize an XML source string, emitting XMLToken values to sink.

Detailed Description

Standalone XML tokenizer for syntax highlighting and source-location-aware editing. Unlike XMLParser::Parse, this tokenizer:

• Does not build an XMLDocument or XMLNode tree.
• Error-recovers on malformed input by emitting ErrorRecovery tokens and synchronizing to the next < or >, rather than aborting with a ParseDiagnostic.
• Does not expand entities — &amp; stays as &amp; in the token stream, so byte offsets match the raw source text.

The token stream is gap-free: concatenating every token's source range reconstructs the original input.

The tokenizer is a template on the sink type so the call is zero-overhead when unused and avoids std::function's per-call indirect dispatch and heap-capture risk.