donner::css::Token Struct Reference

A CSS token, which are created as a first step when parsing a CSS string. More...

#include "donner/css/Token.h"

Classes
struct	AtKeyword
	<at-keyword-token>, representing @ followed by an identifier. More...
struct	BadString
	<bad-string-token>, which is generated when a string contains an unescaped newline. More...
struct	BadUrl
	<bad-url-token>, which represents an invalid url() function. More...
struct	CDC
	<CDC-token>, which represents --> in the source. More...
struct	CDO
	<CDO-token>, which represents <!-- in the source. More...
struct	CloseCurlyBracket
	<}-token>, which represents } in the source. More...
struct	CloseParenthesis
	<)-token>, which represents ) in the source. More...
struct	CloseSquareBracket
	<]-token>, which represents ] in the source. More...
struct	Colon
	<colon-token>, which represents ':' in the source. More...
struct	Comma
	<comma-token>, which represents ',' in the source. More...
struct	CurlyBracket
	<{-token>, which represents { in the source. More...
struct	Delim
	<delim-token>, which contains a single character. More...
struct	Dimension
	<dimension-token>, which represents a dimension such as '50px'. More...
struct	EofToken
	<EOF-token>, which marks the end of the input stream and is always output at the end of a token list. More...
struct	ErrorToken
	Special error token, used to mark named parsing errors. More...
struct	Function
	<function-token>, which indicates the start of a function call. More...
struct	Hash
	<hash-token>, representing a CSS identifier that starts with a #. More...
struct	Ident
	<ident-token>, which represents a CSS identifier, which is an unquoted string. More...
struct	Number
	<number-token>, which represents a number, either integer or floating point. More...
struct	Parenthesis
	<(-token>, which represents ( in the source. More...
struct	Percentage
	<percentage-token>, which represents a percentage such as '50'. More...
struct	Semicolon
	<semicolon-token>, which represents ';' in the source. More...
struct	SquareBracket
	<[-token>, which represents [ in the source. More...
struct	String
	<string-token>, which represents a quoted string, either with double or single quotes ("foo" or ‘'foo’`). More...
struct	Url
	<url-token>, which represents a url() function. More...
struct	Whitespace
	<whitespace-token>, which contains one or more whitespace characters in the source. More...

Public Types
using	TokenValue
	Variant containing all supported token types.

Public Member Functions
	Token (TokenValue &&value, size_t offset)
	Construct a new Token object, taking ownership of a TokenValue, at a given offset within the source string.
TokenIndex	tokenIndex () const
	Returns the token type.
parser::FileOffset	offset () const
	Returns the offset within the source string where this token starts.
template<typename T >
bool	is () const
	Check if the token is of the given type.
template<typename T >
T &	get () &
	Get the token value as a reference.
template<typename T >
const T &	get () const &
	Get the token value as a const-reference.
template<typename T >
T &&	get () &&
	Get the token value as an rvalue-reference for move semantics.
template<typename T >
T *	tryGet ()
	Get the token value as a pointer, or nullptr if the token is not of the given type.
template<typename T >
const T *	tryGet () const
	Get the token value as a const-pointer, or nullptr if the token is not of the given type.
template<typename Visitor >
auto	visit (Visitor &&visitor) const
	Visit the token value using a visitor, which internally uses std::visit.
bool	isParseError () const
	Returns true if this token is a type of parse error.
bool	operator== (const Token &other) const
	Equality operator.

Static Public Member Functions
template<typename T , TokenIndex index = 0>
static constexpr TokenIndex	indexOf ()
	At compile-time, return the TokenIndex of a given token type, which can be used to uniquely identify a token.

Friends
std::ostream &	operator<< (std::ostream &os, const Token &token)
	Ostream output operator.

Detailed Description

A CSS token, which are created as a first step when parsing a CSS string.

See https://www.w3.org/TR/css-syntax-3/#tokenization for more details.

Member Typedef Documentation

TokenValue

using donner::css::Token::TokenValue

Initial value:

 
      std::variant<Ident, Function, AtKeyword, Hash, String, BadString, Url, BadUrl, Delim, Number,
                   Percentage, Dimension, Whitespace, CDO, CDC, Colon, Semicolon, Comma,
                   SquareBracket, Parenthesis, CurlyBracket, CloseSquareBracket, CloseParenthesis,
                   CloseCurlyBracket, ErrorToken, EofToken>

Variant containing all supported token types.

Constructor & Destructor Documentation

Token()

donner::css::Token::Token ( TokenValue && value, size_t offset )

inline

Construct a new Token object, taking ownership of a TokenValue, at a given offset within the source string.

This allows creating a Token from any CSS Tokens documented above.

For example, to create a token of a given type:

auto token = Token(Token::String("test"), 0);

The tokenizer automatically creates tokens using this API.

Parameters

value	Token value.
offset	Offset within the source string where this token starts.

Member Function Documentation

get() [1/3]

template<typename T >

T & donner::css::Token::get ( ) &

inline

Get the token value as a reference.

Example usage:

Token::Ident& ident = token.get<Token::Ident>();

Template Parameters

T	Token type, which must be one of the values in the TokenValue variant.

Precondition

The token must be of the given type, i.e. is<T>() must be true.

get() [2/3]

template<typename T >

T && donner::css::Token::get ( ) &&

inline

Get the token value as an rvalue-reference for move semantics.

Example usage:

Token::Ident ident = std::move(token.get<Token::Ident>());

Template Parameters

T	Token type, which must be one of the values in the TokenValue variant.

Precondition

The token must be of the given type, i.e. is<T>() must be true.

get() [3/3]

template<typename T >

const T & donner::css::Token::get ( ) const &

inline

Get the token value as a const-reference.

Example usage:

const RcString value = token.get<Token::Ident>().value;

Template Parameters

T	Token type, which must be one of the values in the TokenValue variant.

Precondition

The token must be of the given type, i.e. is<T>() must be true.

indexOf()

template<typename T , TokenIndex index = 0>

static constexpr TokenIndex donner::css::Token::indexOf ( )

inlinestaticconstexpr

At compile-time, return the TokenIndex of a given token type, which can be used to uniquely identify a token.

Example usage:

const TokenIndex index = Token::indexOf<Token::Ident>();

Template Parameters

T	Token class, such as Token::Ident or Token::Function. For a full list of valid values, see the members of the TokenValue variant.
index	Internally used to track the current index, do not manually specify.

Returns

constexpr TokenIndex The index of the given token type.

is()

template<typename T >

bool donner::css::Token::is ( ) const

inline

Check if the token is of the given type.

Example usage:

if (token.is<Token::Ident>()) {
  const Token::Ident& ident = token.get<Token::Ident>();
  // ...
}

Template Parameters

T	Token type, which must be one of the values in the TokenValue variant.

tokenIndex()

TokenIndex donner::css::Token::tokenIndex ( ) const

inline

Returns the token type.

For example, to compare this token type against a known type:

if (token.tokenIndex() == Token::indexOf<Token::Ident>()) {
 // ...
}

tryGet() [1/2]

template<typename T >

T * donner::css::Token::tryGet ( )

inline

Get the token value as a pointer, or nullptr if the token is not of the given type.

This is a convenience method for is<T>() ? &get<T>() : nullptr.

Example usage:

if (Token::Ident* ident = token.tryGet<Token::Ident>()) {
 // ident is mutable here.
 // ...
}

Template Parameters

T	Token type, which must be one of the values in the TokenValue variant.

tryGet() [2/2]

template<typename T >

const T * donner::css::Token::tryGet ( ) const

inline

Get the token value as a const-pointer, or nullptr if the token is not of the given type.

This is a convenience method for is<T>() ? &get<T>() : nullptr.

Example usage:

if (const Token::Ident* ident = token.tryGet<Token::Ident>()) {
 // ...
}

Template Parameters

T	Token type, which must be one of the values in the TokenValue variant.

visit()

template<typename Visitor >

auto donner::css::Token::visit ( Visitor && visitor ) const

inline

Visit the token value using a visitor, which internally uses std::visit.

For example:

token.visit([](auto&& t) {
  using Type = std::remove_cvref_t<decltype(t)>;
 
 if constexpr (std::is_same_v<Type, Token::Ident>) {
   // t is an Ident
 } else if constexpr (std::is_same_v<Type, Token::Function>) {
   // ...
 }
});

Template Parameters

Visitor

Visitor type, which should be a callable type that accepts a reference to the token, for example: void(auto&&).

Parameters

visitor

Visitor callback, which will be invoked with the token as a parameter. For example: [](auto&& t) { ... }.

---

The documentation for this struct was generated from the following file:

• donner/css/Token.h