CSS Tokens

Raw CSS tokens, which are created as a first step when parsing a CSS string. More...

Classes
struct	donner::css::Token::Ident
	<ident-token>, which represents a CSS identifier, which is an unquoted string. More...
struct	donner::css::Token::Function
	<function-token>, which indicates the start of a function call. More...
struct	donner::css::Token::AtKeyword
	<at-keyword-token>, representing @ followed by an identifier. More...
struct	donner::css::Token::Hash
	<hash-token>, representing a CSS identifier that starts with a #. More...
struct	donner::css::Token::String
	<string-token>, which represents a quoted string, either with double or single quotes ("foo" or 'foo'). More...
struct	donner::css::Token::BadString
	<bad-string-token>, which is generated when a string contains an unescaped newline. More...
struct	donner::css::Token::Url
	<url-token>, which represents a url() function. More...
struct	donner::css::Token::BadUrl
	<bad-url-token>, which represents an invalid url() function. More...
struct	donner::css::Token::Delim
	<delim-token>, which contains a single character. More...
struct	donner::css::Token::Number
	<number-token>, which represents a number, either integer or floating point. More...
struct	donner::css::Token::Percentage
	<percentage-token>, which represents a percentage such as '50'. More...
struct	donner::css::Token::Dimension
	<dimension-token>, which represents a dimension such as '50px'. More...
struct	donner::css::Token::Whitespace
	<whitespace-token>, which contains one or more whitespace characters in the source. More...
struct	donner::css::Token::CDO
	<CDO-token>, which represents <! More...
struct	donner::css::Token::CDC
	<CDC-token>, which represents --> in the source. More...
struct	donner::css::Token::Colon
	<colon-token>, which represents ':' in the source. More...
struct	donner::css::Token::Semicolon
	<semicolon-token>, which represents ';' in the source. More...
struct	donner::css::Token::Comma
	<comma-token>, which represents ',' in the source. More...
struct	donner::css::Token::SquareBracket
	<[-token>, which represents [ in the source. More...
struct	donner::css::Token::Parenthesis
	<(-token>, which represents ( in the source. More...
struct	donner::css::Token::CurlyBracket
	<{-token>, which represents { in the source. More...
struct	donner::css::Token::CloseSquareBracket
	<]-token>, which represents ] in the source. More...
struct	donner::css::Token::CloseParenthesis
	<)-token>, which represents ) in the source. More...
struct	donner::css::Token::CloseCurlyBracket
	<}-token>, which represents } in the source. More...
struct	donner::css::Token::ErrorToken
	Special error token, used to mark named parsing errors. More...
struct	donner::css::Token::EofToken
	<EOF-token>, which marks the end of the input stream and is always output at the end of a token list. More...

Detailed Description

Raw CSS tokens, which are created as a first step when parsing a CSS string.

See https://www.w3.org/TR/css-syntax-3/#tokenization for more details.

Tokens are created by donner::css::parser::details::Tokenizer, which is automatically used within the CSS parser suite.

A full list of tokens can be found in the TokenValue variant list.

---

Class Documentation

donner::css::Token::Ident

struct donner::css::Token::Ident

<ident-token>, which represents a CSS identifier, which is an unquoted string.

For example, div, color, red are all identifiers.

See https://www.w3.org/TR/css-syntax-3/#ident-token-diagram for the railroad diagram.

• Identifiers can begin with a-zA-Z_, non-ascii, -, or --.
• After the first character, they contain a-zA-Z0-9_-, non-ascii, or escape sequences such as \u1234.

Collaboration diagram for donner::css::Token::Ident:

Public Member Functions
	Ident (RcString value)
	Create an identifier token.
bool	operator== (const Ident &other) const =default
	Equality operator.
Public Attributes
RcString	value
	Identifier value.
Friends
std::ostream &	operator<< (std::ostream &os, const Ident &obj)
	Ostream output operator.

Constructor & Destructor Documentation

Ident()

donner::css::Token::Ident::Ident ( RcString value )

inlineexplicit

Create an identifier token.

Parameters

value

Identifier value.

Friends And Related Symbol Documentation

operator<<

std::ostream & operator<< ( std::ostream & os, const Ident & obj )

friend

Ostream output operator.

Example output:

Ident(red)

Parameters

os	Output stream.
obj	Object to output.

donner::css::Token::Function

struct donner::css::Token::Function

<function-token>, which indicates the start of a function call.

For rgb(255, 0, 0), the function token would be created for the rgb( part, and the name would be rgb.

Note that for url, an unquoted url(foo) is parsed as a Url, while a quoted value like url("foo") is parsed as a Function token.

Collaboration diagram for donner::css::Token::Function:

Public Member Functions
	Function (RcString name)
	Create a Function with the given name.
bool	operator== (const Function &other) const =default
	Equality operator.
Public Attributes
RcString	name
	Function name, not including the '(' character.
Friends
std::ostream &	operator<< (std::ostream &os, const Function &obj)
	Ostream output operator.

Constructor & Destructor Documentation

Function()

donner::css::Token::Function::Function ( RcString name )

inlineexplicit

Create a Function with the given name.

Parameters

name	Function name, not including the '(' character.

Friends And Related Symbol Documentation

operator<<

std::ostream & operator<< ( std::ostream & os, const Function & obj )

friend

Ostream output operator.

Example output:

Function(url)

Parameters

os	Output stream.
obj	Object to output.

donner::css::Token::AtKeyword

struct donner::css::Token::AtKeyword

<at-keyword-token>, representing @ followed by an identifier.

For example, @media and @import.

See https://www.w3.org/TR/css-syntax-3/#at-keyword-token-diagram for the railroad diagram.

Collaboration diagram for donner::css::Token::AtKeyword:

Public Member Functions
	AtKeyword (RcString value)
	Create an AtKeyword with the given value.
bool	operator== (const AtKeyword &other) const =default
	Equality operator.
Public Attributes
RcString	value
	The value, not including the '@' character.
Friends
std::ostream &	operator<< (std::ostream &os, const AtKeyword &obj)
	Ostream output operator.

Constructor & Destructor Documentation

AtKeyword()

donner::css::Token::AtKeyword::AtKeyword ( RcString value )

inlineexplicit

Create an AtKeyword with the given value.

Parameters

value

Value, not including the '@' character.

Friends And Related Symbol Documentation

operator<<

std::ostream & operator<< ( std::ostream & os, const AtKeyword & obj )

friend

Ostream output operator.

Example output:

AtKeyword(media)

Parameters

os	Output stream.
obj	Object to output.

donner::css::Token::Hash

struct donner::css::Token::Hash

<hash-token>, representing a CSS identifier that starts with a #.

For example, #foo and #fff.

See https://www.w3.org/TR/css-syntax-3/#hash-token-diagram for the railroad diagram.

Collaboration diagram for donner::css::Token::Hash:

Public Types
enum class	Type {   Unrestricted ,   Id }
	Hash type, which is set to Type::Unrestricted by default, and Type::Id if the hash would be a valid identifier per the rules on Token::Ident. More...
Public Member Functions
	Hash (Type type, RcString name)
	Create a Hash token.
bool	operator== (const Hash &other) const =default
	Equality operator.
Public Attributes
Type	type
	Hash type, defaults to unrestricted if not otherwise set.
RcString	name
	The name, not including the '#' character.
Friends
std::ostream &	operator<< (std::ostream &os, const Hash &obj)
	Ostream output operator.

Member Enumeration Documentation

Type

enum class donner::css::Token::Hash::Type

strong

Hash type, which is set to Type::Unrestricted by default, and Type::Id if the hash would be a valid identifier per the rules on Token::Ident.

Enumerator
Unrestricted	The default type.
Id	The hash value is a valid identifier, starting with a-zA-Z_, non-ascii, -, or --.

Constructor & Destructor Documentation

Hash()

donner::css::Token::Hash::Hash ( Type type, RcString name )

inline

Create a Hash token.

Parameters

type	Hash type, which should be Type::Unrestricted by default.
name	Hash name, not including the '#' character.

Friends And Related Symbol Documentation

operator<<

std::ostream & operator<< ( std::ostream & os, const Hash & obj )

friend

Ostream output operator.

Example output:

Hash(unrestricted: foo)

Parameters

os	Output stream.
obj	Object to output.

donner::css::Token::String

struct donner::css::Token::String

<string-token>, which represents a quoted string, either with double or single quotes ("foo" or 'foo').

Collaboration diagram for donner::css::Token::String:

Public Member Functions
	String (RcString value)
	Create a String token.
bool	operator== (const String &other) const =default
	Equality operator.
Public Attributes
RcString	value
	String value, not including quotes.
Friends
std::ostream &	operator<< (std::ostream &os, const String &obj)
	Ostream output operator.

Constructor & Destructor Documentation

String()

donner::css::Token::String::String ( RcString value )

inlineexplicit

Create a String token.

Parameters

value

String value, not including quotes.

Friends And Related Symbol Documentation

operator<<

std::ostream & operator<< ( std::ostream & os, const String & obj )

friend

Ostream output operator.

Example output:

String("foo")

Parameters

os	Output stream.
obj	Object to output.

donner::css::Token::BadString

struct donner::css::Token::BadString

<bad-string-token>, which is generated when a string contains an unescaped newline.

For example, tokenizing 'foo\\nbar' would result in a BadString token with value foo.

Collaboration diagram for donner::css::Token::BadString:

Public Member Functions
	BadString (RcString value)
	Create a BadString token.
bool	operator== (const BadString &other) const =default
	Equality operator.
Public Attributes
RcString	value
	Valid part of the string before parsing hit an unescaped newline.
Friends
std::ostream &	operator<< (std::ostream &os, const BadString &obj)
	Ostream output operator.

Constructor & Destructor Documentation

BadString()

donner::css::Token::BadString::BadString ( RcString value )

inlineexplicit

Create a BadString token.

Parameters

value

The part of the string parsed before hitting the unescaped newline, not including the newline.

Friends And Related Symbol Documentation

operator<<

std::ostream & operator<< ( std::ostream & os, const BadString & obj )

friend

Ostream output operator.

Example output:

BadString("foo")

Parameters

os	Output stream.
obj	Object to output.

donner::css::Token::Url

struct donner::css::Token::Url

<url-token>, which represents a url() function.

For example, url(foo.png), where the Url value is 'foo.png'.

NOTE: This expects that the contents of the url() are not quoted, if they are quoted this will tokenize as a Function instead.

Collaboration diagram for donner::css::Token::Url:

Public Member Functions
	Url (RcString value)
	Create a Url token.
bool	operator== (const Url &other) const =default
	Equality operator.
Public Attributes
RcString	value
	Url value, not including the surrounding 'url(' and ')' characters.
Friends
std::ostream &	operator<< (std::ostream &os, const Url &obj)
	Ostream output operator.

Constructor & Destructor Documentation

Url()

donner::css::Token::Url::Url ( RcString value )

inlineexplicit

Create a Url token.

Parameters

value

Url value, not including the surrounding 'url(' and ')' characters.

Friends And Related Symbol Documentation

operator<<

std::ostream & operator<< ( std::ostream & os, const Url & obj )

friend

Ostream output operator.

Example output:

Url(foo.png)

Parameters

os	Output stream.
obj	Object to output.

donner::css::Token::BadUrl

struct donner::css::Token::BadUrl

<bad-url-token>, which represents an invalid url() function.

For example, url(whitespace in middle), url(() or url(not\u001Fprintable) will all result in a BadUrl token.

Bad URLs may be created when:

• There is whitespace in the middle of the url, such as url(foo bar.png).
• There is an extra '(' in the URL, such as url(foo(bar.png).
• There is a non-printable character in the URL, such as url(foo\u001Fbar.png).

NOTE: This expects that the contents of the url() are not quoted, if they are quoted this will tokenize as a Function instead.

Public Member Functions
bool	operator== (const BadUrl &) const
	Equality operator.
Friends
std::ostream &	operator<< (std::ostream &os, const BadUrl &)
	Ostream output operator, which always outputs "BadUrl".

donner::css::Token::Delim

struct donner::css::Token::Delim

<delim-token>, which contains a single character.

These are typically symbol characters, such as +, -, *, '.', '!', etc, since other characters would create a Token::Ident.

Delim tokens include the ! on !important rules and combinators in selector lists, such as > in parent > child.

Public Member Functions
	Delim (char value)
	Create a Delim token.
bool	operator== (const Delim &other) const =default
	Equality operator.
Public Attributes
char	value
	The delim character value.
Friends
std::ostream &	operator<< (std::ostream &os, const Delim &obj)
	Ostream output operator.

Constructor & Destructor Documentation

Delim()

donner::css::Token::Delim::Delim ( char value )

inlineexplicit

Create a Delim token.

Parameters

value

The delim character value.

Friends And Related Symbol Documentation

operator<<

std::ostream & operator<< ( std::ostream & os, const Delim & obj )

friend

Ostream output operator.

Example output:

Delim(+)

Parameters

os	Output stream.
obj	Object to output.

donner::css::Token::Number

struct donner::css::Token::Number

<number-token>, which represents a number, either integer or floating point.

The token captures both the parsed number and the original string value.

Collaboration diagram for donner::css::Token::Number:

Public Member Functions
	Number (double value, RcString valueString, NumberType type)
	Create a Number token.
bool	operator== (const Number &other) const
	Equality operator.
Public Attributes
double	value
	The parsed number value.
RcString	valueString
	The original string value.
NumberType	type
	The type of number, either integer or floating point.
Friends
std::ostream &	operator<< (std::ostream &os, const Number &obj)
	Ostream output operator.

Constructor & Destructor Documentation

Number()

donner::css::Token::Number::Number ( double value, RcString valueString, NumberType type )

inline

Create a Number token.

Parameters

value	The parsed number value.
valueString	The original string value.
type	The type of number, either integer or floating point.

Friends And Related Symbol Documentation

operator<<

std::ostream & operator<< ( std::ostream & os, const Number & obj )

friend

Ostream output operator.

Example output:

Number(1.5, str='1.5', number)

Parameters

os	Output stream.
obj	Object to output.

donner::css::Token::Percentage

struct donner::css::Token::Percentage

<percentage-token>, which represents a percentage such as '50'.

The token's value scaled so that 100 is equivalent to '100'.

Collaboration diagram for donner::css::Token::Percentage:

Public Member Functions
	Percentage (double value, RcString valueString, NumberType type)
	Create a Percentage token.
bool	operator== (const Percentage &other) const
	Equality operator.
Public Attributes
double	value
	The percentage multiplied by 100, 100% -> 100.0.
RcString	valueString
	The original string value.
NumberType	type
	The type of number, either integer or floating point.
Friends
std::ostream &	operator<< (std::ostream &os, const Percentage &obj)
	Ostream output operator.

Constructor & Destructor Documentation

Percentage()

donner::css::Token::Percentage::Percentage ( double value, RcString valueString, NumberType type )

inline

Create a Percentage token.

Parameters

value	The percentage multiplied by 100, 100% -> 100.0
valueString	The original string value.
type	The type of number, either integer or floating point.

Friends And Related Symbol Documentation

operator<<

std::ostream & operator<< ( std::ostream & os, const Percentage & obj )

friend

Ostream output operator.

Example output:

Percentage(100, str='100%', integer)

Parameters

os	Output stream.
obj	Object to output.

donner::css::Token::Dimension

struct donner::css::Token::Dimension

<dimension-token>, which represents a dimension such as '50px'.

The token contains the parsed number, parsed unit (if it is a known suffix), as well as the raw strings for both the number and unit suffix.

Collaboration diagram for donner::css::Token::Dimension:

Public Member Functions
	Dimension (double value, RcString suffixString, std::optional< Lengthd::Unit > suffixUnit, RcString valueString, NumberType type)
	Create a Dimension token.
bool	operator== (const Dimension &other) const
	Equality operator.
Public Attributes
double	value
	The parsed number value.
RcString	suffixString
	Raw string of the unit suffix, e.g. 'px'.
std::optional< Lengthd::Unit >	suffixUnit
	The parsed unit of the suffix, if known. If the input string has an invalid suffix, and donner::parser::LengthParser failed to identify it, this will be std::nullopt.
RcString	valueString
	The original string of the value number.
NumberType	type
	The type of number, either integer or floating point.
Friends
std::ostream &	operator<< (std::ostream &os, const Dimension &obj)
	Ostream output operator.

Constructor & Destructor Documentation

Dimension()

donner::css::Token::Dimension::Dimension ( double value, RcString suffixString, std::optional< Lengthd::Unit > suffixUnit, RcString valueString, NumberType type )

inline

Create a Dimension token.

Parameters

value	The parsed number value.
suffixString	The unit suffix, such as 'px'.
suffixUnit	The parsed unit, if it is a known suffix.
valueString	The original string value.
type	The type of number, either integer or floating point.

Friends And Related Symbol Documentation

operator<<

std::ostream & operator<< ( std::ostream & os, const Dimension & obj )

friend

Ostream output operator.

Example output:

Dimension(1.5px, str='1.5px', number)

Parameters

os	Output stream.
obj	Object to output.

donner::css::Token::Whitespace

struct donner::css::Token::Whitespace

<whitespace-token>, which contains one or more whitespace characters in the source.

These include ' ', '\t', '\n', '\r\n', '\r', and '\f'.

See https://www.w3.org/TR/css-syntax-3/#whitespace for the railroad diagram.

Collaboration diagram for donner::css::Token::Whitespace:

Public Member Functions
	Whitespace (RcString value)
	Create a Whitespace token.
bool	operator== (const Whitespace &other) const =default
	Equality operator.
Public Attributes
RcString	value
	The whitespace characters.
Friends
std::ostream &	operator<< (std::ostream &os, const Whitespace &obj)
	Ostream output operator.

Constructor & Destructor Documentation

Whitespace()

donner::css::Token::Whitespace::Whitespace ( RcString value )

inlineexplicit

Create a Whitespace token.

Parameters

value

The whitespace characters.

Friends And Related Symbol Documentation

operator<<

std::ostream & operator<< ( std::ostream & os, const Whitespace & obj )

friend

Ostream output operator.

Example output:

Whitespace(' ', len=1)

Parameters

os	Output stream.
obj	Object to output.

donner::css::Token::CDO

struct donner::css::Token::CDO

<CDO-token>, which represents <!

-- in the source.

Public Member Functions
bool	operator== (const CDO &) const
	Equality operator.
Friends
std::ostream &	operator<< (std::ostream &os, const CDO &)
	Ostream output operator, which outputs CDO.

donner::css::Token::CDC

struct donner::css::Token::CDC

<CDC-token>, which represents --> in the source.

Public Member Functions
bool	operator== (const CDC &) const
	Equality operator.
Friends
std::ostream &	operator<< (std::ostream &os, const CDC &)
	Ostream output operator, which outputs CDC.

donner::css::Token::Colon

struct donner::css::Token::Colon

<colon-token>, which represents ':' in the source.

Public Member Functions
bool	operator== (const Colon &) const
	Equality operator.
Friends
std::ostream &	operator<< (std::ostream &os, const Colon &)
	Ostream output operator, which outputs Colon.

donner::css::Token::Semicolon

struct donner::css::Token::Semicolon

<semicolon-token>, which represents ';' in the source.

Public Member Functions
bool	operator== (const Semicolon &) const
	Equality operator.
Friends
std::ostream &	operator<< (std::ostream &os, const Semicolon &)
	Ostream output operator, which outputs Semicolon.

donner::css::Token::Comma

struct donner::css::Token::Comma

<comma-token>, which represents ',' in the source.

Public Member Functions
bool	operator== (const Comma &) const
	Equality operator.
Friends
std::ostream &	operator<< (std::ostream &os, const Comma &)
	Ostream output operator, which outputs Comma.

donner::css::Token::SquareBracket

struct donner::css::Token::SquareBracket

<[-token>, which represents [ in the source.

Public Member Functions
bool	operator== (const SquareBracket &) const
	Equality operator.
Friends
std::ostream &	operator<< (std::ostream &os, const SquareBracket &)
	Ostream output operator, which outputs SquareBracket.

donner::css::Token::Parenthesis

struct donner::css::Token::Parenthesis

<(-token>, which represents ( in the source.

Public Member Functions
bool	operator== (const Parenthesis &) const
	Equality operator.
Friends
std::ostream &	operator<< (std::ostream &os, const Parenthesis &)
	Ostream output operator, which outputs Parenthesis.

donner::css::Token::CurlyBracket

struct donner::css::Token::CurlyBracket

<{-token>, which represents { in the source.

Public Member Functions
bool	operator== (const CurlyBracket &) const
	Equality operator.
Friends
std::ostream &	operator<< (std::ostream &os, const CurlyBracket &)
	Ostream output operator, which outputs CurlyBracket.

donner::css::Token::CloseSquareBracket

struct donner::css::Token::CloseSquareBracket

<]-token>, which represents ] in the source.

Public Member Functions
bool	operator== (const CloseSquareBracket &) const
	Equality operator.
Friends
std::ostream &	operator<< (std::ostream &os, const CloseSquareBracket &)
	Ostream output operator, which outputs CloseSquareBracket.

donner::css::Token::CloseParenthesis

struct donner::css::Token::CloseParenthesis

<)-token>, which represents ) in the source.

Public Member Functions
bool	operator== (const CloseParenthesis &) const
	Equality operator.
Friends
std::ostream &	operator<< (std::ostream &os, const CloseParenthesis &)
	Ostream output operator, which outputs CloseParenthesis.

donner::css::Token::CloseCurlyBracket

struct donner::css::Token::CloseCurlyBracket

<}-token>, which represents } in the source.

Public Member Functions
bool	operator== (const CloseCurlyBracket &) const
	Equality operator.
Friends
std::ostream &	operator<< (std::ostream &os, const CloseCurlyBracket &)
	Ostream output operator, which outputs CloseCurlyBracket.

donner::css::Token::ErrorToken

struct donner::css::Token::ErrorToken

Special error token, used to mark named parsing errors.

Public Types
enum class	Type {   EofInString ,   EofInComment ,   EofInUrl }
	Error type. More...
Public Member Functions
	ErrorToken (Type type)
	Construct a new Error Token object.
bool	operator== (const ErrorToken &other) const =default
	Equality operator.
Public Attributes
Type	type
	Error type.
Friends
std::ostream &	operator<< (std::ostream &os, const ErrorToken &obj)
	Ostream output operator.

Member Enumeration Documentation

Type

enum class donner::css::Token::ErrorToken::Type

strong

Error type.

Enumerator
EofInString	An EOF was hit when parsing a quoted string, such as "string<eof>.
EofInComment	An EOF was hit when parsing a comment, such as /* comment<eof>.
EofInUrl	An EOF was hit when parsing a URL, such as url(<eof>.

Constructor & Destructor Documentation

ErrorToken()

donner::css::Token::ErrorToken::ErrorToken ( Type type )

inlineexplicit

Construct a new Error Token object.

Parameters

type	Error type.

Friends And Related Symbol Documentation

operator<<

std::ostream & operator<< ( std::ostream & os, const ErrorToken & obj )

friend

Ostream output operator.

Example output:

ErrorToken(EofInString)

Parameters

os	Output stream.
obj	Object to output.

donner::css::Token::EofToken

struct donner::css::Token::EofToken

<EOF-token>, which marks the end of the input stream and is always output at the end of a token list.

Public Member Functions
bool	operator== (const EofToken &) const
	Equality operator.
Friends
std::ostream &	operator<< (std::ostream &os, const EofToken &)
	Ostream output operator, which outputs EofToken.