Module

std::toml::token

Records

record TomlSpan

Byte span into the original TOML source text.

Spans are half-open byte ranges: start is inclusive and end is exclusive. They intentionally use byte offsets so diagnostics can point into UTF-8 input without depending on display width or terminal rendering.

Example

let span: TomlSpan = TomlSpan::new(4, 12);
Members
start: u64

Inclusive byte offset where the span begins.

end: u64

Exclusive byte offset where the span ends.

static new(start: u64, end: u64): TomlSpan

Creates a new source span from byte offsets.

The constructor does not validate ordering. Lexer and parser code should pass stable offsets from the original source cursor.

Arguments

  • start: inclusive byte offset.
  • end: exclusive byte offset.

Example

let empty: TomlSpan = TomlSpan::new(0, 0);
let key: TomlSpan = TomlSpan::new(0, 7);
record TomlToken

Owned TOML token with original source span.

Tokens own their lexeme text. This keeps parser cursors valid after lexer helper strings go out of scope and makes snapshots deterministic.

Example

let token: TomlToken = TomlToken::new(
  TomlTokenKind::IDENTIFIER,
  "package",
  TomlSpan::new(0, 7),
);
Members
kind: TomlTokenKind

Token category.

lexeme: String

Owned token text after lexer normalization.

span: TomlSpan

Byte span in the original input.

static create(kind: TomlTokenKind, lexeme: String, span: TomlSpan): TomlToken

Creates a token by cloning an owned String value.

Use this overload when lexer code built a String while decoding escapes. The caller keeps ownership of the original string.

Example

let text: String = String::create("value");
let token: TomlToken = TomlToken::create(TomlTokenKind::STRING, text, TomlSpan::new(1, 8));
static create(kind: TomlTokenKind, lexeme: String, span: TomlSpan): TomlToken

Creates a token by cloning an owned String value.

Use this overload when lexer code built a String while decoding escapes. The caller keeps ownership of the original string.

Example

let text: String = String::create("value");
let token: TomlToken = TomlToken::create(TomlTokenKind::STRING, text, TomlSpan::new(1, 8));
static new(kind: TomlTokenKind, lexeme: str, span: TomlSpan): TomlToken

Creates a token by copying a borrowed string slice into owned storage.

Arguments

  • kind: token category.
  • lexeme: borrowed token text to copy.
  • span: source byte span.

Example

let token: TomlToken = TomlToken::new(TomlTokenKind::BOOLEAN, "true", TomlSpan::new(0, 4));

Enums

enum TomlTokenKind

Token categories produced by the TOML lexer.

These are syntactic categories. Semantic scalar classification happens later when parser code builds TomlValue records.

Members
LEFT_BRACKET
RIGHT_BRACKET
LEFT_BRACE
RIGHT_BRACE
DOT
EQUALS
COMMA
NEWLINE
IDENTIFIER
STRING
INTEGER
FLOAT
DATE_TIME
BOOLEAN
EOF