A2UIAgent.BlockLexer (A2UIAgent v0.1.1)

Splits model output into conversational text and the raw blocks between a format's tags.

iex> A2UIAgent.BlockLexer.tokenize("Hi <a2ui>root = Text(\"</a2ui>\")</a2ui> bye")
[{:text, "Hi"}, {:a2ui_raw, ~s|root = Text("</a2ui>")|, true}, {:text, "bye"}]

Inside a block, a closing tag counts only outside string literals and single-line comments: the scanner follows strings (with backslash escapes and triple-quoted forms) and comments until their end. The opening tag may carry attributes, and both tags ignore case. A block whose closing tag never comes is returned with final? false.

Text and blocks are trimmed, and a Markdown code fence around either is dropped; text that is empty after that is left out.

Summary

Types

raw_part()

A part of an unwrapped response: text or an uncompiled block.

Functions

clean(text)

Trims text and drops a Markdown code fence around it.

tokenize(content, options \\ [])

Tokenizes content. Options: :tag (the tag name, default "a2ui"), :string_delimiters (default " and ') and :comments (the markers that start a single-line comment, default #).

Types

raw_part()

(since 0.1.0)

@type raw_part() :: {:text, String.t()} | {:a2ui_raw, String.t(), boolean()}

A part of an unwrapped response: text or an uncompiled block.

Functions

clean(text)

(since 0.1.0)

@spec clean(String.t()) :: String.t()

Trims text and drops a Markdown code fence around it.

iex> A2UIAgent.BlockLexer.clean("```express\nroot = Text(\"x\")\n```")
~s|root = Text("x")|

tokenize(content, options \\ [])

(since 0.1.0)

@spec tokenize(String.t(), keyword()) :: [raw_part()]

Tokenizes content. Options: :tag (the tag name, default "a2ui"), :string_delimiters (default " and ') and :comments (the markers that start a single-line comment, default #).