# `A2UIAgent.BlockLexer`
[🔗](https://github.com/a2ui-elixir/a2ui/blob/v0.1.1/umbrella/apps/a2ui_agent/lib/a2ui_agent/block_lexer.ex#L1)

Splits model output into conversational text and the raw blocks
between a format's tags.

    iex> A2UIAgent.BlockLexer.tokenize("Hi <a2ui>root = Text(\"</a2ui>\")</a2ui> bye")
    [{:text, "Hi"}, {:a2ui_raw, ~s|root = Text("</a2ui>")|, true}, {:text, "bye"}]

Inside a block, a closing tag counts only outside string literals and
single-line comments: the scanner follows strings (with backslash
escapes and triple-quoted forms) and comments until their end. The
opening tag may carry attributes, and both tags ignore case. A block
whose closing tag never comes is returned with `final?` false.

Text and blocks are trimmed, and a Markdown code fence around either
is dropped; text that is empty after that is left out.

# `raw_part`
*since 0.1.0* 

```elixir
@type raw_part() :: {:text, String.t()} | {:a2ui_raw, String.t(), boolean()}
```

A part of an unwrapped response: text or an uncompiled block.

# `clean`
*since 0.1.0* 

```elixir
@spec clean(String.t()) :: String.t()
```

Trims `text` and drops a Markdown code fence around it.

    iex> A2UIAgent.BlockLexer.clean("```express\nroot = Text(\"x\")\n```")
    ~s|root = Text("x")|

# `tokenize`
*since 0.1.0* 

```elixir
@spec tokenize(String.t(), keyword()) :: [raw_part()]
```

Tokenizes `content`. Options: `:tag` (the tag name, default `"a2ui"`),
`:string_delimiters` (default `"` and `'`) and `:comments` (the markers
that start a single-line comment, default `#`).

---

*Consult [api-reference.md](api-reference.md) for complete listing*