# `OpenUI.Parser.Lexer`
[🔗](https://github.com/ideia4it/openui/blob/v0.1.1/lib/openui/parser/lexer.ex#L1)

Turns OpenUI Lang source into tokens (language.md, section 1.3).

Lexing never fails: a character that matches no rule is skipped. Tokens:

  * `{:ident, name}`: an identifier, lowercase or uppercase (the parser
    decides whether it is a component);
  * `{:state, "$name"}` and `{:at, "Name"}` for `$name` and `@Name`;
  * `{:string, value}`, or `{:string, value, :unterminated}` when the text
    ends inside it;
  * `{:number, number}`, `{:literal, true | false | nil}`;
  * `{:op, text}` for every operator and punctuation mark;
  * `:newline`.

A number with an exponent marker and no exponent digits (`1e`) is not a
usable number in the spec; here it takes the value of its mantissa.

# `token`

```elixir
@type token() ::
  {:ident, String.t()}
  | {:state, String.t()}
  | {:at, String.t()}
  | {:string, String.t()}
  | {:string, String.t(), :unterminated}
  | {:number, number()}
  | {:literal, true | false | nil}
  | {:op, String.t()}
  | :newline
```

# `tokenize`

```elixir
@spec tokenize(String.t()) :: [token()]
```

Returns the tokens of `source`.

---

*Consult [api-reference.md](api-reference.md) for complete listing*