Lexical Analysis

Status: Complete (Phase 1) Crate: compiler/lexical-analysis Entry Point: pub fn tokenize(input: &str) -> Result<Vec<Token>, LexError>

Overview

The lexical analysis feature slice is responsible for converting raw Neuro source code into a stream of tokens. It implements a complete lexer with Unicode support, multiple number bases, string literals with escape sequences, and comprehensive error reporting.

Architecture

This slice follows the Vertical Slice Architecture pattern:

• Self-contained: No dependencies on other feature slices
• Infrastructure only: Depends only on shared-types for common types
• Public API: Single entry point (tokenize)
• Internal implementation: All internals are pub(crate)

Features

Token Types Supported

Keywords

func · val · mut · const · as · if · else · return · true · false · while · loop · for · in · break · continue · struct · enum · impl · trait · dyn · import · export · module · match · where · type · newtype · unsafe · move · self · Self

Type names (i32, f64, bool, string, char, …) are ordinary identifiers, not keywords: they are resolved by the type checker, which is what lets a newtype or a type alias introduce one.

String Literals and Interpolation

A string token carries a StringValue: Plain(String) for a literal without holes, Interp(Vec<InterpChunk>) when it contains at least one {expr} hole. Each chunk is either Text(String) (already unescaped) or Hole { source, span } (the raw expression text plus its location). The parser hands the chunks to expression parsing; see string interpolation for the user-facing syntax and format mini-language. An unterminated { hole is the lexer's UnterminatedInterpolation error.

A triple-quoted """…""" block string decodes to the same StringValue, so nothing downstream of the lexer distinguishes the two forms. Logos matches only the opening delimiter — it has no non-greedy repetition, so a regex ending in """ would run to the last one in the file — and a callback scans the body, strips the closing delimiter's indentation, and reuses the ordinary chunk decoder. Dedent drops characters from an indexed (offset, char) view rather than rebuilding the text, which is how holes inside a block string keep true source spans. See triple-quoted strings for the dedent rules and their errors.

Operators

• Arithmetic: +, -, *, /, %
• Compound assignment: +=, -=, *=, /=, %=
• Comparison: ==, !=, <, >, <=, >=
• Logical: &&, ||, !
• Bitwise: &, |, ^, ~, << (there is no >> token; right shift is the .shr(n) method, because >> is reserved for function composition)
• Fallible: ?? (coalesce), ? (propagate)
• Assignment: =
• Other: @ (attributes), -> (return type), => (match arm), :: (path and turbofish), .. / ..= (ranges), . (member access)

Delimiters

( ) · { } · [ ] · , · : · ;

; is tokenized only so a stray semicolon can be reported as an unexpected token; Neuro statements are newline-terminated. Newlines are themselves tokens (TokenKind::Newline), because the parser needs them to find statement boundaries.

Literals

Integers (multiple bases):

142          // Decimal20b1010      // Binary30o52        // Octal40x2A        // Hexadecimal

Floats:

13.1421.0e1032.5e-3

Strings (with escape sequences):

1"hello world"2"line 1\nline 2"3"tab\there"4"quote: \""5"unicode: \u{1F600}"6"hex: \xAB"

Booleans:

1true2false

Identifiers

• Unicode support (XID_Start + XID_Continue)
• Examples: myVar, _private, 计算, café

Comments

1// Line comment2 3/*4 * Block comment5 * Can span multiple lines6 */7 8/* Block comments nest: /* this inner one */ and the outer is still open. */

Nesting means a block comment ends only at the */ that unwinds it to depth zero, so a block already containing a comment can be commented out wholesale. Each /* therefore needs its own */; a file that ends while a comment is still open is LexError::UnterminatedBlockComment. A comment body is raw text — /* and */ inside a string or char literal within it are still counted, exactly as // already swallows a quote to end of line.

Span Tracking

Every token includes precise source location information:

1pub struct Token {2    pub kind: TokenKind,3    pub span: Span,  // start and end byte positions4}

This enables:

• Accurate error reporting
• IDE features (go-to-definition, hover)
• Debugging information in generated code

Usage

Basic Example

1use lexical_analysis::tokenize;2 3let source = r#"4    func add(a: i32, b: i32) -> i32 {5        return a + b6    }7"#;8 9let tokens = tokenize(source)?;10for token in tokens {11    println!("{:?} at {:?}", token.kind, token.span);12}

Error Handling

1use lexical_analysis::{tokenize, LexError};2 3let source = "val x = \"unterminated string";4match tokenize(source) {5    Ok(tokens) => println!("Success: {} tokens", tokens.len()),6    Err(LexError::UnterminatedString { span }) => {7        eprintln!("Error at {:?}: unterminated string", span);8    }9    Err(e) => eprintln!("Lexical error: {}", e),10}

Error Types

1pub enum LexError {2    UnexpectedChar { character: char, span: Span },3    UnterminatedString { span: Span },4    InvalidNumber { text: String, span: Span },5    InvalidEscape { escape: String, span: Span },6    InvalidCharLiteral { literal: String, span: Span },7    UnterminatedBlockComment { span: Span },8    UnterminatedInterpolation { span: Span },9}

All errors include span information for precise error reporting.

Implementation Details

Technology

• Lexer generator: logos 0.14
• Unicode support: unicode-ident for identifier validation

Performance

• Zero-copy tokenization where possible
• Lazy evaluation of token values

Testing

Test categories:

• Keywords and identifiers
• All operator types
• Number literals (all bases, floats)
• String literals and escape sequences
• Comments (line and block)
• Error cases (invalid syntax, unterminated strings, bad escapes)

Example test:

1#[test]2fn tokenize_string_with_escapes() {3    let input = r#""hello\nworld\t\u{1F600}""#;4    let tokens = tokenize(input).unwrap();5    assert_eq!(tokens.len(), 1);6    match &tokens[0].kind {7        TokenKind::String(s) => {8            assert!(s.contains('\n'));9            assert!(s.contains('\t'));10        }11        _ => panic!("Expected string token"),12    }13}

Design Decisions

Why logos?

• Performance: Generates optimized DFA-based lexer
• Simplicity: Declarative regex-based token definitions
• Maintenance: Easy to add new token types
• Error handling: Integrated error recovery

Unicode Support

Neuro embraces Unicode for identifiers to support international developers:

• Follows UAX#31 (Unicode Identifier Syntax)
• XID_Start for first character
• XID_Continue for subsequent characters

String Escape Sequences

Supports common escape sequences for developer convenience:

• \n, \r, \t - Common whitespace
• \", \\ - Quote and backslash
• \0 - Null character
• \xNN - Hex byte (2 digits)
• \u{NNNN} - Unicode codepoint (1-6 hex digits)

API Reference

Public Functions

1/// Tokenize a Neuro source file into a token stream2pub fn tokenize(input: &str) -> Result<Vec<Token>, LexError>

Public Types

1pub struct Token {2    pub kind: TokenKind,3    pub span: Span,4}5 6pub enum TokenKind {7    // Keywords8    Func, Val, Mut, If, Else, Return,9 10    // Literals (integer and float also have suffixed forms)11    Integer(i64),12    Float(f64),13    String(StringValue),   // Plain(..) or Interp(..), see above14    Char(char),15    Boolean(bool),16 17    // Identifiers and lifetimes18    Identifier(String),19    Lifetime(String),20 21    // Operators (many variants)...22}

The excerpt above shows representative variants; the full set lives in tokens.rs.

Integration Points

Downstream Consumers

• syntax-parsing: Consumes token stream for AST generation
• LSP server (Phase 8): Uses tokens for syntax highlighting

Dependencies

• shared-types: Span type for source locations
• No dependencies on other feature slices (maintains slice independence)

Future Enhancements

• Token stream caching for incremental compilation
• Better error recovery (continue lexing after an error)
• Documentation comment tokens (///, /**)

Nothing links TokenKind to neuro-language-support/syntaxes/neuro.tmLanguage.json, so any change to the token set has to update that editor grammar by hand in the same commit. tests/tmlanguage_sync.rs checks what it can without a tokenizer — that every keyword is covered, that the grammar's keyword rule invents none of its own, and that the rules naming a declaration are ordered ahead of the keyword rule so they stay reachable. tools/tmlanguage_scopes.mjs prints the scopes the grammar actually assigns to a source file, for the rules those checks cannot reach.