End-to-End Compilation

Status: Implemented · AST → typed HIR → LLVM Slice: compiler/neurc (orchestrator) Dependencies: lexical-analysis, syntax-parsing, module-resolution, argument-binding, semantic-analysis, hir-lowering, llvm-backend

Overview

The neurc compiler driver provides end-to-end compilation from Neuro source files (.nr) to native executables. neurc is the only crate permitted to depend on every feature slice; it owns pipeline orchestration and contains no feature business logic itself (VSA).

Architecture

Compilation Pipeline

1Source File (.nr)2    ↓3┌──────────────────────────────────────────────────────────────┐4│ 1. Read Source (fs::read_to_string)                          │5├──────────────────────────────────────────────────────────────┤6│ 2. Lexical Analysis + Parsing (syntax_parsing::parse)        │7│    - Tokenization (logos)                                     │8│    - AST construction (Pratt + statement parser)             │9├──────────────────────────────────────────────────────────────┤10│ 3. Module Resolution (module_resolution::resolve_program)    │11│    - Expands every module the root file reaches into one     │12│      program: `.nr` files, `mod.nr` directories, inline      │13│      `module { }` blocks, imports and re-exports             │14│    - Verifies qualified paths and item/field visibility,     │15│      then erases the qualifiers into one flat namespace      │16│    - The driver prepends the prelude here, unless the root    │17│      file opted out with `@no_prelude`                       │18├──────────────────────────────────────────────────────────────┤19│ 3b. Argument Binding (argument_binding::bind_arguments)      │20│    - Reorders named arguments into declaration order, once   │21│      every module is merged and every callee is visible      │22├──────────────────────────────────────────────────────────────┤23│ 4. Semantic Analysis (semantic_analysis::type_check)         │24│    - Type checking, scope resolution                         │25│    - Emits warnings (e.g. lints)                             │26├──────────────────────────────────────────────────────────────┤27│ 5. HIR Lowering (hir_lowering::lower_program)                 │28│    - AST → typed High-Level IR (neuro-hir)                   │29│    - Every expression carries its resolved type             │30├──────────────────────────────────────────────────────────────┤31│ 6. Code Generation (llvm_backend::compile)                   │32│    - Consumes the typed HIR directly                         │33│    - LLVM IR generation (inkwell / LLVM 23)                  │34│    - Object code emission                                     │35├──────────────────────────────────────────────────────────────┤36│ 7. Write Object File (tempfile)                              │37│    - Temporary `.o` / `.obj`; removed after linking          │38├──────────────────────────────────────────────────────────────┤39│ 8. Link Executable                                           │40│    - System linker driver invocation + C runtime linking     │41└──────────────────────────────────────────────────────────────┘42    ↓43Native Executable (`.exe` on Windows, no extension on Unix)

The typed HIR (neuro-hir) is the stable, backend-agnostic contract between the frontend (parser + type checker) and the backends. llvm-backend consumes it, and so does mlir-backend. HIR lowering outlines every tensor operation into a function of its own, and neurc asks mlir-backend for each one's body, integer checks included. The LLVM backend, which has no tensor loops of its own, links those in and calls them through the program's own tensor ABI. On Linux it also lowers each @gpu function's body, and each @kernel function's per-thread body, through the MLIR GPU dialects to an NVIDIA or AMD kernel, and the LLVM backend links that kernel in with its launcher and the GPU runtime. See the HIR Lowering, LLVM Backend and MLIR Backend component docs.

Implementation

Core Function: compile_file

1fn compile_file(input: &Path, output: Option<&Path>, optimization: u8, emit: EmitKind) -> Result<PathBuf>

Purpose: Orchestrates the complete compilation pipeline from source file to artifact, and returns the path it wrote.

Stages, in order: read source → resolve modules and parse (module_resolution::resolve_program, with the parser and the prelude injected by the driver) → argument_binding::bind_arguments → semantic_analysis::type_check → hir_lowering::lower_program → check that main exists (unless --emit asks for an object or IR) → llvm_backend::compile → write the object file into a temporary directory → link. The rationale for this order lives in the neurc CONTEXT.md.

Error Handling Strategy:

• Uses anyhow::Context for error-chain construction; each stage adds contextual information.
• Fail-fast: stops at the first error, prints a detailed message to stderr, exits non-zero.
• The type checker reports every type error it finds before the run stops; parse errors stop at the first one.

Example Error Output (a program with a type mismatch):

1Type errors found in "concat.nr":2error: cannot apply binary operator + to types string and i323 --> concat.nr:2:134  |52 |     val s = "count: " + 16  |             ^^^^^^^^^^^^^7 8Compilation failed: Type checking failed9  Caused by (1): 1 type error(s) found

check vs compile

neurc check runs stages 1 through 5 (read, resolve + parse, bind arguments, type-check, HIR lowering) and stops; it validates a program (including that it lowers cleanly to HIR) without producing a binary. neurc compile runs the full pipeline. neurc run calls compile_file with an output path inside a temporary directory, runs the result, and exits with the program's own status.

--emit obj stops one step short of the linker and writes the object file to the output path instead. It carries no main requirement, because an object may be a library; the default --emit exe still demands an entry point. See CLI Usage for linking one into a shared library a foreign DLPack consumer can call.

--emit llvm-ir stops one step earlier still and writes the textual LLVM module, with the host data layout and triple and after -O's pass pipeline. It carries no main requirement either. See CLI Usage.

Linking

The driver shells out to a linker driver (a C compiler front-ending the real linker, which brings the C runtime and startup code). On Unix it always invokes cc. On Windows it tries, in order: clang, then lld-link, then MSVC cl.exe (which locates the real link.exe).

CLI Integration

neurc check   <INPUT>            # Stages 1-5: resolve + parse, bind, type-check, lower to HIR
neurc compile <INPUT> [OPTIONS]  # Full pipeline to a native binary
neurc run     <INPUT> [-O <N>]   # Compile into a temporary directory and run

neurc check   <INPUT>            # Stages 1-5: resolve + parse, bind, type-check, lower to HIR
neurc compile <INPUT> [OPTIONS]  # Full pipeline to a native binary
neurc run     <INPUT> [-O <N>]   # Compile into a temporary directory and run

Options (for compile):

• -o, --output <FILE>: output executable path (defaults to the input filename, .exe on Windows)
• -O <LEVEL>: optimization level (0 to 3)
• --emit <exe|obj|llvm-ir>: artifact to write (defaults to exe)

Examples:

neurc check   examples/basics/hello.nr
neurc compile examples/basics/hello.nr
neurc compile examples/basics/hello.nr -o bin/hello
RUST_LOG=debug neurc compile examples/basics/hello.nr   # debug logging

neurc check   examples/basics/hello.nr
neurc compile examples/basics/hello.nr
neurc compile examples/basics/hello.nr -o bin/hello
RUST_LOG=debug neurc compile examples/basics/hello.nr   # debug logging

Every flag is described in the CLI Usage Guide.

Exit Codes

Under run, a successful build exits with the program's own status instead.

Testing

End-to-end coverage lives in compiler/neurc/tests/. The per-feature suites under tests/suite/ (arrays.rs, drop_destructors.rs, hir_lowering.rs, …) compile and run real programs, asserting exit codes and output. tests/examples.rs builds and runs every program in examples/ against its pinned exit code and stdout, and tests/architecture_tests.rs enforces the slice dependency rules. cargo test --workspace runs all of it.

Known Limitations

1. Debug information: no DWARF/PDB generation yet.
2. Flat namespace: module resolution merges every file into one namespace; two modules declaring the same top-level name collide rather than nesting.
3. System toolchain required for linking (clang/MSVC on Windows, gcc/clang on Unix); no bundled linker.

Future Enhancements

Planned on the Quick Roadmap: debug information (-g), incremental compilation with a persistent cache, and LTO defaults for release builds.

Setup

LLVM 23 with LLVM_SYS_231_PREFIX set is required to build the compiler. See the Installation Guide for per-platform instructions (Linux, macOS, Windows), MLIR included, since every build needs it. Common build problems are covered in Troubleshooting.

References

• CONTRIBUTING.md, development guidelines and architecture rules
• CHANGELOG.md, version history
• compiler/neurc/src/main.rs, implementation
• Installation Guide, toolchain setup