Rebuilding the Parser: Why We Changed and What It Enables
Over the past few weeks, we’ve been evolving the Luma parser to support richer language features like:
- Chained method calls and expressions
- Lambda expressions with inline or block bodies
- Interpolated strings
- Complex expression precedence handling
- Dot access + indexing + slicing
- Functional constructs like
.filter(x -> x > 1),.walk(...) -> {}
But as we began introducing these features, we hit a wall: our old parser was brittle - adding one new construct often broke several others.
What Was Wrong?
Our previous parser treated expressions in isolation, using a “greedy” or shallow parsing approach. It lacked:
- Robust precedence handling
- Context-aware parsing (e.g., dot chains after lambdas)
- Encapsulation of feature-specific logic (e.g., walk blocks)
- And importantly, no tolerance for future growth - small additions caused large breakages.
The Shift: Precedence-Driven Expression Parsing
To fix this, we rewrote core parts of the parser with Pratt-style precedence parsing, and introduced modular expression combinators like:
- parseMaybeLambdaOrPrimary()
- parseChainedExpr()
- parseWalkExpr()
- parseArgList()
This allowed us to separate concerns:
- Primary expressions handle literals, identifiers, and parenthesized expressions.
- Chained expressions handle post-fix operators like
.method()or[index]. - Precedence-aware expressions now correctly parse
a + b * c - d.
This structure now lets us write:
nums.filter(x -> x > 1)[0].to_str()… and have it parse without panic.
Resilience by Design
Thanks to skipType, isVarDecl, and our block-aware parseProgram, we’re now parsing:
- Variable declarations
- Assignments
- Expressions with arbitrary depth
- Functional flows like
list.walk(...) -> {} - Inline lambdas
…all while supporting comments, optional types, and built-in functions like .get() or .type().
Future-Proofing
This refactor sets us up to confidently add:
- User-defined functions (
fn greet(name: str) { ... }) - Pattern matching
- Richer type inference
- Macros or meta constructs
- And even bytecode optimizations at compile time
With a clean, composable parsing structure, we can now build up instead of constantly patching over.