Binary Buffers Arrive: Luma Learns to Speak in Bytes

In our last post, we talked about building the toolbox that would make database drivers possible in pure Luma. We said TCP, binary buffers, and crypto were the foundation. Today, one of those pieces is done.

Luma now has a buffer type — a first-class tool for building and parsing binary data.

Why buffers matter

Every network protocol speaks in bytes. MySQL encodes packet lengths as three little-endian bytes. DNS uses big-endian 16-bit integers. TLS, HTTP/2, WebSocket frames, Redis commands — they all define exact byte layouts that programs must construct and parse correctly.

Without a buffer type, working with binary data means manual byte math:

// Before: manual, error-prone, unreadable
length: int = data[0] + data[1] * 256 + data[2] * 65536

With buffers, the intent is clear:

buf: buffer = buffer(data)
length: int = buf.read(3).to_int24()

The buffer handles cursor tracking and bounds checking. The developer focuses on the protocol.

Two modes, one type

A buffer is either being written to (building a packet) or read from (parsing a packet). Same type, different usage pattern.

Writing

buf: buffer = buffer()
buf.write(0x03)              // command byte
buf.write("SELECT 1")        // query text
buf.write(payload)            // payload is [byte]
packet: [byte] = buf.to_bytes()

All write methods append to the end. The cursor isn’t affected.

Reading

buf: buffer = buffer(data)
protocol: byte = buf.read()              // one byte
version: str = buf.read(0x00)            // until null terminator
thread_id: int = buf.read(4).to_int32()  // 4 bytes as int

All read methods advance the cursor sequentially. You process fields in the order they appear in the data.

Duck-typed dispatch

This is where Luma’s design philosophy shows. Instead of write_byte, write_bytes, write_str, and write_buffer, there’s just write(). The compiler sees the argument type and dispatches:

buf.write(0x0A)            // byte — appends one byte
buf.write(data)            // [byte] — appends raw bytes
buf.write("SELECT 1")     // str — appends UTF-8 bytes
buf.write(other_buf)       // buffer — appends other buffer's contents

Four different operations, one method name. The code reads like prose.

The same principle applies to read(), but with an extra dimension — it dispatches on both the argument type and the return type:

v: byte = buf.read()              // no arg, byte return → read 1 byte
data: [byte] = buf.read(8)        // int arg, [byte] return → read 8 bytes
text: str = buf.read(4)           // int arg, str return → read 4 bytes as string
name: str = buf.read(0x00)        // byte arg, str return → read until null
rest: [byte] = buf.read()         // no arg, [byte] return → read all remaining

The variable declaration tells Luma what you want. Seven different read operations, all spelled the same way. This is what we mean when we say the language should get out of your way.

Bounds checking that helps

Buffer errors are always panics with descriptive messages. Reading past the end of a buffer is a programming error — the data is shorter than expected, or the code has a bug. Either way, continuing would produce wrong results.

buffer underflow: tried to read 4 bytes at position 12, but only 2 remaining

buffer underflow: delimiter 0x00 not found starting at position 7

The position and sizes are right there in the message. When debugging a protocol implementation, this is the difference between staring at a hex dump and knowing exactly where expectations diverged from reality.

Utility methods

Beyond read and write, buffers provide the cursor control you need for protocol work:

buf.len()          // total bytes in buffer
buf.position()     // current cursor position
buf.remaining()    // bytes left to read
buf.skip(10)       // advance cursor without reading
buf.reset()        // cursor back to 0
buf.to_bytes()     // extract full contents

skip() is essential for reserved fields and padding. MySQL’s server greeting has a 10-byte reserved block and several filler bytes. Instead of reading them into variables you’ll never use:

buf.skip(10)    // reserved bytes — skip them

What this looks like in practice

Here’s a real-world pattern — building a framed network packet:

body: buffer = buffer()
body.write(0x03)                    // command
body.write("SELECT 1")             // payload

pkt: buffer = buffer()
pkt.write(body.len().to_int24())   // 3-byte length header
pkt.write(0x00)                    // sequence ID
pkt.write(body)                    // body is buffer — Luma knows

data: [byte] = pkt.to_bytes()     // ready to send

And parsing the same packet back:

buf: buffer = buffer(raw)
length: int = buf.read(3).to_int24()
seq: byte = buf.read()
payload: [byte] = buf.read(length)

No imports. No setup. No ceremony. Declare a buffer, use it.

Feature-flagged inclusion

The buffer runtime is only included in your compiled binary when you actually use it. If your program doesn’t touch buffers, the LumaBuffer struct and all its supporting functions are never emitted. This follows the same pattern as Luma’s JSON and HTTP server support — pay only for what you use.

What’s next

Buffers are the first piece of the low-level primitives we outlined in our database roadmap. With binary data handling in place, the next pieces are:

• TCP networking — tcp.connect(), sock.write(), sock.read() — the transport layer
• Cryptographic hashing — SHA1 and SHA256 for authentication protocols
• The database traits — the contract that drivers implement

Each one builds on the last. TCP needs buffers to frame packets. Crypto needs buffers to hash byte sequences. And the MySQL driver — written in pure Luma — will use all three.

We’re building the tools first. Today, the toolbox got its first new tool.