solenopsisbot's avatar: an anime girl with violet eyes and flowers in her hair

solenopsisbot

Viola Founder, Solexsis

I build language models from the bytes up, and computer friends to talk through them. Right now that mostly means Braid. Also: computers, programming, and anime.

Research & Focus

  • Architectures
  • Social AI
  • Finetuning
  • RL
scroll

Now Solexsis Research

Braid, a language model that reads raw bytes

No tokenizer, and not a transformer. Braid is a causal U-Net over the byte stream: it does most of its work with cheap convolutions and only attends inside two bounded windows. The background of this page is a braid too, which is not a coincidence.

  1. Bytes, n-gram tables, conv stem every byte
  2. Pool ÷4, shift one slot every 4th
  3. Attention, 2 KB reach every 16th
  4. Expand ×4, gated skip every 4th
  5. 256-byte sliding attention every byte
  6. Byte logits + exact copy-4 every byte
Attention is 0.7% of the FLOPs. The shift after each pool is the only thing keeping it causal, and it needs no mask.
Parameters
295M
Vocabulary
256 bytes
Context
8,192 bytes
Decode state
2.44 MB, flat

Against a matched transformer

  • 3.07× training throughput
  • 2.23× less peak memory
  • 2.99× less energy

At equal data the transformer still finishes 0.019 bits per byte better. The win is time, memory and energy, not quality.

~30M params each, same FineWeb bytes, one RTX 3090, three seeds.

Projects

Everything else I'm building

Contact

Get in touch