01
Train fluency. Retrieve knowledge.
A compact language model built and trained on consumer hardware. Follow the data preparation, model architecture, and experiments behind a 369-million-parameter model, with search and tool use as future research directions.
- Consumer-GPU pretraining
- 32-layer transformer
- Reviewed data and reproducible training