A 518-million-parameter Llama-style language model, pretrained from random weights on the same legal and financial corpus as its 125M sibling. Give it the start of a sentence and it continues in the legal register.
This is a base model, not a chatbot. It was trained on next-token prediction only, so it continues text rather than answering questions. Prompt it with the opening of a sentence and watch it complete the thought.
The honest quality metric is held-out validation perplexity: 7.91 (lower is better), reached over 5 epochs of pretraining. It speaks the legal register fluently (case-citation phrasing, procedural language) but it does not know facts reliably: at ~2 bits per parameter, 518M parameters hold only about 129MB of usable knowledge. Four times the 125M's capacity, still far short of knowing case law. Grounded facts need retrieval (RAG).
Why we quote bits‑per‑byte, not just perplexity. This model uses a 32K tokenizer; the 125M uses 16K, and perplexity is measured per token, so the two perplexities are not directly comparable: a bigger vocabulary packs more text into each token, making each prediction intrinsically harder. That gap is loudest early in training, when a raw perplexity comparison actively misleads: at step 1000 this model read perplexity 19.4 against the 125M's 16.5 at the same step, yet in bits‑per‑byte the two were already neck and neck. Bits‑per‑byte divides by the UTF‑8 bytes of the underlying text, so it is tokenizer‑invariant. On that honest footing the 500M finishes ahead of the 125M on both measures: 0.588 bits/byte vs 0.632, and 7.91 perplexity vs 8.36.
Corpus: US case law (~40%), SEC filings (~40%), educational web text (~20%). First call may take ~15–30s while the model wakes from idle.
| Stage | Cost | Share |
|---|---|---|
| Data pipeline32K tokenizer + re-tokenize (CPU); corpus cleaning is shared with the 125M | $1.35 | 1.2% |
| Pretraining5 epochs on 8x B200, 10.4B tokens seen, incl. smoke runs | $115.69 | 98.8% |
| Total to assemble | $117.04 | 100% |
Almost all of it is one line: 5 epochs of pretraining on 8x B200. Building the corpus and the tokenizer that fed it cost $1.35; the GPUs that learned from it cost $115.69.
Figures are actual invoiced Modal usage (GPU + CPU + memory) for this model’s lineage, not estimates. Serving is billed separately and scales to zero.