mlnomad/goat-v-482m-c4-seed0

GOAT-V 482M · C4 1× Chinchilla · seed 0 (PyTorch)

GOAT-V attention: no Q/K projections, V projection kept, no value embeddings. The Yat kernel acts directly on RoPE'd head slices of the residual stream; attention scores are L1-normalized rather than softmax. Combined with the YatNMN-Softplus MLP (scalar bias + learnable α). Trained on allenai/c4 for 1× Chinchilla on a single TPU v6e-8.

x_heads = RoPE(x.reshape(B, T, H, D))
dots    = x_heads @ x_heads^T
dist²   = ||x_i||² + ||x_j||² − 2·dots
scores  = (dots + softplus(b))² / (dist² + softplus(ε))
scores  = L1_normalize(scores)              # strict causal j<i
v       = c_v(x).reshape(B, T, H_kv, D)
y       = scores @ v                         # value projection retained
y       = c_proj(y)

Architecture

Params	482.3M
Depth / n_embd / heads	d=22 / 1408 / 22
Attention	Yat kernel on RoPE'd residual head slices (no Q/K), V projection kept
MLP	YatNMN-Softplus (scalar bias + learnable α)
Value embeds	none
Tokens	9.65B (1× Chinchilla, 20× params)
Data	allenai/c4 (en)
Final loss	2.9694 (smoothed 3.0007)
Hardware	TPU v6e-8 (europe-west4-a, TRC), fp32, ~20.6h

Sister run

This is the C4 counterpart to mlnomad/yatnmn-full-d22-chinchilla-pytorch, the FineWeb-Edu version of the same architecture (final loss 2.57). The two differ only in pretraining corpus, so the pair isolates the effect of dataset choice on the GOAT-V architecture.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "mlnomad/goat-v-482m-c4-seed0",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")

ids = tokenizer("The capital of France is", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=30, do_sample=True, temperature=0.8, top_p=0.9)
print(tokenizer.decode(out[0], skip_special_tokens=True))

License

Apache 2.0.