Luna 3-230m
Our smartest model so far. Small enough to run on most devices, large enough to actually be intelligent.
Abstract
Luna 3 is a family of models with an upgraded context window and better architecture. These models are designed to perform well on complex tasks and provide high intelligence daily.
Luna was trained on an NVIDIA RTX 3050 6GB using BF16 precision and AdamW. Across five epochs, its perplexity dropped from 200 to 1.9/strong>. The training run took about 5 hours.
This is the current top model in the AobanLabs lineup. It can hold a conversation, continue text, and attempt general language tasks. It can also confidently produce something that makes absolutely no sense.
Model Release
The model files, custom implementation, and tokenizer are available on Hugging Face.
View Luna 3 on Hugging Face Try AobanBotLuna can generate repetitive, inaccurate, incomplete, or completely unrelated text. It's intended for research, experimentation, and casual conversation. However, Luna 3 is smart enough to generate coherent-like advice, please do not trust it... it will probably not tell the truth.
License: MIT
Architecture
Luna uses a causal Transformer architecture. Each generated token can attend to the preceding context, allowing the model to predict and generate text one token at a time.
Its design includes RMSNorm, causal scaled dot-product attention, a GELU feed-forward network with a 4× expansion, RoPE, and untied input and output embeddings.
Each attention head operates on 64 dimensions:
A consumer GPU normally has 2-16GB of VRAM, which cannot train large, frontier models. However, we used CPU offloading and fragmentation to increase training speed and help decrease memory usage.
Tokenization
Luna uses luna_tokenizer.json, included
in the Hugging Face repository. Its configured
vocabulary contains 100k tokens.
The tokenizer converts text into units the model can process. A familiar word might be one token, while a longer or less common word could be split into several. The tokenizer is an important part of the model's setup.
One Hundred Thousand entries. Somewhere in there is probably a token for whatever the model says when it forgets the question.
Training Details
Luna's reported training run used BF16 precision,
the AdamW optimizer, a learning rate of
3e-4, and a batch size of
32. Training took approximately
5 hours on an RTX 3050 6GB.
Luna was trained on 6 million tokens.
Training hyperparameters
| Hyperparameter | Value |
|---|---|
| Precision | BF16 |
| Optimizer | AdamW |
| Learning rate | 3e-4 |
| Batch size | 24 |
| GPU | NVIDIA RTX 3050 6GB |
| Training duration | Approximately 5 hours |
Training results
The model card reports the following average losses and perplexity values on an average. Training values could not be recovered.
| Epoch | Average loss | Perplexity |
|---|---|---|
| 1 | ~4 | ~300 |
| 2 | ~3 | ~70 |
| 3 | ~2.5 | ~30 |
| 4 | ~2.3 | ~17 |
| 5 | ~1.9 | ~12 |
The repository metadata lists
OpenAssistant/oasst1 and
HuggingFaceFW/fineweb. The training
notes specifically mention a subset of OASST1.
Training Philosophy
Luna focuses on English conversational text. Its intended uses include chatbots, casual dialogue, stories, text continuation, research, and fine-tuning.
The documented setup uses FlashAttention for attention computation and AdamW for optimization. Luna was built to be practical to experiment with on consumer hardware, without requiring a data center to get started.
It ran on an RTX 3050 6GB. No server farm. No room full of humming racks. Just a GPU, a training run, and the occasional reason to check the logs.
Getting Started
You can load Luna with Transformers and its custom model implementation. The example below uses the generation settings documented with the release.
Requirements
- Python and PyTorch
- Transformers and Hugging Face Hub
- The Tokenizers library
- Sufficient RAM or GPU memory for loading and generation
This example loads modeling_luna.py
and enables trust_remote_code=True.
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
REPO = "Aobangaming/Luna-3-230m"
tok = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForCausalLM.from_pretrained(
REPO, trust_remote_code=True, dtype=torch.bfloat16
).to("cuda").eval()
prompt = tok.apply_chat_template(
[{"role": "user", "content": "Hi"}],
tokenize=False, add_generation_prompt=True,
)
ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to("cuda")
with torch.no_grad():
out = model.generate(**ids, max_new_tokens=512)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True).strip())
If loading fails, ig ur cooked. Just ask a coding assistant or smth.
Capabilities and Limitations
Luna can continue text, generate responses, and attempt general language tasks. The model may struggle with instructions, long conversations, and unfamiliar topics.
Fine-tuning can adapt Luna for particular characters, chatbots, or tasks. Applications should credit AobanLabs, and include simple, coherent instructions.
The release focuses on English. However, Luna 3 is one of our Multilingual models, being able to speak around 3 languages. Please do not try speaking other languages with it, as this is a learned pattern. It has not been trained on those 3 languages fully.
Environmental Impact
AobanLabs uses cheap, low electricity models to reduce Co2 emissions. 0.17 kg CO₂e
What's Next?
Future work on Luna could focus on instruction following, more consistent conversations, better evaluation, context handling, and faster inference.
The goal is straightforward: make the model more useful without making it impractical to run. More training, better data, and careful testing are all part of that.
We're still working toward the day Luna answers
2 + 3 with 5 every time.
For now, mathematics remains an experimental feature.