How to prompt small language models
Small models punish vague prompts. Here is how to get useful answers from a 1B to 8B model running on your phone.
Most prompt advice is written for big cloud models. Those can handle a messy prompt and still figure out what you meant. Small models can’t. A 3B model on your phone needs you to be clear, and that changes how you write.
Put the task first
Put the task in the first sentence instead of the background. “Rewrite this email to be shorter” works better than explaining why the email exists and who reads it.
A big model will find your point in the noise, but a small model grabs the first thing it sees and runs with it, so the first thing it reads should be the instruction.
Keep the prompt short
Small models have small context windows and get lost when you fill them up, so every sentence you paste in should matter.
If you want a summary, give it:
- the text
- the word limit
That’s enough. You don’t need disclaimers or a polite setup, because the model was never offended anyway.
One task at a time
This is the biggest difference from cloud models. Ask a big model to translate a paragraph, review the translation, and suggest a better ending, and it does all three.
A small model does the first one, messes up the second, and forgets the third. You get much better results by asking once, taking the answer, and only then moving to the next request. Those follow-ups are free since the model runs on your phone.
Name the format
Tell the model the shape you want:
- “Answer in three bullet points”
- “Write a short paragraph under 80 words”
- “Give me a table with two columns”
“Keep it brief and organized” is too vague. Small models follow clear format rules well and guess badly.
Skip long roleplay
Telling a big model to act as a veteran editor can change the output. On a small model, a long persona just eats context.
A short one is fine, like “you are a strict editor.” A paragraph of backstory is wasted space.
Use temperature when you can
If your app has a temperature setting, use it:
- Lower for factual work and clean rewriting
- Higher for brainstorming
That one setting does more than most prompt tweaks.
The simple rule
Small models reward clear prompts and punish extra words. Cut it down to instruction, material, and format, and a model that fits on your phone will do more than you expect.
FAQ
Why do small models need different prompts?
Small models have less capacity to infer intent from messy prompts. They work best when the task, source material, and desired format are stated clearly up front.
How long should a prompt be for a small model?
Keep prompts short. Small models have limited context windows, so every extra sentence competes with the actual task and can reduce answer quality.
Can small models handle multiple tasks in one prompt?
Not reliably. Small models usually complete the first task and lose track of follow-up instructions. Ask one question at a time for better results.
What temperature setting works best for small models?
Use lower temperature for factual rewriting and structured answers. Use higher temperature for brainstorming and creative variations.