Context and Thinking Tokens in a Small LLM

Grading

Overview

A support desk receives thousands of messages a day. Each message has to reach the right team. In this assignment you turn Qwen3-0.6B, a small instruction-tuned language model, into a router. It reads a customer message and answers with one of 27 intents in JSON. The messages come from the Bitext customer support data set. You measure how many tokens the router uses for each message. This includes the tokens sent to the model and the tokens it generates before its answer. Part 1 is about the context. You show the model labeled examples before each message. You measure how accuracy, prompt length, time and GPU memory change as the number of examples grows from none to 432. You then fine-tune the model with LoRA so that it routes with no examples. You compare prompting and fine-tuning on accuracy and cost. Finally, you hide the message in an inbox of up to 400 other messages. For the base model and the fine-tuned router, you measure accuracy when the message is at the start, in the middle and at the end of the context. Part 2 is about reasoning. Qwen3 can write its reasoning in a <think> block before it answers. You measure whether reasoning improves the routing, what it costs in generated tokens and time, and what happens when you turn it on for the router you fine-tuned in Part 1, whose training data never contained any reasoning. Before you change anything, all cells run from top to bottom and produce the baseline. The baseline uses no few-shot examples, a single message instead of an inbox, and an answer with the reasoning still attached. The questions are graded on what you measure and how you explain it. A result that contradicts the literature is a valid result if your measurement is sound and you can say why it differs.

Background

Production systems often use language models to classify support tickets, emails, search queries and content flags. Each request uses input and output tokens. Both parts of this assignment measure those token counts. Brown et al. showed that a large language model can learn a task from examples placed in its prompt, with no change to its weights (Brown et al., 2020). Each example increases the prompt length on every request. Longer prompts take more processing time and memory. The choice between prompting and fine-tuning with LoRA (Hu et al., 2021) affects both accuracy and cost. Mosbach et al. compared the two using the same models and the same number of examples. Their results depended on model size and the amount of data (arXiv:2305.16938). A model can lose accuracy on prompts much shorter than its advertised context window. Liu et al. found that models answer best from information at the start or the end of a long context (Liu et al., 2023). Hsieh et al. found that many long-context models fall well short of their advertised length on retrieval and multi-step tasks (Hsieh et al., 2024). Part 1 makes these comparisons for one routing task and measures the cost of each option. Chain-of-thought prompting asks a model to write its reasoning before answering. It improves accuracy on many multi-step problems (Wei et al., 2022). Models such as Qwen3 can switch between thinking and non-thinking modes (arXiv:2505.09388). Reasoning adds generated tokens to every request. Those tokens increase API charges and response time. Sprague et al. analyzed more than 100 papers and ran their own experiments. They found that chain-of-thought helps mainly on math and symbolic reasoning, with little benefit elsewhere (arXiv:2409.12183). Part 2 tests that finding on a classification task. It also measures what fine-tuning on answers without reasoning does to the reasoning of the model.

What you hand in

• The notebook, with every cell output saved, every TODO completed and every question answered in its markdown cell.

What you will need

You need a GPU runtime. In Colab, open Runtime, select Change runtime type, and choose a T4 GPU before starting. The longest prompts, 16 examples per intent and about 12,000 tokens, peak at 7.6 GB of GPU memory, which fits on a T4. The complete solution ran in 12 minutes on an RTX PRO 6000; a T4 takes several times longer. A free Colab session can end early. The fine-tuned router of Part 1 is saved to disk, so Part 2 can reload it as long as the runtime’s files are still there.

Assignment notebook

Submitting

Commit the notebook to your private clone of the class repo under assignments/assignment-4. Save every cell output before committing. A notebook without outputs is graded as though its cells never ran. Submit the complete GitHub URL for that directory to Canvas or Brightspace. Do not submit the repository root. The assignment submission guide explains the remaining requirements, including how to set up your clone, add the TAs as collaborators, and create the README required in each assignment folder.

References

• Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., et al. (2020). Language Models are Few-Shot Learners. arXiv [cs.CL].
• Hsieh, C., Sun, S., Kriman, S., Acharya, S., Rekesh, D., et al. (2024). RULER: What’s the real context size of your long-context language models?. arXiv [cs.CL].
• Hu, E., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., et al. (2021). LoRA: Low-Rank Adaptation of large language models. arXiv [cs.CL].
• Liu, N., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., et al. (2023). Lost in the middle: How language models use long contexts. arXiv [cs.CL].
• Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv [cs.CL].

---