Theme 3: Fine-Tuning Gemma, Starter Kit

This starter kit demonstrates the full fine-tuning workflow on Gemma 4, Google’s open-weight model, using Unsloth on a free Colab GPU:

Baseline: measure how the plain Gemma model performs on a task.
Fine-tune: train a small set of additional weights (LoRA) on examples of the task.
Compare: measure the fine-tuned model on the same test set and compare the results.
Application: show both models side by side in a web application.
Sample task: IT support ticket triage. Given a support ticket, the model must reply with JSON containing the ticket’s category and priority. The plain model does not know the required categories or format. The fine-tuned model learns them from about 180 examples. This makes the effect of fine-tuning easy to measure.

Your team will replace the sample task with the problem statement you selected (3A or 3B). Problem 3B uses images; see the section “Adapting for images (Problem 3B)” near the end of this notebook.

Before you start: select a GPU

This notebook runs the Gemma model inside Google Colab and needs a GPU. Select Runtime → Change runtime type → T4 GPU → Save. The free T4 GPU is sufficient.

No API key is required. Gemma 4 is released under the Apache 2.0 licence and downloads without a Hugging Face account.

How to use this notebook

Run the cells in order, from top to bottom, the first time (select a cell and press Shift + Enter).
Comments marked STUDENT TODO show the intended places to make changes.
After changing a cell, re-run that cell and any cells below it that depend on it.
If the notebook stops working as expected, select Runtime → Restart session and run all cells again from the top. The installation cell does not need to be repeated after a restart unless the runtime was disconnected.
Step 0: Installation (run once)

%%capture

Installs Unsloth and compatible library versions (takes 2-4 minutes; output is hidden).

import os
if “COLAB_” not in “”.join(os.environ.keys()):
!pip install unsloth
else:
!pip install sentencepiece protobuf “datasets==4.3.0” “huggingface_hub>=0.34.0” hf_transfer
!pip install –no-deps unsloth_zoo bitsandbytes accelerate peft trl triton unsloth
!unsloth install-kernels
!pip install –no-deps –upgrade “torchao>=0.16.0”
!pip install –no-deps transformers==5.5.0 “tokenizers>=0.22.0,<=0.23.0”
!pip install “huggingface_hub>=1.5.0,<2.0”
!pip install –no-deps –upgrade timm
!pip install “gradio>=6,<7” pandas

import torch
if torch.cuda.is_available():
print(f”GPU ready: {torch.cuda.get_device_name(0)}”)
else:
raise RuntimeError(“No GPU found. Select Runtime → Change runtime type → T4 GPU, then run the notebook again from the top.”)

GPU ready: Tesla T4
Step 1: Load the plain Gemma model

MAX_SEQ_LENGTH = 1024 # the maximum length of one training example (prompt + answer) in tokens

from unsloth import FastModel

STUDENT TODO: the Gemma model to use.

“unsloth/gemma-4-E2B-it” Gemma 4 E2B (about 2B effective parameters). Fits comfortably on a free T4. (default)

“unsloth/gemma-4-E4B-it” Gemma 4 E4B (about 4B effective parameters). More capable, slower, uses more memory.

MODEL_NAME = “unsloth/gemma-4-E2B-it”

model, tokenizer = FastModel.from_pretrained(
model_name = MODEL_NAME,
dtype = None, # selected automatically
max_seq_length = MAX_SEQ_LENGTH,
load_in_4bit = True, # 4-bit quantisation reduces memory use
full_finetuning = False,
)
print(f”Loaded {MODEL_NAME}”)

🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning.
🦥 Unsloth Zoo will now patch everything to make training faster!
==((====))== Unsloth 2026.10.8: Fast Gemma4 patching. Transformers: 5.5.0.
\ /| Tesla T4. Num GPUs = 1. Max memory: 14.563 GB. Platform: Linux.
O^O/ _/ \ Torch: 2.11.0+cu130. CUDA: 7.5. CUDA Toolkit: 13.0. Triton: 3.6.0
\ / Bfloat16 = FALSE. FA [Xformers = 0.0.35. FA2 = False]
“-____-” Free license: http://github.com/unslothai/unsloth
Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored!
Unsloth: Using float16 precision for gemma4 won’t work! Using float32.
Loaded unsloth/gemma-4-E2B-it

def generate(user_text, system_text=None, max_new_tokens=256, temperature=0.0):
“”“Sends one message to Gemma and returns its reply as text.
temperature = 0.0 gives the same answer every time; higher values give more varied answers.”””
messages = []
if system_text:
messages.append({“role”: “system”, “content”: [{“type”: “text”, “text”: system_text}]})
messages.append({“role”: “user”, “content”: [{“type”: “text”, “text”: user_text}]})

inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt = True,
    tokenize = True,
    return_dict = True,
    return_tensors = "pt",
).to("cuda")

sampling = {"do_sample": True, "temperature": temperature, "top_p": 0.95, "top_k": 64} if temperature > 0 else {"do_sample": False}
outputs = model.generate(**inputs, max_new_tokens=max_new_tokens, use_cache=True, **sampling)
new_tokens = outputs[0][inputs["input_ids"].shape[1]:]
return tokenizer.decode(new_tokens, skip_special_tokens=True).strip()

print(generate(“In one sentence, what is fine-tuning in machine learning?”, max_new_tokens=80))

Fine-tuning in machine learning is the process of taking a pre-trained model and further training it on a smaller, task-specific dataset to adapt it for a particular application.
Step 2: The task and the dataset

Each example is a pair: an input (the prompt the model receives) and the output the model should produce.

The sample dataset is generated in code. The test set uses different ticket wording from the training set, so the evaluation measures whether the model has learned the task rather than memorised the examples.

import random
from datasets import load_dataset
from google.colab import userdata

Retrieve your saved Hugging Face token

hf_token = userdata.get(‘HF_TOKEN’)

Load the Hindi version of the IndicCMix dataset

data = load_dataset(“ai4bharat/IndicCMix”, data_files=“hi.parquet”, split=“train”, token=hf_token)
data = data.remove_columns([“audio”])
print(len(data), “examples loaded.”)

LANGUAGE = “Hindi”

Format the data into the exact prompt structure the AI needs

pairs = [{“input”: f”Rewrite this in casual romanized {LANGUAGE}, as people write in chat messages.\n\nText: {r[‘english’]}”,
“output”: r[“romanized_casual”]} for r in data]

Shuffle and split the data: 30 examples held back for the test set

random.Random(0).shuffle(pairs)
test_examples, train_examples = pairs[:30], pairs[30:]

print(f”{len(train_examples)} training examples, {len(test_examples)} test examples.\n”)
print(“Example input:\n”, train_examples[0][“input”])
print(“\nExample output:\n”, train_examples[0][“output”])

104800 examples loaded.
104770 training examples, 30 test examples.

Example input:
Rewrite this in casual romanized Hindi, as people write in chat messages.

Text: AI algorithms are being used in student behavioral analysis to identify patterns and provide personalized feedback via email notifications.

Example output:
AI algorithms ka upayog student behavioral analysis mein kiya ja raha hai pattern identify karne aur personalized feedback dene ke liye email notifications ke zariye.
Using your own dataset

Your dataset must become two lists of {“input”: …, “output”: …} pairs: one for training and one for testing. Keep the test examples out of the training set. Examples:

From a CSV file with columns “input” and “output” (upload it using the folder icon in the left sidebar):

import pandas as pd
rows = pd.read_csv(“my_data.csv”).to_dict(“records”)
random.Random(0).shuffle(rows)
test_examples, train_examples = rows[:30], rows[30:]

Problem 3A: AI4Bharat IndicCMix (gated dataset).

1. Sign in at huggingface.co and accept the terms on https://huggingface.co/datasets/ai4bharat/IndicCMix

2. Create an access token at https://huggingface.co/settings/tokens

3. Add it in Colab Secrets (key icon, left sidebar) as HF_TOKEN, with Notebook access enabled.

from datasets import load_dataset
data = load_dataset(“ai4bharat/IndicCMix”, data_files=“ka.parquet”, split=“train”) # STUDENT TODO: “ka.parquet” for Kannada or “hi.parquet” for Hindi
data = data.remove_columns([“audio”]) # the text columns are all that is needed
print(len(data), “examples”)
LANGUAGE = “Kannada” # STUDENT TODO: “Kannada” or “Hindi”, matching the file above
pairs = [{“input”: f”Rewrite this in casual romanized {LANGUAGE}, as people write in chat messages.\n\nText: {r[‘english’]}”,
“output”: r[“romanized_casual”]} for r in data]
random.Random(0).shuffle(pairs)
test_examples, train_examples = pairs[:30], pairs[30:]
For Problem 3A, the outputs are text rather than JSON. Replace score() in the next step with a text-similarity measure, for example chrF:

!pip install -q sacrebleu
import sacrebleu
def score(prediction, expected):
return {“chrf”: sacrebleu.sentence_chrf(prediction, [expected]).score}
Then change the metric names in evaluate() (valid_json, category_correct, priority_correct) to chrf.

For Problem 3B (images), see “Adapting for images (Problem 3B)” near the end of this notebook.

Step 3: Baseline evaluation of the plain model

score() compares one model output with the expected output. evaluate() runs the model on every test example and reports the results.

!pip install -q sacrebleu
import pandas as pd
import sacrebleu

def score(prediction, expected):
“”“Measures how closely the AI’s translation matches the expected Hinglish output using the chrF metric.”””
return {“chrf”: sacrebleu.sentence_chrf(prediction, [expected]).score}

def evaluate(examples, label, generate_fn=None):
generate_fn = generate_fn or generate
rows = []
for i, ex in enumerate(examples, start=1):
prediction = generate_fn(ex[“input”], max_new_tokens=128)
rows.append({
“input”: ex[“input”].split(“\n\n”, 1)[-1],
“expected”: ex[“output”],
label: prediction,
**score(prediction, ex[“output”])
})
print(f”\r{label}: {i}/{len(examples)} evaluated”, end=””)

df = pd.DataFrame(rows)
# Track the chrF metric instead of the old JSON metrics
metrics = df[["chrf"]].mean().round(1)
print(f"\n{label} results (chrF score - higher is better):\n{metrics.to_string()}")
return df, metrics

Run the evaluation on the untrained model

baseline_df, baseline_metrics = evaluate(test_examples, “plain_model”)
baseline_df[[“input”, “expected”, “plain_model”]].head(5)

plain_model: 30/30 evaluated
plain_model results (chrF score - higher is better):
chrf 42.3
input expected plain_model
0 Text: The District and Sessions Court, Balodab… Balodabazar ke District aur Sessions Court ka … Here are a few options, depending on how casua…
1 Text: The Climate Change Knowledge Centre offe… Climate Change Knowledge Centre apne kid’s cor… Here are a few options, depending on how super…
2 Text: The tax treatment of Provident Fund with… Bharat ke corporate sector mein bahut se salar… Here are a few options, depending on how casua…
3 Text: In Meghalaya, local chefs showcased thei… Meghalaya mein, local chefs ne apni culinary s… Here are a few options, depending on how super…
4 Text: Local communities participate in Faceboo… Local samuday Facebook groups mein forest mana… Here are a few options, depending on how casua…
Step 4: Fine-tune with LoRA

LoRA (Low-Rank Adaptation) leaves the original model weights unchanged and trains a small set of additional weights. This makes fine-tuning possible on a free GPU in a few minutes.

STUDENT TODO: training settings.

LORA_RANK = 8 # size of the LoRA adapters; larger can learn more but may overfit
TRAIN_STEPS = 150 # number of training steps; increase for larger datasets
LEARNING_RATE = 2e-4

model = FastModel.get_peft_model(
model,
finetune_vision_layers = False, # text only
finetune_language_layers = True,
finetune_attention_modules = True,
finetune_mlp_modules = True,
r = LORA_RANK,
lora_alpha = LORA_RANK,
lora_dropout = 0,
bias = “none”,
random_state = 3407,
)

from unsloth.chat_templates import get_chat_template, train_on_responses_only
tokenizer = get_chat_template(tokenizer, chat_template = “gemma-4”)

from datasets import Dataset
def to_text(example):
conversation = [{“role”: “user”, “content”: example[“input”]},
{“role”: “assistant”, “content”: example[“output”]}]
text = tokenizer.apply_chat_template(conversation, tokenize=False, add_generation_prompt=False)
return {“text”: text.removeprefix(“”)}

train_dataset = Dataset.from_list(train_examples).map(to_text)
print(train_dataset[0][“text”])

<|turn>user
Rewrite this in casual romanized Hindi, as people write in chat messages.

Text: AI algorithms are being used in student behavioral analysis to identify patterns and provide personalized feedback via email notifications.<turn|>
<|turn>model
AI algorithms ka upayog student behavioral analysis mein kiya ja raha hai pattern identify karne aur personalized feedback dene ke liye email notifications ke zariye.<turn|>

from trl import SFTTrainer, SFTConfig

trainer = SFTTrainer(
model = model,
tokenizer = tokenizer,
train_dataset = train_dataset,
eval_dataset = None,
args = SFTConfig(
dataset_text_field = “text”,
per_device_train_batch_size = 1,
gradient_accumulation_steps = 4,
warmup_steps = 5,
max_steps = TRAIN_STEPS,
learning_rate = LEARNING_RATE,
logging_steps = 5,
optim = “adamw_8bit”,
weight_decay = 0.001,
lr_scheduler_type = “linear”,
seed = 3407,
report_to = “none”,
),
)
trainer = train_on_responses_only(trainer) # learn only from the expected outputs, not the prompts

trainer_stats = trainer.train()
print(f”Training finished in {trainer_stats.metrics[‘train_runtime’] / 60:.1f} minutes.”)

Unsloth: Switching to float32 training since model cannot work with float16
🦥 Unsloth: Padding-free auto-enabled, enabling faster training.
Unsloth: Auto-detected instruction_part = ‘<|turn>user\n’ and response_part = ‘<|turn>model\n’
The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer’s values. Updated tokens: {‘bos_token_id’: 2}.
==((====))== Unsloth - 2x faster free finetuning | Num GPUs used = 1
\ /| Num examples = 104,770 | Num Epochs = 1 | Total steps = 150
O^O/ _/ \ Batch size per device = 1 | Gradient accumulation steps = 4
\ / Data Parallel GPUs = 1 | Total batch size (1 x 4 x 1) = 4
“-____-” Trainable parameters = 12,668,928 of 5,135,846,944 (0.25% trained)
[150/150 07:24, Epoch 0/1]
Step Training Loss
5 2.156894
10 1.441767
15 1.243124
20 0.865204
25 0.714547
30 0.864452
35 0.979577
40 0.799337
45 0.605699
50 0.741940
55 0.689378
60 0.810827
65 0.584502
70 0.785260
75 0.509081
80 0.650948
85 0.803978
90 0.805613
95 0.702236
100 0.770449
105 0.569147
110 0.533606
115 0.650095
120 0.666676
125 0.575326
130 0.868773
135 0.544110
140 0.553693
145 0.630942
150 0.566253
Training finished in 7.9 minutes.
Step 5: Evaluate the fine-tuned model and compare

finetuned_df, finetuned_metrics = evaluate(test_examples, “fine_tuned_model”)

comparison = pd.DataFrame({“Plain Gemma (%)”: baseline_metrics, “Fine-tuned Gemma (%)”: finetuned_metrics})
comparison[“Change (points)”] = comparison[“Fine-tuned Gemma (%)”] - comparison[“Plain Gemma (%)”]
print(comparison.to_string())

side_by_side = baseline_df[[“input”, “expected”, “plain_model”]].copy()
side_by_side[“fine_tuned_model”] = finetuned_df[“fine_tuned_model”]
side_by_side.head(10)

fine_tuned_model: 30/30 evaluated
fine_tuned_model results (chrF score - higher is better):
chrf 71.8
Plain Gemma (%) Fine-tuned Gemma (%) Change (points)
chrf 42.3 71.8 29.5
input expected plain_model fine_tuned_model
0 Text: The District and Sessions Court, Balodab… Balodabazar ke District aur Sessions Court ka … Here are a few options, depending on how casua… Balodabazar ki District aur Sessions Court ko …
1 Text: The Climate Change Knowledge Centre offe… Climate Change Knowledge Centre apne kid’s cor… Here are a few options, depending on how super… Climate Change Knowledge Centre apne kid’s cor…
2 Text: The tax treatment of Provident Fund with… Bharat ke corporate sector mein bahut se salar… Here are a few options, depending on how casua… Provident Fund ki withdrawal ka tax treatment …
3 Text: In Meghalaya, local chefs showcased thei… Meghalaya mein, local chefs ne apni culinary s… Here are a few options, depending on how super… Meghalaya mein, local chefs ne apni culinary s…
4 Text: Local communities participate in Faceboo… Local samuday Facebook groups mein forest mana… Here are a few options, depending on how casua… Local communities Facebook groups mein forest …
5 Text: The denim brand’s social media campaign … Us denim brand ki social media campaign viral … Here are a few options, depending on how super… Us denim brand ka social media campaign viral …
6 Text: Effective leadership in the legal sector… Kaanooni kshetra mein prabhavi netritva ke liy… Here are a few options, depending on how casua… Legal sector mein effective leadership ke liye…
7 Text: Italy collaborates with India on museum … Italy ne museum e-ticketing par India ke saath… Here are a few options, depending on how casua… Italy India ke saath museum e-ticketing par mi…
8 Text: Literature festivals celebrate Northeast… Literature festivals mein Northeast India ke f… Here are a few options, depending on the exact… Literature festivals Northeast India ki lok pa…
9 Text: They were invariably guided by the Manu,… Ve hamesha Manu, Yagnavalkya, aur dusre Smriti… Here are a few options, depending on the exact… Woh hamesha Manu, Yagnavalkya, aur doosre Smri…

Save the comparison table to a CSV file

side_by_side.to_csv(“model_comparison.csv”, index=False)

Automatically trigger the download to your computer

from google.colab import files
files.download(“model_comparison.csv”)

Step 6: Launch the comparison application

The application sends the same input to the plain model (by temporarily switching off the LoRA adapters) and to the fine-tuned model. Run this cell to open it below and create a public link.

Install the required libraries

!pip install -q gTTS transformers librosa

import gradio as gr
from gtts import gTTS
import os
from transformers import pipeline

gr.close_all()

Load the highly accurate Whisper Small model on the CPU

print(“Loading Voice Recognition model (whisper-small)…”)
transcriber = pipeline(“automatic-speech-recognition”, model=“openai/whisper-small”, device=“cpu”)

APP_TITLE = “Hinglish AI Translator 3.0 🚀🎙️”
APP_DESCRIPTION = “Speak English, choose your vibe, and get a Hinglish voice reply!”

def generate_plain(user_text, **kwargs):
try:
with model.disable_adapter():
return generate(user_text, **kwargs)
except Exception as e:
return f”(Plain model output unavailable: {e})”

def on_translate(text_input, audio_input, vibe):
final_text = “”

# 1. Try to transcribe the audio first
if audio_input is not None:
    try:
        transcription = transcriber(audio_input)["text"].strip()
        if transcription:
            final_text = transcription
    except Exception as e:
        print(f"Transcription error: {e}")

# 2. If audio was empty (too quiet) or failed, fall back to the text box
if not final_text and text_input and text_input.strip():
    final_text = text_input.strip()

# 3. If BOTH are empty, throw a helpful error
if not final_text:
    raise gr.Error("I couldn't hear any speech! Please try speaking louder, or switch to the Text tab.")

# Dynamic Prompting based on the "Vibe"
if vibe == "Gen-Z Slang":
    prompt_style = "super casual, Gen-Z chat style with heavy slang"
elif vibe == "Polite & Respectful":
    prompt_style = "polite, respectful chat style using 'aap'"
else:
    prompt_style = "casual romanized Hindi"

prompt = f"Rewrite this in {prompt_style}, as people write in chat messages.\n\nText: {final_text}"

# Generate text outputs
baseline_text = generate_plain(prompt, max_new_tokens=128)
tuned_text = generate(prompt, max_new_tokens=128)

# Text-to-Speech Magic (Audio Generation)
audio_path = "hinglish_audio.mp3"
try:
    tts = gTTS(text=tuned_text, lang='en', tld='co.in')
    tts.save(audio_path)
except:
    audio_path = None

return final_text, baseline_text, tuned_text, audio_path

Upgraded Premium UI

with gr.Blocks() as demo:
gr.Markdown(f”{APP_TITLE}”)
gr.Markdown(f”{APP_DESCRIPTION}”)

with gr.Row():
    with gr.Column(scale=1):
        gr.Markdown("### ⚙️ Input & Settings")

        with gr.Tabs():
            with gr.TabItem("⌨️ Text"):
                input_box = gr.Textbox(label="English Input", lines=3, placeholder="Type an English sentence here...")
            with gr.TabItem("🎙️ Voice"):
                audio_in = gr.Audio(sources=["microphone"], type="filepath", label="Speak clearly into your mic")

        vibe_radio = gr.Radio(
            choices=["Standard Chat", "Gen-Z Slang", "Polite & Respectful"],
            value="Standard Chat",
            label="Choose the Vibe"
        )
        translate_btn = gr.Button("🚀 Translate & Compare", variant="primary")

    with gr.Column(scale=2):
        gr.Markdown("### 📊 Results")
        with gr.Group():
            tuned_out = gr.Textbox(label="✨ Fine-Tuned Gemma (Hinglish)", lines=3)
            audio_out = gr.Audio(label="🔊 Hear the Hinglish Translation", type="filepath", interactive=False)

        with gr.Accordion("See Baseline Model Failure (For Judges)", open=False):
            plain_out = gr.Textbox(label="❌ Plain Gemma (Fails at code-mixing)", lines=3)

translate_btn.click(
    fn=on_translate,
    inputs=[input_box, audio_in, vibe_radio],
    outputs=[input_box, plain_out, tuned_out, audio_out]
)

demo.launch(share=True, theme=gr.themes.Soft(primary_hue=“orange”))

Closing server running on port: 7860
Loading Voice Recognition model (whisper-small)…
Colab notebook detected. To show errors in colab notebook, set debug=True in launch()

• Running on public URL: https://ff937ae3cdff12d650.gradio.live

This share link is temporary and will last for up to 1 week (best effort). For free permanent hosting and GPU upgrades, run gradio deploy from the terminal in the working directory to deploy to Hugging Face Spaces (https://huggingface.co/spaces)
Step 7 (optional): Save the fine-tuned adapters

The LoRA adapters are small (tens of megabytes). Save them before the Colab session ends if you want to reuse the model.

model.save_pretrained(“gemma_4_lora”)
tokenizer.save_pretrained(“gemma_4_lora”)
!zip -qr gemma_4_lora.zip gemma_4_lora
print(“Saved. Download gemma_4_lora.zip from the folder icon in the left sidebar.”)

Saved. Download gemma_4_lora.zip from the folder icon in the left sidebar.
Adapting for images (Problem 3B)

Fine-tuning on receipt images uses Unsloth’s vision workflow. The steps are the same (baseline, fine-tune, compare, build), but the model loading, the data format and the trainer settings change. The code below is based on Unsloth’s official Gemma 4 E2B vision notebook: https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma4_(E2B)-Vision.ipynb

1. Load the model for images (instead of Step 1):

from unsloth import FastVisionModel
model, processor = FastVisionModel.from_pretrained(
“unsloth/gemma-4-E2B-it”,
load_in_4bit = True, # keeps memory within a free T4
use_gradient_checkpointing = “unsloth”,
)
2. Load the receipts dataset and format each example as a conversation with an image:

import json
from datasets import load_dataset
train_data = load_dataset(“jsdnrs/ICDAR2019-SROIE”, split=“train”)
test_data = load_dataset(“jsdnrs/ICDAR2019-SROIE”, split=“test”)

INSTRUCTION = “Extract the company, date, address and total from this receipt. Reply with JSON only.”

def to_conversation(sample):
target = json.dumps({k: sample[“entities”][k] for k in [“company”, “date”, “address”, “total”]})
return {“messages”: [
{“role”: “user”, “content”: [{“type”: “text”, “text”: INSTRUCTION},
{“type”: “image”, “image”: sample[“image”]}]},
{“role”: “assistant”, “content”: [{“type”: “text”, “text”: target}]},
]}

train_conversations = [to_conversation(s) for s in train_data]
3. Generate from an image (use this for the baseline and for the fine-tuned evaluation):

def generate_from_image(image, instruction=INSTRUCTION, max_new_tokens=256):
messages = [{“role”: “user”, “content”: [{“type”: “image”}, {“type”: “text”, “text”: instruction}]}]
input_text = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(image.convert(“RGB”), input_text, add_special_tokens=False, return_tensors=“pt”).to(“cuda”)
outputs = model.generate(**inputs, max_new_tokens=max_new_tokens, use_cache=True, do_sample=False)
return processor.decode(outputs[0][inputs[“input_ids”].shape[1]:], skip_special_tokens=True).strip()
4. Add LoRA adapters and train (instead of Step 4):

model = FastVisionModel.get_peft_model(
model,
finetune_vision_layers = True, finetune_language_layers = True,
finetune_attention_modules = True, finetune_mlp_modules = True,
r = 16, lora_alpha = 16, lora_dropout = 0, bias = “none”, random_state = 3407,
)

from unsloth.trainer import UnslothVisionDataCollator
from trl import SFTTrainer, SFTConfig
trainer = SFTTrainer(
model = model,
train_dataset = train_conversations,
processing_class = processor.tokenizer,
data_collator = UnslothVisionDataCollator(model, processor),
args = SFTConfig(
per_device_train_batch_size = 1, gradient_accumulation_steps = 4,
max_grad_norm = 0.3, warmup_ratio = 0.03, max_steps = 60,
learning_rate = 2e-4, logging_steps = 5, optim = “adamw_8bit”,
weight_decay = 0.001, lr_scheduler_type = “cosine”, seed = 3407,
output_dir = “outputs”, report_to = “none”,
# required for vision fine-tuning:
remove_unused_columns = False, dataset_text_field = “”,
dataset_kwargs = {“skip_prepare_dataset”: True}, max_length = 2048,
),
)
trainer.train()
5. Score each field: parse the JSON reply with parse_json() and compare each of the four fields with the annotation (for example ignoring case and extra spaces). Report the accuracy for each field for the plain and fine-tuned models. Use a subset of the 361 test receipts if GPU time is limited.

Next steps

Re-read your selected problem statement (3A or 3B) in the Theme 3 document.
Prepare a training set and a separate test set.
Run the baseline evaluation, fine-tune, and compare. Record the settings used for every run.
Build the application around your fine-tuned model.
Attempt items from the Good to have section of the problem statement for additional credit.
Note on fair comparison: the plain model receives the same short instruction as the fine-tuned model. A stronger baseline gives the plain model a detailed prompt with the categories and format. Comparing against both baselines is listed under “Good to have”.

Troubleshooting

Issue Resolution
“No GPU found” Select Runtime → Change runtime type → T4 GPU, then run all cells from the top.
CUDA out of memory Select Runtime → Restart session and run the cells again. Reduce MAX_SEQ_LENGTH or max_new_tokens, or use the E2B model.
NameError: name ‘…’ is not defined A cell was skipped. Run all cells above in order.
The first answer is slow Expected. The model is compiled on first use. Later answers are faster.
Installation errors Select Runtime → Disconnect and delete runtime, then run the notebook again from the top.
GPU usage limit reached Colab limits free GPU time per account. Continue with another team member’s Google account.
The public link no longer works Links expire when the Colab session disconnects. Re-run the launch cell to create a new link.