Fun Sunday exercise, how much useful information I can squeeze in a tiny AI model. My goal was to take a general-purpose language model and turn it into an expert on a topic vital such as agriculture.
Here are my steps:
1. The Foundation: Google’s Gemma 270M
I started with this small but powerful but compact base model, gemma-3-270m-it. At just 550 MB, it’s a brilliant piece of engineering that can run on consumer-grade hardware.I am using my laptop.
2. The Technique: Parameter-Efficient Fine-Tuning (PEFT) with LoRA
Instead of retraining the entire model (which is slow and resource-intensive), I used a technique called LoRA. Think of it like adding a small, highly specialized “expert module” to the model’s existing brain. The original model’s knowledge remains, but we efficiently “teach” it a new skill, in this case agricultural information.
3. The Curriculum: The Agriculture Q&A Dataset
I used the KisanVaani/agriculture-qa dataset to teach the model the nuances of farming, crops, pests, and soil.
4. The Result
After a 15m training session, the new “expert module” I created was only 45 MB! That’s right. For just 45 MB, I layered deep agricultural knowledge onto a powerful base model. This process has created a specialized AI assistant that is more accurate and relevant for agricultural queries than the original.
Model output:
— Loading Model and Tokenizer —
Model and tokenizer loaded successfully.
Dataset loaded successfully from ‘/home/abdoulaye/aiplayground/agriculture_qa_dataset’.
Device set to use cuda:0
— Testing Base Model Performance —
— Test Question 1 —
Question:
which maize disease survive well in warm and humid weather.
Original Answer:
Gray leaf spot
Generated Answer (Base Model):
The maize disease, also known as the maize blight, is a fungal disease that can affect maize plants, particularly in areas with high humidity and high temperatures. It’s a common problem in many parts of the world, and it can be difficult to control.
————————————–
— Test Question 2 —
Question:
how can overuse of pesticides destroy soil nutrients?
Original Answer:
Pesticides can kill beneficial soil microorganisms and reduce soil biodiversity, which can lead to nutrient depletion and reduced soil fertility.
Generated Answer (Base Model):
Overuse of pesticides can be a serious threat to soil nutrients, which are essential for plant growth, soil health, and overall ecosystem function. Here are some ways pesticides can negatively impact soil nutrients:
* **Reduced nutrient availability:** Pesticides can disrupt the natural nutrient cycle, leading to nutrient deficiencies and reduced plant growth.
* **Soil degradation:** Pesticides can cause soil erosion, compaction, and altered soil structure, weakening the soil’s ability to retain nutrients.
* **Reduced plant health:** Pesticides can suppress plant growth, leading to stunted development, reduced yields, and increased susceptibility to disease.
* **Soil contamination:** Pesticides can contaminate soil with harmful chemicals, which can harm soil microorganisms, leading to soil degradation and reduced nutrient availability.
* **Impact on plant physiology:** Pesticides can affect plant physiology, including nutrient uptake, metabolism, and stress tolerance.
* **Altered soil pH:** Pesticides can alter soil pH, which can affect the availability of essential nutrients.
This quick experiment shows small AI models can provide practical solutions. By using efficient models like Gemma and smart techniques like LoRA, we can build tools that understand various local contexts.
The power to build specialized AI is here, and I’m excited to see what people will build in my region.
For those interested in the technical details, I used the Hugging Face Transformers library to handle the model and the PEFT library’s implementation of LoRA for efficient training. You can learn more about them at the links below:
* For the Hugging Face `transformers` library: This is the main documentation, the central hub for everything related to the library.
* https://huggingface.co/docs/transformers (https://huggingface.co/docs/transformers)
* For LoRA and PEFT (Parameter-Efficient Fine-Tuning): This link goes directly to the Hugging Face documentation for the peft library, which is what you used to implement LoRA.
* https://huggingface.co/docs/peft/conceptual_guides/lora (https://huggingface.co/docs/peft/conceptual_guides/lora)
#AI #LLM #FineTuning #Gemma #PEFT #LoRA #DemocratizeAI #AIforGood #TechInAfrica #GhanaTech #NLP #MachineLearning