commiting changes with clean history

.github/workflows/sync_to_hub.yml

@@ -0,0 +1,19 @@
+name: Sync to Hugging Face Hub
+on:
+  push:
+    branches: [main]
+
+  workflow_dispatch:
+
+jobs:
+  sync-to-hub:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+        with:
+          fetch-depth: 0
+          lfs: true
+      - name: Push to hub
+        env:
+          HF_TOKEN: ${{ secrets.HUGGINGFACE_SYNC_TOKEN }}
+        run: git push https://schmuelling:[email protected]/spaces/schmuelling/hopsworks_chat main

.gitignore

@@ -0,0 +1,9 @@
+*.gguf
+*.pdf
+*.pyc
+__pycache__/
+.env
+.ipynb_checkpoints/
+venv/
+.DS_Store
+.content

.gradio/certificate.pem

@@ -0,0 +1,31 @@
+-----BEGIN CERTIFICATE-----
+MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAw
+TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
+cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4
+WhcNMzUwNjA0MTEwNDM4WjBPMQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJu
+ZXQgU2VjdXJpdHkgUmVzZWFyY2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBY
+MTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAK3oJHP0FDfzm54rVygc
+h77ct984kIxuPOZXoHj3dcKi/vVqbvYATyjb3miGbESTtrFj/RQSa78f0uoxmyF+
+0TM8ukj13Xnfs7j/EvEhmkvBioZxaUpmZmyPfjxwv60pIgbz5MDmgK7iS4+3mX6U
+A5/TR5d8mUgjU+g4rk8Kb4Mu0UlXjIB0ttov0DiNewNwIRt18jA8+o+u3dpjq+sW
+T8KOEUt+zwvo/7V3LvSye0rgTBIlDHCNAymg4VMk7BPZ7hm/ELNKjD+Jo2FR3qyH
+B5T0Y3HsLuJvW5iB4YlcNHlsdu87kGJ55tukmi8mxdAQ4Q7e2RCOFvu396j3x+UC
+B5iPNgiV5+I3lg02dZ77DnKxHZu8A/lJBdiB3QW0KtZB6awBdpUKD9jf1b0SHzUv
+KBds0pjBqAlkd25HN7rOrFleaJ1/ctaJxQZBKT5ZPt0m9STJEadao0xAH0ahmbWn
+OlFuhjuefXKnEgV4We0+UXgVCwOPjdAvBbI+e0ocS3MFEvzG6uBQE3xDk3SzynTn
+jh8BCNAw1FtxNrQHusEwMFxIt4I7mKZ9YIqioymCzLq9gwQbooMDQaHWBfEbwrbw
+qHyGO0aoSCqI3Haadr8faqU9GY/rOPNk3sgrDQoo//fb4hVC1CLQJ13hef4Y53CI
+rU7m2Ys6xt0nUW7/vGT1M0NPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNV
+HRMBAf8EBTADAQH/MB0GA1UdDgQWBBR5tFnme7bl5AFzgAiIyBpY9umbbjANBgkq
+hkiG9w0BAQsFAAOCAgEAVR9YqbyyqFDQDLHYGmkgJykIrGF1XIpu+ILlaS/V9lZL
+ubhzEFnTIZd+50xx+7LSYK05qAvqFyFWhfFQDlnrzuBZ6brJFe+GnY+EgPbk6ZGQ
+3BebYhtF8GaV0nxvwuo77x/Py9auJ/GpsMiu/X1+mvoiBOv/2X/qkSsisRcOj/KK
+NFtY2PwByVS5uCbMiogziUwthDyC3+6WVwW6LLv3xLfHTjuCvjHIInNzktHCgKQ5
+ORAzI4JMPJ+GslWYHb4phowim57iaztXOoJwTdwJx4nLCgdNbOhdjsnvzqvHu7Ur
+TkXWStAmzOVyyghqpZXjFaH3pO3JLF+l+/+sKAIuvtd7u+Nxe5AW0wdeRlN8NwdC
+jNPElpzVmbUq4JUagEiuTDkHzsxHpFKVK7q4+63SM1N95R1NbdWhscdCb+ZAJzVc
+oyi3B43njTOQ5yOf+1CceWxG1bQVs5ZufpsMljq4Ui0/1lvh+wjChP4kqKOJ2qxq
+4RgqsahDYVvTH9w7jXbyLeiNdd8XM2w9U/t7y0Ff/9yi0GE44Za4rF2LN9d11TPA
+mRGunUHBcnWEvgJBQl9nJEiU0Zsnvgc/ubhPgXRR4Xq37Z0j4r7g1SgEEzwxA57d
+emyPxgcYxn/eR44/KJ4EBs+lVDR3veyJm+kXQ99b21/+jh5Xos1AnX5iItreGCc=
+-----END CERTIFICATE-----

LAB_DESCRIPTION.MD

@@ -0,0 +1,335 @@
+# Lab 2: Parameter Efficient Fine-Tuning (PEFT) of Large Language Models
+
+**Course:** ID2223 / HT2025  
+**Students:** Sebastian Schmuelling, Ramin Darudi  
+
+---
+
+## Overview
+
+This project implements Parameter Efficient Fine-Tuning (PEFT) using LoRA (Low-Rank Adaptation) to fine-tune large language models on the FineTome-100k instruction dataset. The fine-tuned models are deployed in a Retrieval-Augmented Generation (RAG) chatbot interface that enables users to query documents indexed in Hopsworks Feature Store.
+
+### Key Features
+
+- **PEFT Fine-Tuning**: Efficient fine-tuning using LoRA with 4-bit quantization
+- **Checkpoint Management**: Automatic checkpointing to HuggingFace Hub for resumable training
+- **Multiple Model Support**: Fine-tuned Llama-3.2-1B and Ministral-3-3B models
+- **RAG System**: Document retrieval using Hopsworks Feature Store and FAISS
+- **CPU-Optimized Inference**: GGUF format models for efficient CPU deployment
+- **Interactive UI**: Gradio-based chatbot with dynamic model selection
+
+---
+
+## Task 1: Fine-Tune a Model and Build a UI
+
+### 1.1 Fine-Tuning Implementation
+
+#### Models Fine-Tuned
+
+1. **Llama-3.2-1B-Instruct**
+   - Base Model: `unsloth/Llama-3.2-1B-Instruct`
+   - Fine-tuned Model: `schmuelling/Llama-3.2-1B-Instruct-finetome`
+   - Training Time: 30.75 minutes on A100 GPU
+   - Peak Memory: 4.477 GB (11.3% of 40GB GPU)
+   - Trainable Parameters: 11,272,192 (0.90% of total parameters)
+
+2. **Ministral-3-3B-Instruct-2512**
+   - Base Model: `unsloth/Ministral-3-3B-Instruct-2512`
+   - Fine-tuned Model: `schmuelling/Ministral-3-3B-Instruct-2512-finetome`
+   - Training Time: 429.2 minutes (~7.2 hours) on A100 GPU
+   - Peak Memory: 6.896 GB (8.7% of 80GB GPU)
+   - Trainable Parameters: 24,707,072 (0.64% of total parameters)
+
+#### Fine-Tuning Process
+
+**Dataset**: FineTome-100k (`mlabonne/FineTome-100k`)
+- 100,000 instruction-following examples
+- Converted from ShareGPT format to HuggingFace chat format using `standardize_sharegpt()`
+- Applied model-specific chat templates (llama-3.1 for Llama, mistral for Ministral)
+
+**Training Configuration**:
+- **Framework**: Unsloth for memory-efficient fine-tuning
+- **Quantization**: 4-bit (BitsAndBytesConfig with NF4 quantization and double quantization)
+- **LoRA Configuration**:
+  - Rank (r): 16
+  - LoRA Alpha: 16
+  - LoRA Dropout: 0
+  - Target Modules: `["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]`
+- **Training Parameters**:
+  - Max Sequence Length: 2048
+  - Batch Size: 2 per device
+  - Gradient Accumulation: 4 steps (effective batch size: 8)
+  - Learning Rate: 2e-4
+  - Learning Rate Scheduler: Linear decay
+  - Warmup Steps: 20
+  - Optimizer: AdamW 8-bit
+  - Weight Decay: 0.001
+  - Epochs: 1
+  - Max Steps: 2000 (Llama), 12,500 (Ministral)
+  - Mixed Precision: bfloat16 on Ampere+ GPUs
+  - Gradient Checkpointing: Enabled ("unsloth" mode)
+
+**Checkpointing Strategy**:
+- Checkpoints saved every 100 steps (Llama) / 1000 steps (Ministral)
+- Automatic push to HuggingFace Hub: `schmuelling/{model_name}-checkpoint`
+- Resume training capability: Automatically detects and loads from checkpoint if repository exists
+- Total checkpoint limit: 3 (oldest checkpoints automatically deleted)
+- Checkpoint strategy: "checkpoint" (push on every save)
+
+**Model Export**:
+- **GGUF Format**: Multiple quantization levels exported for CPU inference
+  - `q4_k_m`: 4-bit quantization (balanced quality/size)
+  - `q8_0`: 8-bit quantization (higher quality)
+  - `q2_k`: 2-bit quantization (smallest size)
+- **Merged 4-bit**: Merged LoRA weights into 4-bit base model for HuggingFace Transformers inference
+- All models pushed to HuggingFace Hub for deployment
+
+#### Training Infrastructure
+
+- **Platform**: Google Colab (free GPU tier)
+- **GPU**: NVIDIA A100 (40GB for Llama, 80GB for Ministral)
+- **Memory Efficiency**: 
+  - 4-bit quantization enabled for memory reduction
+  - LoRA adapters: 0.90% trainable parameters for Llama, 0.64% for Ministral
+  - Gradient checkpointing enabled for additional memory savings
+  - Peak memory usage: 4.477 GB (Llama), 6.896 GB (Ministral)
+
+### 1.2 RAG System Implementation
+
+The RAG (Retrieval-Augmented Generation) system enables the chatbot to answer questions based on indexed documents.
+
+**Document Indexing** (`index_content.ipynb`):
+- **Document Loader**: LangChain DoclingLoader for PDF processing
+- **Document**: "Building Machine Learning Systems with a Feature Store.pdf"
+- **Chunking Strategy**: HybridChunker with semantic chunking
+- **Embeddings Model**: `sentence-transformers/all-MiniLM-L6-v2` (384 dimensions)
+- **Storage**: Hopsworks Feature Store (`book_embeddings` feature group, version 1)
+- **Index**: FAISS IndexFlatIP (Inner Product with L2 normalization) for similarity search
+- **Indexed Chunks**: 1,333 document chunks
+
+**Retrieval Process**:
+1. User query is encoded into embedding vector using SentenceTransformer
+2. FAISS performs cosine similarity search (L2-normalized inner product)
+3. Top-k chunks retrieved (default: 10 chunks, configurable in `rag_prompt.yml`)
+4. Context assembled with separator (`\n\n`) and passed to LLM
+
+**RAG Prompt Template** (`prompts/rag_prompt.yml`):
+- System prompt: Defines assistant role for Hopsworks documentation
+- Context injection: Retrieved document chunks inserted into prompt
+- Generation parameters:
+  - max_tokens: 256
+  - temperature: 0.7
+  - stop_sequences: `["Question:", "\n\n"]`
+
+### 1.3 User Interface
+
+**Gradio Application** (`app.py`):
+- **Model Selection**: Dropdown menus for repository and model selection
+- **Dynamic Loading**: Models loaded on-demand from HuggingFace Hub using `llama-cpp-python`
+- **Chat Interface**: Streaming responses with conversation history using `gr.ChatInterface`
+- **Status Display**: Real-time feedback on model loading and operations
+- **Model Information**: Displays model description, repository, and file details
+
+**Features**:
+- Multiple model support from different repositories (configured in `models_config.json`)
+- CPU-optimized inference using `llama-cpp-python` with GGUF models
+- Streaming text generation for better UX
+- Example prompts for quick testing
+- Error handling and user-friendly messages
+- Automatic installation of `llama-cpp-python` at runtime
+
+**Deployment**:
+- Deployed to HuggingFace Spaces
+- Environment variables configured via Space secrets (`HOPSWORKS_API_KEY`)
+- Automatic model downloading on first load
+- Supports GGUF format models for CPU inference
+
+---
+
+## Task 2: Improve Pipeline Scalability and Model Performance
+
+### 2.1 Model-Centric Improvements
+
+#### Hyperparameter Configuration
+
+**Learning Rate Scheduling**:
+- Linear learning rate decay implemented
+- Warmup steps: 20 for stable training start
+- Learning rate: 2e-4 (standard for LoRA fine-tuning)
+
+**LoRA Configuration**:
+- **Rank (r=16)**: Selected for balance between model capacity and parameter efficiency
+- **Alpha (16)**: Set equal to rank for optimal scaling
+- **Target Modules**: Selected attention and MLP layers (`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`) for maximum impact
+- **Dropout**: 0 (optimized for Unsloth)
+
+**Training Efficiency Optimizations**:
+- **Gradient Accumulation (4)**: Effective batch size of 8 with minimal memory overhead
+- **Mixed Precision**: bfloat16 on Ampere+ GPUs (automatic detection)
+- **Optimizer**: AdamW 8-bit for memory efficiency
+- **Gradient Checkpointing**: "unsloth" mode for memory savings
+
+#### Architecture Choices
+
+**Quantization Strategy**:
+- **4-bit Quantization**: NF4 quantization with double quantization
+- **Benefits**: Significant memory reduction enabling training on free Colab GPUs
+- **Implementation**: BitsAndBytesConfig with `load_in_4bit=True`
+
+**Model Selection**:
+- Tested two model sizes: 1B (Llama) and 3B (Ministral) parameters
+- Both models fine-tuned successfully with same LoRA configuration
+- Models exported in multiple GGUF quantization levels for different use cases
+
+### 2.2 Data-Centric Improvements
+
+#### Dataset Used
+
+**FineTome-100k Dataset**:
+- Source: `mlabonne/FineTome-100k` from HuggingFace
+- Size: 100,000 instruction-following examples
+- Format: ShareGPT format (converted to HuggingFace standard format)
+- Quality: High-quality, diverse instruction-following examples
+- Processing: Standardized using `unsloth.chat_templates.standardize_sharegpt()`
+
+**Data Preprocessing**:
+- ShareGPT format conversion to HuggingFace chat format
+- Application of model-specific chat templates
+- Batched processing with `dataset.map()` for efficiency
+
+#### Evaluation Framework
+
+**Evaluation Script** (`evaluation/evaluate_models.py`):
+- **Perplexity Calculation**: Measures model's prediction confidence
+  - Lower perplexity = better model
+  - Calculated on held-out test set (50 examples from FineTome-100k)
+  - Compares base model vs fine-tuned model
+- **Memory Efficiency Tracking**: Prints model size and parameter counts
+- **Implementation**: Uses 4-bit quantization for both models during evaluation
+
+**Evaluation Setup**:
+- Base Model: `unsloth/Llama-3.2-1B-Instruct`
+- Fine-tuned Model: `schmuelling/Llama-3.2-1B-Instruct-finetome`
+- Test Set: 50 examples from FineTome-100k dataset
+- Metrics: The finetuned model displayed an average of 2.3% improvement over base instruct model over 10 evaluation runs.
+
+- Ministral-3 Model could not be evaluated due to how new it is and not being part of the transformer library, something we had not taken into account. 
+
+### 2.3 Pipeline Scalability Improvements
+
+#### Training Scalability
+
+**Checkpoint Management**:
+- Automatic checkpointing to HuggingFace Hub every 100/1000 steps
+- Resume from checkpoint capability (automatic detection)
+- Checkpoint versioning with limit of 3 checkpoints
+
+**Model Versioning**:
+- Versioned models on HuggingFace Hub
+- Multiple quantization formats for different deployment scenarios
+- Separate checkpoint and final model repositories
+
+#### Inference Scalability
+
+**Model Optimization**:
+- GGUF quantization for CPU inference (q2_k, q4_k_m, q8_0)
+- Multiple quantization levels for quality/speed trade-offs
+- Merged 4-bit model for HuggingFace Transformers inference
+- CPU-optimized inference using `llama-cpp-python`
+
+**Model Loading**:
+- Dynamic model loading from HuggingFace Hub
+- On-demand downloading (models not stored in Space)
+- Support for multiple model repositories via configuration
+
+#### RAG System Scalability
+
+**Index Implementation**:
+- FAISS IndexFlatIP for exact similarity search
+- L2 normalization for cosine similarity
+- Efficient retrieval with configurable top-k
+
+**Embedding System**:
+- SentenceTransformer embeddings (`all-MiniLM-L6-v2`, 384 dimensions)
+- Stored in Hopsworks Feature Store for persistence
+- FAISS index built in-memory for fast retrieval
+
+**Retrieval Configuration**:
+- Configurable number of retrieved chunks (default: 10)
+- Configurable context separator
+- Real-time retrieval and context assembly
+
+---
+
+## Technical Architecture
+
+### System Components
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                    Fine-Tuning Pipeline                      │
+├─────────────────────────────────────────────────────────────┤
+│ 1. Load Base Model (4-bit quantized)                        │
+│ 2. Add LoRA Adapters (r=16, alpha=16)                      │
+│ 3. Load FineTome-100k Dataset                               │
+│ 4. Train with Checkpointing (every 100/1000 steps)          │
+│ 5. Export to GGUF Format (q2_k, q4_k_m, q8_0)              │
+│ 6. Push to HuggingFace Hub                                  │
+└─────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────┐
+│                    RAG System                                │
+├─────────────────────────────────────────────────────────────┤
+│ 1. Document Indexing (DoclingLoader)                        │
+│ 2. Embedding Generation (SentenceTransformers)              │
+│ 3. Storage (Hopsworks Feature Store)                        │
+│ 4. FAISS Index for Retrieval (IndexFlatIP)                 │
+└─────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────┐
+│                    Inference Pipeline                        │
+├─────────────────────────────────────────────────────────────┤
+│ 1. User Query                                                │
+│ 2. Query Embedding (SentenceTransformer)                    │
+│ 3. FAISS Retrieval (Top-k chunks)                           │
+│ 4. Context Assembly                                          │
+│ 5. LLM Generation (GGUF model via llama-cpp-python)         │
+│ 6. Stream Response to User                                  │
+└─────────────────────────────────────────────────────────────┘
+```
+
+### File Structure
+
+```
+rag_finetune_LLM/
+├── app.py                          # Gradio UI application
+├── models_config.json              # Model configuration
+├── prompts/
+│   └── rag_prompt.yml             # RAG prompt template
+├── finetuning/
+│   ├── Finetune_notebook_Llama.ipynb      # Llama fine-tuning
+│   └── Finetune_notebook_ministral.ipynb  # Ministral fine-tuning
+├── evaluation/
+│   └── evaluate_models.py         # Model evaluation script
+├── index_content.ipynb            # Document indexing notebook
+├── requirements.txt               # Python dependencies
+├── README.md                      # HuggingFace Space config
+├── README_SETUP.md                # Setup instructions
+└── LAB_DESCRIPTION.md             # This file
+```
+
+---
+
+## Conclusion
+
+This project successfully demonstrates Parameter Efficient Fine-Tuning (PEFT) using LoRA on large language models, achieving memory and computational savings while maintaining model quality. The implementation includes:
+
+- Efficient fine-tuning with checkpointing and resume capability
+- Multiple model support (Llama and Ministral)
+- RAG system with Hopsworks Feature Store integration
+- Production-ready UI deployed on HuggingFace Spaces
+- Comprehensive documentation and evaluation framework
+
+---
+
+**Last Updated**: December 2025  

README.md

@@ -0,0 +1,11 @@
+---
+title: Hopsworks RAG ChatBot
+emoji: 🤖
+colorFrom: blue
+colorTo: purple
+sdk: gradio
+sdk_version: 4.44.1
+app_file: app.py
+python_version: "3.10"
+pinned: false
+---

README_SETUP.md

@@ -0,0 +1,248 @@
+# Setup and Deployment Guide
+
+This guide walks you through setting up and deploying the Hopsworks RAG ChatBot to HuggingFace Spaces.
+
+## Table of Contents
+1. [Prerequisites](#prerequisites)
+2. [Local Setup](#local-setup)
+3. [Indexing Documents](#indexing-documents)
+4. [Configuring Models](#configuring-models)
+5. [Deploying to HuggingFace Spaces](#deploying-to-huggingface-spaces)
+6. [Syncing with GitHub](#syncing-with-github)
+7. [Testing](#testing)
+8. [Troubleshooting](#troubleshooting)
+
+---
+
+## Prerequisites
+
+Before you begin, ensure you have:
+
+- **Python 3.10** installed locally
+- **Git** installed
+- **Hopsworks Account**: Sign up at [hopsworks.ai](https://www.hopsworks.ai/)
+- **HuggingFace Account**: Sign up at [huggingface.co](https://huggingface.co/)
+- **PDF Documents** you want to index for RAG
+
+---
+
+## Local Setup
+
+### 1. Clone the Repository
+
+```bash
+git clone <your-repo-url>
+cd rag_finetune_LLM
+```
+
+### 2. Create Virtual Environment
+
+```bash
+python3.10 -m venv venv
+source venv/bin/activate  # On Windows: venv\Scripts\activate
+```
+
+### 3. Install Dependencies
+
+```bash
+pip install -r requirements.txt
+```
+
+### 4. Configure Environment Variables
+
+Create a `.env` file in the root directory:
+
+```bash
+# .env
+HOPSWORKS_API_KEY=your_hopsworks_api_key_here
+```
+
+**Get your Hopsworks API Key:**
+1. Go to [Hopsworks](https://www.hopsworks.ai/)
+2. Navigate to your project
+3. Click on your profile → Settings → API Keys
+4. Create a new API key and copy it
+
+---
+
+## Indexing Documents
+
+### 1. Add Your PDF Document
+
+Place your PDF file in the project directory (e.g., `content/your_content.pdf`)
+
+### 2. Update the Indexing Notebook
+
+Open `index_content.ipynb` and update the PDF path:
+
+```python
+PDF_PATH = "content/your_content.pdf"  # Update this
+```
+
+### 3. Run the Notebook
+
+Execute all cells in `index_content.ipynb`:
+
+```bash
+jupyter notebook index_content.ipynb
+```
+
+This will:
+- Load and chunk your PDF using Docling
+- Generate embeddings with sentence-transformers
+- Upload to Hopsworks Feature Store as `content` feature group
+
+**Note:** This only needs to be done once. The embeddings will be available for all deployments.
+
+---
+
+## Configuring Models
+
+### 1. Edit Model Configuration
+
+Update `models_config.json` with your models.
+
+
+
+### 2. Model Format Requirements
+
+- Models should be in **GGUF format** (for CPU-optimized inference unless you have GPUs)
+- Hosted on HuggingFace Hub
+
+---
+
+## Deploying to HuggingFace Spaces
+
+### Method 1: Direct Git Push (Recommended)
+
+#### 1. Create a New Space
+
+1. Go to [HuggingFace Spaces](https://huggingface.co/spaces)
+2. Click **"Create new Space"**
+3. Configure:
+   - **Name**: `your-rag-chatbot`
+   - **SDK**: Gradio
+   - **Hardware**: CPU basic (free tier works fine)
+   - **Visibility**: Public or Private
+
+#### 2. Get Your HuggingFace Token
+
+1. Go to [HuggingFace Settings → Tokens](https://huggingface.co/settings/tokens)
+2. Click **"New token"**
+3. Give it a name (e.g., "spaces-deploy")
+4. Select **Write** permission
+5. Copy the token
+
+#### 3. Connect Your Repository
+
+```bash
+# Add HuggingFace Space as remote
+```bash
+git remote add space https://YOUR_USERNAME:[email protected]/spaces/your-username/your-rag-chatbot
+```
+
+#### 4. Configure Secrets
+
+In your Space settings on HuggingFace:
+
+1. Go to **Settings** → **Repository secrets**
+2. Add the following secret:
+   - **Name**: `HOPSWORKS_API_KEY`
+   - **Value**: Your Hopsworks API key
+
+#### 5. Wait for Build
+
+The Space will automatically build and deploy. This may take a couple of minutes.
+
+---
+
+### Method 2: GitHub Sync (Automatic)
+
+#### 1. Enable GitHub Actions
+
+The repository includes `.github/workflows/sync_to_huggingface.yaml` for automatic syncing.
+
+#### 2. Add GitHub Secrets
+
+In your GitHub repository:
+
+1. Go to **Settings** → **Secrets and variables** → **Actions**
+2. Add:
+   - **Name**: `HF_TOKEN`
+   - **Value**: Your HuggingFace write token
+
+**Get your HuggingFace Token:**
+1. Go to [HuggingFace Settings → Tokens](https://huggingface.co/settings/tokens)
+2. Create a new token with **write** permissions
+3. Copy the token
+
+#### 3. Update Workflow File (if needed)
+
+Edit `.github/workflows/sync_to_huggingface.yaml` and update:
+
+```yaml
+env:
+          HF_TOKEN: ${{ secrets.HUGGINGFACE_SYNC_TOKEN }} #leave this
+          HF_SPACE_URL: https://huggingface.co/spaces/your-username/your-space-name
+```
+
+#### 4. Automatic Syncing
+
+Now, every push to your `main` branch will automatically sync to HuggingFace Spaces!
+
+```bash
+git add .
+git commit -m "Update model configuration"
+git push origin main  # Automatically syncs to HF Spaces
+```
+
+---
+
+## Testing
+
+### Local Testing
+
+Before deploying, test locally:
+
+```bash
+python app.py
+```
+
+This will:
+1. Install llama-cpp-python at runtime
+2. Connect to Hopsworks and load embeddings
+3. Launch Gradio interface at a local host (exact url can be found in the command line)
+
+### Testing on HuggingFace Spaces
+
+1. Open your Space URL: `https://huggingface.co/spaces/your-username/your-space-name`
+2. Select a model from the dropdown
+3. Click **"Load Model"** (wait 1-3 minutes for first load)
+4. Once loaded, ask a question related to your documents
+5. Verify the response uses context from your indexed documents
+
+---
+
+## Configuration Reference
+
+### README.md (Space Configuration)
+
+### models_config.json
+
+Defines available models in the dropdown:
+
+```json
+{
+  "models": [
+    {
+      "name": "Display Name",           // Shown in dropdown
+      "repo_id": "username/repo",       // HuggingFace model repository
+      "filename": "model.gguf",         // GGUF file in the repo
+      "description": "Model description" // Shown in UI
+    }
+  ]
+}
+```
+
+
+**Happy Deploying! 🚀**

app.py

@@ -0,0 +1,233 @@
+import subprocess
+subprocess.run("pip install llama-cpp-python==0.3.15", shell=True, check=True)
+
+import gradio as gr
+import hopsworks
+from sentence_transformers import SentenceTransformer
+from llama_cpp import Llama
+import faiss
+import numpy as np
+import os
+import json
+import yaml
+from dotenv import load_dotenv
+
+# 1. Load Environment Variables & Validation
+load_dotenv()
+
+HOPSWORKS_API_KEY = os.getenv("HOPSWORKS_API_KEY")
+
+if not HOPSWORKS_API_KEY:
+    raise ValueError("HOPSWORKS_API_KEY not found in environment variables.")
+
+# Load models configuration
+with open("models_config.json", "r") as f:
+    models_config = json.load(f)
+
+# Load RAG prompt configuration
+with open("prompts/rag_prompt.yml", "r") as f:
+    prompt_config = yaml.safe_load(f)
+
+# Global variable to store the current LLM
+llm = None
+
+print("Initializing embeddings and connecting to Hopsworks...")
+# 
+try:
+    embeddings = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
+
+    project = hopsworks.login(api_key_value=HOPSWORKS_API_KEY)
+    fs = project.get_feature_store()
+    book_fg = fs.get_feature_group("book_embeddings", version=1)
+
+    df = book_fg.read()
+    
+    if df.empty:
+        raise ValueError("Feature group 'book_embeddings' is empty.")
+
+    texts = df['text'].tolist()
+    raw_embeddings = [emb if isinstance(emb, list) else emb.tolist() for emb in df['embedding']]
+    embedding_vectors = np.array(raw_embeddings, dtype='float32')
+
+    dimension = embedding_vectors.shape[1]
+    index = faiss.IndexFlatIP(dimension)
+    
+    faiss.normalize_L2(embedding_vectors)
+    index.add(embedding_vectors)
+
+    print("Embeddings and FAISS index initialized.")
+
+except Exception as e:
+    print(f"Critical Error during initialization: {e}")
+    index = None
+
+# Function to load a model dynamically
+def load_model(repo_name, model_name, progress=gr.Progress()):
+    global llm
+    try:
+        progress(0, desc="Initializing...")
+
+        # Find the repository
+        repo = next((r for r in models_config["repositories"] if r["name"] == repo_name), None)
+        if not repo:
+            return f"Error: Repository '{repo_name}' not found in config."
+
+        # Find the model within the repository
+        model = next((m for m in repo["models"] if m["name"] == model_name), None)
+        if not model:
+            return f"Error: Model '{model_name}' not found in repository."
+
+        print(f"Loading model: {model['name']}...")
+        print(f"Repo: {repo['repo_id']}, File: {model['filename']}")
+
+        progress(0.3, desc=f"Downloading/Loading {model['name']}...")
+
+        llm = Llama.from_pretrained(
+            repo_id=repo["repo_id"],
+            filename=model["filename"],
+            n_ctx=2048,
+            n_threads=4,
+            n_gpu_layers=-1,
+            verbose=False
+        )
+
+        progress(1.0, desc="Complete!")
+        return f"✅ Model '{model_name}' loaded successfully!"
+
+    except Exception as e:
+        llm = None
+        return f"❌ Error loading model: {str(e)}"
+
+def retrieve_context(query, k=None):
+    if index is None:
+        return "Error: Search index not initialized."
+
+    # Use k from prompt config if not specified
+    if k is None:
+        k = prompt_config["rag"]["num_retrieved_chunks"]
+
+    query_embedding = embeddings.encode(query).astype('float32').reshape(1, -1)
+    faiss.normalize_L2(query_embedding)
+
+    distances, indices = index.search(query_embedding, k)
+
+    retrieved_texts = []
+    for i in indices[0]:
+        if 0 <= i < len(texts):
+            retrieved_texts.append(texts[i])
+
+    # Use separator from prompt config
+    separator = prompt_config["rag"]["context_separator"]
+
+    print(f"Retrieved {len(retrieved_texts)} context chunks for the query.")
+    print("Similarities:", distances)
+    return separator.join(retrieved_texts)
+
+def respond(message, history):
+    """
+    Generator function for streaming response.
+    gr.ChatInterface passes 'message' and 'history' automatically.
+    """
+    if llm is None:
+        yield "System Error: Models failed to load. Check console logs."
+        return
+
+    # Retrieve context using config settings
+    context = retrieve_context(message)
+
+    # Build prompt from template
+    prompt = prompt_config["template"].format(
+        context=context,
+        question=message
+    )
+
+    # Get generation parameters from config
+    gen_params = prompt_config["generation"]
+
+    output = llm(
+        prompt,
+        max_tokens=gen_params["max_tokens"],
+        temperature=gen_params["temperature"],
+        stop=gen_params["stop_sequences"],
+        stream=True
+    )
+
+    partial_message = ""
+    for chunk in output:
+        text_chunk = chunk["choices"][0]["text"]
+        partial_message += text_chunk
+        yield partial_message
+
+with gr.Blocks(title="Hopsworks RAG ChatBot") as demo:
+    gr.Markdown("<h1 style='text-align: center; color: #1EB382'>Hopsworks ChatBot</h1>")
+
+    # Model Selection Section
+    with gr.Row():
+        repo_dropdown = gr.Dropdown(
+            choices=[r["name"] for r in models_config["repositories"]],
+            label="Select Repository",
+            value=models_config["repositories"][0]["name"],
+            scale=2
+        )
+        model_dropdown = gr.Dropdown(
+            choices=[m["name"] for m in models_config["repositories"][0]["models"]],
+            label="Select Model",
+            value=models_config["repositories"][0]["models"][0]["name"],
+            scale=2
+        )
+        load_button = gr.Button("Load Model", variant="primary", scale=1)
+
+    status_box = gr.Textbox(
+        label="Status",
+        value="⚠️ Please select a repository and model, then click 'Load Model'",
+        interactive=False
+    )
+
+    # Model info display
+    model_info = gr.Markdown("")
+
+    # Chat Interface
+    chat_interface = gr.ChatInterface(
+        fn=respond,
+        chatbot=gr.Chatbot(height=400),
+        textbox=gr.Textbox(placeholder="Ask a question about your documents...", container=False, scale=7),
+        examples=["What is the main topic of the documents?", "Summarize the key points."],
+        cache_examples=False,
+    )
+
+    # Function to update model dropdown when repository changes
+    def update_model_choices(repo_name):
+        repo = next((r for r in models_config["repositories"] if r["name"] == repo_name), None)
+        if repo and repo["models"]:
+            model_choices = [m["name"] for m in repo["models"]]
+            return gr.Dropdown(choices=model_choices, value=model_choices[0])
+        return gr.Dropdown(choices=[], value=None)
+
+    # Function to update model info display
+    def update_model_info(repo_name, model_name):
+        repo = next((r for r in models_config["repositories"] if r["name"] == repo_name), None)
+        if not repo:
+            return ""
+
+        model = next((m for m in repo["models"] if m["name"] == model_name), None)
+        if model:
+            return f"**{model['name']}**\n\n{model['description']}\n\n Repository: `{repo['repo_id']}`\n\n File: `{model['filename']}`"
+        return ""
+
+    # Event handlers
+    repo_dropdown.change(update_model_choices, inputs=[repo_dropdown], outputs=[model_dropdown])
+    repo_dropdown.change(update_model_info, inputs=[repo_dropdown, model_dropdown], outputs=[model_info])
+    model_dropdown.change(update_model_info, inputs=[repo_dropdown, model_dropdown], outputs=[model_info])
+    load_button.click(load_model, inputs=[repo_dropdown, model_dropdown], outputs=[status_box])
+
+    # Load default model info on startup
+    demo.load(
+        lambda: update_model_info(
+            models_config["repositories"][0]["name"],
+            models_config["repositories"][0]["models"][0]["name"]
+        ),
+        outputs=[model_info]
+    )
+
+if __name__ == "__main__":
+    demo.launch(share=True)

evaluation/evaluate_models.py

@@ -0,0 +1,109 @@
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
+from datasets import load_dataset
+from tqdm import tqdm
+from datetime import datetime
+
+class ModelEvaluator:
+    def __init__(self, base_model_name, finetuned_model_name):
+        print("Loading models...")
+
+        # Tokenizer (identical for both models)
+        print("Loading tokenizer...")
+        self.tokenizer = AutoTokenizer.from_pretrained(finetuned_model_name)
+
+        # 4-bit config
+        bnb_config = BitsAndBytesConfig(
+            load_in_4bit=True,
+            bnb_4bit_use_double_quant=True,
+            bnb_4bit_quant_type="nf4",
+            bnb_4bit_compute_dtype=torch.bfloat16
+        )
+
+        # Base model
+        print("Loading base model in 4-bit...")
+        self.base_model = AutoModelForCausalLM.from_pretrained(
+            base_model_name,
+            device_map="auto",
+            quantization_config=bnb_config,
+            trust_remote_code=True
+        )
+        self.print_model_size(self.base_model, "Base Model")
+
+        # Finetuned model
+        print("Loading fine-tuned model in 4-bit...")
+        self.finetuned_model = AutoModelForCausalLM.from_pretrained(
+            finetuned_model_name,
+            device_map="auto",
+            quantization_config=bnb_config,
+            trust_remote_code=True
+        )
+        self.print_model_size(self.finetuned_model, "Fine-Tuned Model")
+
+        print("Models loaded successfully!\n")
+
+    def print_model_size(self, model, name: str):
+        total_params = sum(p.numel() for p in model.parameters())
+        trainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
+
+        # Estimate memory footprint
+        param_bytes = 0
+        for p in model.parameters():
+            if hasattr(p, "quant_state"):  # bitsandbytes quantized
+                param_bytes += p.numel() * 0.5  
+            else:
+                param_bytes += p.numel() * p.element_size()
+
+        size_mb = param_bytes / (1024**2)
+
+        print(f"\n[{name}]")
+        print(f"  • Total parameters:     {total_params:,}")
+        print(f"  • Trainable parameters: {trainable_params:,}")
+        print(f"  • Approx size:          {size_mb:.2f} MB\n")
+
+    def calculate_perplexity(self, model, texts):
+        model.eval()
+        total_loss = 0
+        total_tokens = 0
+
+        print("Calculating perplexity...")
+        for text in tqdm(texts):
+            enc = self.tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
+            input_ids = enc.input_ids.to(model.device)
+
+            with torch.no_grad():
+                outputs = model(input_ids, labels=input_ids)
+                loss = outputs.loss
+                total_loss += loss.item() * input_ids.size(1)
+                total_tokens += input_ids.size(1)
+
+        ppl = torch.exp(torch.tensor(total_loss / total_tokens))
+        return ppl.item()
+
+def main():
+    BASE_MODEL = "unsloth/Llama-3.2-1B-Instruct"
+    FINETUNED_MODEL = "schmuelling/Llama-3.2-1B-Instruct-finetome"
+
+    print("Loading dataset...")
+    dataset = load_dataset("mlabonne/FineTome-100k", split="train[:100]")
+
+    test_texts = [
+        item["conversations"][0]["value"]
+        for item in dataset
+        if len(item["conversations"]) > 0
+    ][:50]
+
+    evaluator = ModelEvaluator(BASE_MODEL, FINETUNED_MODEL)
+
+    print("\n=== PERPLEXITY EVALUATION ===")
+    base_ppl = evaluator.calculate_perplexity(evaluator.base_model, test_texts)
+    ft_ppl = evaluator.calculate_perplexity(evaluator.finetuned_model, test_texts)
+
+    improvement = ((base_ppl - ft_ppl) / base_ppl) * 100
+
+    print(f"\nBase Model Perplexity:       {base_ppl:.2f}")
+    print(f"Fine-Tuned Model Perplexity: {ft_ppl:.2f}")
+    print(f"Improvement:                 {improvement:.2f}%")
+
+if __name__ == "__main__":
+    main()

index_content.ipynb

@@ -0,0 +1,335 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/opt/anaconda3/envs/rag_llm/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
+      "  from .autonotebook import tqdm as notebook_tqdm\n"
+     ]
+    }
+   ],
+   "source": [
+    "import os\n",
+    "import hopsworks\n",
+    "from sentence_transformers import SentenceTransformer\n",
+    "import numpy as np\n",
+    "import pandas as pd\n",
+    "from langchain_docling import DoclingLoader\n",
+    "from langchain_docling.loader import ExportType\n",
+    "from docling.chunking import HybridChunker\n",
+    "\n",
+    "os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "PDF_PATH = \"content/Building+Machine+Learning+Systems+with+a+Feature+Store.pdf\"\n",
+    "EMBED_MODEL_ID = \"sentence-transformers/all-MiniLM-L6-v2\"\n",
+    "EXPORT_TYPE = ExportType.DOC_CHUNKS"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "2025-12-02 19:43:33,611 INFO: detected formats: [<InputFormat.PDF: 'pdf'>]\n",
+      "2025-12-02 19:43:33,861 INFO: Going to convert document batch...\n",
+      "2025-12-02 19:43:33,863 INFO: Initializing pipeline for StandardPdfPipeline with options hash e15bc6f248154cc62f8db15ef18a8ab7\n",
+      "2025-12-02 19:43:33,913 WARNING: The plugin langchain_docling will not be loaded because Docling is being executed with allow_external_plugins=false.\n",
+      "2025-12-02 19:43:33,914 INFO: Loading plugin 'docling_defaults'\n",
+      "2025-12-02 19:43:33,926 INFO: Registered picture descriptions: ['vlm', 'api']\n",
+      "2025-12-02 19:43:33,981 WARNING: The plugin langchain_docling will not be loaded because Docling is being executed with allow_external_plugins=false.\n",
+      "2025-12-02 19:43:33,982 INFO: Loading plugin 'docling_defaults'\n",
+      "2025-12-02 19:43:34,010 INFO: Registered ocr engines: ['auto', 'easyocr', 'ocrmac', 'rapidocr', 'tesserocr', 'tesseract']\n",
+      "2025-12-02 19:43:42,281 INFO: Auto OCR model selected ocrmac.\n",
+      "2025-12-02 19:43:42,299 WARNING: The plugin langchain_docling will not be loaded because Docling is being executed with allow_external_plugins=false.\n",
+      "2025-12-02 19:43:42,299 INFO: Loading plugin 'docling_defaults'\n",
+      "2025-12-02 19:43:42,323 INFO: Registered layout engines: ['docling_layout_default', 'docling_experimental_table_crops_layout']\n",
+      "2025-12-02 19:43:42,347 INFO: Accelerator device: 'mps'\n",
+      "2025-12-02 19:43:57,889 WARNING: The plugin langchain_docling will not be loaded because Docling is being executed with allow_external_plugins=false.\n",
+      "2025-12-02 19:43:57,907 INFO: Loading plugin 'docling_defaults'\n",
+      "2025-12-02 19:43:57,919 INFO: Registered table structure engines: ['docling_tableformer']\n",
+      "2025-12-02 19:44:40,325 INFO: Accelerator device: 'mps'\n",
+      "2025-12-02 19:44:41,261 INFO: Processing document Building+Machine+Learning+Systems+with+a+Feature+Store.pdf\n",
+      "2025-12-02 19:51:45,276 INFO: Finished converting document Building+Machine+Learning+Systems+with+a+Feature+Store.pdf in 491.52 sec.\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Token indices sequence length is longer than the specified maximum sequence length for this model (1143 > 512). Running this sequence through the model will result in indexing errors\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Loaded 1333 document chunks\n"
+     ]
+    }
+   ],
+   "source": [
+    "loader = DoclingLoader(\n",
+    "    file_path=PDF_PATH,\n",
+    "    export_type=EXPORT_TYPE,\n",
+    "    chunker=HybridChunker(tokenizer=EMBED_MODEL_ID),\n",
+    ")\n",
+    "\n",
+    "docs = loader.load()\n",
+    "print(f\"Loaded {len(docs)} document chunks\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "page_content='Praise for Building Machine Learning Systems with a Feature Store\n",
+      "It' s easy to be lost in quality metrics land and forget about the crucial systems aspect to ML. Jim does a great job explaining those aspects and gives a lot of practical tips on how to survive a long deployment.\n",
+      "-Hannes Mühleisen, cocreator of DuckDB\n",
+      "Building machine learning systems in production has historically involved a lot of black magic and undocumented learnings. Jim Dowling is doing a great service to ML practitioners by sharing the best practices and putting together clear step-by-step guide.' metadata={'source': 'content/Building+Machine+Learning+Systems+with+a+Feature+Store.pdf', 'dl_meta': {'schema_name': 'docling_core.transforms.chunker.DocMeta', 'version': '1.0.0', 'doc_items': [{'self_ref': '#/texts/7', 'parent': {'$ref': '#/body'}, 'children': [], 'content_layer': 'body', 'label': 'text', 'prov': [{'page_no': 1, 'bbox': {'l': 97.75, 't': 162.01999999999998, 'r': 432.0, 'b': 126.02999999999997, 'coord_origin': 'BOTTOMLEFT'}, 'charspan': [0, 213]}]}, {'self_ref': '#/texts/8', 'parent': {'$ref': '#/body'}, 'children': [], 'content_layer': 'body', 'label': 'text', 'prov': [{'page_no': 1, 'bbox': {'l': 264.75, 't': 122.13, 'r': 432.0, 'b': 110.03200000000004, 'coord_origin': 'BOTTOMLEFT'}, 'charspan': [0, 38]}]}, {'self_ref': '#/texts/9', 'parent': {'$ref': '#/body'}, 'children': [], 'content_layer': 'body', 'label': 'text', 'prov': [{'page_no': 2, 'bbox': {'l': 81.2, 't': 608.02, 'r': 432.0, 'b': 572.03, 'coord_origin': 'BOTTOMLEFT'}, 'charspan': [0, 256]}]}], 'headings': ['Praise for Building Machine Learning Systems with a Feature Store'], 'origin': {'mimetype': 'application/pdf', 'binary_hash': 2591788756701469466, 'filename': 'Building+Machine+Learning+Systems+with+a+Feature+Store.pdf'}}}\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(docs[1])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Created 1333 splits\n",
+      "Sample: Praise for Building Machine Learning Systems with a Feature Store\n",
+      "I witnessed the rise of feature st...\n"
+     ]
+    }
+   ],
+   "source": [
+    "if EXPORT_TYPE == ExportType.DOC_CHUNKS:\n",
+    "    splits = docs\n",
+    "else:\n",
+    "    from langchain_text_splitters import MarkdownHeaderTextSplitter\n",
+    "    splitter = MarkdownHeaderTextSplitter(\n",
+    "        headers_to_split_on=[\n",
+    "            (\"#\", \"Header_1\"),\n",
+    "            (\"##\", \"Header_2\"),\n",
+    "            (\"###\", \"Header_3\"),\n",
+    "        ],\n",
+    "    )\n",
+    "    splits = [split for doc in docs for split in splitter.split_text(doc.page_content)]\n",
+    "\n",
+    "print(f\"Created {len(splits)} splits\")\n",
+    "print(f\"Sample: {splits[0].page_content[:100]}...\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "2025-12-02 19:52:07,229 INFO: Use pytorch device_name: mps\n",
+      "2025-12-02 19:52:07,232 INFO: Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n"
+     ]
+    }
+   ],
+   "source": [
+    "embeddings = SentenceTransformer(EMBED_MODEL_ID)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Batches: 100%|██████████| 42/42 [00:18<00:00,  2.31it/s]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Created 1333 embeddings\n"
+     ]
+    }
+   ],
+   "source": [
+    "texts = [split.page_content for split in splits]\n",
+    "metadatas = [split.metadata for split in splits]\n",
+    "\n",
+    "vectors = embeddings.encode(texts, show_progress_bar=True, batch_size=32)\n",
+    "print(f\"Created {len(vectors)} embeddings\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "2025-12-02 19:52:44,050 INFO: Initializing external client\n",
+      "2025-12-02 19:52:44,064 INFO: Base URL: https://c.app.hopsworks.ai:443\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "UserWarning: The installed hopsworks client version 4.4.2 may not be compatible with the connected Hopsworks backend version 4.2.2. \n",
+      "To ensure compatibility please install the latest bug fix release matching the minor version of your backend (4.2) by running 'pip install hopsworks==4.2.*'\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "2025-12-02 19:52:47,302 INFO: Python Engine initialized.\n",
+      "\n",
+      "Logged in to project, explore it here https://c.app.hopsworks.ai:443/p/1271977\n"
+     ]
+    }
+   ],
+   "source": [
+    "project = hopsworks.login()\n",
+    "fs = project.get_feature_store()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Created dataframe with 1333 rows\n"
+     ]
+    }
+   ],
+   "source": [
+    "data = []\n",
+    "for i, (text, vector, metadata) in enumerate(zip(texts, vectors, metadatas)):\n",
+    "    data.append({\n",
+    "        'id': i,\n",
+    "        'text': text,\n",
+    "        'page': metadata.get('page', metadata.get('page_number', 0)),\n",
+    "        'embedding': vector\n",
+    "    })\n",
+    "\n",
+    "df = pd.DataFrame(data)\n",
+    "print(f\"Created dataframe with {len(df)} rows\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Feature Group created successfully, explore it at \n",
+      "https://c.app.hopsworks.ai:443/p/1271977/fs/1258579/fg/1790385\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Uploading Dataframe: 100.00% |██████████| Rows 1333/1333 | Elapsed Time: 00:01 | Remaining Time: 00:00\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Launching job: book_embeddings_2_offline_fg_materialization\n",
+      "Job started successfully, you can follow the progress at \n",
+      "https://c.app.hopsworks.ai:443/p/1271977/jobs/named/book_embeddings_2_offline_fg_materialization/executions\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "(Job('book_embeddings_2_offline_fg_materialization', 'SPARK'), None)"
+      ]
+     },
+     "execution_count": 9,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "book_fg = fs.get_or_create_feature_group(\n",
+    "    name=\"book_embeddings\",\n",
+    "    version=2,\n",
+    "    primary_key=[\"id\"],\n",
+    "    description=\"Book text chunks with embeddings\"\n",
+    ")\n",
+    "\n",
+    "book_fg.insert(df)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "rag_llm",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.14"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

models_config.json

@@ -0,0 +1,52 @@
+{
+  "repositories": [
+    {
+      "name": "Unsloth - Qwen3 4B",
+      "repo_id": "unsloth/Qwen3-4B-Instruct-2507-GGUF",
+      "models": [
+        {
+          "name": "Qwen3 4B IQ4_XS",
+          "filename": "en3-4B-Instruct-2507-IQ4_XS.gguf",
+          "description": "4-bit quantization. Very slow on CPU (free CPU inference not recommended)."
+        }
+      ]
+    },
+    {
+      "name": "HuggingFace TB - SmolLM2 1.7B",
+      "repo_id": "HuggingFaceTB/SmolLM2-1.7B-Instruct-GGUF",
+      "models": [
+        {
+          "name": "SmolLM2 1.7B Q4_K_M",
+          "filename": "smollm2-1.7b-instruct-q4_k_m.gguf",
+          "description": "Lightweight 1.7B parameter model, fast inference (Recommended)"
+        }
+      ]
+    },
+    {
+      "name": "Unsloth - Qwen3 1.7B",
+      "repo_id": "unsloth/Qwen3-1.7B-GGUF",
+      "models": [
+        {
+          "name": "Qwen3 1.7B IQ4_XS",
+          "filename": "Qwen3-1.7B-IQ4_XS.gguf",
+          "description": "Good balance between performance and speed"
+        }
+      ]
+    },
+    {
+      "name": "Unsloth - Ministral 3 3B Instruct",
+      "repo_id": "unsloth/Ministral-3-3B-Instruct-2512-GGUF",
+      "models": [
+        {
+          "name": "Ministral 3 3B Instruct IQ4_NL",
+          "filename": "Ministral-3-3B-Instruct-2512-IQ4_NL.gguf",
+          "description": "4-bit quantization of Ministral 3B Instruct model"
+        }
+      ]
+    }
+  ]
+}
+
+
+
+

prompts/rag_prompt.yml

@@ -0,0 +1,29 @@
+system_prompt: "You are a helpful AI assistant that answers questions based on the provided context from documents."
+
+template: |
+  You are a Hopsworks assistant that helps users with questions related to Hopsworks documentatin and usage.
+  Rules:
+  -Use the providedcontext to answer the question. 
+  -If you don't know the answer, say you don't know. 
+  -If the user is just chatting general, respond accordingly.
+  -If the user has questions that are code related please provide code snippets in python.
+
+
+  Context:
+  {context}
+
+  Question: {question}
+
+
+# Hyperparameters
+generation:
+  max_tokens: 256
+  temperature: 0.7
+  stop_sequences:
+    - "Question:"
+    - "\n\n"
+
+# RAG settings
+rag:
+  num_retrieved_chunks: 10  
+  context_separator: "\n\n" 

requirements.txt

@@ -0,0 +1,12 @@
+gradio>=6.0.0
+langchain
+langchain-docling
+sentence-transformers
+hopsworks[python] == 4.4.*
+python-dotenv
+faiss-cpu
+numpy
+pandas
+pyyaml
+
+