📝 Major documentation overhaul: Consolidated all docs into professional README.md

.env.example

@@ -1,13 +1,32 @@
-# Pinecone Configuration
+# ===================================
+# PaperBOT Environment Configuration
+# ===================================
+# Copy this file to .env and fill in your API keys
+# cp .env.example .env
+
+# --------------------------
+# REQUIRED: Pinecone Vector Database
+# Get your API key: https://www.pinecone.io/
+# --------------------------
 PINECONE_API_KEY=your_pinecone_api_key_here
 
-# Google AI (Gemini) Configuration
+# --------------------------
+# REQUIRED: Google AI (Gemini)
+# Get your API key: https://aistudio.google.com/
+# --------------------------
 GOOGLE_API_KEY=your_google_ai_api_key_here
 
-# HuggingFace Token (optional, for some models)
+# --------------------------
+# OPTIONAL: HuggingFace Token
+# Get your token: https://huggingface.co/settings/tokens
+# Required for some gated models
+# --------------------------
 HF_TOKEN=your_huggingface_token_here
 
-# Application Settings (optional)
-# MAX_UPLOAD_SIZE=52428800  # 50MB in bytes
-# BATCH_SIZE=32
-# MODEL_TYPE=quality  # Options: fast, balanced, quality
+# --------------------------
+# OPTIONAL: Application Settings
+# Uncomment and modify as needed
+# --------------------------
+# MAX_UPLOAD_SIZE=15728640      # 15MB in bytes (default)
+# BATCH_SIZE=32                  # Chunks per batch
+# MODEL_TYPE=quality             # Options: fast, balanced, quality

CHANGELOG.md

@@ -1,6 +1,40 @@
-# CHANGELOG
+# Changelog
 
-## Version 2.0.0 - Major Upgrade (January 2026)
+All notable changes to PaperBOT are documented here.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
+
+---
+
+## [2.1.0] - January 2025
+
+### Added
+- **Document Preview** - In-browser preview for all document types
+- **Preloaded Files Preview** - Preview button for files in data/ folder
+- **File Size Warnings** - Warning dialog for files over 5MB
+- **Model Pre-warming** - Embedding model loads on server startup
+- **Curated Fallback Responses** - Beautiful output when API quota exceeded
+- **Retry Logic** - Exponential backoff for transient API failures
+
+### Changed
+- Reduced max file size from 50MB to 15MB for better performance
+- Optimized chunk size to 300 words (from 200)
+- Improved error messages with troubleshooting tips
+- Enhanced UI with Bootstrap 5
+
+### Fixed
+- Pinecone 40KB metadata limit error (chunks now enforced to 8KB)
+- Google API 429 quota errors now handled gracefully
+- Memory leaks during batch processing
+- Encoding issues with special characters
+
+### Security
+- API keys stored only in .env (not tracked by git)
+- Added comprehensive .gitignore
+
+---
+
+## [2.0.0] - January 2025 (Major Upgrade)
 
 ### 🎉 New Features
 

FEATURES.md

@@ -1,162 +0,0 @@
-# PaperBOT Features
-
-## 🚀 Core Features
-
-### 1. **Multi-Format Document Support**
-Upload and process various document formats:
-- PDF documents
-- Microsoft Word (.docx, .doc)
-- Plain text files (.txt)
-- Markdown files (.md)
-- CSV data files
-- JSON files
-- Excel spreadsheets (.xlsx, .xls)
-
-### 2. **Parallel Processing**
-- **Batch Embedding**: Process multiple chunks simultaneously
-- **Multi-threaded Upload**: Non-blocking document processing
-- **Optimized Memory Usage**: Automatic memory cleanup between batches
-- **Fast Processing**: 30-50 chunks/second on average hardware
-
-### 3. **Smart Semantic Search**
-- **Vector Database**: Powered by Pinecone for fast similarity search
-- **Top-K Retrieval**: Retrieves 10 most relevant chunks
-- **Relevance Scoring**: Shows confidence scores for retrieved content
-- **Namespace Isolation**: Each document in separate namespace for accuracy
-
-### 4. **Advanced Memory Management**
-- **Automatic Garbage Collection**: Clears memory every 5 batches
-- **Memory Monitoring**: Real-time memory usage tracking
-- **Batch Processing**: Configurable batch sizes (default: 32 chunks)
-- **Resource Cleanup**: Automatic cleanup after processing
-
-### 5. **Intelligent Q&A System**
-- **RAG (Retrieval Augmented Generation)**: Combines search with AI generation
-- **Customizable Responses**: Choose style and length
-- **Context-Aware**: Only answers from uploaded document
-- **Fallback Mechanisms**: Multiple fallback strategies if main pipeline fails
-
-### 6. **User-Friendly Interface**
-- **Drag & Drop Upload**: Easy file upload
-- **Progress Tracking**: Real-time upload and processing progress
-- **Document Management**: View current document, delete when done
-- **Response Customization**: Select explanation style and length
-- **Formatted Answers**: Markdown rendering with syntax highlighting
-
-## 📊 Performance Features
-
-### Configurable Processing Modes
-Choose between speed and quality in `QASystem/config.py`:
-
-#### Fast Mode
-```python
-CURRENT_MODEL = "fast"
-BATCH_SIZE = 64
-```
-- ⚡ 5-10x faster processing
-- 💾 Lower memory usage
-- ✅ Great for large documents (100+ pages)
-
-#### Balanced Mode
-```python
-CURRENT_MODEL = "balanced"
-BATCH_SIZE = 32
-```
-- ⚖️ Good balance of speed and quality
-- 📄 Recommended for most documents
-
-#### Quality Mode
-```python
-CURRENT_MODEL = "quality"
-BATCH_SIZE = 16
-```
-- 🎯 Highest accuracy
-- 📚 Best for technical/academic papers
-
-## 🔒 Security & Reliability
-
-### Document Isolation
-- Each document stored in separate namespace
-- Previous documents automatically cleared
-- No mixing of different document content
-
-### Error Handling
-- Comprehensive try-catch blocks
-- Graceful fallbacks for API failures
-- Detailed error messages with troubleshooting tips
-- Server-side validation for file types and sizes
-
-### API Management
-- Cached embedding models (load once, use many times)
-- Optimized API calls to reduce costs
-- Timeout protection (2-minute max per upload)
-
-## 💡 Smart Features
-
-### 1. Model Warm-up
-Application pre-loads embedding model on startup for instant first upload
-
-### 2. Progress Callbacks
-Real-time progress updates during document processing:
-- 10%: Upload started
-- 30%: File received
-- 50%: Document store initialized  
-- 70%: Embedding in progress
-- 90%: Writing to database
-- 100%: Complete
-
-### 3. Adaptive Chunking
-Intelligent document splitting:
-- Word-based splitting for better context
-- Configurable chunk size (default: 250 words)
-- Overlap between chunks (default: 50 words)
-- Preserves paragraph structure
-
-### 4. Response Styles
-Choose from multiple explanation styles:
-- **Simple & Intuitive**: Easy to understand
-- **Balanced**: Mix of detail and clarity
-- **Detailed & Technical**: In-depth technical explanations
-- **Academic**: Formal academic writing
-
-### 5. Response Lengths
-Control response verbosity:
-- **Short**: 1 paragraph summary
-- **Medium**: 2-3 paragraphs (default)
-- **Comprehensive**: Detailed multi-paragraph response
-
-## 🛠️ Technical Stack
-
-- **Framework**: FastAPI (high-performance async web framework)
-- **Vector DB**: Pinecone (scalable vector search)
-- **RAG Framework**: Haystack AI (modular NLP framework)
-- **Embeddings**: Sentence Transformers (state-of-the-art models)
-- **LLM**: Google Gemini 1.5 Flash (fast, accurate generation)
-- **Frontend**: Vanilla JS with Bootstrap & SweetAlert2
-- **Processing**: Concurrent.futures for parallel execution
-
-## 📈 Performance Metrics
-
-Typical performance on mid-range hardware (tested):
-- **Upload Speed**: 1-3 seconds for file transfer
-- **Processing Speed**: 30-50 chunks/second
-- **Query Response**: 2-5 seconds end-to-end
-- **Memory Usage**: 500MB-2GB depending on document size
-
-## 🎯 Use Cases
-
-1. **Academic Research**: Quickly understand complex papers
-2. **Technical Documentation**: Extract information from manuals
-3. **Business Reports**: Analyze CSV/Excel data with AI
-4. **Code Documentation**: Process Markdown documentation
-5. **Data Analysis**: Query JSON/CSV datasets naturally
-6. **Meeting Notes**: Search through text/DOCX notes
-
-## 🔄 Continuous Improvements
-
-The codebase includes:
-- Comprehensive error logging
-- Performance monitoring
-- Memory usage tracking
-- Detailed console output for debugging
-- Modular design for easy enhancements

FIXES_APPLIED.md

@@ -1,281 +0,0 @@
-# Large File Upload Fixes - Implementation Summary
-
-## Problem Identified
-The application was unable to handle file uploads around 5MB due to multiple issues:
-1. Missing file size validation and limits in FastAPI
-2. No streaming upload support for large files
-3. Insufficient error handling for edge cases
-4. Memory management issues during processing
-5. Frontend timeout issues
-6. Lack of proper progress tracking
-
-## Solutions Implemented
-
-### 1. **FastAPI/Backend Improvements** ([app.py](app.py))
-
-#### A. Enhanced Upload Configuration
-- **Added FastAPI upload size limits**: Configured `MAX_UPLOAD_SIZE = 50MB`
-- **Increased timeouts**: Set `timeout_keep_alive=600` (10 minutes) for large file processing
-- **Added connection limits**: `limit_concurrency=10` to prevent resource exhaustion
-- **Graceful shutdown**: `timeout_graceful_shutdown=30` for clean server stops
-
-#### B. Streaming File Upload
-```python
-# Old approach: Direct file copy (can fail for large files)
-shutil.copyfileobj(file.file, buffer)
-
-# New approach: Streaming with size validation
-while chunk := await file.read(1024 * 1024):  # Read 1MB at a time
-    file_size += len(chunk)
-    if file_size > MAX_FILE_SIZE:
-        return error  # Early termination
-    chunks.append(chunk)
-```
-
-**Benefits**:
-- Handles files of any size up to 50MB
-- Validates size during upload (not after)
-- Prevents memory overflow
-- Better error messages
-
-#### C. Comprehensive Error Handling
-- **File validation**: Checks for empty files, missing files, invalid types
-- **Size validation**: Real-time size checking during upload
-- **Processing errors**: Proper cleanup on failure
-- **HTTP status codes**: 400, 413, 500 for different error types
-- **Detailed error messages**: User-friendly error descriptions
-
-#### D. File Size Reporting
-- Shows file size in MB in success messages
-- Displays processing time
-- Tracks upload progress
-
-### 2. **Document Processing Improvements** ([QASystem/ingestion.py](QASystem/ingestion.py))
-
-#### A. Enhanced Validation
-```python
-# Input validation
-- Checks if file exists before processing
-- Validates document store is available
-- Verifies extracted content is not empty
-- Validates chunk creation success
-```
-
-#### B. Better Error Recovery
-- **Batch failure tolerance**: Continues processing if <20% of batches fail
-- **Partial success handling**: Accepts results if >50% chunks succeed
-- **Detailed error logging**: Full stack traces for debugging
-- **Memory cleanup**: Automatic cleanup on errors
-
-#### C. Improved Progress Tracking
-```python
-# Progress indicators
-✓ File read successfully
-✓ Extracted content (X documents, Y chars)
-✓ Created Z chunks
-✓ Batch N/M complete
-✓ Wrote to Pinecone
-```
-
-#### D. Memory Management
-- **Periodic cleanup**: Every 3 batches for large files
-- **Memory monitoring**: Tracks usage before/after operations
-- **Resource reporting**: Shows memory delta in statistics
-
-### 3. **Configuration Optimization** ([QASystem/config.py](QASystem/config.py))
-
-#### Updated Settings for 5MB Files
-```python
-# Large File Detection
-LARGE_FILE_THRESHOLD = 3MB  # Changed from 2MB
-# Better catches 5MB files for optimization
-
-# Chunk Settings
-LARGE_FILE_CHUNK_LENGTH = 350  # Optimized from 400
-LARGE_FILE_BATCH_SIZE = 20     # Optimized from 24
-```
-
-**Why these values?**
-- 350 words per chunk: Balance between context and speed
-- Batch size 20: Prevents memory issues on quality model
-- 3MB threshold: Catches most research papers (5-10MB range)
-
-### 4. **Frontend Improvements** ([templates/index.html](templates/index.html))
-
-#### A. Better Validation
-```javascript
-// File size display
-console.log(`File: ${file.name}, Size: ${fileSizeMB}MB`);
-
-// Empty file check
-if (file.size === 0) {
-    // Show error
-}
-
-// Size validation with actual size shown
-text: `File size is ${fileSizeMB}MB. Max is 50MB.`
-```
-
-#### B. Enhanced User Feedback
-- Shows exact file size in error messages
-- Real-time progress polling
-- Better error descriptions
-- Loading indicators
-
-## Key Features Added
-
-### 1. **Streaming Upload**
-- Reads files in 1MB chunks
-- Validates size during upload
-- Prevents memory overflow
-- Handles files up to 50MB
-
-### 2. **Comprehensive Validation**
-✓ File type validation  
-✓ File size validation (client + server)  
-✓ Empty file detection  
-✓ Content extraction validation  
-✓ Embedding validation  
-
-### 3. **Error Recovery**
-✓ Partial batch success tolerance  
-✓ Automatic cleanup on failure  
-✓ Detailed error messages  
-✓ Graceful degradation  
-
-### 4. **Progress Tracking**
-✓ Real-time upload progress  
-✓ Processing stage indicators  
-✓ Batch completion tracking  
-✓ Final statistics report  
-
-### 5. **Memory Optimization**
-✓ Streaming file reads  
-✓ Periodic garbage collection  
-✓ Memory usage monitoring  
-✓ Batch size optimization  
-
-## Testing Recommendations
-
-### Test Cases to Verify
-1. **Small files** (< 1MB): Should process quickly
-2. **Medium files** (1-3MB): Standard processing
-3. **Large files** (3-10MB): Optimized settings activated
-4. **Max size** (50MB): Should work but warn if approaching limit
-5. **Oversized** (> 50MB): Should reject with clear error
-6. **Empty files**: Should reject with error
-7. **Invalid types**: Should reject with supported formats list
-
-### Expected Behavior
-```
-File Size    | Chunk Size | Batch Size | Expected Time
--------------|------------|------------|---------------
-< 1MB        | 300 words  | 16 chunks  | < 30s
-1-3MB        | 300 words  | 16 chunks  | 30-60s
-3-10MB       | 350 words  | 20 chunks  | 1-3 min
-10-50MB      | 350 words  | 20 chunks  | 3-10 min
-```
-
-## Performance Improvements
-
-### Before Fixes
-- ❌ Files >5MB: Failed silently or timeout
-- ❌ No size validation until after upload
-- ❌ Poor error messages
-- ❌ Memory issues with large files
-- ❌ No progress tracking
-
-### After Fixes
-- ✅ Files up to 50MB: Full support
-- ✅ Size validation during upload
-- ✅ Clear, actionable error messages
-- ✅ Optimized memory usage
-- ✅ Real-time progress tracking
-- ✅ Automatic cleanup on errors
-- ✅ Detailed processing statistics
-
-## Files Modified
-
-1. **[app.py](app.py)** - Main application
-   - Added streaming upload
-   - Enhanced error handling
-   - Improved validation
-   - Better configuration
-
-2. **[QASystem/ingestion.py](QASystem/ingestion.py)** - Document processing
-   - Better error recovery
-   - Enhanced validation
-   - Memory optimization
-   - Progress tracking
-
-3. **[QASystem/config.py](QASystem/config.py)** - Configuration
-   - Optimized thresholds
-   - Better chunk sizes
-   - Improved batch sizes
-
-4. **[templates/index.html](templates/index.html)** - Frontend
-   - Better validation
-   - Enhanced error messages
-   - File size display
-
-## How to Use
-
-1. **Start the server**:
-   ```bash
-   python app.py
-   ```
-
-2. **Upload a file**:
-   - Drag and drop or click "Choose File"
-   - Files up to 50MB supported
-   - Watch progress in real-time
-
-3. **Monitor progress**:
-   - Console shows detailed processing steps
-   - Frontend shows upload percentage
-   - Statistics displayed on completion
-
-## Troubleshooting
-
-### If upload still fails:
-
-1. **Check file size**: Must be < 50MB
-2. **Check file type**: PDF, DOCX, TXT, etc.
-3. **Check console logs**: Look for specific errors
-4. **Check memory**: Ensure system has >2GB free
-5. **Check network**: Stable connection required
-6. **Check Pinecone**: API key and index must be valid
-
-### Common Issues:
-
-**Issue**: "File too large"  
-**Solution**: File exceeds 50MB, compress or split it
-
-**Issue**: "Empty file"  
-**Solution**: File has no content, check source
-
-**Issue**: "Processing timeout"  
-**Solution**: File is very large/complex, try splitting it
-
-**Issue**: "No chunks embedded"  
-**Solution**: Check embedding model and Pinecone connection
-
-## Performance Tips
-
-1. **For fastest processing**: Use files < 3MB
-2. **For large documents**: Consider splitting into chapters
-3. **For better quality**: Use smaller chunk sizes (edit config.py)
-4. **For faster speed**: Use "fast" model in config.py
-5. **For maximum compatibility**: Use PDF format
-
-## Summary
-
-All issues related to uploading ~5MB files have been resolved with:
-- ✅ Streaming upload support
-- ✅ Comprehensive validation
-- ✅ Better error handling
-- ✅ Memory optimization
-- ✅ Progress tracking
-- ✅ Detailed logging
-
-The application now handles files from 1KB to 50MB reliably with appropriate error messages and recovery mechanisms at every stage.

INSTALLATION.md

@@ -1,110 +0,0 @@
-# PaperBOT Installation Guide
-
-## Prerequisites
-- Python 3.9 or higher
-- Pinecone account and API key
-- Google AI (Gemini) API key
-
-## Quick Start
-
-### 1. Clone or Download the Repository
-```bash
-cd PaperBOT
-```
-
-### 2. Create Virtual Environment (Recommended)
-```bash
-# Windows
-python -m venv venv
-venv\Scripts\activate
-
-# Linux/Mac
-python3 -m venv venv
-source venv/bin/activate
-```
-
-### 3. Install Dependencies
-```bash
-pip install -r requirements.txt
-```
-
-### 4. Configure Environment Variables
-1. Copy `.env.example` to `.env`:
-   ```bash
-   # Windows
-   copy .env.example .env
-   
-   # Linux/Mac
-   cp .env.example .env
-   ```
-
-2. Edit `.env` and add your API keys:
-   ```
-   PINECONE_API_KEY=your_actual_pinecone_key
-   GOOGLE_API_KEY=your_actual_google_ai_key
-   ```
-
-### 5. Setup Pinecone Index
-1. Log in to [Pinecone Console](https://app.pinecone.io/)
-2. Create a new index with these settings:
-   - **Name**: `paperbot`
-   - **Dimensions**: `1024`
-   - **Metric**: `cosine`
-   - **Pod Type**: `Starter` or `s1`
-
-### 6. Run the Application
-```bash
-python app.py
-```
-
-The application will start on `http://localhost:8000`
-
-## Supported File Formats
-- **PDF** (.pdf)
-- **Word Documents** (.docx, .doc)
-- **Text Files** (.txt)
-- **Markdown** (.md)
-- **CSV** (.csv)
-- **JSON** (.json)
-- **Excel** (.xlsx, .xls)
-
-## Performance Optimization
-
-### For Faster Processing
-Edit `QASystem/config.py`:
-```python
-CURRENT_MODEL = "fast"  # Use fast embedding model
-BATCH_SIZE = 64         # Increase batch size
-```
-
-### For Better Quality
-Edit `QASystem/config.py`:
-```python
-CURRENT_MODEL = "quality"  # Use high-quality model
-CHUNK_SETTINGS = {
-    "split_length": 200,   # Smaller chunks for precision
-    "split_overlap": 75
-}
-```
-
-## Troubleshooting
-
-### Out of Memory Errors
-- Reduce `BATCH_SIZE` in `config.py` to 16 or 8
-- Use `CURRENT_MODEL = "fast"` for smaller memory footprint
-
-### Slow Upload Times
-- Increase `BATCH_SIZE` for parallel processing
-- Use `CURRENT_MODEL = "fast"` for faster embedding
-
-### API Rate Limits
-- Wait a moment between requests
-- Check your API key quotas
-
-## System Requirements
-- **RAM**: Minimum 4GB (8GB+ recommended)
-- **Storage**: 2GB free space for models
-- **Internet**: Required for API calls
-
-## Need Help?
-Check the console output (F12 in browser) for detailed error messages.

QUICKSTART.md

@@ -1,227 +0,0 @@
-# PaperBOT Quick Reference
-
-## 🚀 Quick Commands
-
-### First Time Setup
-```bash
-# 1. Create virtual environment
-python -m venv venv
-
-# 2. Activate (Windows)
-venv\Scripts\activate
-
-# 2. Activate (Linux/Mac)
-source venv/bin/activate
-
-# 3. Install dependencies
-pip install -r requirements.txt
-
-# 4. Configure environment
-cp .env.example .env
-# Edit .env with your API keys
-
-# 5. Run application
-python app.py
-```
-
-### Daily Usage
-```bash
-# Windows
-start.bat
-
-# Linux/Mac
-chmod +x start.sh
-./start.sh
-```
-
-## 📝 API Keys Setup
-
-### Pinecone
-1. Go to https://www.pinecone.io/
-2. Sign up/Login
-3. Create new API key
-4. Copy to `.env` file
-
-### Google AI (Gemini)
-1. Go to https://makersuite.google.com/app/apikey
-2. Create API key
-3. Copy to `.env` file
-
-### Pinecone Index Setup
-1. Login to Pinecone Console
-2. Click "Create Index"
-3. Settings:
-   - Name: `paperbot`
-   - Dimensions: `1024`
-   - Metric: `cosine`
-   - Cloud: Any (AWS/GCP/Azure)
-   - Region: Choose nearest
-
-## ⚙️ Configuration Options
-
-### Speed vs Quality (`QASystem/config.py`)
-
-**Fast Mode** (Recommended for large docs):
-```python
-CURRENT_MODEL = "fast"
-BATCH_SIZE = 64
-```
-
-**Quality Mode** (Recommended for technical papers):
-```python
-CURRENT_MODEL = "quality"
-BATCH_SIZE = 16
-```
-
-## 🎯 Supported File Types
-
-| Format | Extension | Notes |
-|--------|-----------|-------|
-| PDF | .pdf | Best for research papers |
-| Word | .docx, .doc | Full text extraction |
-| Text | .txt | Plain text |
-| Markdown | .md | Preserves formatting |
-| CSV | .csv | Tabular data |
-| Excel | .xlsx, .xls | Spreadsheets |
-| JSON | .json | Structured data |
-
-## 🔧 Troubleshooting
-
-### Issue: Out of Memory
-**Solution**: Reduce batch size in `config.py`
-```python
-BATCH_SIZE = 8  # Lower value
-```
-
-### Issue: Slow Upload
-**Solution**: Use fast model
-```python
-CURRENT_MODEL = "fast"
-```
-
-### Issue: API Rate Limit
-**Solution**: Wait 1-2 minutes between uploads
-
-### Issue: Can't Connect to Server
-**Solution**: Check if port 8000 is free
-```bash
-# Windows
-netstat -ano | findstr :8000
-
-# Linux/Mac
-lsof -i :8000
-```
-
-## 📊 Performance Tips
-
-### For Large Documents (100+ pages)
-```python
-# config.py
-CURRENT_MODEL = "fast"
-BATCH_SIZE = 64
-CHUNK_SETTINGS = {
-    "split_length": 400,
-    "split_overlap": 50
-}
-```
-
-### For Technical Papers
-```python
-# config.py
-CURRENT_MODEL = "quality"
-BATCH_SIZE = 16
-CHUNK_SETTINGS = {
-    "split_length": 200,
-    "split_overlap": 75
-}
-```
-
-### For CSV/Excel Files
-- Keep files under 10,000 rows for best performance
-- Remove unnecessary columns before upload
-- Use CSV format for faster processing
-
-## 🎨 UI Features
-
-### Response Styles
-- **Simple & Intuitive**: Easy explanations
-- **Balanced**: Mix of detail and clarity
-- **Detailed & Technical**: In-depth analysis
-- **Academic**: Formal writing
-
-### Response Lengths
-- **Short**: 1 paragraph
-- **Medium**: 2-3 paragraphs (recommended)
-- **Comprehensive**: Detailed multi-paragraph
-
-## 🔍 Example Questions
-
-### For Research Papers
-- "What is the main contribution of this paper?"
-- "Explain the methodology used"
-- "What are the key findings?"
-- "How does this compare to previous work?"
-
-### For Data Files (CSV/Excel)
-- "What are the main trends in this data?"
-- "Summarize the statistics"
-- "What columns are available?"
-
-### For Documentation
-- "How do I install this software?"
-- "Explain the configuration options"
-- "What are the prerequisites?"
-
-## 📈 Monitoring
-
-### Memory Usage
-Check console output for memory stats:
-```
-📊 Statistics:
-• Memory: 1250.5MB
-• Processing speed: 45 chunks/sec
-```
-
-### Performance Metrics
-- Upload: 1-3 seconds
-- Processing: 30-50 chunks/sec
-- Query: 2-5 seconds
-
-## 🆘 Getting Help
-
-1. Check console output (terminal)
-2. Open browser console (F12)
-3. Review error messages
-4. Check [INSTALLATION.md](INSTALLATION.md)
-5. Review [FEATURES.md](FEATURES.md)
-
-## 🔄 Updates & Maintenance
-
-### Update Dependencies
-```bash
-pip install --upgrade -r requirements.txt
-```
-
-### Clear Cache
-```bash
-# Delete uploads
-rm -rf uploads/*
-
-# Clear Python cache
-find . -type d -name __pycache__ -exec rm -r {} +
-```
-
-### Reset Database
-Delete all vectors in Pinecone console or via code
-
-## 🎓 Best Practices
-
-1. **One Document at a Time**: Upload new doc clears previous
-2. **Appropriate Model**: Use fast for speed, quality for accuracy
-3. **Clear Questions**: Be specific in your queries
-4. **File Size**: Keep under 50MB for best performance
-5. **Internet**: Stable connection required for API calls
-
----
-
-**Need more help?** Check the full documentation in [README.md](README.md)

QUICK_REFERENCE.md

@@ -1,351 +0,0 @@
-# 🚀 Large File Upload Issue - RESOLVED
-
-## Executive Summary
-
-**Problem**: Application unable to upload files around 5MB  
-**Root Cause**: Multiple issues in upload pipeline  
-**Status**: ✅ **FULLY RESOLVED**  
-**Tested Up To**: 50MB files  
-
----
-
-## 🎯 What Was Fixed
-
-### Critical Issues Resolved
-
-1. **❌ No file size validation** → ✅ Multi-layer size validation (client + server)
-2. **❌ No streaming upload** → ✅ Chunked streaming for large files  
-3. **❌ Poor error handling** → ✅ Comprehensive error recovery
-4. **❌ Memory issues** → ✅ Optimized memory management
-5. **❌ No progress tracking** → ✅ Real-time progress updates
-6. **❌ Timeout problems** → ✅ Extended timeouts (10 min)
-7. **❌ Silent failures** → ✅ Detailed error messages
-
----
-
-## 📊 Performance Comparison
-
-### Before Fixes
-```
-File Size | Status
-----------|--------
-< 1 MB    | ✅ Works
-1-3 MB    | ⚠️  Sometimes
-3-5 MB    | ❌ Fails
-> 5 MB    | ❌ Fails
-```
-
-### After Fixes
-```
-File Size | Status      | Time
-----------|-------------|--------
-< 1 MB    | ✅ Works    | < 30s
-1-3 MB    | ✅ Works    | 30-60s
-3-5 MB    | ✅ Works    | 1-2 min
-5-10 MB   | ✅ Works    | 2-5 min
-10-50 MB  | ✅ Works    | 5-10 min
-> 50 MB   | ⛔ Rejected | N/A
-```
-
----
-
-## 🔧 Technical Changes
-
-### 1. Backend ([app.py](app.py))
-
-**Upload Endpoint Rewrite**
-- ✅ Streaming file upload (1MB chunks)
-- ✅ Real-time size validation
-- ✅ Comprehensive error handling
-- ✅ Automatic cleanup on failure
-- ✅ Detailed logging
-
-**Server Configuration**
-- ✅ 10-minute timeout
-- ✅ 50MB request limit
-- ✅ Connection limiting
-- ✅ Graceful shutdown
-
-### 2. Processing ([QASystem/ingestion.py](QASystem/ingestion.py))
-
-**Validation Improvements**
-- ✅ File existence check
-- ✅ Content validation
-- ✅ Chunk creation verification
-- ✅ Embedding validation
-
-**Error Recovery**
-- ✅ Batch failure tolerance (20%)
-- ✅ Partial success handling (50%)
-- ✅ Memory cleanup on error
-- ✅ Full stack traces
-
-**Memory Management**
-- ✅ Periodic garbage collection
-- ✅ Memory usage tracking
-- ✅ Batch size optimization
-- ✅ Resource monitoring
-
-### 3. Configuration ([QASystem/config.py](QASystem/config.py))
-
-**Optimized for 5MB Files**
-- ✅ 3MB threshold (was 2MB)
-- ✅ 350-word chunks (balanced)
-- ✅ Batch size 20 (memory-safe)
-
-### 4. Frontend ([templates/index.html](templates/index.html))
-
-**User Experience**
-- ✅ File size display
-- ✅ Empty file detection
-- ✅ Better error messages
-- ✅ Real-time progress
-
----
-
-## 🎮 How to Test
-
-### Quick Test (5MB file)
-
-1. **Start server**:
-   ```bash
-   python app.py
-   ```
-
-2. **Upload 5MB PDF**:
-   - Open http://localhost:8000
-   - Choose a ~5MB research paper
-   - Click upload
-
-3. **Verify**:
-   - ✅ Progress bar shows updates
-   - ✅ Console shows "Large file detected"
-   - ✅ Processing completes in 1-2 minutes
-   - ✅ Success message shows file size
-   - ✅ Can ask questions immediately
-
-### Expected Console Output
-```
-📥 Upload endpoint called - Filename: paper.pdf
-✓ File type validated: .pdf
-✓ File read successfully: 5.23MB
-✓ File saved to: uploads\paper.pdf
-
-📄 Starting document ingestion: paper.pdf
-   ⚡ Large file detected - using optimized settings:
-      Chunk length: 350 words
-      Batch size: 20 chunks
-   ✓ Created 156 chunks
-   ✓ Batch 1/8 complete (20 chunks, +15.2MB)
-   🧹 Memory cleanup: 485.3MB
-   ...
-   ✅ Ingestion completed successfully!
-   • Size: 5.23MB
-   • Chunks: 156
-   • Time: 94.3s
-```
-
----
-
-## 📚 Documentation
-
-| File | Purpose |
-|------|---------|
-| **[FIXES_APPLIED.md](FIXES_APPLIED.md)** | Detailed technical explanation of all changes |
-| **[TESTING_GUIDE.md](TESTING_GUIDE.md)** | Comprehensive testing procedures and benchmarks |
-| **This File** | Quick reference and summary |
-
----
-
-## ✅ Verification Checklist
-
-### Upload Pipeline
-- [x] File type validation
-- [x] File size validation (client-side)
-- [x] File size validation (server-side)
-- [x] Streaming upload support
-- [x] Empty file detection
-- [x] Error message clarity
-
-### Processing Pipeline
-- [x] Document extraction
-- [x] Chunk creation
-- [x] Embedding generation
-- [x] Database storage
-- [x] Memory management
-- [x] Error recovery
-
-### User Experience
-- [x] Progress tracking
-- [x] Clear error messages
-- [x] File size display
-- [x] Processing statistics
-- [x] Success confirmation
-- [x] Immediate usability
-
----
-
-## 🚨 Known Limits
-
-| Limit | Value | Reason |
-|-------|-------|--------|
-| **Max File Size** | 50MB | Memory constraints |
-| **Upload Timeout** | 10 min | Very large file processing |
-| **Min File Size** | > 0 bytes | Must have content |
-| **Supported Formats** | PDF, DOCX, TXT, etc. | Converter availability |
-
----
-
-## 🐛 Troubleshooting
-
-### Issue: Upload fails for 5MB file
-**Check**:
-1. Console for specific error
-2. File type is supported
-3. File is not corrupted
-4. Pinecone API key is valid
-5. Internet connection is stable
-
-### Issue: Slow processing
-**Solutions**:
-1. Use "fast" model in config.py
-2. Increase chunk size
-3. Close other applications
-4. Check system has >2GB RAM free
-
-### Issue: Memory error
-**Solutions**:
-1. Reduce batch size in config.py
-2. Use "fast" model (uses less memory)
-3. Increase system RAM
-4. Process smaller files
-
----
-
-## 🎓 For Developers
-
-### Key Design Decisions
-
-**Why streaming upload?**
-- Handles files larger than available RAM
-- Validates size during upload (not after)
-- Better user experience (shows progress)
-
-**Why 3MB threshold?**
-- Research papers typically 5-10MB
-- Activates optimizations early enough
-- Prevents memory issues on medium files
-
-**Why 350-word chunks?**
-- Balance between context and speed
-- Works well with quality model
-- Optimal for most research papers
-
-**Why batch size 20?**
-- Prevents out-of-memory errors
-- Good balance with quality model
-- Allows frequent cleanup
-
-### Code Architecture
-
-```
-Client Upload
-    ↓
-[Streaming Validation] → Size check every 1MB
-    ↓
-[File Storage] → Save to uploads/
-    ↓
-[Document Extraction] → PDF/DOCX/etc to text
-    ↓
-[Chunk Creation] → 300-350 word chunks
-    ↓
-[Batch Embedding] → 16-20 chunks at a time
-    ↓
-[Vector Storage] → Pinecone write
-    ↓
-[Memory Cleanup] → Garbage collection
-    ↓
-Success!
-```
-
----
-
-## 📈 Metrics
-
-### Success Rates (Expected)
-
-| File Size | Success Rate | Avg Time |
-|-----------|-------------|----------|
-| < 1MB     | 99%         | 25s      |
-| 1-3MB     | 98%         | 50s      |
-| 3-5MB     | 95%         | 100s     |
-| 5-10MB    | 90%         | 180s     |
-| 10-50MB   | 85%         | 360s     |
-
-*Lower success rates for larger files due to network/system variability*
-
-### Error Distribution (Fixed)
-
-| Error Type | Before | After |
-|------------|--------|-------|
-| File too large | 60% | 0% |
-| Timeout | 25% | 2% |
-| Memory error | 10% | 1% |
-| Network error | 5% | 5% |
-
----
-
-## 🔐 Security Notes
-
-- ✅ File type whitelist enforced
-- ✅ File size limits enforced
-- ✅ Path traversal prevented
-- ✅ Automatic cleanup on error
-- ✅ No arbitrary code execution
-- ✅ API key not exposed
-
----
-
-## 🎉 Final Notes
-
-### What You Can Do Now
-
-✅ Upload research papers up to 50MB  
-✅ Get detailed progress updates  
-✅ See clear error messages  
-✅ Process large documents reliably  
-✅ Handle multiple file formats  
-✅ Monitor memory usage  
-✅ Track processing statistics  
-
-### What's Improved
-
-✅ **Reliability**: 95%+ success rate for 5MB files  
-✅ **Performance**: Optimized settings activate automatically  
-✅ **User Experience**: Real-time progress and clear errors  
-✅ **Error Recovery**: Automatic cleanup and retry capability  
-✅ **Monitoring**: Detailed logging for debugging  
-
----
-
-## 📞 Support
-
-If issues persist:
-
-1. Check **[TESTING_GUIDE.md](TESTING_GUIDE.md)** for specific test cases
-2. Review **[FIXES_APPLIED.md](FIXES_APPLIED.md)** for technical details  
-3. Check console output for specific errors
-4. Verify system requirements (RAM, disk space)
-5. Test with smaller files first
-
----
-
-**Status**: ✅ Production Ready  
-**Tested**: Files from 1KB to 50MB  
-**Confidence**: High  
-**Next Steps**: Deploy and monitor real-world usage
-
----
-
-*All changes have been implemented, tested, and documented.*  
-*The application now handles large file uploads reliably and efficiently.*

README.md

@@ -1,271 +1,490 @@
-# 🤖 PaperBOT - AI Research Assistant
+<p align="center">
+  <img src="https://img.shields.io/badge/Python-3.9+-blue?style=for-the-badge&logo=python&logoColor=white" alt="Python">
+  <img src="https://img.shields.io/badge/FastAPI-0.128+-00a393?style=for-the-badge&logo=fastapi&logoColor=white" alt="FastAPI">
+  <img src="https://img.shields.io/badge/Haystack-2.22+-1C3D5A?style=for-the-badge" alt="Haystack">
+  <img src="https://img.shields.io/badge/Pinecone-Vector_DB-6b21a8?style=for-the-badge" alt="Pinecone">
+  <img src="https://img.shields.io/badge/Google_Gemini-AI-4285F4?style=for-the-badge&logo=google&logoColor=white" alt="Gemini">
+  <img src="https://img.shields.io/badge/License-MIT-green?style=for-the-badge" alt="License">
+</p>
+
+<h1 align="center">🤖 PaperBOT</h1>
+<h3 align="center">AI-Powered Research Paper Assistant</h3>
+
+<p align="center">
+  <b>Upload any document and ask questions in natural language. Get AI-powered answers grounded in your document's content.</b>
+</p>
+
+<p align="center">
+  <a href="#-features">Features</a> •
+  <a href="#-quick-start">Quick Start</a> •
+  <a href="#-usage">Usage</a> •
+  <a href="#%EF%B8%8F-configuration">Configuration</a> •
+  <a href="#-api-reference">API</a> •
+  <a href="#-architecture">Architecture</a>
+</p>
 
-PaperBOT is an intelligent document analysis system that uses **RAG (Retrieval Augmented Generation)** to help you understand and query research papers, documents, and data files. Upload any supported document and ask questions in natural language!
+---
+
+## 🎯 What is PaperBOT?
+
+PaperBOT is a **Retrieval-Augmented Generation (RAG)** application that allows you to upload research papers, documents, or data files and have intelligent conversations about their content. Unlike generic chatbots, PaperBOT answers are **always grounded in your uploaded document**, preventing hallucinations and ensuring accuracy.
+
+### Key Highlights
+
+- 📄 **Multi-format Support** — PDF, DOCX, TXT, MD, CSV, JSON, Excel
+- 🚀 **Fast Processing** — Parallel embedding with optimized chunking
+- 🎯 **Accurate Answers** — RAG ensures responses come from your document
+- 🎨 **Beautiful UI** — Modern, responsive interface with progress tracking
+- 🔒 **Privacy First** — Your documents stay on your infrastructure
+
+---
+
+## ✨ Features
+
+<table>
+<tr>
+<td width="50%">
+
+### 📚 Document Processing
+- **9 file formats** supported
+- Smart text chunking (300 words/chunk)
+- Parallel batch embedding
+- Metadata size enforcement for Pinecone
+
+</td>
+<td width="50%">
+
+### 🧠 AI-Powered Q&A
+- Semantic search with Pinecone
+- Google Gemini 2.0 Flash for generation
+- Curated fallback responses
+- Customizable response styles
+
+</td>
+</tr>
+<tr>
+<td width="50%">
+
+### ⚡ Performance
+- Model pre-warming on startup
+- Configurable speed/quality tradeoff
+- Memory-efficient processing
+- Up to 15MB file support
 
-## ✨ Key Features
+</td>
+<td width="50%">
 
-- 📄 **Multi-Format Support**: PDF, DOCX, DOC, TXT, MD, CSV, JSON, XLSX, XLS
-- ⚡ **Parallel Processing**: Fast document ingestion with multi-threaded embedding
-- 🧠 **Smart Semantic Search**: Powered by Pinecone vector database
-- 💾 **Memory Management**: Optimized batch processing with automatic cleanup
-- 🎯 **RAG Pipeline**: Combines retrieval with Google Gemini for accurate answers
-- 🎨 **Beautiful UI**: Modern, responsive interface with drag-and-drop upload
-- 🔧 **Configurable**: Choose between speed and quality modes
+### 🎨 User Experience
+- Drag-and-drop file upload
+- Real-time progress tracking
+- In-browser document preview
+- Preloaded files support
+
+</td>
+</tr>
+</table>
+
+---
 
 ## 🚀 Quick Start
 
 ### Prerequisites
-- Python 3.9+
-- Pinecone API key ([Get it here](https://www.pinecone.io/))
-- Google AI API key ([Get it here](https://makersuite.google.com/app/apikey))
+
+| Requirement | Version | Purpose |
+|-------------|---------|---------|
+| Python | 3.9+ | Runtime |
+| Pinecone Account | Free tier | Vector database |
+| Google AI API Key | Free tier | LLM generation |
 
 ### Installation
 
-1. **Clone the repository**
-   ```bash
-   git clone <your-repo-url>
-   cd PaperBOT
-   ```
-
-2. **Create virtual environment**
-   ```bash
-   python -m venv venv
-   # Windows
-   venv\Scripts\activate
-   # Linux/Mac
-   source venv/bin/activate
-   ```
-
-3. **Install dependencies**
-   ```bash
-   pip install -r requirements.txt
-   ```
-
-4. **Configure environment**
-   ```bash
-   # Copy example env file
-   cp .env.example .env
-   
-   # Edit .env and add your API keys
-   PINECONE_API_KEY=your_key_here
-   GOOGLE_API_KEY=your_key_here
-   ```
-
-5. **Setup Pinecone Index**
-   - Create index named `paperbot`
-   - Dimensions: `1024`
-   - Metric: `cosine`
-
-6. **Run the application**
-   ```bash
-   python app.py
-   ```
-
-7. **Open in browser**
-   ```
-   http://localhost:8000
-   ```
+```bash
+# 1. Clone the repository
+git clone https://github.com/vikash-48413/PaperBOT.git
+cd PaperBOT
 
-## 📖 Usage
+# 2. Create virtual environment
+python -m venv venv
 
-1. **Upload Document**: Drag & drop or click to upload (PDF, DOCX, CSV, etc.)
-2. **Wait for Processing**: Progress bar shows upload and embedding status
-3. **Ask Questions**: Type your question in natural language
-4. **Customize Response**: Select explanation style and length
-5. **Get Answers**: AI generates context-aware answers from your document
+# Windows
+venv\Scripts\activate
 
-## 🏗️ Architecture
+# Linux/Mac
+source venv/bin/activate
 
-### Tech Stack
-- **Backend**: FastAPI (async Python web framework)
-- **Vector DB**: Pinecone (cloud-native vector database)
-- **RAG Framework**: Haystack AI
-- **Embeddings**: Sentence Transformers (BAAI/bge-large-en-v1.5)
-- **LLM**: Google Gemini 1.5 Flash
-- **Frontend**: HTML/CSS/JS with Bootstrap
+# 3. Install dependencies
+pip install -r requirements.txt
 
-### Processing Pipeline
+# 4. Configure environment
+cp .env.example .env
+# Edit .env with your API keys (see Configuration section)
 
+# 5. Run the application
+python app.py
 ```
-Document Upload
-    ↓
-File Validation & Storage
-    ↓
-Format Detection & Conversion
-    ↓
-Text Extraction
-    ↓
-Chunking (250 words, 50 overlap)
-    ↓
-Parallel Batch Embedding (32 chunks/batch)
-    ↓
-Vector Storage (Pinecone)
-    ↓
-Ready for Queries!
+
+### Access the Application
+
+Open your browser and navigate to:
+```
+http://localhost:8000
 ```
 
-### Query Pipeline
+---
+
+## 📖 Usage
+
+### Basic Workflow
 
 ```
-User Query
-    ↓
-Query Embedding
-    ↓
-Semantic Search (Top 10 chunks)
-    ↓
-Context Assembly
-    ↓
-LLM Generation (Gemini)
-    ↓
-Formatted Answer
+┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
+│  Upload Document │ ──▶ │  Ask Questions  │ ──▶ │   Get Answers   │
+│  (PDF, DOCX...) │     │  (Natural Lang) │     │  (AI-Powered)   │
+└─────────────────┘     └─────────────────┘     └─────────────────┘
+```
+
+### Step-by-Step
+
+1. **Upload a Document**
+   - Drag & drop or click to select a file
+   - Supported: PDF, DOCX, DOC, TXT, MD, CSV, JSON, XLSX, XLS
+   - Maximum size: 15MB (recommended: under 5MB for fast processing)
+
+2. **Wait for Processing**
+   - Progress bar shows upload and embedding status
+   - Processing time: ~30s for 1MB, ~2-3min for 5MB
+
+3. **Ask Questions**
+   - Type your question in natural language
+   - Example: "What is the main contribution of this paper?"
+
+4. **Customize Response**
+   - **Style**: Simple, Balanced, or Technical
+   - **Length**: Short, Medium, or Comprehensive
+
+5. **Preview Document**
+   - Click the 👁️ Preview button to view documents in-browser
+   - No download required
+
+### Using Preloaded Files
+
+Place documents in the `data/` folder to make them available as preloaded options:
+
+```bash
+# Add a paper to preloaded files
+cp your-paper.pdf data/
 ```
 
+---
+
 ## ⚙️ Configuration
 
-Edit [`QASystem/config.py`](QASystem/config.py) for performance tuning:
+### Environment Variables
 
-### Fast Mode (5-10x faster)
-```python
-CURRENT_MODEL = "fast"
-BATCH_SIZE = 64
-CHUNK_SETTINGS = {
-    "split_length": 400,
-    "split_overlap": 50
-}
+Create a `.env` file in the project root:
+
+```env
+# Required: Pinecone Vector Database
+PINECONE_API_KEY=your_pinecone_api_key_here
+
+# Required: Google AI (Gemini)
+GOOGLE_API_KEY=your_google_api_key_here
+
+# Optional: HuggingFace (for some models)
+HF_TOKEN=your_huggingface_token_here
 ```
 
-### Quality Mode (best accuracy)
+### Getting API Keys
+
+| Service | Link | Notes |
+|---------|------|-------|
+| Pinecone | [pinecone.io](https://www.pinecone.io/) | Free tier: 1 index, 100K vectors |
+| Google AI | [aistudio.google.com](https://aistudio.google.com/) | Free tier: 60 requests/min |
+| HuggingFace | [huggingface.co](https://huggingface.co/) | Optional, for gated models |
+
+### Pinecone Index Setup
+
+Create an index with these settings:
+
+| Setting | Value |
+|---------|-------|
+| **Name** | `paperbot` |
+| **Dimensions** | `1024` |
+| **Metric** | `cosine` |
+| **Cloud** | Any (AWS, GCP, Azure) |
+
+### Performance Tuning
+
+Edit `QASystem/config.py` to adjust:
+
 ```python
-CURRENT_MODEL = "quality"
-BATCH_SIZE = 16
+# Embedding model (must match Pinecone dimensions)
+CURRENT_MODEL = "quality"  # Options: "fast", "balanced", "quality"
+
+# Chunk settings
 CHUNK_SETTINGS = {
-    "split_length": 200,
-    "split_overlap": 75
+    "split_by": "word",
+    "split_length": 300,    # Words per chunk
+    "split_overlap": 15,    # Overlap between chunks
 }
+
+# Batch size for embeddings
+BATCH_SIZE = 32  # Higher = faster, but uses more memory
 ```
 
-## 📊 Performance
+---
 
-Tested on mid-range hardware (16GB RAM, i7 CPU):
+## 📡 API Reference
+
+### Endpoints
+
+| Method | Endpoint | Description |
+|--------|----------|-------------|
+| `GET` | `/` | Main web interface |
+| `POST` | `/upload_document` | Upload and process a document |
+| `POST` | `/get_result` | Ask a question |
+| `GET` | `/document_status` | Check current document status |
+| `GET` | `/preview_document` | Preview current document |
+| `GET` | `/preview_file/{filename}` | Preview any file |
+| `POST` | `/delete_document` | Delete current document |
+| `GET` | `/preloaded_files` | List preloaded files |
+| `POST` | `/load_preloaded_file` | Load a preloaded file |
+| `GET` | `/model_status` | Check if embedding model is ready |
+
+### Example: Ask a Question (cURL)
+
+```bash
+curl -X POST "http://localhost:8000/get_result" \
+  -H "Content-Type: application/x-www-form-urlencoded" \
+  -d "question=What is attention mechanism?&style=Balanced&length=Medium"
+```
+
+### Response Format
 
-| Metric | Performance |
-|--------|------------|
-| Upload Speed | 1-3 seconds |
-| Processing Speed | 30-50 chunks/sec |
-| Query Response | 2-5 seconds |
-| Memory Usage | 500MB - 2GB |
+```json
+{
+  "answer": "The attention mechanism allows the model to focus on relevant parts of the input...",
+  "source_file": "attention_paper.pdf"
+}
+```
 
-**Processing Times (typical)**:
-- 10-page PDF: ~15-30 seconds
-- 50-page PDF: ~60-90 seconds
-- 100-page PDF: ~2-3 minutes
+---
 
-## 🎯 Use Cases
+## 🏗️ Architecture
 
-- 📚 **Academic Research**: Understand complex papers quickly
-- 📊 **Data Analysis**: Query CSV/Excel files naturally
-- 📝 **Documentation**: Search technical docs and manuals
-- 💼 **Business**: Analyze reports and presentations
-- 🔬 **Research**: Extract insights from scientific papers
+### System Overview
 
-## 🔧 Advanced Features
+```
+┌──────────────────────────────────────────────────────────────────┐
+│                           PaperBOT                                │
+├──────────────────────────────────────────────────────────────────┤
+│  ┌─────────────┐    ┌──────────────┐    ┌─────────────────────┐  │
+│  │   FastAPI   │───▶│  Haystack    │───▶│  Google Gemini      │  │
+│  │   Server    │    │  Pipeline    │    │  (LLM Generation)   │  │
+│  └─────────────┘    └──────────────┘    └─────────────────────┘  │
+│         │                  │                                      │
+│         ▼                  ▼                                      │
+│  ┌─────────────┐    ┌──────────────┐                             │
+│  │  Document   │    │   Pinecone   │                             │
+│  │  Converters │    │  Vector DB   │                             │
+│  └─────────────┘    └──────────────┘                             │
+└──────────────────────────────────────────────────────────────────┘
+```
 
-### Memory Management
-- Automatic garbage collection every 5 batches
-- Real-time memory monitoring
-- Configurable batch sizes
-- Resource cleanup after processing
+### Tech Stack
 
-### Semantic Search
-- Vector similarity search with Pinecone
-- Top-K retrieval with relevance scoring
-- Namespace isolation per document
-- Fallback retrieval strategies
+| Component | Technology | Purpose |
+|-----------|------------|---------|
+| **Backend** | FastAPI + Uvicorn | Async web server |
+| **RAG Framework** | Haystack 2.22 | Pipeline orchestration |
+| **Embeddings** | Sentence Transformers | BAAI/bge-large-en-v1.5 |
+| **Vector DB** | Pinecone | Semantic search |
+| **LLM** | Google Gemini 2.0 Flash | Answer generation |
+| **Frontend** | HTML/CSS/JS + Bootstrap | User interface |
 
-### Error Handling
-- Comprehensive validation
-- Graceful API failure fallbacks
-- Detailed error messages
-- Automatic retry mechanisms
+### Processing Pipeline
+
+```
+Document Upload
+      │
+      ▼
+┌─────────────────┐
+│ File Validation │  ← Check type, size
+└────────┬────────┘
+         │
+         ▼
+┌─────────────────┐
+│ Format Converter│  ← PDF, DOCX, Excel → Text
+└────────┬────────┘
+         │
+         ▼
+┌─────────────────┐
+│ Text Chunking   │  ← 300 words/chunk
+└────────┬────────┘
+         │
+         ▼
+┌─────────────────┐
+│ Size Enforcement│  ← Ensure <8KB per chunk
+└────────┬────────┘
+         │
+         ▼
+┌─────────────────┐
+│ Batch Embedding │  ← 32 chunks/batch
+└────────┬────────┘
+         │
+         ▼
+┌─────────────────┐
+│ Pinecone Upload │  ← Store vectors
+└─────────────────┘
+```
+
+---
 
 ## 📁 Project Structure
 
 ```
 PaperBOT/
-├── app.py                          # FastAPI application
-├── requirements.txt                # Python dependencies
-├── setup.py                        # Package setup
-├── .env.example                    # Environment template
-├── INSTALLATION.md                 # Detailed setup guide
-├── FEATURES.md                     # Feature documentation
-├── QASystem/
+├── app.py                 # Main FastAPI application
+├── QASystem/              # Core RAG system
 │   ├── __init__.py
-│   ├── config.py                   # Configuration settings
-│   ├── ingestion.py                # Document processing
-│   ├── retrieval_and_generation.py # RAG pipeline
-│   └── utils.py                    # Utilities
-├── templates/
-│   └── index.html                  # Web interface
-├── uploads/                        # Temporary file storage
-└── data/                           # Sample documents
+│   ├── config.py          # Configuration settings
+│   ├── ingestion.py       # Document processing & embedding
+│   ├── retrieval_and_generation.py  # Q&A pipeline
+│   └── utils.py           # Pinecone utilities
+├── templates/             # HTML templates
+│   └── index.html         # Main UI
+├── data/                  # Preloaded documents
+├── uploads/               # User uploads (gitignored)
+├── requirements.txt       # Python dependencies
+├── .env.example           # Environment template
+├── start.bat              # Windows launcher
+├── start.sh               # Linux/Mac launcher
+└── LICENSE                # MIT License
 ```
 
-## 🛠️ Troubleshooting
+---
 
-### Out of Memory
-- Reduce `BATCH_SIZE` to 8 or 16
-- Use `CURRENT_MODEL = "fast"`
-- Process smaller documents
+## 🧪 Testing
 
-### Slow Processing
-- Increase `BATCH_SIZE` to 64
-- Use `CURRENT_MODEL = "fast"`
-- Check internet connection
+### Run System Tests
 
-### API Errors
-- Verify API keys in `.env`
-- Check Pinecone index configuration
-- Ensure sufficient API quotas
+```bash
+# Test all components
+python test_system.py
+
+# Test Pinecone connection
+python test_pinecone.py
+```
+
+### Manual Testing Checklist
+
+1. **Upload Test**: Upload a small PDF (<1MB)
+2. **Query Test**: Ask "What is this document about?"
+3. **Preview Test**: Click the preview button
+4. **Delete Test**: Delete the document
+
+---
 
-### Upload Fails
-- Check file size (max 50MB)
-- Verify file format is supported
-- Check console (F12) for details
+## 🔧 Troubleshooting
 
-## 📚 Documentation
+### Common Issues
 
-- [Installation Guide](INSTALLATION.md) - Detailed setup instructions
-- [Features](FEATURES.md) - Comprehensive feature list
-- [Configuration](QASystem/config.py) - Performance tuning options
+<details>
+<summary><b>❌ "Pinecone index not found"</b></summary>
+
+Create the index in Pinecone console:
+- Name: `paperbot`
+- Dimensions: `1024`
+- Metric: `cosine`
+</details>
+
+<details>
+<summary><b>❌ "Google API quota exceeded"</b></summary>
+
+The free tier has rate limits. Either:
+- Wait a few minutes for quota reset
+- Upgrade to paid tier
+- Use curated fallback (automatic)
+</details>
+
+<details>
+<summary><b>❌ "File too large" error</b></summary>
+
+Maximum file size is 15MB. For faster processing:
+- Keep files under 5MB
+- Split large documents into chapters
+</details>
+
+<details>
+<summary><b>❌ Server not starting</b></summary>
+
+1. Check if port 8000 is in use
+2. Verify virtual environment is activated
+3. Check all dependencies: `pip install -r requirements.txt`
+</details>
+
+<details>
+<summary><b>❌ "Model dimension mismatch"</b></summary>
+
+The embedding model dimension must match Pinecone index:
+- `fast` model → 384 dimensions
+- `balanced` model → 768 dimensions
+- `quality` model → 1024 dimensions
+
+Either recreate the Pinecone index or change `CURRENT_MODEL` in config.py
+</details>
+
+---
 
 ## 🤝 Contributing
 
-Contributions are welcome! Areas for improvement:
-- Additional file format support
-- More embedding model options
-- Advanced retrieval strategies
-- UI/UX enhancements
-- Performance optimizations
+Contributions are welcome! Please:
 
-## 📄 License
+1. Fork the repository
+2. Create a feature branch: `git checkout -b feature/amazing-feature`
+3. Commit changes: `git commit -m 'Add amazing feature'`
+4. Push to branch: `git push origin feature/amazing-feature`
+5. Open a Pull Request
 
-This project is open source. Feel free to use and modify.
+### Development Setup
 
-## 🙏 Acknowledgments
+```bash
+# Install development dependencies
+pip install -r requirements-dev.txt
 
-- **Haystack AI** - RAG framework
-- **Pinecone** - Vector database
-- **Google AI** - Gemini LLM
-- **Sentence Transformers** - Embedding models
+# Run tests
+pytest
 
-## 📞 Support
+# Format code
+black . --line-length 100
+```
+
+---
+
+## 📜 License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+---
+
+## 👤 Author
+
+**Vikash**
+- GitHub: [@vikash-48413](https://github.com/vikash-48413)
+- Email: vikash17052005@gmail.com
+
+---
+
+## 🙏 Acknowledgments
 
-For issues and questions:
-1. Check [INSTALLATION.md](INSTALLATION.md) for setup help
-2. Review [FEATURES.md](FEATURES.md) for usage details
-3. Check console output for error details
-4. Open an issue on GitHub
+- [Haystack](https://haystack.deepset.ai/) — RAG framework
+- [Pinecone](https://www.pinecone.io/) — Vector database
+- [Google AI](https://ai.google.dev/) — Gemini LLM
+- [Sentence Transformers](https://www.sbert.net/) — Embeddings
+- [FastAPI](https://fastapi.tiangolo.com/) — Web framework
 
 ---
 
-**Made with ❤️ for researchers, students, and knowledge workers**
+<p align="center">
+  <b>⭐ Star this repository if you find it useful!</b>
+</p>

TESTING_GUIDE.md

@@ -1,326 +0,0 @@
-# Testing Guide - Large File Upload Fixes
-
-## Quick Test Checklist
-
-### ✅ Pre-Flight Checks
-1. Server starts without errors
-2. Pinecone connection works
-3. No Python dependency issues
-
-### ✅ File Upload Tests
-
-#### Test 1: Small File (< 1MB)
-- **File**: Any PDF < 1MB
-- **Expected**: Fast upload (< 30s)
-- **Check**: Progress bar, success message, file size shown
-
-#### Test 2: Medium File (1-3MB)
-- **File**: Research paper 1-3MB
-- **Expected**: Normal processing (30-60s)
-- **Check**: Optimized settings NOT activated
-
-#### Test 3: Large File (3-5MB) ⭐ PRIMARY TEST
-- **File**: Research paper ~5MB
-- **Expected**: Optimized processing (1-2 min)
-- **Check**: Console shows "Large file detected - using optimized settings"
-- **Verify**: 
-  - Chunk length: 350 words
-  - Batch size: 20 chunks
-  - Memory cleanup messages appear
-
-#### Test 4: Very Large File (5-10MB)
-- **File**: Long research paper 5-10MB
-- **Expected**: Slower but successful (2-5 min)
-- **Check**: All chunks processed, no memory errors
-
-#### Test 5: Maximum Size (45-50MB)
-- **File**: Very large document
-- **Expected**: Works but takes time (5-10 min)
-- **Warning**: May be slow, monitor console
-
-#### Test 6: Oversized File (> 50MB)
-- **File**: Any file > 50MB
-- **Expected**: Clear error message
-- **Error Text**: "File size (XXmb) exceeds maximum allowed size (50MB)"
-- **HTTP Code**: 413 (Payload Too Large)
-
-### ✅ Error Handling Tests
-
-#### Test 7: Empty File
-- **File**: Create empty .pdf file
-- **Expected**: Error "File is empty"
-- **HTTP Code**: 400
-
-#### Test 8: Wrong File Type
-- **File**: .exe, .zip, .mp3, etc.
-- **Expected**: Error listing supported formats
-- **Supported**: PDF, DOCX, DOC, TXT, MD, CSV, JSON, XLSX, XLS
-
-#### Test 9: Corrupted PDF
-- **File**: Damaged PDF file
-- **Expected**: Error during content extraction
-- **Check**: Proper error message, cleanup occurs
-
-### ✅ Performance Tests
-
-#### Test 10: Memory Usage
-1. Note starting memory (Task Manager)
-2. Upload 5MB file
-3. Check memory during processing
-4. Verify memory cleanup after completion
-5. **Expected**: Memory returns near baseline
-
-#### Test 11: Multiple Uploads
-1. Upload file A (5MB)
-2. Wait for completion
-3. Delete file A
-4. Upload file B (5MB)
-5. **Expected**: Both work, no accumulation errors
-
-#### Test 12: Upload Cancellation
-1. Start uploading large file
-2. Refresh page mid-upload
-3. **Expected**: Cleanup occurs, no orphaned files
-
-### ✅ Progress Tracking Tests
-
-#### Test 13: Progress Updates
-1. Upload 5MB file
-2. Watch console output
-3. **Check for**:
-   - ✓ File type validated
-   - ✓ File read successfully: X.XXmb
-   - ✓ Cleared previous uploads
-   - ✓ File saved to: uploads/...
-   - ✓ Initializing document store
-   - ✓ Converting file
-   - ✓ Extracted content
-   - ✓ Created X chunks
-   - ✓ Batch N/M complete
-   - ✓ Memory cleanup
-   - ✓ Wrote X chunks to Pinecone
-   - ✅ Ingestion completed successfully
-
-#### Test 14: Frontend Progress
-1. Upload any file
-2. Watch progress bar
-3. **Check**:
-   - Shows percentage
-   - Updates in real-time
-   - Shows "Complete! ✓" at end
-
-## Console Output Examples
-
-### ✅ Successful 5MB Upload
-```
-📥 Upload endpoint called - Filename: research_paper.pdf
-✓ File type validated: .pdf
-📦 Reading file in chunks...
-✓ File read successfully: 5.23MB
-🧹 Clearing previous uploads...
-✓ File saved to: uploads\research_paper.pdf
-📤 Upload started: research_paper.pdf (5.23MB)
-🔧 Initializing document store...
-📊 Processing document...
-
-📄 Starting document ingestion: research_paper.pdf
-   💾 Server baseline memory: 450.2MB
-   ✓ Cleared existing documents from vector store
-   🔄 Converting .pdf file...
-   ✓ Extracted content (1 document(s), 45230 chars)
-   📊 File size: 5.23MB
-   ⚡ Large file detected - using optimized settings:
-      Chunk length: 350 words
-      Batch size: 20 chunks
-   ✂️  Splitting into chunks...
-   ✓ Created 156 chunks
-   🧠 Embedding chunks with parallel processing...
-   📦 Processing 156 chunks in 8 batches (batch_size=20)
-   ✓ Batch 1/8 complete (20 chunks, +15.2MB)
-   ✓ Batch 2/8 complete (20 chunks, +14.8MB)
-   ✓ Batch 3/8 complete (20 chunks, +15.1MB)
-   🧹 Memory cleanup: 485.3MB
-   ✓ Batch 4/8 complete (20 chunks, +15.0MB)
-   ...
-   ✓ Batch 8/8 complete (16 chunks, +12.1MB)
-   ✓ Successfully embedded 156 chunks
-   💾 Writing to vector database...
-   ✓ Wrote 156 chunks to Pinecone
-
-✅ Ingestion completed successfully!
-   📊 Statistics:
-   • Document: research_paper.pdf
-   • Format: PDF
-   • Size: 5.23MB
-   • Chunks created: 156
-   • Time taken: 94.32 seconds
-   • Speed: 1.7 chunks/sec
-   • Memory used: +45.3MB (450.2MB → 495.5MB)
-⏱️  Total processing time: 94.3s
-✅ Upload completed: research_paper.pdf
-```
-
-### ❌ Failed Upload - File Too Large
-```
-📥 Upload endpoint called - Filename: huge_file.pdf
-✓ File type validated: .pdf
-📦 Reading file in chunks...
-❌ Error: File size (52.34MB) exceeds maximum (50MB)
-```
-
-### ❌ Failed Upload - Empty File
-```
-📥 Upload endpoint called - Filename: empty.pdf
-✓ File type validated: .pdf
-📦 Reading file in chunks...
-✓ File read successfully: 0.00MB
-❌ Error: File is empty
-```
-
-## Browser Console Checks
-
-### Successful Upload
-```javascript
-File selected: File {name: "paper.pdf", size: 5485760}
-File: paper.pdf, Size: 5.23MB
-Starting upload for: paper.pdf
-Sending request to /upload_document
-Processing progress: 30%
-Processing progress: 50%
-Processing progress: 70%
-Processing progress: 90%
-```
-
-### Failed Upload
-```javascript
-File: huge.pdf, Size: 52.34MB
-❌ File too large error shown
-```
-
-## What to Look For
-
-### ✅ SUCCESS Indicators
-- Progress bar completes to 100%
-- Green success notification
-- File appears in "Current Document" section
-- Console shows "✅ Ingestion completed successfully"
-- Can ask questions immediately
-
-### ❌ FAILURE Indicators
-- Red error notification
-- Clear error message
-- Console shows "❌" errors
-- File automatically cleaned up
-- Can upload again without issues
-
-## Performance Benchmarks
-
-| File Size | Expected Time | Chunks | Memory Usage |
-|-----------|---------------|--------|---------------|
-| 0.5 MB    | 15-20s        | 40-60  | +20-30MB      |
-| 1 MB      | 25-35s        | 80-100 | +30-40MB      |
-| 3 MB      | 50-70s        | 120-140| +40-60MB      |
-| 5 MB      | 80-120s       | 150-180| +50-80MB      |
-| 10 MB     | 150-240s      | 280-320| +80-120MB     |
-
-*Times vary based on system specs and model choice*
-
-## Common Issues & Solutions
-
-### Issue: "Timeout after 10 minutes"
-**Cause**: File too complex or system too slow  
-**Solution**: 
-- Try smaller file
-- Use "fast" model in config.py
-- Increase chunk size in config.py
-
-### Issue: "Memory Error"
-**Cause**: Insufficient RAM  
-**Solution**:
-- Close other applications
-- Increase batch size in config.py
-- Use "fast" model (uses less memory)
-
-### Issue: "No chunks embedded"
-**Cause**: Pinecone connection issue  
-**Solution**:
-- Check PINECONE_API_KEY in .env
-- Verify internet connection
-- Check Pinecone index exists
-
-### Issue: "File not found after upload"
-**Cause**: Permission or path issue  
-**Solution**:
-- Check uploads/ directory exists
-- Verify write permissions
-- Check disk space
-
-## Quick Debug Commands
-
-### Check Server Logs
-```bash
-# Look for errors in console
-# Search for ❌ or "ERROR" or "Exception"
-```
-
-### Check File Size
-```bash
-# Windows PowerShell
-(Get-Item "path\to\file.pdf").Length / 1MB
-```
-
-### Check Memory Usage
-```bash
-# Windows Task Manager
-# Look for python.exe process
-```
-
-### Check Uploads Directory
-```bash
-# PowerShell
-Get-ChildItem uploads\
-```
-
-## Success Criteria
-
-For a 5MB file upload to be considered successful:
-
-1. ✅ File uploads without errors
-2. ✅ Optimized settings activate (console shows message)
-3. ✅ All chunks process successfully
-4. ✅ Data written to Pinecone
-5. ✅ Memory cleanup occurs
-6. ✅ Processing completes in < 5 minutes
-7. ✅ Can ask questions and get answers
-8. ✅ Statistics show correct file size
-
-## Report Template
-
-When reporting issues, include:
-
-```
-File Details:
-- Name: _______
-- Size: _______
-- Type: _______
-
-Error:
-- Message: _______
-- HTTP Code: _______
-- When occurred: _______
-
-Console Output:
-[Paste last 20-30 lines]
-
-Browser Console:
-[Paste any errors]
-
-System:
-- OS: _______
-- RAM: _______
-- Python Version: _______
-```
-
----
-
-**Need Help?** Check [FIXES_APPLIED.md](FIXES_APPLIED.md) for detailed technical information.

UPGRADE_SUMMARY.md

@@ -1,324 +0,0 @@
-# 🎯 PaperBOT v2.0 - Upgrade Summary
-
-## ✅ Completed Improvements
-
-### 1. Multi-Format File Support ✅
-**What Changed:**
-- Added support for 7+ file formats beyond just PDF/TXT
-- New formats: DOCX, DOC, MD, CSV, JSON, XLSX, XLS
-
-**Implementation:**
-- Created converter functions for each format
-- Added `python-docx` for Word documents
-- Added `pandas` for CSV/Excel files
-- Integrated Markdown support
-- JSON parsing with proper formatting
-
-**Files Modified:**
-- `QASystem/ingestion.py` - Added converter functions
-- `app.py` - Updated file validation
-- `templates/index.html` - Updated UI file types
-- `requirements.txt` - Added dependencies
-
----
-
-### 2. Parallel Processing ✅
-**What Changed:**
-- Implemented batch processing for embeddings
-- Added multi-threaded document upload
-- Concurrent chunk processing
-
-**Implementation:**
-- `process_chunks_parallel()` function for batch embedding
-- ThreadPoolExecutor in `app.py` for async upload
-- Configurable batch size (default: 32 chunks)
-- Progress tracking throughout pipeline
-
-**Performance Gain:**
-- 3-5x faster processing
-- 30-50 chunks/second (up from 10-15)
-- Non-blocking UI during upload
-
-**Files Modified:**
-- `QASystem/ingestion.py` - Parallel processing logic
-- `app.py` - Async upload handling
-
----
-
-### 3. Memory Management ✅
-**What Changed:**
-- Real-time memory monitoring
-- Automatic garbage collection
-- Batch-wise memory cleanup
-- Memory usage reporting
-
-**Implementation:**
-- Added `psutil` for memory monitoring
-- `get_memory_usage()` function
-- `clear_memory()` for garbage collection
-- Cleanup every 5 batches
-- Memory stats in console output
-
-**Memory Optimization:**
-- Prevents memory leaks
-- Handles large documents (100+ pages)
-- Configurable batch sizes for memory control
-
-**Files Modified:**
-- `QASystem/ingestion.py` - Memory management functions
-- `requirements.txt` - Added psutil
-
----
-
-### 4. Enhanced Semantic Search ✅
-**What Changed:**
-- Increased retrieval candidates from 5 to 10
-- Added cached embedders for faster queries
-- Implemented fallback retrieval strategies
-- Added relevance scoring display
-
-**Implementation:**
-- `get_text_embedder()` with caching
-- Enhanced `get_result()` function
-- Multiple fallback mechanisms
-- Relevance score calculation
-- Better error handling
-
-**Search Quality:**
-- Better context coverage
-- More accurate answers
-- Faster query responses (2-5s)
-- Graceful degradation on errors
-
-**Files Modified:**
-- `QASystem/retrieval_and_generation.py` - Enhanced retrieval
-
----
-
-### 5. Progress Tracking & Error Handling ✅
-**What Changed:**
-- Real-time progress updates
-- Comprehensive error messages
-- Detailed console logging
-- Better user feedback
-
-**Implementation:**
-- Progress callbacks in ingestion
-- Try-catch blocks everywhere
-- Detailed error traces
-- User-friendly error messages
-- Troubleshooting hints
-
-**User Experience:**
-- Clear status updates
-- Informative error messages
-- Debug information in console
-- Recovery suggestions
-
-**Files Modified:**
-- `app.py` - Progress tracking
-- `QASystem/ingestion.py` - Logging
-- `templates/index.html` - UI feedback
-
----
-
-### 6. Dependencies Updated ✅
-**What Changed:**
-- Added 8 new packages
-- Updated requirements.txt
-- Created .env.example
-
-**New Dependencies:**
-```
-python-docx      # Word documents
-pandas           # Data processing
-openpyxl         # Excel files
-psutil           # Memory monitoring
-tqdm             # Progress bars
-markdown         # Markdown support
-jinja2           # Templates
-```
-
-**Files Modified:**
-- `requirements.txt` - Complete dependencies
-- `.env.example` - Environment template
-
----
-
-### 7. Comprehensive Documentation ✅
-**What Changed:**
-- Created 6 new documentation files
-- Added helper scripts
-- Created quick reference guides
-
-**New Files:**
-- `README.md` - Complete project overview
-- `INSTALLATION.md` - Setup guide
-- `FEATURES.md` - Feature documentation
-- `QUICKSTART.md` - Quick reference
-- `CHANGELOG.md` - Version history
-- `.env.example` - Config template
-- `test_system.py` - System test script
-- `start.bat` / `start.sh` - Startup scripts
-
----
-
-## 📊 Performance Comparison
-
-| Metric | Before (v1.0) | After (v2.0) | Improvement |
-|--------|---------------|--------------|-------------|
-| Upload Speed | 5-10s | 1-3s | **3x faster** |
-| Processing Speed | 10-15 chunks/s | 30-50 chunks/s | **3-4x faster** |
-| Query Response | 5-10s | 2-5s | **2x faster** |
-| Memory Usage | Unoptimized | Monitored & Optimized | **Better** |
-| File Formats | 2 (PDF, TXT) | 7+ formats | **4x more** |
-| Error Handling | Basic | Comprehensive | **Much better** |
-
----
-
-## 🔧 Key Code Improvements
-
-### Ingestion Pipeline (Before)
-```python
-# Simple pipeline, no parallelization
-indexing.run({"converter": {"sources": [file]}})
-```
-
-### Ingestion Pipeline (After)
-```python
-# Parallel processing with memory management
-documents = convert_to_documents(file)  # Multi-format
-chunks = split_documents(documents)
-embedded = process_chunks_parallel(chunks)  # Parallel!
-write_to_store(embedded)
-clear_memory()  # Memory cleanup
-```
-
-### Retrieval (Before)
-```python
-# Simple retrieval, top_k=5
-retriever = PineconeEmbeddingRetriever(top_k=5)
-```
-
-### Retrieval (After)
-```python
-# Enhanced retrieval with caching and fallbacks
-embedder = get_text_embedder()  # Cached!
-retriever = PineconeEmbeddingRetriever(top_k=10)  # More candidates
-# + Multiple fallback strategies
-```
-
----
-
-## 📁 Project Structure (Updated)
-
-```
-PaperBOT/
-├── 📄 app.py                          # FastAPI app (enhanced)
-├── 📄 requirements.txt                # Updated dependencies
-├── 📄 setup.py                        # Package setup
-├── 📄 .env.example                    # NEW: Config template
-├── 📄 .gitignore                      # Git ignore rules
-├── 📄 test_system.py                  # NEW: System test
-├── 📄 start.bat                       # NEW: Windows startup
-├── 📄 start.sh                        # NEW: Linux/Mac startup
-│
-├── 📚 Documentation (NEW)
-│   ├── README.md                      # Complete overview
-│   ├── INSTALLATION.md                # Setup guide
-│   ├── FEATURES.md                    # Feature docs
-│   ├── QUICKSTART.md                  # Quick reference
-│   └── CHANGELOG.md                   # Version history
-│
-├── 📂 QASystem/
-│   ├── __init__.py
-│   ├── config.py                      # Performance config
-│   ├── ingestion.py                   # Enhanced with parallel processing
-│   ├── retrieval_and_generation.py    # Enhanced semantic search
-│   └── utils.py                       # Utilities
-│
-├── 📂 templates/
-│   └── index.html                     # Updated UI
-│
-├── 📂 uploads/                        # Temp storage
-│   └── .gitkeep                       # NEW
-│
-└── 📂 data/                           # Sample docs
-```
-
----
-
-## 🎯 How to Use the Upgrades
-
-### 1. Install New Dependencies
-```bash
-pip install -r requirements.txt
-```
-
-### 2. Configure Performance (Optional)
-Edit `QASystem/config.py`:
-```python
-# For speed
-CURRENT_MODEL = "fast"
-BATCH_SIZE = 64
-
-# For quality
-CURRENT_MODEL = "quality"
-BATCH_SIZE = 16
-```
-
-### 3. Upload New File Types
-- Drag & drop any supported format
-- System auto-detects and converts
-- Watch progress bar for status
-
-### 4. Monitor Performance
-- Check console for memory stats
-- View processing speed (chunks/sec)
-- See relevance scores in answers
-
----
-
-## 🚀 Next Steps
-
-### To Run the Application:
-```bash
-# Option 1: Use startup script
-start.bat  # Windows
-./start.sh  # Linux/Mac
-
-# Option 2: Manual start
-python app.py
-```
-
-### To Test the System:
-```bash
-python test_system.py
-```
-
-### To Read Documentation:
-- Quick start: `QUICKSTART.md`
-- Full guide: `README.md`
-- Features: `FEATURES.md`
-- Installation help: `INSTALLATION.md`
-
----
-
-## 📝 Summary
-
-✅ **7 Major Improvements Completed**
-✅ **6 New Documentation Files**
-✅ **3-5x Performance Improvement**
-✅ **7+ File Formats Supported**
-✅ **Optimized Memory Management**
-✅ **Enhanced Semantic Search**
-✅ **Production-Ready Code**
-
-The application is now:
-- ⚡ **Faster** - Parallel processing, cached models
-- 🧠 **Smarter** - Better retrieval, relevance scoring
-- 💪 **Stronger** - Memory management, error handling
-- 📚 **More Capable** - 7+ file formats
-- 🎯 **Better Documented** - Comprehensive guides
-
-**Status: READY FOR PRODUCTION USE! 🎉**

data/.gitkeep

@@ -1,2 +0,0 @@
-# This file ensures the data directory is tracked by Git
-# Preloaded PDF files can be placed here

uploads/.gitkeep

@@ -1,2 +0,0 @@
-# This file ensures the uploads directory is tracked by Git
-# User-uploaded files will be stored here but are ignored by .gitignore