Introduction

Building a local AI assistant that respects privacy while delivering real utility is a significant technical challenge. This tutorial walks through the architectural decisions, implementation patterns, and lessons learned from building Xoul—a local AI agent for Windows that runs entirely on consumer hardware without sending data to external APIs.

Unlike cloud-based assistants, Xoul uses local LLMs (Large Language Models) running via Ollama, executes tasks in an isolated QEMU virtual machine, and maintains a three-tier memory system. The entire setup requires no terminal commands and runs on GPUs with as little as 5GB VRAM.

Prerequisites

Before starting this implementation, ensure you have:

• Windows 10/11 with administrative privileges
• NVIDIA GPU with 5GB+ VRAM (tested on GTX 2080)
• 50GB free disk space for VM and models
• Basic understanding of Python (async/await patterns helpful)
• Familiarity with LLM concepts and tool calling

Software dependencies installed automatically:

• QEMU (via winget)
• Ollama for Windows
• Python 3.10+
• Ubuntu Cloud image (downloaded during setup)

Learning Objectives

By completing this tutorial, you will understand:

• How to architect a local AI agent with proper security isolation
• Implementing a two-phase agent loop (Planning → Execution) to improve local model reliability
• Building a multi-model tool call parser that handles diverse LLM output formats
• Designing a three-tier memory system for conversational context
• Creating one-click installation flows for non-technical users

Step-by-Step Implementation Guide

Step 1: Design the Core Architecture

The fundamental architectural decision is separation of concerns: the LLM inference runs on the Windows host while agent execution happens in an isolated Linux VM. This provides both security (malicious commands can't damage the host) and functionality (full Linux toolchain access).

Architecture components:

• Host Layer: Ollama server, desktop client, VM manager
• VM Layer: Agent server, tool executors, memory database
• Communication: SSH tunnel + port forwarding (2222→22, 3000→3000)

The VM accesses the host's Ollama instance via the special QEMU gateway address 10.0.2.2:11434. This allows the agent to make inference requests without exposing ports externally.

Step 2: Implement Automated Setup

The setup_env.ps1 PowerShell script (751 lines) handles the entire installation process. The critical insight: users must never open a terminal.

Setup flow implementation:

# Detect GPU VRAM via nvidia-smi
$vram = (nvidia-smi --query-gpu=memory.total --format=csv,noheader,nounits | Select-Object -First 1)

if ($vram -lt 6000) {
    Write-Host "Recommending Q4 4-bit quantized model (3.5GB VRAM)"
    $model = "qwen2.5:7b-instruct-q4_K_M"
} elseif ($vram -lt 12000) {
    Write-Host "Recommending Q8 8-bit quantized model (7GB VRAM)"
    $model = "qwen2.5:7b-instruct-q8_0"
} else {
    Write-Host "Recommending full precision 30B model"
    $model = "qwen2.5:30b-instruct"
}

The script automatically handles QEMU installation via winget, downloads the Ubuntu Cloud image, and configures SSH key pairs. A critical gotcha: QEMU on Windows fails with paths containing spaces or non-ASCII characters. Solution: convert to Windows 8.3 short path format.

function Get-ShortPath($longPath) {
    $fso = New-Object -ComObject Scripting.FileSystemObject
    $folder = $fso.GetFolder($longPath)
    return $folder.ShortPath
}

Step 3: Build the VM Manager

The vm_manager.py module (979 lines) manages the complete VM lifecycle. The key method is setup(), which orchestrates image download, cloud-init configuration, and first boot.

class VMManager:
    async def setup(self):
        # Download Ubuntu Cloud image
        if not self.image_path.exists():
            await self._download_cloud_image()

    # Create user-data for cloud-init
    cloud_config = {
        'users': [{
            'name': 'xoul',
            'ssh_authorized_keys': [self.ssh_public_key],
            'sudo': 'ALL=(ALL) NOPASSWD:ALL'
        }],
        'packages': ['python3-pip', 'git', 'sqlite3'],
        'runcmd': [
            'systemctl enable ssh',
            'mkdir -p /home/xoul/share'
        ]
    }
    
    # Boot VM with cloud-init
    await self._boot_with_cloudinit(cloud_config)
    
    # Wait for SSH availability
    await self._wait_for_ssh(timeout=120)</code></pre><p>File sharing between Windows and VM uses SCP exclusively, targeting a single <code>share/</code> directory. Initial attempts with 9P VirtIO filesystem mounts proved unreliable on Windows, so the implementation defaults to the simpler SCP approach despite slightly higher latency.</p><h3>Step 4: Implement the Two-Phase Agent Loop</h3><p>The agent loop in <code>server.py</code> (400+ lines) is the intelligence core. Initial implementations used a simple "get response → execute tools" pattern, which failed catastrophically with local models due to poor tool calling reliability.</p><p><strong>Solution: Split into Planning and Execution phases.</strong></p><pre><code>async def _run_agent_loop(self, message: str, conversation_id: str):
# Phase 1: Planning
planning_prompt = f"""Analyze this request and create an execution plan:

{message}
Available tools: {self._format_tool_descriptions()}
Respond with:

What information you need

Which tools you'll use

Expected order of execution"""
 plan = await self.llm.generate(planning_prompt)
Phase 2: Execution
 execution_prompt = f"""Execute this plan:


{plan}
User request: {message}
Call the necessary tools to complete the task."""
response = await self.llm.generate_with_tools(
    execution_prompt,
    tools=self.tool_registry.get_tools()
)

# Execute tool calls with guardrails
results = await self._execute_with_guardrails(response.tool_calls)

# Phase 3: Synthesis
final_answer = await self.llm.generate(f"""Based on these results:

{results}
Provide a clear answer to: {message}""")
return final_answer</code></pre><p>This architectural change doubled the perceived success rate of tool execution. The planning phase forces the model to think through its approach before committing to tool calls.</p><h3>Step 5: Add Execution Guardrails</h3><p>Local models make predictable mistakes. Guardrails prevent these at the code level:</p><p><strong>Duplicate Call Prevention:</strong></p><pre><code>def _deduplicate_tool_calls(self, tool_calls: list) -> list:
seen = set()
unique_calls = []

for call in tool_calls:
    # Create deterministic key from tool name + arguments
    key = f"{call.name}:{json.dumps(call.args, sort_keys=True)}"
    
    if key not in seen:
        seen.add(key)
        unique_calls.append(call)
    else:
        # Inject cached result
        call.result = self._get_cached_result(key)
        call.from_cache = True
        unique_calls.append(call)

return unique_calls</code></pre><p><strong>Auto-Fetch Chain:</strong> After web search, automatically fetch the top result to prevent hallucination from title-only information.</p><pre><code>async def _execute_with_auto_fetch(self, tool_call):
result = await self._execute_tool(tool_call)

if tool_call.name == 'web_search' and result.get('results'):
    top_url = result['results'][0]['url']
    
    # Automatically fetch full content
    fetch_call = ToolCall(
        name='fetch_url',
        args={'url': top_url}
    )
    
    content = await self._execute_tool(fetch_call)
    result['top_result_content'] = content

return result</code></pre><p><strong>Context Compaction:</strong> After each turn, compress verbose tool execution logs to prevent context window explosion.</p><pre><code>def compact_after_turn(self, messages: list) -> list:
compacted = []

for msg in messages:
    if msg['role'] == 'tool':
        # Compress tool output
        if len(msg['content']) > 1000:
            msg['content'] = msg['content'][:500] + 
                "\n[...truncated...]\n" + 
                msg['content'][-500:]
    
    # Remove ephemeral system prompts
    if msg.get('ephemeral'):
        continue
        
    compacted.append(msg)

return compacted</code></pre><h3>Step 6: Build the Multi-Model Tool Call Parser</h3><p>Different LLMs use incompatible tool calling formats. The <code>tool_call_parser.py</code> implements a fallback chain that tries model-specific parsers first, then falls back to generic strategies.</p><pre><code>class ToolCallParser:
def parse(self, model_name: str, response: str) -> list[ToolCall]:
    # Detect model family
    if 'qwen' in model_name.lower():
        parser = self._parse_qwen_xml
    elif 'glm' in model_name.lower():
        parser = self._parse_glm_function
    elif 'nemotron' in model_name.lower():
        parser = self._parse_nemotron_xml
    else:
        parser = self._parse_generic
    
    # Try specific parser
    try:
        calls = parser(response)
        if calls:
            return calls
    except Exception as e:
        logger.warning(f"Primary parser failed: {e}")
    
    # Fallback chain
    for fallback in [self._parse_json_array, 
                     self._parse_xml_any, 
                     self._parse_function_notation]:
        try:
            calls = fallback(response)
            if calls:
                return calls
        except:
            continue
    
    return []

def _parse_qwen_xml(self, text: str) -> list[ToolCall]:
    pattern = r'<tool_call>\s*<name>(.*?)</name>\s*<args>(.*?)</args>\s*</tool_call>'
    matches = re.finditer(pattern, text, re.DOTALL)
    
    calls = []
    for match in matches:
        name = match.group(1).strip()
        args_json = match.group(2).strip()
        
        try:
            args = json.loads(args_json)
            calls.append(ToolCall(name=name, args=args))
        except json.JSONDecodeError:
            continue
    
    return calls</code></pre><p>This design means users can switch models in <code>config.json</code> and tool calling continues working without code changes. llm-tool-calling</p><h3>Step 7: Implement Three-Tier Memory</h3><p>The memory system in <code>memory_tools.py</code> (601 lines) uses a cognitive psychology-inspired architecture: STM → MTM → LTM.</p><p><strong>Short-Term Memory (STM):</strong> Records every conversation turn immediately to SQLite. Capped at 5000 characters per entry.</p><pre><code>async def record_stm(self, conversation_id: str, role: str, content: str):
# Truncate if needed
if len(content) > 5000:
    content = content[:5000] + "...[truncated]"

await self.db.execute(
    "INSERT INTO stm (conversation_id, role, content, timestamp) VALUES (?, ?, ?, ?)",
    (conversation_id, role, content, time.time())
)</code></pre><p><strong>Mid-Term Memory (MTM):</strong> After 10+ minutes of inactivity, summarize the conversation session into key points.</p><pre><code>async def consolidate_to_mtm(self, conversation_id: str):
# Get recent STM entries
stm_entries = await self.db.fetch_all(
    "SELECT * FROM stm WHERE conversation_id = ? AND timestamp > ?",
    (conversation_id, time.time() - 600)
)

if not stm_entries:
    return

# Summarize via LLM
conversation_text = "\n".join([f"{e['role']}: {e['content']}" for e in stm_entries])

summary_prompt = f"""Summarize this conversation into 3-5 key points:

{conversation_text}
Focus on:

User preferences mentioned

Important facts or decisions

Action items or commitments"""
  summary = await self.llm.generate(summary_prompt)
Store in MTM
  await self.db.execute(
  "INSERT INTO mtm (conversation_id, summary, timestamp) VALUES (?, ?, ?)",
  (conversation_id, summary, time.time())
  )

Long-Term Memory (LTM): Explicit facts stored via remember() tool. Searchable via vector embeddings.

async def remember(self, key: str, value: str, category: str = "general"):

Generate embedding for semantic search
  embedding = await self.embedding_model.encode(f"{key}: {value}")
  await self.db.execute(
  "INSERT INTO ltm (key, value, category, embedding, timestamp) VALUES (?, ?, ?, ?, ?)",
  (key, value, category, embedding.tobytes(), time.time())
  )


async def recall(self, query: str, limit: int = 5) -> list:
    query_embedding = await self.embedding_model.encode(query)
# Cosine similarity search
results = await self.db.fetch_all(
    """SELECT key, value, category, 
       cosine_similarity(embedding, ?) as score
       FROM ltm
       ORDER BY score DESC
       LIMIT ?""",
    (query_embedding.tobytes(), limit)
)

return results</code></pre><p>The three tiers balance performance (STM is fast, raw SQL), context preservation (MTM maintains conversation flow), and long-term knowledge retention (LTM enables semantic recall). vector-databases</p><h3>Step 8: Create the Desktop Client</h3><p>The desktop client connects to the agent server's API on port 3000. Implementation uses a lightweight HTTP client with streaming support for real-time responses.</p><pre><code>class DesktopClient:
async def send_message(self, message: str):
    async with aiohttp.ClientSession() as session:
        async with session.post(
            'http://localhost:3000/api/chat',
            json={'message': message, 'conversation_id': self.conversation_id},
            timeout=aiohttp.ClientTimeout(total=300)
        ) as response:
            # Stream response chunks
            async for chunk in response.content.iter_chunked(1024):
                data = json.loads(chunk)
                
                if data.get('type') == 'tool_call':
                    self.display_tool_execution(data)
                elif data.get('type') == 'response':
                    self.display_response(data['content'])
                elif data.get('type') == 'error':
                    self.display_error(data['message'])</code></pre><h2>Troubleshooting Common Issues</h2><h3>VM Won't Start or SSH Connection Fails</h3><p><strong>Symptom:</strong> QEMU process starts but SSH connection times out or VM doesn't respond.</p><p><strong>Solutions:</strong></p><ul><li>Check for spaces or non-ASCII characters in installation path. Use the short path converter function.</li><li>Verify port 2222 isn't already in use: <code>netstat -ano | findstr :2222</code></li><li>Check QEMU process logs: <code>vm_manager.py</code> writes to <code>logs/qemu.log</code></li><li>Ensure cloud-init completed: SSH into VM manually and check <code>/var/log/cloud-init.log</code></li></ul><h3>Local Model Not Calling Tools Correctly</h3><p><strong>Symptom:</strong> Model generates text responses instead of tool calls, or calls wrong tools repeatedly.</p><p><strong>Solutions:</strong></p><ul><li>Verify tool descriptions are clear and include examples in the prompt</li><li>Check that the model supports tool calling (not all local models do)</li><li>Enable planning phase: ensure <code>USE_PLANNING_PHASE=true</code> in config</li><li>Increase temperature slightly (0.7-0.8) for more creative tool exploration</li><li>Try a larger model if VRAM allows (7B → 14B significant improvement)</li></ul><h3>Context Window Overflow</h3><p><strong>Symptom:</strong> Agent becomes slow or fails after 5-10 conversation turns.</p><p><strong>Solutions:</strong></p><ul><li>Verify <code>compact_after_turn()</code> is being called after each response</li><li>Reduce <code>MAX_CONTEXT_MESSAGES</code> in config (default 50, try 30)</li><li>Enable MTM consolidation: set <code>MTM_CONSOLIDATE_INTERVAL=600</code></li><li>Clear STM periodically: <code>conversation_id</code> rotation after long sessions</li></ul><h3>High VRAM Usage or Out of Memory</h3><p><strong>Symptom:</strong> System becomes unresponsive or Ollama crashes during generation.</p><p><strong>Solutions:</strong></p><ul><li>Switch to a smaller quantized model (Q8 → Q4, or 7B → 3B)</li><li>Reduce context window: <code>num_ctx=4096</code> instead of 8192 in Ollama modelfile</li><li>Disable parallel requests: <code>MAX_CONCURRENT_REQUESTS=1</code></li><li>Close other GPU-intensive applications</li><li>Monitor with: <code>nvidia-smi dmon -s u</code></li></ul><h2>Best Practices and Optimization</h2><h3>Model Selection Strategy</h3><p>Choose models based on your hardware and use case:</p><ul><li><strong>5-8GB VRAM:</strong> Qwen2.5:7B Q4 quantized. Good balance of speed and capability for general tasks.</li><li><strong>8-12GB VRAM:</strong> Qwen2.5:7B Q8 or GLM-4:9B. Better reasoning, acceptable speed.</li><li><strong>12-24GB VRAM:</strong> Qwen2.5:14B or Nemotron:22B. Strong tool calling and complex reasoning.</li><li><strong>24GB+ VRAM:</strong> Qwen2.5:30B full precision. Near GPT-3.5 level performance on many tasks.</li></ul><p>Test with the installer's auto-recommendation first, then upgrade if needed. ollama-models</p><h3>Prompt Engineering for Local Models</h3><p>Local models require more explicit prompting than GPT-4:</p><ul><li><strong>Always provide examples</strong> of tool usage in the system prompt</li><li><strong>Use structured formats:</strong> XML or JSON schema in instructions</li><li><strong>Break complex tasks</strong> into explicit steps in the planning phase</li><li><strong>Include constraints:</strong> "Call only one tool at a time" if model over-calls</li><li><strong>Add negative examples:</strong> "Do NOT call web_search for math problems"</li></ul><h3>Memory Management</h3><p>Optimal memory configuration depends on usage patterns:</p><ul><li><strong>Frequent short sessions:</strong> Aggressive STM→MTM consolidation (5min interval)</li><li><strong>Long continuous sessions:</strong> Higher STM retention (15min) but frequent compaction</li><li><strong>Knowledge-intensive use:</strong> Liberal use of explicit <code>remember()</code> calls</li><li><strong>Privacy-sensitive data:</strong> Implement LTM encryption at rest</li></ul><h3>Security Hardening</h3><p>For production deployments beyond personal use:</p><ul><li>Enable disk encryption for VM image: <code>qemu-img convert -O qcow2 -o encryption=on</code></li><li>Implement rate limiting on API endpoints</li><li>Add authentication layer (JWT tokens) to desktop client</li><li>Restrict VM network access to localhost only</li><li>Regular automated backups of LTM database</li></ul><h3>Performance Optimization</h3><p>For faster response times:</p><ul><li><strong>Preload models:</strong> Keep Ollama warm with periodic health checks</li><li><strong>Cache embeddings:</strong> Store frequent query embeddings in Redis</li><li><strong>Async everything:</strong> Use <code>asyncio.gather()</code> for parallel tool execution where safe</li><li><strong>SSD for VM image:</strong> Place <code>ubuntu.qcow2</code> on NVMe if available</li><li><strong>Increase VM RAM allocation</strong> if host has 32GB+ (default 4GB → 8GB)</li></ul><h2>Next Steps and Extensions</h2><h3>Adding Custom Tools</h3><p>Extend the agent's capabilities by registering new tools in <code>tool_registry.py</code>:</p><pre><code>@tool_registry.register

async def read_email(count: int = 10) -> list:
    """Fetch recent emails from local mailbox.
Args:
    count: Number of emails to retrieve (default 10)

Returns:
    List of email objects with subject, sender, body
"""
# Implementation here
pass</code></pre><p>The decorator automatically generates tool descriptions for the LLM. Follow the pattern: clear docstring, type hints, and explicit argument descriptions.</p><h3>Multi-Agent Collaboration</h3><p>Scale beyond single-agent by implementing agent-to-agent communication:</p><ul><li>Specialist agents (code, research, data analysis) with dedicated models</li><li>Message bus for inter-agent communication (RabbitMQ or Redis Pub/Sub)</li><li>Coordinator agent that routes requests to specialists</li><li>Shared LTM database with agent-specific namespaces</li></ul><h3>Voice Interface</h3><p>Add speech input/output for hands-free interaction:</p><ul><li>Whisper for local speech-to-text (runs on CPU)</li><li>Piper TTS for natural voice synthesis</li><li>Wake word detection with Porcupine</li><li>Streaming audio pipeline to minimize latency</li></ul><h3>Mobile Client</h3><p>Build iOS/Android apps that connect to the local agent:</p><ul><li>Tailscale for secure remote access without port forwarding</li><li>React Native or Flutter for cross-platform development</li><li>Push notifications for proactive agent alerts</li><li>Offline mode with local message queuing</li></ul><h3>Test-Driven Evolution</h3><p>Implement the "evolving AI" concept from the source material:</p><ul><li>Maintain test suite of user requests + expected behaviors</li><li>Run tests after prompt changes to catch regressions</li><li>Automatic A/B testing of prompt variations</li><li>User feedback loop: thumbs up/down → prompt refinement</li></ul><h2>Conclusion</h2><p>Building a local AI assistant that genuinely respects privacy while remaining useful requires careful architectural choices. The key insights from this implementation:</p><p><strong>Isolation is non-negotiable.</strong> Running agent code in a VM prevents catastrophic host damage and provides a full Linux environment. The overhead is minimal on modern hardware.</p><p><strong>Local models need guardrails.</strong> The two-phase planning loop, duplicate detection, auto-fetch chains, and context compaction aren't optional—they're what make local LLMs viable for agent applications.</p><p><strong>One-click setup matters.</strong> Technical users underestimate how much terminal usage intimidates average users. Automating the entire installation flow dramatically expands your potential user base.</p><p><strong>Memory architecture is foundational.</strong> A three-tier system that balances immediate recall, session continuity, and long-term knowledge retention creates the persistent personality that distinguishes assistants from chatbots.</p><p>The complete implementation runs on consumer hardware (GTX 2080 tested), costs nothing per request, and keeps all personal data local. While local models lag behind GPT-4 in raw capability, the architecture compensates through smarter prompting, better tool integration, and robust error handling.</p><p>For developers interested in the full source code, the Xoul project is open source. The implementation demonstrates that privacy-respecting AI assistants are technically feasible today—no cloud APIs required.</p><aside class="info-box key-takeaway"><strong>Attribution:</strong> Based on the development story shared by Kim Namhyun on DEV Community, documenting the 9-day build process of Xoul, a local AI assistant for Windows users.</aside>