AI & LLM Integration

📋Learning Objectives

• Understand the LLM Provider Abstraction architecture
• Configure and use multiple LLM providers (OpenAI, Anthropic, Azure, Ollama)
• Learn how dynamic model routing selects cost-effective, policy-compliant models
• Implement budget enforcement with soft and hard limits
• Make streaming and non-streaming LLM requests from plugins
• Track costs and understand pricing models

🏗️LLM Provider Architecture

The BFF provides a unified /api/llm/chat/completions endpoint that abstracts multiple providers with built-in authorization, budget controls, and failover.

Key Features

🤖Supported Providers

🔀Dynamic Model Routing

When a requested model is blocked by policy or over budget, the BFF automatically finds an alternative using a weighted score:

• 70% Cost — cheaper models ranked higher
• 20% Latency — faster response times preferred
• 10% Error Rate — more reliable providers preferred

Response Headers

💰Budget Enforcement

Budget limits prevent runaway costs. The BFF tracks spending in Redis and checks PDP policies.

Budget Configuration

# PDP policy with budget constraints
rules:
  - effect: permit
    resource: "llm:openai:chat"
    action: "invoke"
    on_permit:
      constraints:
        model:
          allow: ["gpt-4o-mini", "gpt-4.1"]
        tokens:
          max_output: 2000
          max_stream: 4096
        spend_budget:
          scope: "user"          # user | team | org
          period: "monthly"      # daily | weekly | monthly
          limit_usd: 25.0        # Hard limit in USD

Budget Response Codes

🔧Plugin SDK Integration

Non-Streaming Request

// src/hooks/useLlm.ts
const sdk = (globalThis as any).pluginSdk;

interface ChatMessage {
  role: 'system' | 'user' | 'assistant';
  content: string;
}

interface ChatOptions {
  model?: string;
  max_tokens?: number;
  temperature?: number;
}

export async function chat(
  messages: ChatMessage[],
  options: ChatOptions = {}
): Promise<string> {
  const response = await sdk.api.fetch('/api/llm/chat/completions', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      model: options.model || 'gpt-4o-mini',
      messages,
      max_tokens: options.max_tokens || 1000,
      temperature: options.temperature || 0.7,
      stream: false
    })
  });

  if (!response.ok) {
    if (response.status === 402) {
      throw new Error('Budget exceeded. Please contact your administrator.');
    }
    if (response.status === 403) {
      throw new Error('You do not have permission to use this model.');
    }
    throw new Error(`LLM request failed: ${response.status}`);
  }

  const data = await response.json();
  
  // Log rerouting info
  const rerouted = response.headers.get('x-aria-model-rerouted');
  const selected = response.headers.get('x-aria-model-selected');
  if (rerouted === 'true') {
    console.log(`Model rerouted to: ${selected}`);
  }

  return data.choices[0].message.content;
}

Streaming Request

// Streaming for real-time UI updates
export async function streamChat(
  messages: ChatMessage[],
  onToken: (token: string) => void,
  options: ChatOptions = {}
): Promise<void> {
  const response = await sdk.api.fetch('/api/llm/chat/completions', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      model: options.model || 'gpt-4o-mini',
      messages,
      max_tokens: options.max_tokens || 1000,
      stream: true
    })
  });

  if (!response.ok) {
    throw new Error(`LLM stream failed: ${response.status}`);
  }

  const reader = response.body?.getReader();
  const decoder = new TextDecoder();

  if (!reader) {
    throw new Error('No response body');
  }

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;

    const chunk = decoder.decode(value, { stream: true });
    const lines = chunk.split('\n').filter(line => line.startsWith('data: '));

    for (const line of lines) {
      const data = line.slice(6); // Remove 'data: ' prefix
      if (data === '[DONE]') return;

      try {
        const parsed = JSON.parse(data);
        const token = parsed.choices?.[0]?.delta?.content;
        if (token) {
          onToken(token);
        }
      } catch (e) {
        // Skip malformed chunks
      }
    }
  }
}

Usage in Component

// src/components/AiAssistant.tsx
const React = (globalThis as any).React;
const { useState } = React;

export function AiAssistant() {
  const [input, setInput] = useState('');
  const [response, setResponse] = useState('');
  const [loading, setLoading] = useState(false);

  const handleSubmit = async () => {
    setLoading(true);
    setResponse('');

    try {
      // Use streaming for better UX
      await streamChat(
        [
          { role: 'system', content: 'You are a helpful assistant.' },
          { role: 'user', content: input }
        ],
        (token) => setResponse(prev => prev + token)
      );
    } catch (err: any) {
      setResponse(`Error: ${err.message}`);
    } finally {
      setLoading(false);
    }
  };

  return (
    <div>
      <textarea
        value={input}
        onChange={(e) => setInput(e.target.value)}
        placeholder="Ask me anything..."
      />
      <button onClick={handleSubmit} disabled={loading}>
        {loading ? 'Thinking...' : 'Send'}
      </button>
      <div className="response">{response}</div>
    </div>
  );
}

permissions:
  api:
    - method: POST
      path: /api/llm/chat/completions

⚙️Provider Configuration

LLM providers are configured in config/providers.yaml. This file defines endpoints, credentials, pricing, circuit breakers, and fallback chains.

Complete Provider Schema

# config/providers.yaml
providers:
  - id: openai-primary              # Unique identifier
    kind: openai                    # Provider type: openai, anthropic, ollama
    name: "OpenAI Primary"          # Human-readable name
    enabled: true                   # Enable/disable this provider
    base_url: https://api.openai.com/v1
    api_key: ${OPENAI_API_KEY}      # Environment variable substitution
    default_model: gpt-4o-mini
    
    supported_features:             # Capabilities
      - chat
      - stream
      - tools
      - vision
      - reasoning
    
    egress:                         # Allowed outbound hosts
      allow:
        - api.openai.com
    
    pricing:                        # USD per token
      gpt-4o-mini:
        input: 0.0000005            # $0.50 per 1M tokens
        output: 0.0000015           # $1.50 per 1M tokens
      gpt-4.1:
        input: 0.000005
        output: 0.000015
    
    circuit_breaker:                # Failover control
      threshold: 5                  # Consecutive failures before opening
      timeout: 60                   # Seconds before half-open state
    
    timeouts:
      default: 30.0                 # Non-streaming timeout
      streaming: 120.0              # Streaming timeout

settings:
  health_check_interval: 30
  config_reload_interval: 300
  
  fallback_chains:                  # Automatic failover
    gpt:
      - openai-primary
      - anthropic-primary
      - ollama-local
    claude:
      - anthropic-primary
      - openai-primary
    default:
      - openai-primary
      - anthropic-primary

Circuit Breaker States

Fallback Chain Behavior

# Request: model="gpt-4o-mini"
# Fallback chain: [openai-primary, anthropic-primary, ollama-local]

# Try openai-primary → Circuit open ❌
# Try anthropic-primary → Success ✅
# Response headers:
#   x-aria-provider: anthropic-primary
#   x-aria-model-rerouted: true

🛡️Classifier Guard

The Classifier Guard analyzes prompts before they reach the LLM, detecting and blocking risky content.

Blocked Categories

Per-Category Actions

# config/classifier.yaml
guard:
  enabled: true
  block_labels:
    - secrets
    - prompt_injection
    - jailbreak
  min_conf: 0.75
  per_label_thresholds:
    secrets: 0.60
    prompt_injection: 0.65

  actions:
    high_cost_request:
      cap_tokens: 2048
      route_model: gpt-4o-mini
      budget:
        hold_multiplier: 1.5
        max_cents: 5000
    
    tools_egress_required:
      disallow_stream: true
      cap_tokens: 1024

🔄Hot-Reload Configuration

Update provider configuration without restarting the BFF:

# 1. Edit the providers file
vim config/providers.yaml

# 2. Trigger reload (no restart needed)
curl -X POST http://bff:8000/api/admin/providers/reload \
  -H "Authorization: Bearer $ADMIN_TOKEN"

# 3. Verify changes
curl http://bff:8000/health/providers | jq

What Happens on Reload

• New providers are added instantly
• Modified providers are replaced atomically
• Removed providers are cleaned up
• In-flight requests complete normally
• Circuit breaker states are preserved

kubectl edit configmap bff-providers-config
kubectl exec deployment/bff -- curl -X POST localhost:8000/api/admin/providers/reload

🔍Troubleshooting

402 Payment Required (Budget Exceeded)

Symptom: {"error": "Budget exceeded for scope user:xyz"}

Fixes:

• Wait for budget reset (daily/weekly/monthly)
• Request budget increase from admin
• Use a cheaper model (e.g., gpt-4o-mini instead of gpt-4.1)

403 Forbidden (Model Not Allowed)

Symptom: {"error": "Model gpt-4.1 not allowed by policy"}

Debug:

# Check PDP policy directly
curl -X POST https://pdp.self.empowernow.ai/access/v1/evaluation \
  -H "Authorization: Bearer $ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "subject": { "type": "user", "id": "current" },
    "resource": { "type": "llm", "id": "openai:chat" },
    "action": { "name": "invoke" },
    "context": { 
      "model": "gpt-4.1",
      "reason_admin": true
    }
  }'

Response Rerouted Unexpectedly

Symptom: Requested gpt-4.1 but got gpt-4o-mini

Check:

• Review x-aria-model-rerouted header
• Check x-aria-budget-remaining — may be near limit
• Check x-aria-decision-id and look up in receipts

Streaming Not Working

Symptom: Response comes all at once, not streamed

Fixes:

• Ensure stream: true in request body
• Don't use await response.json() — use response.body.getReader()
• Some models/providers don't support streaming (check features)

❓Knowledge Check

const reader = response.body.getReader();
const decoder = new TextDecoder();

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  
  const chunk = decoder.decode(value, { stream: true });
  // Parse SSE "data: {...}" lines
}

📝Day 22 Checkpoint

• Understand LLM Provider Abstraction architecture
• Know the supported providers and their pricing
• Understand how dynamic model routing works
• Know the difference between 402 and 403 errors
• Can make non-streaming LLM requests from plugins
• Can make streaming LLM requests from plugins
• Understand budget enforcement and holds