Skip to content

Admin Guide

Introduction

LLM-Inspect is an enterprise solution for secure and observable AI interactions. This guide provides comprehensive instructions for system administrators to deploy, manage, and maintain the LLM-Inspect platform. The system uses Docker containers for deployment and includes various services for authentication, monitoring, and security.

Installation

Prerequisites

  • Docker Engine (20.10.x or newer)
  • Docker Compose (2.x or newer)
  • Minimum 32GB RAM
  • 8+ CPU cores
  • 100GB available storage (SSD recommended)
  • Valid SSL certificates
  • Network access to required services
  • A locally running LLM with at-least 70B parameters

Installation Steps

  1. Extract the provided zip file:

  2. Run the installer script:

Basic installation:

./llminspect-cli -p /full/path/to/llminspect -o install

Installation with SSL/TLS:

./llminspect-cli -p /full/path/to/llminspect -o install \
    --cert-file /path/to/certificate.crt \
    --cert-key /path/to/private.key \
    --domain your-domain.com

Important Notes: - The installer script is designed for Linux systems only - The installer should only be used for initial installation - After installation, use standard Docker Compose commands for daily operations - The -p flag must point to the full path where you extracted the files

SSL/TLS Configuration: - The following flags must be used together for proper SSL/TLS setup: - --cert-file: Full path to your SSL certificate file - --cert-key: Full path to your SSL private key file - --domain: Domain name that matches the CN in your certificate - If any of these three SSL parameters are provided, all three must be provided - If none are provided, the installation will proceed with basic SSL/TLS configuration

Environment Configuration: - By default, the installer will prompt you for various configuration values during installation - To bypass interactive configuration, you can use the --env-file parameter:

./llminspect-cli -p /full/path/to/llminspect -o install --env-file /path/to/env.file
- The env file should contain all required environment variables in KEY=VALUE format - Using an env file is recommended for automated deployments or when you want to maintain consistent configurations

For stopping the services:

./llminspect-cli -p /full/path/to/llminspect -o stop

⚠️ IMPORTANT: If Installation Fails ⚠️ If the installer encounters an error and fails to complete, you MUST run the clean operation before attempting to install again:

./llminspect-cli -p /full/path/to/llminspect -o clean
This will revert all changes made by the failed installation attempt and allow you to start fresh.

Parameter Summary: - -p, --project: (Required) Full path to the extracted llminspect directory - -o, --operation: (Required) One of: - 'install': Performs initial installation - 'stop': Stops running services - 'clean': Reverts all file changes made by the installer, restoring the installation directory to its original state. Useful for: - Starting fresh after a failed installation - Recovering from installer-generated configuration errors - Resetting the environment before attempting a new installation - Cleaning up partial changes if installation was interrupted - --env-file: (Optional) Path to environment file for automated configuration - SSL/TLS group (all three required if using SSL): - --domain: Domain name matching certificate CN - --cert-file: Path to SSL certificate - --cert-key: Path to SSL private key - --license: (Optional) To see the license of open-source services used in llminspect

Architecture Overview

LLM-Inspect consists of several key components: - LLM Gateway (LLM-GW): Central entry point for AI model interactions - iGuard: Security validation service - Keycloak: Authentication and user management - Grafana & Prometheus: Monitoring and metrics - MongoDB: Primary database - Redis: Caching layer - Vector DB: Specialised database for AI operations - Tempo: Distributed tracing backend

Configuration

Configuring LLM-Gateway

The LLM-Gateway requires specific configuration parameters to function properly. These configurations can be set through environment variables and volume mounts in your docker-compose.override.yaml file.

Required Environment Variables

Category Variable Description
Django Settings DJANGO_DB_ENGINE Specifies the database backend for Django (SQLite3 in this case)
DJANGO_DB_NAME Path to the SQLite database file
ALLOWED_HOSTS List of hosts/domains that can serve the Django application
Service URLs INSPECT_HEXAGON_URL URL for the Inspect Hexagon service
ISANITIZE_URL URL for the iSanitize service
LOCAL_LLM_URL URL for the local LLM service endpoint
GEMINI_PROXY_URL URL for the Gemini-OpenAI proxy service
IGUARD_SERVICE_URL URL for the iGuard load balancer service
REDIS_URL Main Redis connection URL
Keycloak Settings KEYCLOAK_URL URL for Keycloak authentication userinfo endpoint
API Keys OPENAI_API_KEY API key for OpenAI services
GEMINI_API_KEY URL for the iSanitize service
INSPECTGPT_API_KEY URL for the iSanitize service
OTEL Settings OTEL_ENABLE Toggle for OpenTelemetry tracing (1=enabled)
OTEL_SERVICE_NAME Service name for OpenTelemetry tracing
OTEL_ENABLE_CONSOLE_EXPORTER Toggle for console export of telemetry data
OTLP_EXPORTER_ENDPOINT Endpoint for OpenTelemetry collector
Miscellaneous ENABLE_HTTPS Toggle for HTTPS protocol
CONFIG_FILE Path to configuration file inside the container
CHUNK_SIZE Size of chunks for streaming LLM responses
Celery Settings CELERY_BROKER_URL Redis URL for Celery message broker
CELERY_RESULT_BACKEND Redis URL for storing Celery task results

Apart from the API keys you can leave everything as default and the deployment will still work.

Note It is recommended that any environment variable you wish to change modify it in the docker-compose.override.yaml file instead of the docker-compose.yml.

Any docker compose configuration defined in the override files always takes precedence.

Essential Volume Mounts

The LLM-Gateway requires specific volumes to be mounted for proper operation:

  1. SSL Certificates

    If ENABLE_HTTPS environment variable is set to true than LLM-GW will read the certificates from the path you mount to /app/Cert/

    volumes:
      - /path/to/your/ssl:/app/Cert
      # by default this is ./ssl:/app/Cert so if you have ssl folder in root directory than no need to change
    

  2. Configuration File

    You need to create conf.yaml file in your desired path on the host and mount it to app/conf.yaml inside the container, any changes in configuration file on the host will automatically reflect inside the container.

    volumes:
      - /path/to/your/llmgw_conf/conf.yaml:/app/conf.yaml
      # by default this is ./llmgw_conf/conf.yaml:/app/conf.yaml
    

  3. Logs Directory

    The LLM-GW container generates logs in /app/LLM.log directory inside the container which you can mount to anywhere in your host directory.

    Note The promtail agent (which collects all logs from host machine and pushed to tempo) needs to know the host location of these logs so if you change the path on the host than you will need to modify the promtail volume mounts aswell.

    volumes:
      - /path/to/your/logs:/app/LLM.log
      # by default this is ./logs://app/LLM.log
    

Logging and Monitoring

The LLM-Gateway generates logs in the /app/LLM.log directory within the container. You can: - Mount this directory to your host system for direct file access - Access logs through the integrated Grafana dashboard - Use the Loki-based logging stack for centralized log management

Additional Configuration

While the above settings are essential, other configuration options are available but should be modified with caution. The default values are optimized for most use cases. If you need to modify additional settings, please consult the technical support team.

Configuring Keycloak

Keycloak in LLM-Inspect can be configured to use custom SSL certificates for secure communication. This section explains how to set up your own certificates.

Environment variables

Category Variable Description
Database Configuration KC_DB Specifies the database type for Keycloak (PostgreSQL in this case)
KC_DB_URL JDBC connection URL for PostgreSQL database
KC_DB_USERNAME Username for database access
KC_DB_PASSWORD Password for database access
Admin Credentials KEYCLOAK_ADMIN Username for the Keycloak admin console
KEYCLOAK_ADMIN_PASSWORD Password for the Keycloak admin console
Feature Flags KC_FEATURES Enables specific Keycloak features (hostname:v2 in this case leave this as it is)
SSL Configuration KC_HTTPS_CERTIFICATE_FILE Path to SSL certificate file inside the container where you have mounted the certificate
KC_HTTPS_CERTIFICATE_KEY_FILE Path to SSL private key file inside the contaienr where you have mounted the key file

Note: As indicated in the comments, sensitive values like passwords should be overridden in a separate override file for security purposes.

SSL Certificate Configuration

  1. Prepare Your Certificates Place your SSL certificates in the designated directory:

    mkdir -p ./keycloak_conf/certs
    cp your-certificate.crt ./keycloak_conf/certs/
    cp your-private-key.key ./keycloak_conf/certs/
    

  2. Update Docker Compose Configuration Add the following volume mounts to your docker-compose.override.yaml:

    services:
      keycloak:
        volumes:
          # <path to certs folder>:/opt/keycloak/certs
          # if you have placed certificates in ./cert folder than leave it as it
          - ./keycloak_conf/certs:/opt/keycloak/certs
        environment:
          KC_HTTPS_CERTIFICATE_FILE: /opt/keycloak/certs/your-certificate.crt
          KC_HTTPS_CERTIFICATE_KEY_FILE: /opt/keycloak/certs/your-private-key.key
          KC_HTTPS_PORT: 8443
        ports:
          - "4116:8443"
    

  3. Verify Certificate Configuration After deploying, verify the SSL configuration:

    curl -v https://your-domain:4116/
    
    You should see your certificate information in the response.

Important Notes

  • Ensure your certificates are in PEM format
  • Certificate files should have appropriate permissions (readable by Keycloak container)
  • For production environments, always use valid certificates from trusted certificate authorities
  • The default port mapping can be adjusted based on your requirements

Configuring Grafana

Grafana in LLM-Inspect can be configured to use custom SSL certificates for secure HTTPS access. Here's how to set up SSL certificates for Grafana.

Environment variables

Category Variable Description
Plugin Configuration GF_INSTALL_PLUGINS Comma-separated list of Grafana plugins to install:
- yesoreyeram-infinity-datasource: A datasource plugin for connecting to APIs and web services
- volkovlabs-form-panel: A panel plugin for creating forms in Grafana
SSL Configuration GF_SERVER_CERT_FILE Path to the SSL certificate file for HTTPS inside the container
GF_SERVER_CERT_KEY Path to the SSL private key file for HTTPS inside the container
Security Settings GF_SECURITY_PASSWORD Admin password for Grafana login

If admin password is not set via env variable than you can set it in the grafana.ini file:

  1. Open the ./grafana_conf/config/grafana.ini
  2. Under the [security] stanza there are two options admin_user and admin_password that you can set.

SSL Certificate Setup

  1. Prepare Your Certificates Place your SSL certificates in the designated directory:

    mkdir -p ./ssl
    cp your-certificate.crt ./ssl/certificate.crt
    cp your-private-key.key ./ssl/private.key
    

  2. Update Docker Compose Configuration Modify your docker-compose.override.yaml to include the SSL certificate mounts:

    services:
      grafana:
        volumes:
          - ./ssl/certificate.crt:/etc/ssl/certificate.crt:ro
          - ./ssl/private.key:/etc/ssl/private.key:ro
          - ./grafana_conf/config/grafana.ini:/etc/grafana/grafana.ini:rw
    

  3. Configure Grafana SSL Settings Update your grafana_conf/config/grafana.ini with the following SSL configuration:

    [server]
    protocol = https
    cert_file = /etc/ssl/certificate.crt
    cert_key = /etc/ssl/private.key
    http_port = 3000
    domain = your-domain.com
    

  4. Verify SSL Configuration After deploying, verify the HTTPS connection:

    curl -v https://your-domain:5116/
    
    You should see your certificate information in the response.

Important Notes

  • Use valid SSL certificates from trusted certificate authorities in production
  • Ensure proper file permissions on certificate files
  • The default Grafana port (5116) can be adjusted in the docker-compose configuration
  • Certificate paths inside the container must match the paths specified in grafana.ini

These configurations will enable secure HTTPS access to your Grafana dashboards.

User Management with Keycloak

Setting Up Keycloak

  1. Access Keycloak admin console:

    https://your-domain:4116/admin
    

  2. Login

    Login with admin credentials set by the two environment variables KC_ADMIN as username and KC_ADMIN_PASSWORD as password

  3. Initial setup:

  4. A realm is already configued with name 'InspectChat' so you don't have to create one (if you don't see one than make sure you have the realm-export.json in ./keycloak_conf)

  5. After selecting the realm go into Clients (you can find this in the left sidebar) there verify that you see a client InspectChat

  6. Select the client by clicking on it.
  7. After selecting it you will see a Credentials tab here you can find the Client Secret.

    The client secret is initially provided as a static value during the first download. While this facilitates initial client provisioning, it is considered a sensitive credential. For enhanced security, it is strongly recommended to generate a new client secret by utilizing the 'Regenerate' function immediately after setup.

    This means any client that is using it will need to know this regenerated client secret for it to work. InspectChat service uses this client secret you can modify client secret in InspectChat by replacing the OPENID_CLIENT_SECRET env variable with new client secret.

Creating Users Manually

  1. Navigate to Users → Add User

Fill required fields:

Username: user.name
Email: [email protected]
First Name: User
Last Name: Name
2. Set initial password in Credentials tab

User Federation with LDAP

  1. Go to User Federation → Add Provider → ldap
  2. Configure LDAP settings:

Fill ldap configuration fields:

Connection URL: ldap://your-ldap-server:389
Bind DN: cn=admin,dc=example,dc=com
Bind Credential: your-password
User DN: ou=users,dc=example,dc=com
  1. Test connection and save
  2. Synchronize users

Daily Operations

Starting and Stopping Services

Start all services:

docker-compose up -d

Stop specific service:

docker-compose stop service-name

Restart service:

docker-compose restart service-name

Monitoring System Health

Check service status:

docker-compose ps
docker stats

Monitor logs:

docker-compose logs -f [service-name]

Troubleshooting

Common Issues and Solutions

  1. Service Failed to Start

    # Check logs
    docker-compose logs [service-name]
    
    # Verify configurations
    docker-compose config
    
    # Check resource usage
    docker stats
    

  2. Authentication Issues

    # Check Keycloak logs
    docker-compose logs keycloak
    
    # Verify network connectivity
    docker network inspect keycloak_keycloak_net
    

Viewing Logs

Access and analyze centralized logs efficiently through Grafana and Docker:

1. Grafana - Centralized Log Visualization

  • Navigate to Explore: Open Grafana and go to the Dashboards section.
  • Select Loki as the Data Source: Choose LLM-Inspect from the list of available dashboards.
  • Go to the logs aggregation section: Scroll down and find the Logs aggregation section
  • Construct Queries to Filter Logs: If you are logged in as admin you can click on the three dots on the log panels and query Loki for logs for example:
    • All logs from a specific container:
      {container="llmgw"}
      
    • Error logs across services:
      {container="iguard"} |= "error"
      
  • Use Grafana's Query Builder: For more complex queries, use Grafana's built-in tools to filter, search, and visualize log data based on time range, log severity, and other metadata.

2. Local Deployment Logs

  • Accessing Logs:
    • Navigate to the root folder where logs are stored:
      cd /logs
      
    • This directory contains raw log files organized by service component and date, allowing for manual inspection when necessary.
  • Log Format and Content: Understand the format of the logs (e.g., JSON, plain text) and the information they contain, such as timestamps, log level, and service identifiers.

3. Docker Container Logs

  • Viewing Real-Time Container Logs:
    docker logs container-name
    
  • Monitoring Logs with Timestamps:
    docker logs -f --timestamps container-name
    
  • Inspecting Recent Log Entries:
    docker logs --tail 100 container-name
    
  • Tips for Effective Log Management:
    • Use docker-compose logs for services managed via Docker Compose:
      docker-compose logs --follow service-name
      
    • Combine logs from all containers in a Docker Compose setup:
      docker-compose logs --tail=50 --follow
      
    • Filter logs by time or keywords using grep or other command-line tools for more specific log analysis.

Additional Tips

  • Setting Up Log Rotation: Ensure that log rotation is configured to manage disk space effectively. Docker and system-level log rotation settings should be verified to prevent service disruption due to filled disk spaces.

Observability with Grafana

Accessing Telemetry Data in Grafana

Accessing Grafana Dashboards

  1. Navigate to the Grafana web interface at:

    https://your-domain:5116
    

  2. Log in using the administrator credentials configured in your grafana.ini configuration file.

  3. From the left navigation sidebar, select "Dashboards" to view available dashboard options.

  4. Locate and open the "LLM-Inspect" dashboard, which provides comprehensive observability and traceability metrics for the system.

Analyzing OpenTelemetry Traces

The LLM-Inspect dashboard includes a dedicated Telemetry section that enables detailed trace analysis:

  1. Within the Telemetry row, you'll find:
  2. A time-series graph showing trace metrics over time
  3. A detailed table listing all captured traces with associated metadata

  1. Each trace entry in the table contains a clickable trace ID. These IDs serve as unique identifiers for individual request traces.

  2. Clicking a trace ID reveals a detailed view of the request journey, including:

    • Complete request path through system services
    • Individual span information and timing
    • Associated span attributes and metadata
    • Service dependencies and interactions

  1. You can also view the span attributes for detail debugging and observability.

    For examle this span shows the request body and headers as they land on the LLM-GW server

This trace visualization allows administrators to: - Track request flow across distributed services - Identify performance bottlenecks - Debug service interactions - Monitor system behavior and health

These telemetry insights are crucial for maintaining optimal system performance and quickly resolving any issues that may arise.

Monitoring User Interactions

Access user activity dashboard: - Message counts - Token usage - Failed validations - Response times

Important Considerations

Security Best Practices

To ensure the security and integrity of your application, follow these best practices for managing access, protecting data, and monitoring potential threats:

  1. Regular Password Rotation
  2. Enforce policies for users and administrators to change their passwords at regular intervals.
  3. Implement strong password requirements (minimum length, complexity) and avoid password reuse.

  4. SSL Certificate Maintenance

  5. Regularly update and renew SSL/TLS certificates before they expire to ensure continuous encryption of data in transit.
  6. Use only certificates from trusted certificate authorities (CAs).
  7. Periodically review and update the cipher suites used to maintain compatibility with current security standards.

  8. Access Control Review

  9. Regularly review user roles and permissions to ensure that users have the minimum level of access required for their role.
  10. Deactivate or remove access for users who no longer need it, such as former employees or users with updated roles.

  11. Monitor Failed Authentication Attempts

  12. Set up monitoring and alerts for failed login attempts to detect potential brute-force attacks or unauthorized access attempts.
  13. Log details of each failed attempt, including the timestamp, IP address, and username, as tracked by Keycloak. Regularly review these logs to identify patterns or suspicious activity.
  14. Implement account lockout mechanisms to temporarily disable accounts after a predefined number of failed attempts, reducing the risk of automated attacks. InspectChat supports this by limiting each user to a maximum of 5 login attempts per minute, helping to prevent excessive failed login attempts.

By following these security best practices, you can help protect your application and data, reduce vulnerabilities, and detect issues early before they become significant risks.

Data Backup and Recovery

Regular backup schedule:

# MongoDB backup
docker-compose exec mongo mongodump --out /backup

# Vector DB backup
docker-compose exec vectordb pg_dump -U myuser mydatabase > backup.sql

MongoDB

Stores critical operational data including audit logs, user interactions, chat messages, and transaction records for InspectChat services. Regular backups are essential to maintain service continuity and comply with audit requirements.

Vector DB

Houses embeddings of SharePoint documents when integration is enabled. Backing up this database is crucial to preserve document search capabilities and maintain the AI's context-aware responses.

PostgreSQL

Contains Keycloak authentication and user management data, including user credentials, roles, and access policies. Regular backups ensure identity management system recovery and maintain secure access control.