Admin Guide

Introduction

LLM-Inspect is an enterprise solution for secure and observable AI interactions. This guide provides comprehensive instructions for system administrators to deploy, manage, and maintain the LLM-Inspect platform. The system uses Docker containers for deployment and includes various services for authentication, monitoring, and security.

Installation

Prerequisites

• Docker Engine (20.10.x or newer)
• Docker Compose (2.x or newer)
• Minimum 32GB RAM
• 8+ CPU cores
• 100GB available storage (SSD recommended)
• Valid SSL certificates
• Network access to required services
• A locally running LLM with at-least 70B parameters

Installation Steps

1. Extract the provided zip file:

2. Run the installer script:

Basic installation:

./llminspect-cli -p /full/path/to/llminspect -o install

Installation with SSL/TLS:

./llminspect-cli -p /full/path/to/llminspect -o install \
    --cert-file /path/to/certificate.crt \
    --cert-key /path/to/private.key \
    --domain your-domain.com

Important Notes: - The installer script is designed for Linux systems only - The installer should only be used for initial installation - After installation, use standard Docker Compose commands for daily operations - The -p flag must point to the full path where you extracted the files

SSL/TLS Configuration: - The following flags must be used together for proper SSL/TLS setup: - --cert-file: Full path to your SSL certificate file - --cert-key: Full path to your SSL private key file - --domain: Domain name that matches the CN in your certificate - If any of these three SSL parameters are provided, all three must be provided - If none are provided, the installation will proceed with basic SSL/TLS configuration

Environment Configuration: - By default, the installer will prompt you for various configuration values during installation - To bypass interactive configuration, you can use the --env-file parameter:

./llminspect-cli -p /full/path/to/llminspect -o install --env-file /path/to/env.file

- The env file should contain all required environment variables in KEY=VALUE format - Using an env file is recommended for automated deployments or when you want to maintain consistent configurations

For stopping the services:

./llminspect-cli -p /full/path/to/llminspect -o stop

⚠️ IMPORTANT: If Installation Fails ⚠️ If the installer encounters an error and fails to complete, you MUST run the clean operation before attempting to install again:

./llminspect-cli -p /full/path/to/llminspect -o clean

This will revert all changes made by the failed installation attempt and allow you to start fresh.

Parameter Summary: - -p, --project: (Required) Full path to the extracted llminspect directory - -o, --operation: (Required) One of: - 'install': Performs initial installation - 'stop': Stops running services - 'clean': Reverts all file changes made by the installer, restoring the installation directory to its original state. Useful for: - Starting fresh after a failed installation - Recovering from installer-generated configuration errors - Resetting the environment before attempting a new installation - Cleaning up partial changes if installation was interrupted - --env-file: (Optional) Path to environment file for automated configuration - SSL/TLS group (all three required if using SSL): - --domain: Domain name matching certificate CN - --cert-file: Path to SSL certificate - --cert-key: Path to SSL private key - --license: (Optional) To see the license of open-source services used in llminspect

Architecture Overview

LLM-Inspect consists of several key components: - LLM Gateway (LLM-GW): Central entry point for AI model interactions - iGuard: Security validation service - Keycloak: Authentication and user management - Grafana & Prometheus: Monitoring and metrics - MongoDB: Primary database - Redis: Caching layer - Vector DB: Specialised database for AI operations - Tempo: Distributed tracing backend

Configuration

Configuring LLM-Gateway

The LLM-Gateway requires specific configuration parameters to function properly. These configurations can be set through environment variables and volume mounts in your docker-compose.override.yaml file.

Required Environment Variables

Category	Variable	Description
Django Settings	DJANGO_DB_ENGINE	Specifies the database backend for Django (SQLite3 in this case)
	DJANGO_DB_NAME	Path to the SQLite database file
	ALLOWED_HOSTS	List of hosts/domains that can serve the Django application
Service URLs	INSPECT_HEXAGON_URL	URL for the Inspect Hexagon service
	ISANITIZE_URL	URL for the iSanitize service
	LOCAL_LLM_URL	URL for the local LLM service endpoint
	GEMINI_PROXY_URL	URL for the Gemini-OpenAI proxy service
	IGUARD_SERVICE_URL	URL for the iGuard load balancer service
	REDIS_URL	Main Redis connection URL
Keycloak Settings	KEYCLOAK_URL	URL for Keycloak authentication userinfo endpoint
API Keys	OPENAI_API_KEY	API key for OpenAI services
	GEMINI_API_KEY	URL for the iSanitize service
	INSPECTGPT_API_KEY	URL for the iSanitize service
OTEL Settings	OTEL_ENABLE	Toggle for OpenTelemetry tracing (1=enabled)
	OTEL_SERVICE_NAME	Service name for OpenTelemetry tracing
	OTEL_ENABLE_CONSOLE_EXPORTER	Toggle for console export of telemetry data
	OTLP_EXPORTER_ENDPOINT	Endpoint for OpenTelemetry collector
Miscellaneous	ENABLE_HTTPS	Toggle for HTTPS protocol
	CONFIG_FILE	Path to configuration file inside the container
	CHUNK_SIZE	Size of chunks for streaming LLM responses
Celery Settings	CELERY_BROKER_URL	Redis URL for Celery message broker
	CELERY_RESULT_BACKEND	Redis URL for storing Celery task results

Apart from the API keys you can leave everything as default and the deployment will still work.

Note It is recommended that any environment variable you wish to change modify it in the docker-compose.override.yaml file instead of the docker-compose.yml.

Any docker compose configuration defined in the override files always takes precedence.

Essential Volume Mounts

The LLM-Gateway requires specific volumes to be mounted for proper operation:

1. SSL Certificates

   If ENABLE_HTTPS environment variable is set to true than LLM-GW will read the certificates from the path you mount to /app/Cert/

   volumes:
     - /path/to/your/ssl:/app/Cert
     # by default this is ./ssl:/app/Cert so if you have ssl folder in root directory than no need to change
   

2. Configuration File

   You need to create conf.yaml file in your desired path on the host and mount it to app/conf.yaml inside the container, any changes in configuration file on the host will automatically reflect inside the container.

   volumes:
     - /path/to/your/llmgw_conf/conf.yaml:/app/conf.yaml
     # by default this is ./llmgw_conf/conf.yaml:/app/conf.yaml
   

3. Logs Directory

   The LLM-GW container generates logs in /app/LLM.log directory inside the container which you can mount to anywhere in your host directory.

   Note The promtail agent (which collects all logs from host machine and pushed to tempo) needs to know the host location of these logs so if you change the path on the host than you will need to modify the promtail volume mounts aswell.

   volumes:
     - /path/to/your/logs:/app/LLM.log
     # by default this is ./logs://app/LLM.log
   

Logging and Monitoring

The LLM-Gateway generates logs in the /app/LLM.log directory within the container. You can: - Mount this directory to your host system for direct file access - Access logs through the integrated Grafana dashboard - Use the Loki-based logging stack for centralized log management

Additional Configuration

While the above settings are essential, other configuration options are available but should be modified with caution. The default values are optimized for most use cases. If you need to modify additional settings, please consult the technical support team.

Configuring Keycloak

Keycloak in LLM-Inspect can be configured to use custom SSL certificates for secure communication. This section explains how to set up your own certificates.

Environment variables

Category	Variable	Description
Database Configuration	KC_DB	Specifies the database type for Keycloak (PostgreSQL in this case)
	KC_DB_URL	JDBC connection URL for PostgreSQL database
	KC_DB_USERNAME	Username for database access
	KC_DB_PASSWORD	Password for database access
Admin Credentials	KEYCLOAK_ADMIN	Username for the Keycloak admin console
	KEYCLOAK_ADMIN_PASSWORD	Password for the Keycloak admin console
Feature Flags	KC_FEATURES	Enables specific Keycloak features (hostname:v2 in this case leave this as it is)
SSL Configuration	KC_HTTPS_CERTIFICATE_FILE	Path to SSL certificate file inside the container where you have mounted the certificate
	KC_HTTPS_CERTIFICATE_KEY_FILE	Path to SSL private key file inside the contaienr where you have mounted the key file

Note: As indicated in the comments, sensitive values like passwords should be overridden in a separate override file for security purposes.

SSL Certificate Configuration

1. Prepare Your Certificates Place your SSL certificates in the designated directory:

   mkdir -p ./keycloak_conf/certs
   cp your-certificate.crt ./keycloak_conf/certs/
   cp your-private-key.key ./keycloak_conf/certs/
   

2. Update Docker Compose Configuration Add the following volume mounts to your docker-compose.override.yaml:

   services:
     keycloak:
       volumes:
         # <path to certs folder>:/opt/keycloak/certs
         # if you have placed certificates in ./cert folder than leave it as it
         - ./keycloak_conf/certs:/opt/keycloak/certs
       environment:
         KC_HTTPS_CERTIFICATE_FILE: /opt/keycloak/certs/your-certificate.crt
         KC_HTTPS_CERTIFICATE_KEY_FILE: /opt/keycloak/certs/your-private-key.key
         KC_HTTPS_PORT: 8443
       ports:
         - "4116:8443"
   

3. Verify Certificate Configuration After deploying, verify the SSL configuration:

   curl -v https://your-domain:4116/
   

   You should see your certificate information in the response.

Important Notes

• Ensure your certificates are in PEM format
• Certificate files should have appropriate permissions (readable by Keycloak container)
• For production environments, always use valid certificates from trusted certificate authorities
• The default port mapping can be adjusted based on your requirements

Configuring Grafana

Grafana in LLM-Inspect can be configured to use custom SSL certificates for secure HTTPS access. Here's how to set up SSL certificates for Grafana.

Environment variables

Category	Variable	Description
Plugin Configuration	GF_INSTALL_PLUGINS	Comma-separated list of Grafana plugins to install: - yesoreyeram-infinity-datasource: A datasource plugin for connecting to APIs and web services - volkovlabs-form-panel: A panel plugin for creating forms in Grafana
SSL Configuration	GF_SERVER_CERT_FILE	Path to the SSL certificate file for HTTPS inside the container
	GF_SERVER_CERT_KEY	Path to the SSL private key file for HTTPS inside the container
Security Settings	GF_SECURITY_PASSWORD	Admin password for Grafana login

If admin password is not set via env variable than you can set it in the grafana.ini file:

1. Open the ./grafana_conf/config/grafana.ini
2. Under the [security] stanza there are two options admin_user and admin_password that you can set.

SSL Certificate Setup

1. Prepare Your Certificates Place your SSL certificates in the designated directory:

   mkdir -p ./ssl
   cp your-certificate.crt ./ssl/certificate.crt
   cp your-private-key.key ./ssl/private.key
   

2. Update Docker Compose Configuration Modify your docker-compose.override.yaml to include the SSL certificate mounts:

   services:
     grafana:
       volumes:
         - ./ssl/certificate.crt:/etc/ssl/certificate.crt:ro
         - ./ssl/private.key:/etc/ssl/private.key:ro
         - ./grafana_conf/config/grafana.ini:/etc/grafana/grafana.ini:rw
   

3. Configure Grafana SSL Settings Update your grafana_conf/config/grafana.ini with the following SSL configuration:

   [server]
   protocol = https
   cert_file = /etc/ssl/certificate.crt
   cert_key = /etc/ssl/private.key
   http_port = 3000
   domain = your-domain.com
   

4. Verify SSL Configuration After deploying, verify the HTTPS connection:

   curl -v https://your-domain:5116/
   

   You should see your certificate information in the response.

Important Notes

• Use valid SSL certificates from trusted certificate authorities in production
• Ensure proper file permissions on certificate files
• The default Grafana port (5116) can be adjusted in the docker-compose configuration
• Certificate paths inside the container must match the paths specified in grafana.ini

These configurations will enable secure HTTPS access to your Grafana dashboards.

User Management with Keycloak

Setting Up Keycloak

1. Access Keycloak admin console:

   https://your-domain:4116/admin
   

2. Login

   Login with admin credentials set by the two environment variables KC_ADMIN as username and KC_ADMIN_PASSWORD as password

3. Initial setup:

4. A realm is already configued with name 'InspectChat' so you don't have to create one (if you don't see one than make sure you have the realm-export.json in ./keycloak_conf)

5. After selecting the realm go into Clients (you can find this in the left sidebar) there verify that you see a client InspectChat

6. Select the client by clicking on it.

7. After selecting it you will see a Credentials tab here you can find the Client Secret.

   The client secret is initially provided as a static value during the first download. While this facilitates initial client provisioning, it is considered a sensitive credential. For enhanced security, it is strongly recommended to generate a new client secret by utilizing the 'Regenerate' function immediately after setup.

   This means any client that is using it will need to know this regenerated client secret for it to work. InspectChat service uses this client secret you can modify client secret in InspectChat by replacing the OPENID_CLIENT_SECRET env variable with new client secret.

Creating Users Manually

1. Navigate to Users → Add User

Fill required fields:

Username: user.name
Email: [email protected]
First Name: User
Last Name: Name

2. Set initial password in Credentials tab

User Federation with LDAP

1. Go to User Federation → Add Provider → ldap
2. Configure LDAP settings:

Fill ldap configuration fields:

Connection URL: ldap://your-ldap-server:389
Bind DN: cn=admin,dc=example,dc=com
Bind Credential: your-password
User DN: ou=users,dc=example,dc=com

1. Test connection and save
2. Synchronize users

Daily Operations

Starting and Stopping Services

Start all services:

docker-compose up -d

Stop specific service:

docker-compose stop service-name

Restart service:

docker-compose restart service-name

Monitoring System Health

Check service status:

docker-compose ps
docker stats

Monitor logs:

docker-compose logs -f [service-name]

Troubleshooting

Common Issues and Solutions

1. Service Failed to Start

   # Check logs
   docker-compose logs [service-name]
   
   # Verify configurations
   docker-compose config
   
   # Check resource usage
   docker stats
   

2. Authentication Issues

   # Check Keycloak logs
   docker-compose logs keycloak
   
   # Verify network connectivity
   docker network inspect keycloak_keycloak_net
   

Viewing Logs

Access and analyze centralized logs efficiently through Grafana and Docker:

1. Grafana - Centralized Log Visualization

• Navigate to Explore: Open Grafana and go to the Dashboards section.
• Select Loki as the Data Source: Choose LLM-Inspect from the list of available dashboards.
• Go to the logs aggregation section: Scroll down and find the Logs aggregation section
• Construct Queries to Filter Logs: If you are logged in as admin you can click on the three dots on the log panels and query Loki for logs for example:
  • All logs from a specific container:

    {container="llmgw"}
    

  • Error logs across services:

    {container="iguard"} |= "error"
    

• Use Grafana's Query Builder: For more complex queries, use Grafana's built-in tools to filter, search, and visualize log data based on time range, log severity, and other metadata.

2. Local Deployment Logs

• Accessing Logs:
  • Navigate to the root folder where logs are stored:

    cd /logs
    

  • This directory contains raw log files organized by service component and date, allowing for manual inspection when necessary.
• Log Format and Content: Understand the format of the logs (e.g., JSON, plain text) and the information they contain, such as timestamps, log level, and service identifiers.

3. Docker Container Logs

• Viewing Real-Time Container Logs:

  docker logs container-name
  

• Monitoring Logs with Timestamps:

  docker logs -f --timestamps container-name
  

• Inspecting Recent Log Entries:

  docker logs --tail 100 container-name
  

• Tips for Effective Log Management:
  • Use docker-compose logs for services managed via Docker Compose:

    docker-compose logs --follow service-name
    

  • Combine logs from all containers in a Docker Compose setup:

    docker-compose logs --tail=50 --follow
    

  • Filter logs by time or keywords using grep or other command-line tools for more specific log analysis.

Additional Tips

• Setting Up Log Rotation: Ensure that log rotation is configured to manage disk space effectively. Docker and system-level log rotation settings should be verified to prevent service disruption due to filled disk spaces.

Observability with Grafana

Accessing Telemetry Data in Grafana

Accessing Grafana Dashboards

1. Navigate to the Grafana web interface at:

   https://your-domain:5116
   

2. Log in using the administrator credentials configured in your grafana.ini configuration file.

3. From the left navigation sidebar, select "Dashboards" to view available dashboard options.

4. Locate and open the "LLM-Inspect" dashboard, which provides comprehensive observability and traceability metrics for the system.

Analyzing OpenTelemetry Traces

The LLM-Inspect dashboard includes a dedicated Telemetry section that enables detailed trace analysis:

1. Within the Telemetry row, you'll find:
2. A time-series graph showing trace metrics over time
3. A detailed table listing all captured traces with associated metadata

1. Each trace entry in the table contains a clickable trace ID. These IDs serve as unique identifiers for individual request traces.

2. Clicking a trace ID reveals a detailed view of the request journey, including:

   • Complete request path through system services
   • Individual span information and timing
   • Associated span attributes and metadata
   • Service dependencies and interactions

1. You can also view the span attributes for detail debugging and observability.

   For examle this span shows the request body and headers as they land on the LLM-GW server

   

This trace visualization allows administrators to: - Track request flow across distributed services - Identify performance bottlenecks - Debug service interactions - Monitor system behavior and health

These telemetry insights are crucial for maintaining optimal system performance and quickly resolving any issues that may arise.

Monitoring User Interactions

Access user activity dashboard: - Message counts - Token usage - Failed validations - Response times

Important Considerations

Security Best Practices

To ensure the security and integrity of your application, follow these best practices for managing access, protecting data, and monitoring potential threats:

1. Regular Password Rotation
2. Enforce policies for users and administrators to change their passwords at regular intervals.

3. Implement strong password requirements (minimum length, complexity) and avoid password reuse.

4. SSL Certificate Maintenance

5. Regularly update and renew SSL/TLS certificates before they expire to ensure continuous encryption of data in transit.
6. Use only certificates from trusted certificate authorities (CAs).

7. Periodically review and update the cipher suites used to maintain compatibility with current security standards.

8. Access Control Review

9. Regularly review user roles and permissions to ensure that users have the minimum level of access required for their role.

10. Deactivate or remove access for users who no longer need it, such as former employees or users with updated roles.

11. Monitor Failed Authentication Attempts

12. Set up monitoring and alerts for failed login attempts to detect potential brute-force attacks or unauthorized access attempts.
13. Log details of each failed attempt, including the timestamp, IP address, and username, as tracked by Keycloak. Regularly review these logs to identify patterns or suspicious activity.
14. Implement account lockout mechanisms to temporarily disable accounts after a predefined number of failed attempts, reducing the risk of automated attacks. InspectChat supports this by limiting each user to a maximum of 5 login attempts per minute, helping to prevent excessive failed login attempts.

By following these security best practices, you can help protect your application and data, reduce vulnerabilities, and detect issues early before they become significant risks.

Data Backup and Recovery

Regular backup schedule:

# MongoDB backup
docker-compose exec mongo mongodump --out /backup

# Vector DB backup
docker-compose exec vectordb pg_dump -U myuser mydatabase > backup.sql

MongoDB

Stores critical operational data including audit logs, user interactions, chat messages, and transaction records for InspectChat services. Regular backups are essential to maintain service continuity and comply with audit requirements.

Vector DB

Houses embeddings of SharePoint documents when integration is enabled. Backing up this database is crucial to preserve document search capabilities and maintain the AI's context-aware responses.

PostgreSQL

Contains Keycloak authentication and user management data, including user credentials, roles, and access policies. Regular backups ensure identity management system recovery and maintain secure access control.