Skip to content

Admin Guide

Introduction

LLM-Inspect is an enterprise solution for secure and observable AI interactions. This guide provides comprehensive instructions for system administrators to deploy, manage, and maintain the LLM-Inspect platform. The system uses Docker containers for deployment and includes various services for authentication, monitoring, and security.

Installation

Prerequisites

  • Docker Engine (20.10.x or newer)
  • Docker Compose (2.x or newer)
  • Minimum 32GB RAM
  • 8+ CPU cores
  • 100GB available storage (SSD recommended)
  • Valid SSL certificates
  • Network access to required services
  • A locally running LLM with at-least 70B parameters

Installation Steps

  1. Extract the provided zip file:

  2. Run the installer script:

Basic Installation (with auto-generated self-signed certificate): 

./llminspect-cli -p /full/path/to/llminspect -o install
The installer will prompt you for an FQDN and automatically create a self-signed certificate.

Installation with Custom SSL/TLS Certificates: 

./llminspect-cli -p /full/path/to/llminspect -o install \
    --cert-file /path/to/certificate.crt \
    --cert-key /path/to/private.key \
    --domain your-domain.com

⚠️ Self-signed certificates (generated in basic installation) are suitable for testing but not recommended for production use. For production, please obtain certificates from a trusted Certificate Authority (CA).

Important Notes: - The installer script is designed for Linux systems only - The installer should only be used for initial installation - After installation, use standard Docker Compose commands for daily operations - The -p flag must point to the full path where you extracted the files

SSL/TLS Configuration: - The following flags must be used together for proper SSL/TLS setup: - --cert-file: Full path to your SSL certificate file - --cert-key: Full path to your SSL private key file - --domain: Domain name that matches the CN in your certificate - If any of these three SSL parameters are provided, all three must be provided - If none are provided, the installation will proceed with basic SSL/TLS configuration

Environment Configuration: - By default, the installer will prompt you for various configuration values during installation

  • To bypass interactive configuration, you can use the --env-file parameter:

./llminspect-cli -p /full/path/to/llminspect --cert-file /path/to/cert.pem --cert-key /path/to/key.pem --domain example.com -o install --env-file /path/to/env.file
- The env file should contain all required environment variables in KEY=VALUE format

  • Using an env file is recommended for automated deployments or when you want to maintain consistent configurations

For stopping the services:

./llminspect-cli -p /full/path/to/llminspect -o stop

⚠️ IMPORTANT: If Installation Fails ⚠️

If the installer encounters an error and fails to complete, you MUST run the clean operation before attempting to install again:

./llminspect-cli -p /full/path/to/llminspect -o clean

This will revert all changes made by the failed installation attempt and allow you to start fresh.

Parameter Summary: - -p, --project: (Required) Full path to the extracted llminspect directory

  • -o, --operation: (Required) One of:

  • 'install': Performs initial installation

  • 'stop': Stops running services

  • 'clean': Reverts all file changes made by the installer, restoring the installation directory to its original state. Useful for:

    • Starting fresh after a failed installation

    • Recovering from installer-generated configuration errors

    • Resetting the environment before attempting a new installation

    • Cleaning up partial changes if installation was interrupted

  • --env-file: (Optional) Path to environment file for automated configuration

  • SSL/TLS group (all three required if using SSL):

  • --domain: Domain name matching certificate CN

  • --cert-file: Path to SSL certificate

  • --cert-key: Path to SSL private key

  • --license: (Optional) To see the license of open-source services used in llminspect

Installation Parameters

Below is a detailed overview of each configuration option, including what value is expected from the user along with examples for clarity.

MongoDB Configuration

  • MONGO_INITDB_ROOT_USERNAME

Explanation: This value sets the root or administrative username for your MongoDB instance.

Example: You might use a username like admin or mongoAdmin.

  • MONGO_INITDB_ROOT_PASSWORD

Explanation: This is the password for the MongoDB root user. It should be strong to protect your database.

Keycloak Configuration

  • KEYCLOAK_ADMIN_PASSWORD

Explanation: This password is for the Keycloak admin account, which manages authentication and access control.

API Keys

  • OPENAI_API_KEY

Explanation: This is your API key for accessing OpenAI services, allowing your application to interact with the OpenAI API.

  • GEMINI_API_KEY

Explanation: This key enables your application to access Gemini services, authorizing your requests to the Gemini API.

Endpoint URLs

  • LLAMA_GUARD_URL

Explanation: Provide the complete URL to your LLama guard instance. This endpoint handles specific tasks such as request validation or generating responses.

Example: http://192.168.1.100:8080/generate (where 192.168.1.100 is the server's IP and 8080 is the port).

  • LLM_BASE_URL

Explanation: This base URL points to your local LLM (Language Learning Model) chat service endpoint for completions.

Example: https://llm.example.com:5000/v1/ (with llm.example.com being your domain and /v1/ indicating the API version).

LLM Gateway Credentials

  • LLMGW_KEY

Explanation: This license key is used to authenticate your deployment with the LLM Gateway, ensuring that your usage is authorized.

Example: An example key might be LLMGW-KEY-9876543210.

  • AUTHCODE

Explanation: This authentication code is required for secure access to the LLM Gateway, adding an extra layer of security.

Example: A sample auth code could be auth1234.

Accessing LLMInspect

After successful installation, the following core services will be available on their respective ports. All services are accessed via HTTPS using the domain you specified during installation.

Core Services Access

1. Keycloak (Authentication Server)

  • Default Port: 4116
  • URL: https://your-domain:4116/admin/master/console
  • Default Credentials:
  • Username: admin
  • Password: The password you provided during installation when prompted for KEYCLOAK_ADMIN_PASSWORD

2. Grafana (Monitoring & Observability)

  • Default Port: 5116
  • URL: https://your-domain:5116
  • Default Credentials:
  • Username: llminspect
  • Password: Set through Keycloak (see below)

To set/reset the Grafana login password:

  1. Log into Keycloak admin console

  2. Select realm InspectChat

  3. Go to Users in the left sidebar

  4. Find and click on user llminspect

  5. Go to Credentials tab

  6. Click "Reset Password"

  7. Set new password and use these credentials to log into Grafana

3. InspectChat (Main Application)

  • Default Port: 6116
  • URL: https://your-domain:6116
  • Credentials: Use the same llminspect user credentials set in Keycloak

4. Admin Panel

  • Default Port: 7116
  • URL: https://your-domain:7116
  • Credentials: Use the same llminspect user credentials set in Keycloak

5. API Gateway

  • Default Port: 8116
  • Base URL: https://your-domain:8116
  • API Endpoints:
  • Chat Completions: https://your-domain:8116/v1/chat/completions
  • Other endpoints follow similar pattern under /v1/

User Management with Keycloak

Creating Users Manually

  1. Navigate to Users → Add User

Fill required fields:

Username: user.name
Email: [email protected]
First Name: User
Last Name: Name
2. Set initial password in Credentials tab

User Federation with LDAP

  1. Go to User Federation → Add Provider → ldap
  2. Configure LDAP settings:

Fill ldap configuration fields:

Connection URL: ldap://your-ldap-server:389
Bind DN: cn=admin,dc=example,dc=com
Bind Credential: your-password
User DN: ou=users,dc=example,dc=com
  1. Test connection and save
  2. Synchronize users

Daily Operations

Starting and Stopping Services

Start all services: 

docker-compose up -d

Stop specific service: 

docker-compose stop service-name

Restart service: 

docker-compose restart service-name

Monitoring System Health

Check service status: 

docker-compose ps
docker stats

Monitor logs: 

docker-compose logs -f [service-name]

Important Considerations

Security Best Practices

To ensure the security and integrity of your application, follow these best practices for managing access, protecting data, and monitoring potential threats:

  1. Regular Password Rotation
  2. Enforce policies for users and administrators to change their passwords at regular intervals.

  3. Implement strong password requirements (minimum length, complexity) and avoid password reuse.

  4. SSL Certificate Maintenance

  5. Regularly update and renew SSL/TLS certificates before they expire to ensure continuous encryption of data in transit.
  6. Use only certificates from trusted certificate authorities (CAs).

  7. Periodically review and update the cipher suites used to maintain compatibility with current security standards.

  8. Access Control Review

  9. Regularly review user roles and permissions to ensure that users have the minimum level of access required for their role.

  10. Deactivate or remove access for users who no longer need it, such as former employees or users with updated roles.

  11. Monitor Failed Authentication Attempts

  12. Set up monitoring and alerts for failed login attempts to detect potential brute-force attacks or unauthorized access attempts.
  13. Log details of each failed attempt, including the timestamp, IP address, and username, as tracked by Keycloak. Regularly review these logs to identify patterns or suspicious activity.
  14. Implement account lockout mechanisms to temporarily disable accounts after a predefined number of failed attempts, reducing the risk of automated attacks. InspectChat supports this by limiting each user to a maximum of 5 login attempts per minute, helping to prevent excessive failed login attempts.

By following these security best practices, you can help protect your application and data, reduce vulnerabilities, and detect issues early before they become significant risks.

Data Backup and Recovery

Regular backup schedule: 

# MongoDB backup
docker-compose exec mongo mongodump --out /backup

# Vector DB backup
docker-compose exec vectordb pg_dump -U myuser mydatabase > backup.sql

MongoDB

Stores critical operational data including audit logs, user interactions, chat messages, and transaction records for InspectChat services. Regular backups are essential to maintain service continuity and comply with audit requirements.

Vector DB

Houses embeddings of SharePoint documents when integration is enabled. Backing up this database is crucial to preserve document search capabilities and maintain the AI's context-aware responses.

PostgreSQL

Contains Keycloak authentication and user management data, including user credentials, roles, and access policies. Regular backups ensure identity management system recovery and maintain secure access control.