Enterprise Architecture

The architecture of LLM-Inspect is designed to offer a robust, secure, and scalable framework that supports both local deployment within enterprise networks and integration with external LLM providers. At the core of the architecture is the LLMGW Service, which acts as the main gateway for managing and processing interactions with LLMs, whether hosted locally or accessed via public APIs. This service ensures that all requests are validated, monitored, and logged, allowing for complete control over data flow and compliance with security protocols.

The system integrates seamlessly with Keycloak for role-based authentication and authorization, supporting secure access for both internal and external users. For identity management, Active Directory can be utilized to fetch and assign roles, ensuring proper user access control across the organization.

Data protection and compliance are achieved through iSanitize and iGuard modules, which provide real-time analysis and validation of user prompts to safeguard against data exposure and to adhere to corporate policies. InspectGPT enhances the platform's security by analyzing potential risks in prompts, including PII detection and content classification.

In terms of observability, Grafana and Loki are integrated to offer detailed monitoring and logging capabilities, allowing administrators to visualize data and track system performance. Logs are collected by a Promtail Agent and can be analyzed in real time, enabling proactive management of security and operational health. For query and data storage needs, MongoDB serves as the primary database, while APISIX operates as the API gateway, managing incoming requests from both GUI Clients and API Clients through InspectChat.

The system supports interaction with local models such as LLaMA 70B as well as public providers like OpenAI, Gemini, and Cohere, making it highly versatile for various LLM-based applications. This comprehensive architecture enables enterprises to harness the power of AI responsibly, maintaining strict control over data security and compliance while providing flexibility for AI model access.