LiteLLM

A proxy and gateway that puts a single OpenAI-compatible endpoint in front of dozens of model providers, absorbing the differences between each vendor's API. It adds virtual keys and budgets per team, request and response logging, caching, rate limiting and optional guardrails, so usage can be tracked and attributed.

Teams adopt it to route traffic between providers, compare models in an evaluation setup, and give developers one internal endpoint instead of scattered keys.

On your own server, the audit trail of every prompt stays with you, provider credentials never reach end users, and the database is a straightforward thing to back up. It can be deployed alongside the rest of your stack.