Inference & Model Serving
Inference tools run large language models efficiently — either self-hosted serving engines you deploy on your own GPUs (vLLM, TGI, SGLang, Ollama) or managed inference APIs that host open models for you (Together, Fireworks, Groq, Replicate, Baseten, Modal). The choice trades operational control and cost at scale against speed of setup.
| Baseten Deploy and scale model inference in production. | Proprietary | Managed | Usage-based | Yes | Managed |
| Fireworks AI Fast managed inference for open models and fine-tunes. | Proprietary | Managed | Usage-based | Yes | Managed |
| Groq Ultra-low-latency inference on custom LPU hardware. | Proprietary | Managed | Usage-based | Yes | Managed |
| LM Studio Desktop app to run local LLMs with a GUI and local server. | Proprietary (Free desktop app (proprietary)) | Self-hosted | Free tier + paid | Yes | Yes |
| Modal Serverless GPU compute for AI workloads. | Proprietary | Managed | Usage-based | You build it | Your choice |
| Ollama Run open LLMs locally with one command. | Open source (MIT) | Self-hosted | Open source (free) | Yes | Yes |
| Replicate Run and deploy any model via a simple API. | Proprietary | Managed | Usage-based | No | Managed |
| SGLang Fast serving engine with structured generation and RadixAttention. | Open source (Apache-2.0) | Self-hosted | Open source (free) | Yes | Yes |
| TensorRT-LLM NVIDIA's optimized inference library for LLMs on GPUs. | Open source (Apache-2.0) | Self-hosted | Open source (free) | Via Triton | Yes |
| Text Generation Inference (TGI) Hugging Face's production LLM serving toolkit. | Open source (Apache-2.0) | Self-hosted, Managed | Open source (free) | Yes | Yes |
| Together AI Managed inference API for open models, OpenAI-compatible. | Proprietary | Managed | Usage-based | Yes | Managed |
| vLLM High-throughput open-source LLM inference and serving engine. | Open source (Apache-2.0) | Self-hosted | Open source (free) | Yes | Yes |
Compiled from public docs and vendor sites; verify current pricing with the vendor. No vendor pays for placement. Each tool links to a fuller profile.
How to choose
Self-host with vLLM/SGLang/TGI when you have GPUs and want the lowest per-token cost at scale or data control; use Ollama or LM Studio for local development. Use managed APIs (Together, Fireworks, Groq) when you want an OpenAI-compatible endpoint without running infrastructure. For live prices of hosted models, see the pricing pages linked below.
FAQ
Self-host or use a managed inference API?
Self-hosting can be cheaper per token at high, steady volume and keeps data in your environment, but you own the GPUs and operations. Managed APIs are faster to start and scale elastically, at a per-token markup.
What does OpenAI-compatible API mean?
The server exposes the same /v1/chat/completions schema as OpenAI, so most SDKs and tools work by only changing the base URL and key.
Where can I see current hosted-model prices?
Our live pricing pages track per-token prices across providers, updated daily — see the links at the bottom of this page.