Model Serving Tools

High-performance C/C++ inference for LLaMA and GGUF models on CPU and GPU.

Local model runner with simple command and HTTP interface.

High-throughput LLM inference engine with PagedAttention.

Hugging Face production server for LLM inference with continuous batching on GPUs.

Self-hosted web interface for local and remote LLMs.

Platform for deploying, serving, and scaling ML and LLM models in production.

Open big data serving engine for vector search, ranking, and ML inference at scale.

Desktop app for running local LLMs with a chat interface.

NVIDIA library for optimized LLM inference on TensorRT.

Open-source LLM gateway for routing, observability, and optimization in Rust.

Open-source distributed inference engine for running LLMs, embeddings, and multimodal models.

Fast inference engine for Transformer models with quantization and batching.

Production inference server for ML and LLM models at scale.

Scalable model serving library built on the Ray distributed framework.

Fast structured generation and serving runtime for LLMs.

Unified framework for packaging and deploying ML models as APIs.

Serverless ML inference platform with fast cold starts and custom runtimes.

Kubernetes-native serverless model serving platform.