A high-throughput and memory-efficient inference and serving engine for LLMs

A fast GPU memory copy library based on NVIDIA GPUDirect RDMA technology

FlashInfer: Kernel Library for LLM Serving

📚200+ Tensor/CUDA Cores Kernels, ⚡️flash-attn-mma, ⚡️hgemm with WMMA, MMA and CuTe (98%~100% TFLOPS of cuBLAS/FA2 🎉🎉).

Fast and memory-efficient exact attention

CUDA Templates for Linear Algebra Subroutines

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficie…