Quantized Attention achieves speedup of 2-5x and 3-11x compared to FlashAttention and xformers, without lossing end-to-end metrics across language, image, and video models.

A Quirky Assortment of CuTe Kernels

verl: Volcano Engine Reinforcement Learning for LLMs

DeepGEMM: clean and efficient FP8 GEMM kernels with fine-grained scaling

Implement Flash Attention using Cute.

Distributed Triton for Parallel Systems

SGLang is a fast serving framework for large language models and vision language models.

DeepEP: an efficient expert-parallel communication library

flash attention tutorial written in python, triton, cuda, cutlass

Large Reasoning Models

Examples of CUDA implementations by Cutlass CuTe

Ongoing research training transformer models at scale

This is the homepage of a new book entitled "Mathematical Foundations of Reinforcement Learning."

O1 Replication Journey: A Strategic Progress Report – Part I

CUDA Templates for Linear Algebra Subroutines

Anole: An Open, Autoregressive and Native Multimodal Models for Interleaved Image-Text Generation

A clean implementation based on AlphaZero for any game in any framework + tutorial + Othello/Gobang/TicTacToe/Connect4 and more

A set of examples around pytorch in Vision, Text, Reinforcement Learning, etc.

Modeling, training, eval, and inference code for OLMo

Official PyTorch Implementation of "Scalable Diffusion Models with Transformers"

Development repository for the Triton language and compiler

DLRover: An Automatic Distributed Deep Learning System

Pre-training code for CrystalCoder 7B LLM

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

how to optimize some algorithm in cuda.

EasyDL: A Kubernetes-native Deep Learning Training Service