How to Make LLM Training Faster with Unsloth and NVIDIA

Unsloth collaborated with NVIDIA to achieve ~25% faster LLM training through three targeted optimizations. First, packed-sequence metadata caching eliminates repeated reconstruction of attention masks and sequence metadata across transformer layers, yielding up to +43.3% forward pass speedup on Qwen3-14B QLoRA SFT. Second, double-buffered activation checkpoint reloading overlaps CPU-to-GPU memory copies with backward compute, delivering 4.6–8.4% step-time improvements on 8B–32B dense models with minimal VRAM overhead. Third, MoE routing is improved by replacing per-expert dynamic torch.where queries with a single bincount pass, reducing CPU-GPU synchronization overhead proportional to num_experts down to a constant, yielding 10–15% speedups on GPT-OSS configurations. All three optimizations target overhead in the 'glue code' around main kernels rather than the kernels themselves, and are automatically enabled on RTX laptops, data center GPUs, and DGX Spark machines after updating Unsloth.