Sparse Interpolated Experts

Unleashing the Power of Meta-tuning for Few-shot Generalization Through Sparse Interpolated Experts. A method that automatically isolates subsets of pre-trained parameters for meta-tuning, achieving state-of-the-art results on challenging Meta-Dataset benchmarks with enhanced OOD generalization.

Automatic Expert Discovery in LLM Upcycling via Sparse Interpolated Mixture-of-Experts. An end-to-end algorithm that fine-tunes dense LLMs into MoE-style models with automatically discovered specialized experts, achieving optimal performance-compute trade-offs.