Compare different open source LLMs

A tool launched by AnyScaleCompute

Aviary is an app that lets you interact with a variety of large language models (LLMs) in a single place. You can compare the outputs of different models directly, rank them by quality, get a cost and latency estimate, and more. In particular it offers good support for Transformer models hosted on Hugging Face and in many cases also supports DeepSpeed inference acceleration.