Does anyone have experience/interest in creating API standards? I think we need an API standard for open source models, like OpenAI Completions/ChatCompletions. This will greatly simplify running benchmarks and evaluations on open-source models, since we don’t need to implement inference code/dialogue templates for each model.
Personally, I use a custom-written local OpenAI-compatible API server to run standard benchmarks. This allows me to easily run benchmark code from different sources just by modifying api_base, since almost all benchmarks support OpenAI models.