Quantitative testing of models on HF

[ochafik]

@ngxson suggested to do a larger sweep of compatibility tests on this llama.cpp PR (cc/ @Vaibhavs10 & @Rocketknight1)

To start with, I've added the models listed in https://aiworld.eu/embed/model/model/treemap (top liked / downloaded) to the tested MODEL_IDS.

Known failures:

• fireworks-ai/llama-3-firefunction-v2 -> Support fireworks-ai/llama-3-firefunction-v2 #7
• ai21labs/AI21-Jamba-1.5-Large -> Support ai21labs/AI21-Jamba-1.5-Large (* expansion operator) #8

Some others are playing hard to get (where are the templates for ai21labs/Jamba-v0.1, apple/OpenELM-1_1B-Instruct , dreamgen/WizardLM-2-7B, xai-org/grok-1?), but otherwise so far so good :-)

[Rocketknight1]

Hi @ochafik, some of the models you tagged as missing templates (like Jamba) are base models that aren't instruction tuned, and so they don't have chat templates!

For the others, I'm not sure - they may be genuinely missing chat templates, and users are just doing formatting manually somehow.

[ngxson]

Hi @ochafik and thanks for pinging. Congrats for the release of minja! 🚀

Re. testing, do you interested in testing in a larger quantity of models? My proposal is:

• We can provide you a dataset having 3 columns: model_id, downloads_count, jinja_template
• You can sort & take top 20 or 50 or 100 by downloads_count (as you like), then run minja test against that smaller set of data

For the models not having template, let's check with @Rocketknight1 to see if he has any solutions.

[ochafik]

some of the models you tagged as missing templates (like Jamba) are base models

@Rocketknight1 Ahhh, makes sense thanks! I’ll test Jamba 1.5 Large asap (edit -> #8), seems to have an interesting template

We can provide you a dataset having 3 columns

@ngxson That would be fabulous, thanks. Happy to let my box crunch through them all (once deduped; I’d be surprised if there were more than a few hundred actually different template contents tbh)