Foundation Model Inference

All

3 repositories

FlexLLMGen
Public archive
Running large language models on a single GPU for throughput-oriented scenarios.
machine-learning deep-learning offloading
machine-learning deep-learning offloading high-throughput opt gpt-3 large-language-models
Python
•
Apache License 2.0
•592•9.4k•52•6•Updated Oct 28, 2024Oct 28, 2024
H2O
Public
[NeurIPS'23] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models.
sparsity high-throughput heavy-hitters
sparsity high-throughput heavy-hitters kv-cache gpt-3 large-language-models
Python
•77•512•35•1•Updated Aug 1, 2024Aug 1, 2024
DejaVu
Public
Python
•45•355•27•0•Updated Apr 2, 2024Apr 2, 2024

ProTip! When viewing an organization's repositories, you can use the props. filter to filter by custom property.