IEEE

MF-Net: Compute-In-Memory SRAM for Multibit Precision Inference Using Memory-Immersed Data Conversion and Multiplication-Free Operators

Abstract: We propose a co-design approach for compute-in-memory inference for deep neural networks (DNN). We use multiplication-free function approximators based on l 1 norm along with a co-adapted ...
GitHub

Sparsity-aware deep learning inference runtime for CPUs

We’re reaching out with heartfelt thanks and important news. Following Neural Magic’s acquisition by Red Hat in January 2025, we’ve shifted our focus to commercial and open-source offerings built ...
GitHub

LoRAX: Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

LoRAX (LoRA eXchange) is a framework that allows users to serve thousands of fine-tuned models on a single GPU, dramatically reducing the cost of serving without compromising on throughput or latency.
IEEE

Measuring and Improving the Energy Efficiency of Large Language Models Inference

Abstract: Recent improvements in the accuracy of machine learning (ML) models in the language domain have propelled their use in a multitude of products and services, touching millions of lives daily.