Model Compression via Dynamic Systems by Assistant Professor Fenglei Fan

Abstract

Recently, the escalating demand for memory and computational resources by large models has presented formidable challenges for their deployment in resource-constrained environments. One prevalent way to serve large models is to develop effective model compression approaches that crop the size of large models while maintaining acceptable performance levels. Currently, the landscape of model compression methodologies predominantly revolves around four fundamental algorithms: pruning, quantization, knowledge distillation, and low-rank approximation whose basic frameworks have been established years ago. However, the compression efficacy of these methods is often capped or challenging to scale based on theoretical analysis. In this talk, we introduce a novel and general-purpose approach, referred to as hyper-compression, that redefines model compression as a problem of parameter representation. Specifically, we extend the concept of hypernets into what we term a ‘hyperfunction'. Then, the hyperfunction is designed based on ergodic theory (ET). This advanced formulation leads to a performant algorithm that offers several distinct advantages, succinctly summarized as PNAS: 1) Preferable compression ratio; 2) No post-hoc retraining; 3) Affordable inference time; and 4) Short compression time. Lastly, in light of the observed stagnation in hardware Moore's Law, we conjecture “Moore's Law of Model Compression", i.e., the efficiency of model compression could double annually in the near future to meet the needs of large model era. We believe that model compression based on hyperfunction can play an important role in “Moore's Law of Model Compression".

Biography

Dr. Fenglei Fan is currently an Assistant Professor with Department of Data Science, City University of Hong Kong. He is the Chief Scientist of the Huawei Key Project. His primary research interests lie in NeuroAI and its applications in model compression and medical imaging. He was the recipients of the IBM AI Horizon Scholarship, the 2021 International Neural Network Society Doctoral Dissertation Award, and He won OlympusMons Pioneering Award, a prestigious award in the field of storage. He has one paper selected as one of few 2024 CVPR Best Paper Award Candidates, one won the IEEE Nuclear and Plasma Society IEEE TRPMS Best Paper Award, and one ESI highly cited paper. He organized special issues in journals like IEEE TRPMS, presented three tutorials in AAAI2023, IJCNN25, and WWW2025, and served as (senior) program committee members in AAAI and IJCAI.