Robust and Tiny Binary Neural Networks using Gradient-based Explainability Methods

Sabih M, Yayla M, Hannig F, Teich J, Chen JJ (2023)

Publication Language: English

Publication Type: Conference contribution, Conference Contribution

Publication year: 2023

Publisher: Association for Computing Machinery (ACM)

City/Town: New York(NY) United States

Pages Range: 87–93

Conference Proceedings Title: EuroMLSys '23: Proceedings of the 3rd Workshop on Machine Learning and System

Event location: Rome, Italy IT

ISBN: 979-8-4007-0084-2

URI: https://dl.acm.org/doi/10.1145/3578356.3592595

DOI: 10.1145/3578356.3592595

Open Access Link: https://dl.acm.org/doi/10.1145/3578356.3592595

Abstract

Binary neural networks (BNNs) are a highly resource-efficient variant of neural networks. The efficiency of BNNs for tiny machine learning (TinyML) systems can be enhanced by structured pruning and making BNNs robust to faults. When used with approximate memory systems, this fault tolerance can be traded off for energy consumption, latency, or cost. For pruning, magnitude-based heuristics are not useful because the weights in a BNN can either be -1 or +1. Global pruning of BNNs has not been studied well so far. Thus, in this paper, we explore gradient-based ranking criteria for pruning BNNs and use them in combination with a sensitivity analysis. For robustness, the state-of-the-art is to train the BNNs with bit-flips in what is known as fault-aware training. We propose a method to guide fault-aware training using gradient-based explainability methods. This allows us to obtain robust and efficient BNNs for deployment on tiny devices. Experiments on audio and image processing applications show that our proposed approach outperforms the existing approaches, making it useful for obtaining efficient and robust models for a slight degradation in accuracy. This makes our approach valuable for many TinyML use cases.

Authors with CRIS profile

Muhammad Sabih Lehrstuhl für Informatik 12 (Hardware-Software-Co-Design) Frank Hannig Lehrstuhl für Informatik 12 (Hardware-Software-Co-Design) Jürgen Teich Lehrstuhl für Informatik 12 (Hardware-Software-Co-Design)

Related research project(s)

Neural Approximate Accelerator Architecture Optimization for DNN Inference on Lightweight FPGAs (NA³Os) May 1, 2024 - April 30, 2027

Involved external institutions

Technische Universität Dortmund DE Germany (DE)

How to cite

APA:

Sabih, M., Yayla, M., Hannig, F., Teich, J., & Chen, J.-J. (2023). Robust and Tiny Binary Neural Networks using Gradient-based Explainability Methods. In Eiko Yoneki, Luigi Nardi (Eds.), EuroMLSys '23: Proceedings of the 3rd Workshop on Machine Learning and System (pp. 87–93). Rome, Italy, IT: New York(NY) United States: Association for Computing Machinery (ACM).

MLA:

Sabih, Muhammad, et al. "Robust and Tiny Binary Neural Networks using Gradient-based Explainability Methods." Proceedings of the EuroMLSys '23: Proceedings of the 3rd Workshop on Machine Learning and Systems, Rome, Italy Ed. Eiko Yoneki, Luigi Nardi, New York(NY) United States: Association for Computing Machinery (ACM), 2023. 87–93.

BibTeX: Download