Too-Hot-to-Handle: Insights into Temperature and Noise Hyperparameters for Differentiable Neural-Architecture-Searches

Conrad J, Wilhelmstatter S, Asthana R, Belagiannis V, Ortmanns M (2024)

Publication Type: Conference contribution

Publication year: 2024

Publisher: Institute of Electrical and Electronics Engineers Inc.

Pages Range: 557-561

Conference Proceedings Title: 2024 IEEE 6th International Conference on AI Circuits and Systems (AICAS)

Event location: Abu Dhabi AE

ISBN: 9798350383638

DOI: 10.1109/AICAS59952.2024.10595971

Abstract

The deployment of neural networks on resource-constrained hardware requires architecture optimization algorithms that respect accuracy and computational cost. One emerging way for performing such an architecture optimization is the differentiable neural-architecture-search (DNAS). DNAS finds the weights of a neural network as well as its topology in a single training. The topology search-space is represented by instantiating different layer-candidates for each layer in parallel, thereby forming a so-called weighted super-network. The instances are connected via a gumbel-softmax function which employs important parameters commonly called temperature τ and noise-scale β. The temperature is scheduled during training and controls the convergence to selecting only a single final candidate per optimized layer. The noise-scale adds a random behavior in trying different candidates before deciding for a final one. The state-of-the-art (SotA) does not define rules for setting those important hyperparameters. Therefore, this work elaborates methods to predict feasible start- and end-temperature as well as noise-scale prior to training and to fine-tune the parameters from metrics obtained during a DNAS.

Authors with CRIS profile

Rohan Asthana Professur für Maschinelles Lernen in der Signalverarbeitung Vasileios Belagiannis Professur für Maschinelles Lernen in der Signalverarbeitung

Involved external institutions

Universität Ulm DE Germany (DE)

How to cite

APA:

Conrad, J., Wilhelmstatter, S., Asthana, R., Belagiannis, V., & Ortmanns, M. (2024). Too-Hot-to-Handle: Insights into Temperature and Noise Hyperparameters for Differentiable Neural-Architecture-Searches. In 2024 IEEE 6th International Conference on AI Circuits and Systems (AICAS) (pp. 557-561). Abu Dhabi, AE: Institute of Electrical and Electronics Engineers Inc..

MLA:

Conrad, Joschua, et al. "Too-Hot-to-Handle: Insights into Temperature and Noise Hyperparameters for Differentiable Neural-Architecture-Searches." Proceedings of the 6th IEEE International Conference on AI Circuits and Systems (AICAS 2024), Abu Dhabi Institute of Electrical and Electronics Engineers Inc., 2024. 557-561.

BibTeX: Download