NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size.

Researchers

Journal

Modalities

Models

Abstract

Neural architecture search (NAS) has attracted much attention and has been illustrated to bring tangible benefits in a large number of applications in the past few years. Architecture topology and architecture size have been regarded as two of the most important aspects for the performance of deep learning models and the community has spawned lots of searching algorithms for both of those aspects of the neural architectures. However, the performance gain from these searching algorithms is achieved under different search spaces and training setups. This makes the overall performance of the algorithms incomparable and the improvement from a sub-module of the searching model unclear. In this paper, we propose NATS-Bench, a unified benchmark on searching for both topology and size, for (almost) any up-to-date algorithm. NATS-Bench includes the search space of 15,625 neural cell candidates for architecture topology and 32,768 for architecture size on three datasets. We analyze the validity of our benchmark in terms of various criteria and performance comparison of all candidates in the search space. We show the versatility of NATS-Bench by benchmarking 13 recent state-of-the-art NAS algorithms. This facilitates a much larger community of researchers to focus on developing better algorithms in a more comparable environment.