Orthogonal Deep Neural Networks.

Researchers

Journal

Modalities

Models

Abstract

In this paper, we introduce the algorithms of Orthogonal Deep Neural Networks (OrthDNNs) to connect with recent interest of spectrally regularized deep learning methods. OrthDNNs are theoretically motivated by generalization analysis of modern DNNs, with the aim to find solution properties of network weights that guarantee better generalization. To this end, we first prove that DNNs are of local isometry on data distributions of practical interest; by using a new covering of the sample space and introducing the local isometry property of DNNs into generalization analysis, we establish a new generalization error bound that is characterized by singular value spectrum of each of networks’ weight matrices. We prove that the optimal bound w.r.t. the degree of isometry is attained when each weight matrix has a spectrum of equal singular values, among which that with orthonormal rows/columns is the most straightforward choice, suggesting the algorithms of OrthDNNs. We present both algorithms of strict and approximate OrthDNNs, and for the later ones we propose Singular Value Bounding, which performs as well as strict OrthDNNs but at a much lower computational cost. We also propose algorithms to make compatible use of batch normalization with OrthDNNs. Extensive comparative studies show the efficacy of OrthDNNs.