Characterizing the Spectrum of the NTK via a Power Series Expansion

Add code
Nov 15, 2022
Michael Murray, Hui Jin, Benjamin Bowman, Guido Montufar

Figure 1 for Characterizing the Spectrum of the NTK via a Power Series Expansion
Figure 2 for Characterizing the Spectrum of the NTK via a Power Series Expansion
Figure 3 for Characterizing the Spectrum of the NTK via a Power Series Expansion
Figure 4 for Characterizing the Spectrum of the NTK via a Power Series Expansion

Share this with someone who'll enjoy it:

Twitter IconFacebook IconLinkedin IconWhatsapp IconMessenger Icon

Under mild conditions on the network initialization we derive a power series expansion for the Neural Tangent Kernel (NTK) of arbitrarily deep feedforward networks in the infinite width limit. We provide expressions for the coefficients of this power series which depend on both the Hermite coefficients of the activation function as well as the depth of the network. We observe faster decay of the Hermite coefficients leads to faster decay in the NTK coefficients. Using this series, first we relate the effective rank of the NTK to the effective rank of the input-data Gram. Second, for data drawn uniformly on the sphere we derive an explicit formula for the eigenvalues of the NTK, which shows faster decay in the NTK coefficients implies a faster decay in its spectrum. From this we recover existing results on eigenvalue asymptotics for ReLU networks and comment on how the activation function influences the RKHS. Finally, for generic data and activation functions with sufficiently fast Hermite coefficient decay, we derive an asymptotic upper bound on the spectrum of the NTK.

* 51 pages, 3 Figures, 1 Table  
View paper onarxiv iconopen_review iconOpenReview

Share this with someone who'll enjoy it:

Twitter IconFacebook IconLinkedin IconWhatsapp IconMessenger Icon