## Accelerated Gradient Descent by Factor-Centering Decomposition

N. N. Schraudolph. Accelerated Gradient Descent by Factor-Centering Decomposition. Technical Report IDSIA-33-98, Istituto Dalle Molle di Studi sull'Intelligenza Artificiale, 1998.

 187.7kB 75.8kB 101.9kB

### Abstract

Gradient factor centering is a new methodology for decomposing neural networks into biased and centered subnets which are then trained in parallel. The decomposition can be applied to any pattern-dependent factor in the network's gradient, and is designed such that the subnets are more amenable to optimization by gradient descent than the original network: biased subnets because of their simplified architecture, centered subnets due to a modified gradient that improves conditioning. The architectural and algorithmic modifications mandated by this approach include both familiar and novel elements, often in prescribed combinations. The framework suggests for instance that shortcut connections---a well-known architectural feature---should work best in conjunction with slope centering, a new technique described herein. Our benchmark experiments bear out this prediction, and show that factor-centering decomposition can speed up learning significantly without adversely affecting the trained network's generalization ability.

### BibTeX Entry

@techreport{facede,
author = {Nicol N. Schraudolph},
title = {\href{http://nic.schraudolph.org/pubs/facede.pdf}{
Factor-Centering Decomposition}},
number = {IDSIA-33-98},
institution = {Istituto Dalle Molle di Studi sull'Intelligenza Artificiale},
address = {Galleria 2, CH-6928 Manno, Switzerland},
year =  1998,
b2h_type = {Other},
b2h_topic = {>Preconditioning},
abstract = {
{\em Gradient factor centering}\/ is a new methodology for decomposing
neural networks into {\em biased}\/ and {\em centered}\/ subnets which
are then trained in parallel.  The decomposition can be applied to any
pattern-dependent factor in the network's gradient, and is designed such
that the subnets are more amenable to optimization by gradient descent
than the original network: biased subnets because of their simplified
architecture, centered subnets due to a modified gradient that improves
conditioning.
The architectural and algorithmic modifications mandated by this
approach include both familiar and novel elements, often in prescribed
combinations.  The framework suggests for instance that {\em shortcut
connections}\/---\,a well-known architectural feature\,---\,should
work best in conjunction with \href{b2hd-slope}{\em slope centering},
a new technique described herein.  Our benchmark experiments bear
out this prediction, and show that factor-centering decomposition
can speed up learning significantly without adversely affecting the
trained network's generalization ability.
}}


Generated by bib2html.pl (written by Patrick Riley) on Thu Sep 25, 2014 12:00:33