Slope Centering: Making Shortcut Weights Effective

N. N. Schraudolph. Slope Centering: Making Shortcut Weights Effective. In Proc. Intl. Conf. Artificial Neural Networks (ICANN), pp. 523–528, Springer Verlag, Berlin, Skövde, Sweden, 1998.

Download

134.5kB	57.3kB	64.5kB

Abstract

Shortcut connections are a popular architectural feature of multi-layer perceptrons. It is generally assumed that by implementing a linear sub-mapping, shortcuts assist the learning process in the remainder of the network. Here we find that this is not always the case: shortcut weights may also act as distractors that slow down convergence and can lead to inferior solutions. This problem can be addressed with slope centering, a particular form of gradient factor centering. By removing the linear component of the error signal at a hidden node, slope centering effectively decouples that node from the shortcuts that bypass it. This eliminates the possibility of destructive interference from shortcut weights, and thus ensures that the benefits of shortcut connections are fully realized.

BibTeX Entry

@inproceedings{Schraudolph98b,
     author = {Nicol N. Schraudolph},
      title = {\href{http://nic.schraudolph.org/pubs/Schraudolph98b.pdf}{
               Slope Centering: Making Shortcut Weights Effective}},
      pages = {523--528},
     editor = {Lars Niklasson and Mikael Bod\'en and Tom Ziemke},
  booktitle =  icann,
    address = {Sk\"ovde, Sweden},
     series = {Perspectives in Neural Computing},
  publisher = {\href{http://www.springer.de/}{Springer Verlag}, Berlin},
       year =  1998,
   b2h_type = {Top Conferences},
  b2h_topic = {>Preconditioning},
   abstract = {
    Shortcut connections are a popular architectural feature of multi-layer
    perceptrons.  It is generally assumed that by implementing a linear
    sub-mapping, shortcuts assist the learning process in the remainder
    of the network.  Here we find that this is not always the case: shortcut
    weights may also act as distractors that slow down convergence and can
    lead to inferior solutions.
    This problem can be addressed with {\em slope centering}\/, a particular
    form of \href{b2hg-facede}{\em gradient factor centering}.  By removing
    the linear component of the error signal at a hidden node, slope centering
    effectively decouples that node from the shortcuts that bypass it.  This
    eliminates the possibility of destructive interference from shortcut
    weights, and thus ensures that the benefits of shortcut connections are
    fully realized.
    }}