chainer.link_hooks.SpectralNormalization¶
-
class
chainer.link_hooks.
SpectralNormalization
(n_power_iteration=1, eps=1e-06, use_gamma=False, factor=None, weight_name='W', name=None)[source]¶ Spectral Normalization link hook implementation.
This hook normalizes a weight using max singular value and this value is computed via power iteration method. Currently, this hook is supposed to be added to
chainer.links.Linear
,chainer.links.EmbedID
,chainer.links.Convolution2D
,chainer.links.ConvolutionND
,chainer.links.Deconvolution2D
, andchainer.links.DeconvolutionND
. However, you can use this to other links like RNNs by specifyingweight_name
. It is highly recommended to add this hook before optimizer setup because this hook add a scaling parametergamma
ifuse_gamma
is True. Otherwise, the registeredgamma
will not be updated.\[\begin{split}\bar{\mathbf{W}} &=& \dfrac{\mathbf{W}}{\sigma(\mathbf{W})} \\ \text{, where} \ \sigma(\mathbf{W}) &:=& \max_{\mathbf{h}: \mathbf{h} \ne 0} \dfrac{\|\mathbf{W} \mathbf{h}\|_2}{\|\mathbf{h}\|_2} = \max_{\|\mathbf{h}\|_2 \le 1} \|\mathbf{W}\mathbf{h}\|_2\end{split}\]See: T. Miyato et. al., Spectral Normalization for Generative Adversarial Networks
- Parameters
n_power_iteration (int) – Number of power iteration. The default value is 1.
eps (float) – Numerical stability in norm calculation. The default value is 1e-6 for the compatibility with mixed precision training. The value used in the author’s implementation is 1e-12.
use_gamma (bool) – If
True
, weight scaling parameter gamma which is initialized by initial weight’s max singular value is introduced.factor (float, None) – Scaling parameter to divide maximum singular value. The default value is 1.0.
weight_name (str) – Link’s weight name to apply this hook. The default value is
'W'
.name (str or None) – Name of this hook. The default value is
'SpectralNormalization'
.
- Variables
Example
There are almost the same but 2 ways to apply spectral normalization (SN) hook to links.
1. Initialize link and SN separately. This makes it easy to handle buffer and parameter of links registered by SN hook.
>>> l = L.Convolution2D(3, 5, 3) >>> hook = chainer.link_hooks.SpectralNormalization() >>> _ = l.add_hook(hook) >>> # Check the shape of the first left singular vector. >>> getattr(l, hook.vector_name).shape (5,) >>> # Delete SN hook from this link. >>> l.delete_hook(hook.name)
2. Initialize both link and SN hook at one time. This makes it easy to define your original
Chain
.>>> # SN hook handles lazy initialization! >>> layer = L.Convolution2D( ... 5, 3, stride=1, pad=1).add_hook( ... chainer.link_hooks.SpectralNormalization())
Methods
-
added
(link)[source]¶ Callback function invoked when the link hook is registered
- Parameters
link (Link) – Link object to which the link hook is registered.
None
if the link hook is registered globally.
-
deleted
(link)[source]¶ Callback function invoked when the link hook is unregistered
- Parameters
link (Link) – Link object to which the link hook is unregistered.
None
if the link hook had been registered globally.
-
__eq__
()¶ Return self==value.
-
__ne__
()¶ Return self!=value.
-
__lt__
()¶ Return self<value.
-
__le__
()¶ Return self<=value.
-
__gt__
()¶ Return self>value.
-
__ge__
()¶ Return self>=value.
Attributes
-
name
= 'SpectralNormalization'¶