Batch Spectral Shrinkage (BSS)

class ftlib.finetune.bss.BatchSpectralShrinkage(k=1)

The regularization term in Catastrophic Forgetting Meets Negative Transfer: Batch Spectral Shrinkage for Safe Transfer Learning (NIPS 2019).

The BSS regularization of feature matrix \(F\) can be described as:

\[L_{bss}(F) = \sum_{i=1}^{k} \sigma_{-i}^2 ,\]

where \(k\) is the number of singular values to be penalized, \(\sigma_{-i}\) is the \(i\)-th smallest singular value of feature matrix \(F\).

All the singular values of feature matrix \(F\) are computed by SVD:

\[F = U\Sigma V^T,\]

where the main diagonal elements of the singular value matrix \(\Sigma\) is \([\sigma_1, \sigma_2, ..., \sigma_b]\).

Parameters

k (int) – The number of singular values to be penalized. Default: 1

Shape:

• Input: \((b, |\mathcal{f}|)\) where \(b\) is the batch size and \(|\mathcal{f}|\) is feature dimension.

• Output: scalar.