Domain Adversarial Methods (DANN, CDAN)

Domain Adversarial Neural Network (DANN)

class dalib.adaptation.dann.DomainAdversarialLoss(domain_discriminator, reduction='mean', grl=None)

The Domain Adversarial Loss proposed in Domain-Adversarial Training of Neural Networks (ICML 2015)

Domain adversarial loss measures the domain discrepancy through training a domain discriminator. Given domain discriminator \(D\), feature representation \(f\), the definition of DANN loss is

\[loss(\mathcal{D}_s, \mathcal{D}_t) = \mathbb{E}_{x_i^s \sim \mathcal{D}_s} log[D(f_i^s)] + \mathbb{E}_{x_j^t \sim \mathcal{D}_t} log[1-D(f_j^t)].\]

Parameters

• domain_discriminator (torch.nn.Module) – A domain discriminator object, which predicts the domains of features. Its input shape is (N, F) and output shape is (N, 1)

• reduction (str, optional) – Specifies the reduction to apply to the output: 'none' | 'mean' | 'sum'. 'none': no reduction will be applied, 'mean': the sum of the output will be divided by the number of elements in the output, 'sum': the output will be summed. Default: 'mean'

• grl (WarmStartGradientReverseLayer, optional) – Default: None.

Inputs:

• f_s (tensor): feature representations on source domain, \(f^s\)

• f_t (tensor): feature representations on target domain, \(f^t\)

• w_s (tensor, optional): a rescaling weight given to each instance from source domain.

• w_t (tensor, optional): a rescaling weight given to each instance from target domain.

Shape:

• f_s, f_t: \((N, F)\) where F means the dimension of input features.

• Outputs: scalar by default. If reduction is 'none', then \((N, )\).

Examples:

>>> from dalib.modules.domain_discriminator import DomainDiscriminator
>>> discriminator = DomainDiscriminator(in_feature=1024, hidden_size=1024)
>>> loss = DomainAdversarialLoss(discriminator, reduction='mean')
>>> # features from source domain and target domain
>>> f_s, f_t = torch.randn(20, 1024), torch.randn(20, 1024)
>>> # If you want to assign different weights to each instance, you should pass in w_s and w_t
>>> w_s, w_t = torch.randn(20), torch.randn(20)
>>> output = loss(f_s, f_t, w_s, w_t)

Conditional Domain Adversarial Network (CDAN)

class dalib.adaptation.cdan.ConditionalDomainAdversarialLoss(domain_discriminator, entropy_conditioning=False, randomized=False, num_classes=-1, features_dim=-1, randomized_dim=1024, reduction='mean')

The Conditional Domain Adversarial Loss used in Conditional Adversarial Domain Adaptation (NIPS 2018)

Conditional Domain adversarial loss measures the domain discrepancy through training a domain discriminator in a conditional manner. Given domain discriminator \(D\), feature representation \(f\) and classifier predictions \(g\), the definition of CDAN loss is

\[\begin{split}loss(\mathcal{D}_s, \mathcal{D}_t) &= \mathbb{E}_{x_i^s \sim \mathcal{D}_s} log[D(T(f_i^s, g_i^s))] \\ &+ \mathbb{E}_{x_j^t \sim \mathcal{D}_t} log[1-D(T(f_j^t, g_j^t))],\\\end{split}\]

where \(T\) is a MultiLinearMap or RandomizedMultiLinearMap which convert two tensors to a single tensor.

Parameters

• domain_discriminator (torch.nn.Module) – A domain discriminator object, which predicts the domains of features. Its input shape is (N, F) and output shape is (N, 1)

• entropy_conditioning (bool, optional) – If True, use entropy-aware weight to reweight each training example. Default: False

• randomized (bool, optional) – If True, use randomized multi linear map. Else, use multi linear map. Default: False

• num_classes (int, optional) – Number of classes. Default: -1

• features_dim (int, optional) – Dimension of input features. Default: -1

• randomized_dim (int, optional) – Dimension of features after randomized. Default: 1024

• reduction (str, optional) – Specifies the reduction to apply to the output: 'none' | 'mean' | 'sum'. 'none': no reduction will be applied, 'mean': the sum of the output will be divided by the number of elements in the output, 'sum': the output will be summed. Default: 'mean'

Note

You need to provide num_classes, features_dim and randomized_dim only when randomized is set True.

Inputs:

• g_s (tensor): unnormalized classifier predictions on source domain, \(g^s\)

• f_s (tensor): feature representations on source domain, \(f^s\)

• g_t (tensor): unnormalized classifier predictions on target domain, \(g^t\)

• f_t (tensor): feature representations on target domain, \(f^t\)

Shape:

• g_s, g_t: \((minibatch, C)\) where C means the number of classes.

• f_s, f_t: \((minibatch, F)\) where F means the dimension of input features.

• Output: scalar by default. If reduction is 'none', then \((minibatch, )\).

Examples:

>>> from dalib.modules.domain_discriminator import DomainDiscriminator
>>> from dalib.adaptation.cdan import ConditionalDomainAdversarialLoss
>>> import torch
>>> num_classes = 2
>>> feature_dim = 1024
>>> batch_size = 10
>>> discriminator = DomainDiscriminator(in_feature=feature_dim * num_classes, hidden_size=1024)
>>> loss = ConditionalDomainAdversarialLoss(discriminator, reduction='mean')
>>> # features from source domain and target domain
>>> f_s, f_t = torch.randn(batch_size, feature_dim), torch.randn(batch_size, feature_dim)
>>> # logits output from source domain adn target domain
>>> g_s, g_t = torch.randn(batch_size, num_classes), torch.randn(batch_size, num_classes)
>>> output = loss(g_s, f_s, g_t, f_t)

class dalib.adaptation.cdan.RandomizedMultiLinearMap(features_dim, num_classes, output_dim=1024)

Random multi linear map

Given two inputs \(f\) and \(g\), the definition is

\[T_{\odot}(f,g) = \dfrac{1}{\sqrt{d}} (R_f f) \odot (R_g g),\]

where \(\odot\) is element-wise product, \(R_f\) and \(R_g\) are random matrices sampled only once and fixed in training.

Parameters

• features_dim (int) – dimension of input \(f\)

• num_classes (int) – dimension of input \(g\)

• output_dim (int, optional) – dimension of output tensor. Default: 1024

Shape:

• f: (minibatch, features_dim)

• g: (minibatch, num_classes)

• Outputs: (minibatch, output_dim)

class dalib.adaptation.cdan.MultiLinearMap

Multi linear map

Shape:

• f: (minibatch, F)

• g: (minibatch, C)

• Outputs: (minibatch, F * C)