当前位置：网站首页>Comparative learning loss function (rince/relic/relicv2)

Comparative learning loss function (rince/relic/relicv2)

2022-07-19 05:47:00 【byzy】

The loss function commonly used in comparative learning is InfoNCE.

One 、Robust Contrastive Learning against Noisy Views（RINCE）

Link to the original text ：https://arxiv.org/pdf/2201.04309.pdf

In this paper, RINCE, A kind of noise （ Such as the excessive enlargement of images 、 Over dubbing of video 、 The video and title are not aligned ） Loss of robustness , And there is no need to explicitly estimate the noise .

The article mentions ：RINCE It's a use. Wasserstein dependency measure The comparative lower bound of mutual information expressed ; and InfoNCE yes KL The lower bound of mutual information expressed by divergence .

Let the data distribution be , The noise data set is $S=\{x,\hat{y}\}$ , The correct label means $\hat{y}_i=y_i$ The probability of is $\eta$ . Then the goal is to minimize

$R_l^\eta(f)=\mathbb{E}_{D_\eta}[l(f(x),\hat{y})]$

among Is the binary cross entropy loss .

Symmetric loss function is robust to noise . Loss function Symmetry satisfies

l(s,1)+l(s,-1)=c (1)

（ Constant ）, among by The resulting forecast score （ Yes The gradient of should also have symmetry ）.

The symmetrical contrast loss has the following form ：

$L(s)=\underbrace{l(s^+,1)}_{\textup{Positive Pair}}+\lambda\underbrace{\sum_{i=1}^Kl(s_i^-,-1)}_{K\textup{ negative pairs}}$ (2)

among s^+ and s^- They are positive / The score of negative sample pairs . The weight $\lambda$ The size of reflects the relative importance of positive and negative samples .

InfoNCE Not satisfied with $s^{+/-}$ Symmetry condition in gradient .

RINCE The losses are as follows ：

$L^{\lambda,q}_{\textup{RINCE}}(s)=\frac{-e^{q\cdot s^+}}{q}+\frac{(\lambda\cdot(e^{s^+}+\sum_{i=1}^Ke^{s^-_i}))^q}{q}$

among and $\lambda$ Both in (0,1] Within the scope of （ Experiments show that $\lambda$ The value of is not sensitive ）.

when ,RINCE Completely satisfied （2） Symmetry of formula （ here , Satisfy （1） Formula and ）：
$L^{\lambda,q=1}_{\textup{RINCE}}(s)=-(1-\lambda)e^{s^+}+\lambda\sum_{i=1}^Ke^{s^-_i}$
At this time, the loss is robust to noise .
Tend to 0 when ,RINCE Asymptotically tends to InfoNCE.

Regardless of Why is it worth , The higher the positive sample score , The lower the negative sample score , The smaller the loss .

In gradient calculation ：
         when ,RINCE Pay more attention to easy-positive The sample of （ Positive samples with high scores ）;
        InfoNCE（） Pay more attention to hard-positive The sample of （ Positive samples with low scores ）;
         Both pay attention to hard-negative The sample of （ Negative samples with high scores ）.

therefore InfoNCE Convergence is faster when there is no noise , and q=1 Of RINCE More robust to noise . In fact, in [0.1,0.5] Within the scope of .

Two 、Representation learning via invariant casual mechanisms（ReLIC）

Link to the original text ：https://arxiv.org/pdf/2010.07922.pdf

This paper divides the data into content and style （ For example, to classify whether the image is a dog , The dog in the image is the content , And background 、 Lighting and other factors are style ）, The expression learned should only be related to the content .

Adopt the scheme of data expansion to change the style while retaining the content （ Like image rotation 、 Change the grayscale 、 Cropping and panning ）, Form a positive sample .

$-\sum_{i=1}^N\sum_{a_{lk}}\log\frac{\exp(\phi(f(x_i^{a_l}),h(x_i^{a_k}))/\tau)}{\sum_{m=1}^M\exp(\phi(f(x_i^{a_l}),h(x_m^{a_k}))/\tau)}+\alpha\sum_{a_{lk},a_{qt}}KL(p^{do(a_{lk})},p^{do(a_{qt})})$

Upper form Represents a neural network ; And relevant , Often take ; $\phi(f(x_i),h(x_j))=\langle g(f(x_i)),g(h(x_j)) \rangle$ , It's a full connection layer . $a_{lk}=(a_l,a_k)$ Expand for a pair ; $p^{do(a_{lk})}$ namely $p^{do(a_{lk})}(Y^R=j|f(x_i))\propto \exp(\phi(f(x_i^{a_l}),h(x_j^{a_k}))/\tau)$ .

The previous item is the usual comparison loss , The latter one is the augmented invariance penalty （ Or invariance loss ）（ That is, the content should not be changed as much as possible ）, This item can reduce the distance between classes .

This paper also explains the reasons for the success of self supervised learning , It proves that ： Set the downstream task set , And the mission Than All tasks in are more detailed . If you pass The expression learned is only related to the content , Then this expression can be generalized to All downstream tasks in .

3、 ... and 、Pushing the limits of self-supervised ResNets: Can we outperform supervised learning without labels on ImageNet?（ReLICv2）

Link to the original text ：https://arxiv.org/pdf/2201.05119.pdf

ReLICv2 and ReLIC The loss function of is similar ：

$l_{\textup{ReLICv2}}(x_i)=\sum_{x^+_i\in P(x_i)}-\alpha\log p(x_i;x_i^+)+\beta D_{KL}(p(x_i)|p(x^+))$

there Namely ReLIC The first term in the formula $\log$ Later content , and $D_{KL}$ Namely ReLIC The second term of the formula in ：
$\log p(x_i;x_i^+)=\log \frac{e^{\phi_\tau(x_i;x_i^+)}}{e^{\phi_\tau(x_i;x_i^+)}+\sum_{x_i^-\in N(x_i)}e^{\phi_\tau(x_i;x_i^-)}}$
$D_{KL}(p(x_i)|p(x_i^+))=\textup{sg}\left [ \mathbb{E}_{p(x_i;x_i^+)}\log p(x_i;x_i^+) \right ]-\mathbb{E}_{p(x_i;x_i^+)}\log p(x_i^+;x_i)$
$\textup{sg}[\cdot]$ Express stop gradient, The degenerate solution can be avoided in the optimization process .

and ReLIC The difference lies in choosing positive and negative samples ： Positive samples are generated by multi-crop augmentation And significance based background removal , And then use the standard SimCLR Expansion plan of （ see Overview of self supervised learning Some examples of comparative learning 2）; Negative samples can be used hard-negative sampling, But this article is in batch It is evenly and randomly selected .

attach ： Two discrete probability distributions KL The divergence ：

$KL(P,Q)=\sum P(x)\log \frac{P(x)}{Q(x)}$