Learning Transferable Visual Models From Natural Language Supervision

At present, I begin to understand the knowledge related to multimodality , Welcome criticism ！

This paper comes from 2021 Year of International Conference on Machine Learning, Sort out and revise the main contents of the paper , Reference resources 【 Paper reading 】CLIP：Learning Transferable Visual Models From Natural Language Supervision ------ Multimodal , Vision , Pre training model _me_yundou The blog of -CSDN Blog Learning Transferable Visual Models From Natural Language Supervision - John_Ran - Blog Garden Two articles .

Thesis title ： Learn transferable visual models from natural language supervision

Research questions ： Combine text data and image data , Put forward CLIP, Use the method of comparative learning to study language - Image pre training , It's an efficient 、 Extensible supervised learning method of natural language .

Research ideas ： Use pictures on the Internet , Training CLIP. After training , Natural language is used to refer to learned visual concepts , Then proceed zero-shot transfer learning.

（1） The first is to build CLIP,CLIP It's actually a pre training model , Including text editor and image editor , Calculate the similarity of text vector and image vector respectively , To predict whether they are a pair , Pictured 1 Shown .CLIP Input the image and text into an image encoder respectively image_encoder And a text encoder text_encoder, Get the vector representation of image and text I-f and T_f . Then map the vector representation of image and text to a joint multi-channel space , Get a new vector representation of image and text that can be directly compared I_e and T_e . Then calculate the distance between the image vector and the text vector cosine Similarity degree . Last , The objective function of comparative learning is to make the similarity of positive sample pairs higher , The similarity of negative sample pairs is low .

  chart 1

 CLIP Jointly train image encoder and text encoder to predict a batch ( Images , Text ) Correct pairing of training examples . At testing time , Learn the text encoder by embedding the name or description of the target dataset class , Synthesize a zero shot linear classifier .CLIP Code as shown 2 Shown ：

  chart 2 

（2） Conduct zero-shot transfer learning

Research process ：1. Build a large enough data set -----》WebImageText(4 100 million texts - The image is right )

                  2. Choose an effective pre training model -----》CLIP

                  3. Select and scale the model ------》 The author chooses two models , One is ResNet-D, Smoothed rect-2 blur                          pooling. take global average pooling Use one attention pooling To improve . Among them the                                   transformer Type of layer , In order to global average-pooled representation As query. second                         vision The structure of is ViT, Few changes ： stay patch embbeding and position embedding combination                             after , Added a layer normalization. Then when it comes to implementation , A little different initialization is used                         Strategy .

                   4. Preliminary training ------》 Trained scale Strategy ,5 individual ResNet,3 individual vit.ResNet-50,  ResNet-101,                           RN50x4, RN50x16, and RN50x64. ViT-B/32, a ViT-B/16, and a ViT-L/14. Finally make                           With 32,768 Of batch size. Used gradient checkpoint. Semi precision .The largest                                   ResNet model, RN50x64, took 18 days to train on 592 V100 GPUs while the                                 largest Vision . Transformer took 12 days on 256 V100 GPUs. One more vit Use                              336 Of pixel resolution.

                    5. utilize CLIP------》 For each data set , Use the names of all classes in the dataset as a set of potential text pairs                         close , And according to CLIP Predict the most likely ( Images 、 Text ) Yes . Besides , Also try for CLIP Provide text prompts                           To help assign tasks , And integrating multiple of these templates to improve performance .

        Data sets and experimental results ： For the performance of the model , The author in 27 Experiments were carried out on two data sets , Found in 16 Better performance on data sets ：

Major innovations ：CLIP It's a pre training model , It's like BERT、GPT、ViT Just like the pre training model . First, these models are trained with a large amount of unlabeled data , Then the trained model can be realized , Enter a piece of text （ Or an image ）, The output text （ Images ） Vector representation of .CLIP and BERT、GPT、ViT The difference is that ,CLIP It's multimodal , Including image processing and text processing , and BERT、GPT Is single text modal ,ViT It's single image mode .

summary ：

About CLIP Some limitations of ：

1. The author thinks that , Only with baseline Draw is not the ultimate goal . Because it is completely supervised with these data sets SOTA Compare to ,CLIP I can't beat them . You need to enlarge the current calculation amount to 1000x To achieve the present SOTA, This is impossible under the current hardware conditions .

2. The author thinks that ,CLIP In some special type especially strong task Not very work. such as , On some fine-grained data sets , Or some more abstract 、 symmetrical task. these task Pictures of the , stay CLIP Of pre-train The data set of appears less . The author thinks that , There are still a lot of it task On ,CLIP I'm guessing .

3. CLIP It works well in many natural image distributions , But in some real out-of-distributiob Is still not very good on the data set , For example OCR On . stay rendered text The performance is quite good , Because this is CLIP Of pre-training It is very common . But in handwritten numeral recognition, it collapsed , Only 88% The accuracy of . Because from semantic and near-duplicate nearest-neighbor retrieval We didn't find .