Learning Transferable Visual Models From Natural Language Supervision

Jointly training an image and text encoder to predict which captions match which images (natural language supervision) can create a general-purpose vision model.

CLIP

  • In traditional computer vision a large-scale dataset of images was created, each image was labeled with a category (e.g. dog, plane, etc.), and a model was trained to classify the images
  • However, there were three issues:
    • Labels are expensive
    • The model only learns the predefined categories
    • The model struggles with new concepts
  • The core idea of CLIP is learning visual representations from text paired with images
  • It is much easier to scale natural language supervision compared to standard crowd-source labeling for image classification
  • Methods which work on natural language can learn passively from the supervision contained in the vast amount of text on the Internet

CLIP

Pre-Training

  • Given a batch of $N$ (image, text) pairs, CLIP is trained to predict which of the $N \times N$ possible (image, text) pairings across a batch actually occurred
  • CLIP learns a multi-modal embedding space by jointly training an image and text encoder to do two things:
    • Maximize the cosine similarity of the image and text embeddings of the $N$ real pairs in the batch while
    • Minimize the cosine similarity of the embeddings of the $N^2 - N$ incorrect pairings
  • The image encoder is a ResNet or a Vision Transformer and the text encoder is a Transformer

Zero-Shot Classification

  • The most famous result from the paper is that CLIP can classify images without being trained on those classes of images
  • If we want to classify an image as either A, B, or C we would create three captions:
    • “A photo of A”
    • “A photo of B”
    • “A photo of C”
    • Note: We use these captions to remain in distribution of captions on the Internet which often take the form of “a photo of ___”
  • We would encode image into $X$ and compute the similarity between $x$ and each caption before taking the caption with the highest similarity

Conclusion

  • CLIP was critical in bridging the gap between image and language into a shared semantic space
  • Breakthrough: if you train a vision model on enough text-image pairs, language becomes an interface for visual understanding