TensorFlow implementation of Barlow Twins (Barlow Twins: Self-Supervised Learning via Redundancy Reduction)

Last update: Sep 14, 2022

Overview

Barlow-Twins-TF

This repository implements Barlow Twins (Barlow Twins: Self-Supervised Learning via Redundancy Reduction) in TensorFlow and demonstrates it on the CIFAR10 dataset.

Summary:

With a ResNet20 as a trunk and a 3-layer MLP (each layer containing 2048 units) and 100 epochs of pre-training, this training notebook can give 62.61% accuracy on the CIFAR10 test set. The pre-training total takes ~23 minutes on a single Tesla V100. There are minor differences from the original implementation. However, the original loss function and the other minor details like having a big enough projection dimension have been maintained.

For details on Barlow Twins, I suggest reading the original paper, it's really well-written.

Loss progress during pre-training

Other notes

Pre-trained model is available here.
To follow the original implementation details as closely as possible, a WarmUpCosine learning rate schedule has been used during pre-training:
During linear evaluation, Cosine Decay has been used.

Acknowledgements

Thanks to Stéphane Deny (one of the authors of the paper) for helping me catch a pesky bug.

Releases(v1.0.0)

v1.0.0(May 10, 2021)

The archive contains a ResNet20 encoder (trunk + projection head) trained using Barlow Twins for 100 epochs with CIFAR10.
Source code(tar.gz)
Source code(zip)
barlow_twins.tar.gz(38.07 MB)

Public implementation of "Learning from Suboptimal Demonstration via Self-Supervised Reward Regression" from CoRL'21

Self-Supervised Reward Regression (SSRR) Codebase for CoRL 2021 paper "Learning from Suboptimal Demonstration via Self-Supervised Reward Regression "

19 Dec 12, 2022

An official PyTorch implementation of the TKDE paper "Self-Supervised Graph Representation Learning via Topology Transformations".

Self-Supervised Graph Representation Learning via Topology Transformations This repository is the official PyTorch implementation of the following pap

2 Oct 31, 2022

Twins: Revisiting the Design of Spatial Attention in Vision Transformers

Twins: Revisiting the Design of Spatial Attention in Vision Transformers Very recently, a variety of vision transformer architectures for dense predic

482 Dec 18, 2022

This repository is the official implementation of Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning (NeurIPS21).

Core-tuning This repository is the official implementation of ``Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regular

18 Dec 17, 2022

PyTorch implementation of the paper Deep Networks from the Principle of Rate Reduction

A repository that shares tuning results of trained models generated by TensorFlow / Keras. Post-training quantization (Weight Quantization, Integer Quantization, Full Integer Quantization, Float16 Quantization), Quantization-aware training. TensorFlow Lite. OpenVINO. CoreML. TensorFlow.js. TF-TRT. MediaPipe. ONNX. [.tflite,.h5,.pb,saved_model,tfjs,tftrt,mlmodel,.xml/.bin, .onnx]

PINTO_model_zoo Please read the contents of the LICENSE file located directly under each folder before using the model. My model conversion scripts ar

2.4k Jan 5, 2023

TensorFlow implementation of Barlow Twins (Barlow Twins: Self-Supervised Learning via Redundancy Reduction)

Related tags

Overview

Barlow-Twins-TF

Loss progress during pre-training

Other notes

Acknowledgements

You might also like...

Public implementation of "Learning from Suboptimal Demonstration via Self-Supervised Reward Regression" from CoRL'21

An official PyTorch implementation of the TKDE paper "Self-Supervised Graph Representation Learning via Topology Transformations".

Twins: Revisiting the Design of Spatial Attention in Vision Transformers

This repository is the official implementation of Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning (NeurIPS21).

PyTorch implementation of the paper Deep Networks from the Principle of Rate Reduction

Official NumPy Implementation of Deep Networks from the Principle of Rate Reduction (2021)

Model-based 3D Hand Reconstruction via Self-Supervised Learning, CVPR2021

TLDR: Twin Learning for Dimensionality Reduction

Releases(v1.0.0)

v1.0.0(May 10, 2021)

Owner

Sayak Paul

Code accompanying the paper Say As You Wish: Fine-grained Control of Image Caption Generation with Abstract Scene Graphs (Chen et al., CVPR 2020, Oral).

This is the repository for the NeurIPS-21 paper [Contrastive Graph Poisson Networks: Semi-Supervised Learning with Extremely Limited Labels].

Interactive Image Segmentation via Backpropagating Refinement Scheme

VarCLR: Variable Semantic Representation Pre-training via Contrastive Learning

A PaddlePaddle version image model zoo.

An OpenAI Gym environment for multi-agent car racing based on Gym's original car racing environment.

Repo for the Tutorials of Day1-Day3 of the Nordic Probabilistic AI School 2021 (https://probabilistic.ai/)

Code for the published paper : Learning to recognize rare traffic sign

Distributed Evolutionary Algorithms in Python

🔥 TensorFlow Code for technical report: "YOLOv3: An Incremental Improvement"

A time series processing library

📖 Deep Attentional Guided Image Filtering

Automatic tool focused on deriving metallicities of open clusters

A lightweight tool to get an AI Infrastructure Stack up in minutes not days.

Collaborative forensic timeline analysis

Official Implementation of CVPR 2022 paper: "Mimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental Learning"

SwinTrack: A Simple and Strong Baseline for Transformer Tracking

SubOmiEmbed: Self-supervised Representation Learning of Multi-omics Data for Cancer Type Classification

💃 VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena

Implementation of "The Power of Scale for Parameter-Efficient Prompt Tuning"