Official Implementation of SimIPU: Simple 2D Image and 3D Point Cloud Unsupervised Pre-Training for Spatial-Aware Visual Representations

Related tags

Deep LearningSimIPU
Overview

Official Implementation of SimIPU

  • SimIPU: Simple 2D Image and 3D Point Cloud Unsupervised Pre-Training for Spatial-Aware Visual Representations
  • Since the code is still waiting for release, if you have any question with reproduction, feel free to contact us. We will try our best to help you.
  • Currently, the core code of SimIPU is implemented in the commercial project. We are trying our best to make the code publicly available.
Comments
  • Question about augmentation

    Question about augmentation

    Hi, I'm a little confused about the data augmentation.

    1. How did you set img_aug when img_moco=True? It seems that we need an 'img_pipeline' in 'simipu_kitti.py', right?
    2. For 3D augmentation, it seems that it is done in this line. So the 3D augmentation is done based on the point features instead the raw points, right? If I want to try moco=True, how to set 3D augmentation? should I do this in the dataset building part? https://github.com/zhyever/SimIPU/blob/5b346e392c161a5e9fdde09b1692656bc7cd3faf/project_cl/decorator/inter_intro_decorator_moco_better.py#L394

    Looking forward to your reply. Many thanks.

    opened by sunnyHelen 2
  • error for env setup:ImportError: cannot import name 'ball_query_ext' from 'mmdet3d.ops.ball_query'

    error for env setup:ImportError: cannot import name 'ball_query_ext' from 'mmdet3d.ops.ball_query'

    Thanks for your insightful paper and clear code repo!

    Hi, I met with the ImportError: cannot import name 'ball_query_ext' from 'mmdet3d.ops.ball_query' when run the command bash tools/dist_train.sh project_cl/configs/simipu/simipu_kitti.py 1 --work_dir ./

    Do you know how to solve it?

    Traceback (most recent call last): File "tools/train.py", line 16, in from mmdet3d.apis import train_model File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/apis/init.py", line 1, in from .inference import (convert_SyncBN, inference_detector, File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/apis/inference.py", line 10, in from mmdet3d.core import (Box3DMode, DepthInstance3DBoxes, File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/core/init.py", line 2, in from .bbox import * # noqa: F401, F403 File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/core/bbox/init.py", line 4, in from .iou_calculators import (AxisAlignedBboxOverlaps3D, BboxOverlaps3D, File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/core/bbox/iou_calculators/init.py", line 1, in from .iou3d_calculator import (AxisAlignedBboxOverlaps3D, BboxOverlaps3D, File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/core/bbox/iou_calculators/iou3d_calculator.py", line 5, in from ..structures import get_box_type File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/core/bbox/structures/init.py", line 1, in from .base_box3d import BaseInstance3DBoxes File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/core/bbox/structures/base_box3d.py", line 5, in from mmdet3d.ops.iou3d import iou3d_cuda File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/ops/init.py", line 5, in from .ball_query import ball_query File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/ops/ball_query/init.py", line 1, in from .ball_query import ball_query File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/ops/ball_query/ball_query.py", line 4, in from . import ball_query_ext ImportError: cannot import name 'ball_query_ext' from 'mmdet3d.ops.ball_query' (/mnt/lustre/xxh/SimIPU-main/mmdet3d/ops/ball_query/init.py)

    I noticed that you once met with the same error. https://github.com/open-mmlab/mmdetection3d/issues/503#issuecomment-847618114

    So, I would like to ask for your help~ Hopefully you have a good solution. :)

    opened by JerryX1110 2
  • A question about eq5 and eq6

    A question about eq5 and eq6

    Thanks for your inspiring work. I have some wonder about eq5 and eq6. As far as I know, After eq5, f should be a tensor which is a global feature with shape (batchsize * 2048 * 1 * 1), how can you sample corresponding image features by projection location? After all, there's no spatial information in f anymore. Or maybe you got features from a previous layer of ResNet? Looking forward to your reply.

    opened by lianchengmingjue 2
  • A question about Tab.5 in Ablation Study

    A question about Tab.5 in Ablation Study

    Thanks for your excellent work first! I have a question about Tab.5 in Ablation Study. Why "Scratch" equals "SimIPU w/o inter-module ", which means that the intra-module is useless?

    opened by Trent-tangtao 1
  • Have you tried not to crop gradient of f^{\alpha} in eq7?

    Have you tried not to crop gradient of f^{\alpha} in eq7?

    Hi, I like your good work! I am wondering have you tried not to crop the gradient of $f^{\alpha}$ in eq7? If you crop the gradient, it seems like the pertaining of the point branch cannot learn anything from the image branch.

    opened by Hiusam 1
  • issues about create_data

    issues about create_data

    Hi, thanks for sharing your great work. I encounter some issues during creating data by running create_data.py First create reduced point cloud for training set [ ] 0/3712, elapsed: 0s, ETA:Traceback (most recent call last): File "tools/create_data.py", line 247, in
    out_dir=args.out_dir)
    File "tools/create_data.py", line 24, in kitti_data_prep
    kitti.create_reduced_point_cloud(root_path, info_prefix)
    File "/mnt/lustre/chenzhuo1/hzha/SimIPU/tools/data_converter/kitti_converter.py", line 374, in create_reduced_point_cloud
    _create_reduced_point_cloud(data_path, train_info_path, save_path)
    File "/mnt/lustre/chenzhuo1/hzha/SimIPU/tools/data_converter/kitti_converter.py", line 314, in _create_reduced_point_cloud
    count=-1).reshape([-1, num_features])
    ValueError: cannot reshape array of size 461536 into shape (6)

    It seems to set the num_features=4 and front_camera_id=2? in this line: https://github.com/zhyever/SimIPU/blob/5b346e392c161a5e9fdde09b1692656bc7cd3faf/tools/data_converter/kitti_converter.py#L291

    I assume doing this can solve the problem but encounter another problem when Create GT Database of KittiDataset
    [ ] 0/3712, elapsed: 0s, ETA:Traceback (most recent call last):
    File "tools/create_data.py", line 247, in
    out_dir=args.out_dir)
    File "tools/create_data.py", line 44, in kitti_data_prep
    with_bbox=True) # for moca
    File "/mnt/lustre/chenzhuo1/hzha/SimIPU/tools/data_converter/create_gt_database.py", line 275, in create_groundtruth_database
    P0 = np.array(example['P0']).reshape(4, 4)
    KeyError: 'P0'

    Can you help me figure out how to solve these issues?

    opened by sunnyHelen 21
Owner
Zhyever
Keep going.
Zhyever
A collection of papers about Transformer in the field of medical image analysis.

A collection of papers about Transformer in the field of medical image analysis.

Junyu Chen 377 Jan 05, 2023
Fast and exact ILP-based solvers for the Minimum Flow Decomposition (MFD) problem, and variants of it.

MFD-ILP Fast and exact ILP-based solvers for the Minimum Flow Decomposition (MFD) problem, and variants of it. The solvers are implemented using Pytho

Algorithmic Bioinformatics Group @ University of Helsinki 4 Oct 23, 2022
Code, environments, and scripts for the paper: "How Private Is Your RL Policy? An Inverse RL Based Analysis Framework"

Privacy-Aware Inverse RL (PRIL) Analysis Framework Code, environments, and scripts for the paper: "How Private Is Your RL Policy? An Inverse RL Based

1 Dec 06, 2021
This repository is the code of the paper "Sparse Spatial Transformers for Few-Shot Learning".

🌟 Sparse Spatial Transformers for Few-Shot Learning This code implements the Sparse Spatial Transformers for Few-Shot Learning(SSFormers). Our code i

chx_nju 38 Dec 13, 2022
Source code for CVPR 2020 paper "Learning to Forget for Meta-Learning"

L2F - Learning to Forget for Meta-Learning Sungyong Baik, Seokil Hong, Kyoung Mu Lee Source code for CVPR 2020 paper "Learning to Forget for Meta-Lear

Sungyong Baik 29 May 22, 2022
A Transformer-Based Feature Segmentation and Region Alignment Method For UAV-View Geo-Localization

University1652-Baseline [Paper] [Slide] [Explore Drone-view Data] [Explore Satellite-view Data] [Explore Street-view Data] [Video Sample] [中文介绍] This

Zhedong Zheng 335 Jan 06, 2023
The DL Streamer Pipeline Zoo is a catalog of optimized media and media analytics pipelines.

The DL Streamer Pipeline Zoo is a catalog of optimized media and media analytics pipelines. It includes tools for downloading pipelines and their dependencies and tools for measuring their performace

8 Dec 04, 2022
Novel and high-performance medical image classification pipelines are heavily utilizing ensemble learning strategies

An Analysis on Ensemble Learning optimized Medical Image Classification with Deep Convolutional Neural Networks Novel and high-performance medical ima

14 Dec 18, 2022
Voice of Pajlada with model and weights.

Pajlada TTS Stripped down version of ForwardTacotron (https://github.com/as-ideas/ForwardTacotron) with pretrained weights for Pajlada's (https://gith

6 Sep 03, 2021
《Where am I looking at? Joint Location and Orientation Estimation by Cross-View Matching》(CVPR 2020)

This contains the codes for cross-view geo-localization method described in: Where am I looking at? Joint Location and Orientation Estimation by Cross-View Matching, CVPR2020.

41 Oct 27, 2022
WatermarkRemoval-WDNet-WACV2021

WatermarkRemoval-WDNet-WACV2021 Thank you for your attention. Citation Please cite the related works in your publications if it helps your research: @

LUYI 63 Dec 05, 2022
HeartRate detector with ArduinoandPython - Use Arduino and Python create a heartrate detector.

Syllabus of Contents Syllabus of Contents Introduction Of Project Features Develop With Python code introduction Installation License Developer Contac

1 Jan 05, 2022
TensorFlow implementation of ENet, trained on the Cityscapes dataset.

segmentation TensorFlow implementation of ENet (https://arxiv.org/pdf/1606.02147.pdf) based on the official Torch implementation (https://github.com/e

Fredrik Gustafsson 248 Dec 16, 2022
CONditionals for Ordinal Regression and classification in tensorflow

Condor Ordinal regression in Tensorflow Keras Tensorflow Keras implementation of CONDOR Ordinal Regression (aka ordinal classification) by Garrett Jen

9 Jul 31, 2022
Conversion between units used in magnetism

convmag Conversion between various units used in magnetism The conversions between base units available are: T - G : 1e4

0 Jul 15, 2021
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

ALBERT ***************New March 28, 2020 *************** Add a colab tutorial to run fine-tuning for GLUE datasets. ***************New January 7, 2020

Google Research 3k Jan 01, 2023
Implementation of ICCV21 paper: PnP-DETR: Towards Efficient Visual Analysis with Transformers

Implementation of ICCV 2021 paper: PnP-DETR: Towards Efficient Visual Analysis with Transformers arxiv This repository is based on detr Recently, DETR

twang 113 Dec 27, 2022
Delving into Localization Errors for Monocular 3D Object Detection, CVPR'2021

Delving into Localization Errors for Monocular 3D Detection By Xinzhu Ma, Yinmin Zhang, Dan Xu, Dongzhan Zhou, Shuai Yi, Haojie Li, Wanli Ouyang. Intr

XINZHU.MA 124 Jan 04, 2023
Official implementation of "UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-wise Perspective with Transformer"

[AAAI2022] UCTransNet This repo is the official implementation of "UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-wise Perspectiv

Haonan Wang 199 Jan 03, 2023
PyTorch implementation of CDistNet: Perceiving Multi-Domain Character Distance for Robust Text Recognition

PyTorch implementation of CDistNet: Perceiving Multi-Domain Character Distance for Robust Text Recognition The unofficial code of CDistNet. Now, we ha

25 Jul 20, 2022