CLIP+NeRF: Fewshot Learning, Putting NeRF on a Diet

Flax/JAX Projects
1

ViT+CLIP+NeRF: Fewshot Learning, Putting NeRF on a Diet

Putting NeRF in a Diet

Is anyone interested in Computer Vision Field? :smiley:
Our team project goal is the implementation of this paper:
Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis :

With this project, We will code a 3D neural scene representation (NeRF: Neural Radiances Field) estimated from a few images. Which is based on extracting the semantic information using a pre-trained visual encoder such as CLIP, a Vision Transformer.

1. Languages and Skills

  • Languages : Python, Pytorch, JAX
  • Other Skills : Git, Github, Cloud Computing Experience

2. Model / Baseline Code

  • ViT, CLIP : https://github.com/openai/CLIP

  • Meatlearning-NeRF (Learned Initialization Paper : JAX Based NeRF Code) :
    https://github.com/tancik/learnit

  • NeRF (Pytorch Based NeRF Code) : https://github.com/yenchenlin/nerf-pytorch

3. Datasets

  • Phototourism dataset:
    https://www.cs.ubc.ca/~kmyi/imw2020/data.html

  • Realistic Synthetic dataset :
    https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1

  • DTU multi-view stereo (MVS) dataset :
    http://roboimagedata.compute.dtu.dk/?page_id=36

4. Challenges

  • JAX based NeRF Code
  • Data Preprocessing
  • CLIP / NeRF training with TPU
  • Augmentation CLIP to NeRF with JAX based code

5. Project Goal Output

The desired project output is an implementation of the Fewshot learning NeRF model which is using CLIP+ViT (JAX based code)

6. References

  • NeRF : https://www.matthewtancik.com/nerf
  • Putting NeRF in a diet : https://arxiv.org/abs/2104.00677
  • Learned Initialization : https://arxiv.org/abs/2012.02189

7. Team Mindset

-Enjoy! :stuck_out_tongue_winking_eye:
-Passionate :fire:
-Collaborative :people_holding_hands:
-Learning Oriented :man_student:

6 Likes
3

@Sasikanth Thank you for your interest :smiley: Welcome!

1 Like
4

Really like the atmosphere here :slight_smile: Looks like one of the more ambitious projects - letā€™s give it a try! @valhalla - maybe you can take a look here as well. Will put it down on the Excel

1 Like
5

Iā€™m interested!

1 Like
7

Welcome guys! :partying_face: Letā€™s make a wonderful project! Please come and join the ā€œputting-nerf-on-a-dietā€ channel on the discord : Flax-HuggingFace-Community-Week @Sasikanth @JYChung @alexlau

2 Likes
8

This is really interesting, and would love to be part of this.

9

Iā€™m interested too!

11

This looks awesome, Iā€™m in

1 Like
12

Nice to meet you all! Our team finishes the team recruiting now! :smile:
Welcome 8 members: @howtowhy @Sasikanth @JYChung @alexlau @syedmech47 @sseung0703 @MrBananaHuman @khalidsaifullaah

@valhalla - Hello, valhalla, Could you put our team members to exel sheet?

4 Likes
13

Sure!

This looks like an ambitious project!

Out of curiosity, how long do you think itā€™ll take to NeRF in JAX? VIT and CLIP are already available in JAX.

Itā€™s important to define the scope such that the project can be finished with the given time and compute budget.

14

@valhalla Thank you for fast response. :grinning: Could I reply after checking more detail on the paper? At now I guess that CLIP and NeRF could be available as JAX code but connecting CLIP and NeRF + Training unified model to new data would be challenge for our team. Also we will consider detail implementation on the paper.

1 Like
15

Could I reply after checking more detail on the paper

Sure!

16

Is there still space left for one more :grinning_face_with_smiling_eyes:?

I have been looking at Few-shot learning for the past month and experimenting with Resnet architectures, would love to try ViT and CLIP.

17

@hassiahk OK hassiahk! If you want to join us please come to discord! :slight_smile: #putting_nerf_on_a_diet channel Flax-HuggingFace-Community-Week
The team recruiting is really finished! now :0!

1 Like
18

added you @hassiahk :slight_smile:

19

Hi,
I think I check this too late thoughā€¦ May I join?

I am a vision researcher in AR industry.
Iā€™m interested in this project!!

20

@george31 Hello could you come in to discord of our channel?

21

@george31 If you want to join our team, please come in discord and participate today meeting
:slight_smile:

22

Yes!! Thanks!

23

@howtowhy Can I get a new invitation link? ā€¦ I thinks the link above got expiredā€¦