CLIP+NeRF: Fewshot Learning, Putting NeRF on a Diet

ViT+CLIP+NeRF: Fewshot Learning, Putting NeRF on a Diet

Putting NeRF in a Diet

Is anyone interested in Computer Vision Field? :smiley:
Our team project goal is the implementation of this paper:
Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis :

With this project, We will code a 3D neural scene representation (NeRF: Neural Radiances Field) estimated from a few images. Which is based on extracting the semantic information using a pre-trained visual encoder such as CLIP, a Vision Transformer.

1. Languages and Skills

  • Languages : Python, Pytorch, JAX
  • Other Skills : Git, Github, Cloud Computing Experience

2. Model / Baseline Code

  • ViT, CLIP : https://github.com/openai/CLIP

  • Meatlearning-NeRF (Learned Initialization Paper : JAX Based NeRF Code) :
    https://github.com/tancik/learnit

  • NeRF (Pytorch Based NeRF Code) : https://github.com/yenchenlin/nerf-pytorch

3. Datasets

  • Phototourism dataset:
    https://www.cs.ubc.ca/~kmyi/imw2020/data.html

  • Realistic Synthetic dataset :
    https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1

  • DTU multi-view stereo (MVS) dataset :
    http://roboimagedata.compute.dtu.dk/?page_id=36

4. Challenges

  • JAX based NeRF Code
  • Data Preprocessing
  • CLIP / NeRF training with TPU
  • Augmentation CLIP to NeRF with JAX based code

5. Project Goal Output

The desired project output is an implementation of the Fewshot learning NeRF model which is using CLIP+ViT (JAX based code)

6. References

  • NeRF : https://www.matthewtancik.com/nerf
  • Putting NeRF in a diet : https://arxiv.org/abs/2104.00677
  • Learned Initialization : https://arxiv.org/abs/2012.02189

7. Team Mindset

-Enjoy! :stuck_out_tongue_winking_eye:
-Passionate :fire:
-Collaborative :people_holding_hands:
-Learning Oriented :man_student:

5 Likes

Hi @howtowhy ,

This appears very interesting and cool project. I would like to participate and contribute in this project. I am hoping, I will be able to learn interesting concepts from this project.

1 Like

@Sasikanth Thank you for your interest :smiley: Welcome!

1 Like

Really like the atmosphere here :slight_smile: Looks like one of the more ambitious projects - let’s give it a try! @valhalla - maybe you can take a look here as well. Will put it down on the Excel

1 Like

I’m interested!

1 Like

This looks super cool!
Could I join this project? I am excited to make this into a demo

2 Likes

Welcome guys! :partying_face: Let’s make a wonderful project! Please come and join the “putting-nerf-on-a-diet” channel on the discord : Flax-HuggingFace-Community-Week @Sasikanth @JYChung @alexlau

2 Likes

This is really interesting, and would love to be part of this.

I’m interested too!

I want to join this project!

This looks awesome, I’m in

1 Like

Nice to meet you all! Our team finishes the team recruiting now! :smile:
Welcome 8 members: @howtowhy @Sasikanth @JYChung @alexlau @syedmech47 @sseung0703 @MrBananaHuman @khalidsaifullaah

@valhalla - Hello, valhalla, Could you put our team members to exel sheet?

4 Likes

Sure!

This looks like an ambitious project!

Out of curiosity, how long do you think it’ll take to NeRF in JAX? VIT and CLIP are already available in JAX.

It’s important to define the scope such that the project can be finished with the given time and compute budget.

@valhalla Thank you for fast response. :grinning: Could I reply after checking more detail on the paper? At now I guess that CLIP and NeRF could be available as JAX code but connecting CLIP and NeRF + Training unified model to new data would be challenge for our team. Also we will consider detail implementation on the paper.

1 Like

Could I reply after checking more detail on the paper

Sure!

Is there still space left for one more :grinning_face_with_smiling_eyes:?

I have been looking at Few-shot learning for the past month and experimenting with Resnet architectures, would love to try ViT and CLIP.

@hassiahk OK hassiahk! If you want to join us please come to discord! :slight_smile: #putting_nerf_on_a_diet channel Flax-HuggingFace-Community-Week
The team recruiting is really finished! now :0!

1 Like

added you @hassiahk :slight_smile:

Hi,
I think I check this too late though… May I join?

I am a vision researcher in AR industry.
I’m interested in this project!!

@george31 Hello could you come in to discord of our channel?