A Text2Text model for semantic generation of building layouts

Flax/JAX Projects
1

The goal of the project would be to fine tune GPT-Neo J 6b on the task of semantic design generation.

The model will learn to transform natural language prompts into geometric descriptions of designs. It will then be deployed in a simple interface that allows for users to immediately see their generated design, creating a semantic generation tool out of Neo.

I already have a dataset of paired annotations-design pairs and have started experimentations with the smaller Neo models as we speak.

Let me know if you’re interested :slight_smile:

6 Likes
2

That’s a cool idea !

Could you post an example of language prompt => geometric descriptions => generated design ? Thanks.

3

Sure! I have loads actually :slight_smile:

These are some outputs from training the smallest neo (125M) on 5% of my total data.

Legend: bedroom=green, bathroom=pink, the other two are kitchen (darker pink) and living room

gptlayout1
gptlayout11978×1472 138 KB

My goal is to understand the effect of scale in this, going all the way up to Neo-J 6b model trained on all the data.

But also, ideally, to incorporate more modalities such as taking advantage of relational data, adding design evaluation as rewards, and other things I can’t think of but others might!

3 Likes
4

Another example

image
image825×755 27.3 KB

5

image
image1212×728 26.2 KB

6

Your experiments look amazing, would love to be a part of this.

1 Like
7

That’s a great project!

1 Like
9

This sounds like a very very cool idea! :exploding_head:

1 Like
10

Hello, thanks for your interest!

yes of course you can join, as soon as I figure out how we do that :slight_smile:

I think we can do that on Monday, if I understood it correctly!

11

@THEODOROS that’s cool idea, would love to be part of this project.

1 Like
12

This sounds very cool @THEODOROS,I would love to be in a project like this.I have some good hands on experience work with transformer based models.

1 Like
13

@THEODOROS let me know if I can be a part of this project,it would be great.

1 Like
14

The idea is very unique @THEODOROS .would love to work for this project

1 Like
15

Thanks everyone! I think joining should be alright, as soon as I figure out how to do it.

I believe Patric mentioned in slack that teams will be made on Monday. The limit, if enough interest, seems to be 10 people although if we do reach that we should consider ways to make this more interesting and explore more angles.

16

Yeah, we’ll start forming teams from Monday, but it looks like you’ve already got a team here :slight_smile:

If there are 10 members, then the team might get 2 TPU VMs :wink:

1 Like
17

officially defining this project :slight_smile:

Putting everybody in the official sheet here. More people can still join! Leave a comment here or on the sheet if you want to change something. @gchhablani I was not sure if you wanted to be part of this project or not, so I added you anyways. Let me know :slight_smile:

1 Like
18

Thannk you @valhalla ! I guess we’ll follow on the slack on how to begin.

1 Like
19

Hey everyone!

It’s a hectic week for me, and I guess it will be for all of us, but it would be nice to meet and discuss ideas if you want. I’ve made our channel in the discord provided by HF here: Flax-HuggingFace-Community-Week

It’s called gpt-building-layouts, feel free to jump in and say hi!

1 Like
20

That’s an amazing idea! If it’s still possible I’d like to join the team @THEODOROS I only able to leave comments in the google spreadsheet. Also, could you please help me to join the slack channel gpt-building-layouts that you mentioned above?