A Text2Text model for semantic generation of building layouts

The goal of the project would be to fine tune GPT-Neo J 6b on the task of semantic design generation.

The model will learn to transform natural language prompts into geometric descriptions of designs. It will then be deployed in a simple interface that allows for users to immediately see their generated design, creating a semantic generation tool out of Neo.

I already have a dataset of paired annotations-design pairs and have started experimentations with the smaller Neo models as we speak.

Let me know if you’re interested :slight_smile:


That’s a cool idea !

Could you post an example of language prompt => geometric descriptions => generated design ? Thanks.

Sure! I have loads actually :slight_smile:

These are some outputs from training the smallest neo (125M) on 5% of my total data.

Legend: bedroom=green, bathroom=pink, the other two are kitchen (darker pink) and living room

My goal is to understand the effect of scale in this, going all the way up to Neo-J 6b model trained on all the data.

But also, ideally, to incorporate more modalities such as taking advantage of relational data, adding design evaluation as rewards, and other things I can’t think of but others might!


Another example

Your experiments look amazing, would love to be a part of this.

1 Like

That’s a great project!

1 Like

Hi @THEODOROS ! I’m located in IST region.
I’d like to work on this project with you. I have some experience in vision models but I’d like to dabble with auto-decoder models. Let me know if you’ve place for one on your team :slight_smile:

1 Like

This sounds like a very very cool idea! :exploding_head:

1 Like

Hello, thanks for your interest!

yes of course you can join, as soon as I figure out how we do that :slight_smile:

I think we can do that on Monday, if I understood it correctly!

@THEODOROS that’s cool idea, would love to be part of this project.

1 Like

This sounds very cool @THEODOROS,I would love to be in a project like this.I have some good hands on experience work with transformer based models.

1 Like

@THEODOROS let me know if I can be a part of this project,it would be great.

1 Like

The idea is very unique @THEODOROS .would love to work for this project

1 Like

Thanks everyone! I think joining should be alright, as soon as I figure out how to do it.

I believe Patric mentioned in slack that teams will be made on Monday. The limit, if enough interest, seems to be 10 people although if we do reach that we should consider ways to make this more interesting and explore more angles.

Yeah, we’ll start forming teams from Monday, but it looks like you’ve already got a team here :slight_smile:

If there are 10 members, then the team might get 2 TPU VMs :wink:

1 Like

officially defining this project :slight_smile:

Putting everybody in the official sheet here. More people can still join! Leave a comment here or on the sheet if you want to change something. @gchhablani I was not sure if you wanted to be part of this project or not, so I added you anyways. Let me know :slight_smile:

1 Like

Thannk you @valhalla ! I guess we’ll follow on the slack on how to begin.

1 Like

Hey everyone!

It’s a hectic week for me, and I guess it will be for all of us, but it would be nice to meet and discuss ideas if you want. I’ve made our channel in the discord provided by HF here: Flax-HuggingFace-Community-Week

It’s called gpt-building-layouts, feel free to jump in and say hi!

1 Like

That’s an amazing idea! If it’s still possible I’d like to join the team @THEODOROS I only able to leave comments in the google spreadsheet. Also, could you please help me to join the slack channel gpt-building-layouts that you mentioned above?