Sentence Similarity for Code Generation related tasks

Beginners
1

I am working on code generation using OpenAI Codex.
Specifically, I’m making a personal project that lets users enter a specific “task” that they want to do related to code generation.
Currently I have around 10 tasks e.g. explain a piece of code, convert some code to a one liner, write unit-tests for a function.
My goal is to recognize the user’s intent. Normally, the approach would be to use a multi-class classification model but due to some time & data constraints, I want to use sentence similarity instead.
Basically my idea is that for each task:
Generate a list of of the 3-4 most common prompts that users can enter for this task
For any user query
Perform a sentence similarity search
For the task with the top similarity score
If score > threshold
recognize this task as the user’s actual intent
Else user intent is not among current set of tasks
My questions are:

  1. Is this approach any good?
  2. Should I use a Passage Ranking or Semantic Textual Similarity model? Or something else?
  3. Is there any relevant dataset on which I can finetune the model to work on prompts specific to code generation?
2

Hey there! There is this blog post from @espejelomar using Sentence Transformers by @nreimers that will answer some of your questions Sentence Transformers for semantic search - a Hugging Face Space by sentence-transformers. I hope this helps :smile: