TGI with guidance generates weird output when asked to answer in a "structured" way

Intermediate
1

Hello, i’m currently using TGI on a project i currently work on, i recently rewrote my prototype to use TGI’s guidance (with pydantic) to force the structure of the model’s output, this is working perfectly and actually solved alot of problems i had.

But there is something that gets very bad, when i ask the model to generate a structured output, like a markdown list, or just when the model wants to add a line break it gets very messy. For the markdown list it just spits nonsense (for some reason it even go into generating html ??), and for the line breaks it ends up as something like this : \\\\\\\n (with as many \ as the model feels like outputting).

For exemple when asking this :

“Can you make a list of cats renown to be good in appartments in a markdown list format please ?”

The model answered this :

Certainly, here is a list of cat breeds that are known to do well in apartments:">">">">">">">* British Shorthair*"> * Scottish Fold*"> * Persian*"> * Exotic Shorthair*"> * Russian Blue*"> * Burmese*"> * Siamese*"> * Sphynx*"> * Bengal*"> * American Shorthair*">

And this is not a one time thing it actually is very difficult to even have the model generates the \\\\\n without adding other characters for no reason, also it is important to note that this behavior appeared at the moment i started using guidance.

I transform my pydantic models in regex since i saw on a github issue that the json type might have ordering issues and since i need ordering i can’t really not use it, but i’m thinking this might be the problem as it might constraint the generation weirdly ? :thinking:

1 Like
2

It might be different, but it might be related to this issue.

1 Like
4

Yeah it’s a bit different but i’m gonna look more into the pr’s code maybe i’ll find a workaround

1 Like
5

So, after some testing (very easy testing actually), it is fore sure not a problem specific to the fact i’m translating the json schema to regex, it seems like TGI has an escaping mechanism with guidance that mess with the generation as sometime i can see some tokens that looks like special tokens (like tool call) for escaping parts of the generation, they appear when the model does not generate them properly, i’m going to look for something like that and will probably open an issue soon :thinking:

1 Like