Learning sets and disabling positional embedding knowledge?

I have an NLP task that I would like to use a transformer against but the position of my tokens is arbitrary; is there any way to disable positional learning during fine-tuning? if not, should I feature engineer my text (basically a set) such that it has some sort of positional relationship?