@BramVanroy I think you can use a single optimizer, but specify a different learning rate for the parameters you want, like so:
optimizer = optim.Adam([
{'params': model.parameters()},
{'params': model.embeddings.pos_tag_embeddings(), 'lr': 5e-5}
], lr=2e-5)