Using Fine-Grained Access Tokens for Inference Endpoints

I have created a couple of endpoints within my organization.

We would like to control access to them indepentently, that is, some users in the organization can use endpoint A, but not endpoint B, and viceversa.

Could this be achieved using Fine-Grained Access Tokens? If not, could you suggest an approach to handle this scenario?

Thanks!