Apologies for crossposting, but I think it’s safe to say that you guys have the most experience of working with transformers. Has anyone looked into models that uses two transformers and takes the dot product in the loss calculation. This is such a problem: tensorflow - Getting nans for gradient - Stack Overflow