Temporal Information

Is there any way we can include the timeliness (timestamp) factor in the transformer model, is it creating sequence temporally ordered or any other feature of the BERT or related transformer on including the temporal information from the data.
thanks