Negative Kl values during PPO training (TRL library)

[naviidmzr]

Hey everyone!
I am trying to train my model using PPO trainer from TRL library. However I receive negative kl values. any Idea where might have gone wrong?
The configs:
generation_kwargs = {
“do_sample”:True,
“top_k”:9,
“max_length”:1024,
“top_p”:0.9,
}

dataset = train_dataset

ppo_config = {“mini_batch_size”: 1,
“batch_size”: 1,
“learning_rate”: 1.41e-5,
}
ppo_trainer = PPOTrainer(config, model, tokenizer = tokenizer, dataset = dataset)