My application relies on Dataset to manage some data.
We have multiple workers loading the same Dataset to do some computation.
Currently, a single worker writes to the Dataset and adds columns to the table.
I need the other workers to have access to the latest data.
In the past, we were able to overwrite the dataset and the workers would simply reload the new version. But we get an error as of 1.16 saying that we can’t overwrite a dataset.
What is the best way to do this? For now, I can disable caching to avoid the error.