In the end I have opted for
imdb_df = pd.DataFrame(imdb)
movie_df = pd.DataFrame(movie_imdb_format)
overall_df = pd.concat([imdb_df, movie_df])
dataset = Dataset.from_pandas(overall_df)
which solves the above problem. Though I am not sure why it was failing.