Huggingface has an interleave datasets function you could check out to combine several datasets together.
And in general, did I decide to solve this problem correctly?
Using more data very well might help, but hard to say without more context. Lots of things can make a model good or bad.