I started as a mixed biologist/chemist/mathematician and in the end, after initially thinking I would do biology, majored in chemistry.  It wasn't easy, especially as I had almost no physics and then had to deal with quantum mechanics and group theory among other nasties, but I've never regretted my decision.  My masters year in the lab was amazing, and I'm very glad now that I chose to do an analytical and numerate degree.

(because it's incredibly useful for my fashion business and all that  )

Joking aside, yes, chemistry is harder than biology but the job market is tough and the more numerate degrees can give you an edge.

PM me if you want to chat further