It's not more convenient to read on a phone. You have made it a habit and then made yourself believe that your habits are rational. Not only for you, but for the rest of the world.
E-Readers are absolutely better for reading than phones, that's why there's a big market for them.
It's wildly more convenient to read on a phone. I prefer the screen, form factor, and overall UX of an e-reader for extended reading sessions, but I haven't turned mine on in years because the phone is just sooooo much more convenient. Most e-readers don't fit in a pocket, and even if they do it's annoying to have to guess when I'll need it and carry the 2nd device. Whenever I have downtime....airport, doctors office, curbisde pickup, wife isn't ready to check out yet, lunch takes 5 minutes in the microwave....that phone/kindle for iOS is always ready to go. I probably do half my reading in 5-15 minute increments of formerly dead time. For a while I even tried switching to the e-reader whenever I sat down to read "for real", but even the relatively painless syncing process wasn't worth the minor UX benefits of the ereader. The phone is my least favorite way to read, but convenience is the one category where it absolutely mops the floor with e-readers(and paper for that matter).
You're not saying it's more convenient to read on a phone. You're saying that it's more convenient to bring your phone.
Sure, but we can all choose our habits. If I'm stuck waiting at a doctor's office, I will read on my phone like everybody else. But in all other situations, when you actively want to read, an e-reader is better.
I'm saying that it's more convenient for me to read on the phone in all situations I can imagine. Even at home....I just sat down, kicked up the recliner, and decided to read a noeel rather than the cookbook on the end table or putting something on TV. The phone is in my pocket and the e-reader is in another room, maybe in need of a charge, maybe in need of a sync. Maybe I should have thought about that before I sat down but I didn't. Sure I could choose to change my habits, but changing habits is difficult and inconvenient. And even once I've succeeded, the new habit takes more time and effort which is inconvenient. That inconvenience is modest and may be totally worth it for the (to your mind significant) benefits of an e-reader. But it is still an inconvenience. First world problems to be sure, but I will not expend 5% more effort for a 2% increase(my number, yours is a lot higher) in reading device quality, and I'm hardly alone in that decision.
It really depends on how you're reading. I find that smartphones are actually more convenient to use for reading lying down in bed, and the screen works better at night too because with modern OLED ones you can use black background with very dim amber text for very comfortable low-light and no-light reading. OTOH eInk is the best option for outdoors reading for sure, and the multi-week battery life is very nice when travelling.