You're not saying it's more convenient to read on a phone. You're saying that it's more convenient to bring your phone.
Sure, but we can all choose our habits. If I'm stuck waiting at a doctor's office, I will read on my phone like everybody else. But in all other situations, when you actively want to read, an e-reader is better.
I'm saying that it's more convenient for me to read on the phone in all situations I can imagine. Even at home....I just sat down, kicked up the recliner, and decided to read a noeel rather than the cookbook on the end table or putting something on TV. The phone is in my pocket and the e-reader is in another room, maybe in need of a charge, maybe in need of a sync. Maybe I should have thought about that before I sat down but I didn't. Sure I could choose to change my habits, but changing habits is difficult and inconvenient. And even once I've succeeded, the new habit takes more time and effort which is inconvenient. That inconvenience is modest and may be totally worth it for the (to your mind significant) benefits of an e-reader. But it is still an inconvenience. First world problems to be sure, but I will not expend 5% more effort for a 2% increase(my number, yours is a lot higher) in reading device quality, and I'm hardly alone in that decision.