I opened my closet this morning and was picking out a T-shirt to wear. I saw a couple of different colours: green, blue, red, lots of black and blacker. But no white. And I realised. I don't have white T-shirts. More than that though - I don't want to have white T-shirts. I tend to get those dirty faster than most.
I put on my T-shirt and peeked out the window. I saw my car in the driveway. I don't have a motorcycle because I don't want a motorcycle. I think they're dangerous.
Not having white T-shirts and not owning a motorcycle are not by happenstance. They're a direct consequence of me not wanting them in the first place. And I'm not afraid or embarrassed to own up in public to the fact that I don't want them at all.
You see, there's a difference between not having something and not wanting to have something.
But when you say that your website doesn't have disabled users, I don't think it's because you don't want them. You actually have no idea if you have them already or not, or what you could do to have them.
Here's the thing.
If you know you could have disabled users and are comfortable putting up on your website "We don't want to have disabled users," then it's absolutely perfectly fine for you not to care about web accessibility.
I somehow doubt you'd do that.