It often depends on the context of which you are being asked. If someone is asked for a good reason (offering assistance or such), then it usually isn't so bad. But if it is for disparaging reasons, that can get pretty annoying.

Sometimes it is because the person asking isn't paying attention, or don't realize someone has vision loss. But there are ways to do it in a non-rude way. For example, I was working with a client, who wanted to get on a specific bus, but that bus stop had different routes stop there. My client asked the driver what the bus number was, and the driver (who didn't even bother looking at the client) just says "What, are you blind? Read the sign". When my client explained he couldn't read the sign (he had a white cane, but wasn't total), the bus driver told him the number without apologizing.

I wrote on color recently as a response on another thread:

https://www.reddit.com/r/Blind/comments/7hikjv/a_description_of_sight_for_the_blindsincebirth/dqrn6g1/

I suppose if you're blind, and someone asks if you're blind, then the appropriate answer would be "yes." There is, of course, often a note of disparagement that goes along with that question that can feel pretty hostile.

Whenever I get the "are you blind" question, it's usually followed by, "is it true...blah blah blah hearing blah blah..." I when they get to, "is it true..." I say,"yup, and read minds too."

I went for awhile where I wasn't asked this question. Then I started working face to face with sighted clients. (I'm a technology trainer and consultant.) I realized that if I didn't have a canned response ready, I could get very annoyed very fast. So I would say something like, yes, I am. I can see enough to know you're in front of me but not enough to know what you look like. That usually is enough of an explanation for most people. I use an iPhone as my work phone, and I'm asked how I use it. Again, I have a response ready. I say, I use gestures to get around the screen, and I hear what the screen is saying in my earpiece. Again, it's enough of an explanation for people.

As for color, I see shades of gray. So I think of colors as lighter or darker than each other. I know what colors mean to people. IE, green means summer. Blue is the color of the sky. Pink and purple are often thought of as colors which women like. Black is the darkest color; white is the lightest. Yellow is bright, like the sun. I know enough to speak about color in visual terms when asked. But color is just a term to me. I guess the best way to explain it would be something like this. I don't listen to jazz. But if a friend told me they liked jazz, I'd say, that's cool. Tell me why you like it. The same goes for color. If someone told me they liked red, I'd say, that's cool. And I'd ask, why do you like the color red? What significance does it hold for you? Although I don't care about color personally, I like to get people talking. So if color is what someone wants to talk about, I'd listen, mainly because it's what's on their mind. And I like to get to know all kinds of people.