Can we stop referring to people as "Aspies"?

I was diagnosed with Aspergers a few years ago. In most areas of life, it's considered offensive to refer to someone based on a characteristic they have or a perceived disadvantage (e.g. - retard, cripple, etc.), why is it okay to refer to someone as an aspie?