I know you weren't trying to use it in a racist way. My IRL friends and I joke like that all the time. But that's because we all know no one in the group will get offended. When in public or around people that aren't apart of our normal circle, we don't say anything like that at all, out of respect to the people around us. We don't know whether or not they'll get upset or not, but it's not worth to test the waters like that. I have a very dark sense of humor, but I don't go around telling the jokes that I personally find funny due to the possibility of offending some. You gotta remember not everyone thinks like you.
So even if your intentions weren't derogatory, it still has the chance of offending.
This will be my last response as I already put my initial support in my previous post.