We eat the same food, we rub elbows, we share almost all of the same space. Most interactions, day to day, there's no cue for who's on what side of the line. That keeps us safe. You're giving them a big visual cue that separates us and them, labels us 'inmate' with one motion.