Is the term “Oy Vey” when used by a non Jewish person offensive

I’ve recently been made aware that the alt right uses oy vey as a derogatory slur but I’m interested to see how many people are aware of this as I was not at all aware.