How do we know found lost groups of Jews are Jews and not Samaritans?

I am under the impression that often discovered groups don’t directly identify as Jews but are identified by practice that is obviously a regional variation of Judaism. How do we know that they’re not part of the Samaritan diaspora that are considered non Jews?