Why can the Temple only be built in Jerusalem?

If someone built another temple somewhere else like Mount Gerizim or Elephantine, why would those be considered halachicly invalid?

Also, why can’t the Temple be built in Shiloh where the Mishkan was?