The reduction in the fitness of progeny from crosses between genetically different populations.