Species barriers, expressed as hybrid inviability and sterility, are often due to epistatic interactions between divergent loci from two lineages. Theoretical models indicate that the strength, direction, and complexity of these genetic interactions can strongly affect the expression of interspecific reproductive isolation and the rates at which new species evolve. Nonetheless, empirical analyses have not quantified the frequency with which loci are involved in interactions affecting hybrid fitness, and whether these loci predominantly interact synergistically or antagonistically, or preferentially involve loci that have strong individual effects on hybrid fitness. We systematically examined the prevalence of interactions between pairs of short chromosomal regions from one species (Solanum habrochaites) co-introgressed into a heterospecific genetic background (Solanum lycopersicum), using lines containing pairwise combinations of 15 chromosomal segments from S. habrochaites in the background of S. lycopersicum (i.e., 95 double introgression lines). We compared the strength of hybrid incompatibility (either pollen sterility or seed sterility) expressed in each double introgression line to the expected additive effect of its two component single introgressions. We found that epistasis was common among co-introgressed regions. Interactions for hybrid dysfunction were substantially more prevalent in pollen fertility compared to seed fertility phenotypes, and were overwhelmingly antagonistic (i.e., double hybrids were less unfit than expected from additive single introgression effects). This pervasive antagonism is expected to attenuate the rate at which hybrid infertility accumulates among lineages over time (i.e., giving diminishing returns as more reproductive isolation loci accumulate), as well as decouple of patterns of accumulation of sterility loci and hybrid incompatibility phenotypes. This decoupling effect might explain observed differences between pollen and seed fertility in their fit to theoretical predictions of the accumulation of isolation loci, including the ‘snowball’ effect.