Environmental estrogens and anti-hormone therapies for breast cancer have diverse tissue- and signaling-pathway-selective outcomes, but how estrogen receptor alpha (ERα) mediates this phenotypic diversity is poorly understood. We implemented a statistical approach to allow unbiased, parallel analyses of multiple crystal structures, and identified subtle perturbations of ERα structure by different synthetic and environmental estrogens. Many of these perturbations were in the sub-Å range, within the noise of the individual structures, but contributed significantly to the activities of synthetic and environmental estrogens. Combining structural perturbation data from many structures with quantitative cellular activity profiles of the ligands enabled identification of structural rules for ligand-specific allosteric signaling-predicting activity from structure. This approach provides a framework for understanding the diverse effects of environmental estrogens and for guiding iterative medicinal chemistry efforts to generate improved breast cancer therapies, an approach that can be applied to understanding other ligand-regulated allosteric signaling pathways.