Obesity is associated with adipose inflammation, defined by macrophages encircling dead adipocytes, as well as extracellular matrix (ECM) remodeling and increased risk of breast cancer. Whether ECM affects macrophage phenotype in obesity is uncertain. A better understanding of this relationship could be strategically important to reduce cancer risk or improve outcomes in the obese. Using clinical samples, computational approaches, and in vitro decellularized ECM models, this study quantified the relative abundance of pro-inflammatory (M1) and anti-inflammatory (M2) macrophages in human breast adipose tissue, determined molecular similarities between obesity and tumor-associated macrophages, and assessed the regulatory effect of obese versus lean ECM on macrophage phenotype. Our results suggest that breast adipose tissue contains more M2- than M1-biased macrophages across all body mass index categories. Obesity further increased M2-biased macrophages but did not affect M1-biased macrophage density. Gene Set Enrichment Analysis suggested that breast tissue macrophages from obese versus lean women are more similar to tumor-associated macrophages. These changes positively correlated with adipose tissue interstitial fibrosis, and in vitro experiments indicated that obese ECM directly stimulates M2-biased macrophage functions. However, mammographic density cannot be used as a clinical indicator of these changes. Collectively, these data suggest that obesity-associated interstitial fibrosis promotes a macrophage phenotype similar to tumor-associated macrophages, which may contribute to the link between obesity and breast cancer.