Despite infiltrating immune cells having an essential function in human disease and patients' responses to treatments, mechanisms influencing variability in infiltration patterns remain unclear. Here, using bulk RNA-seq data from 46 tissues in the Genotype-Tissue Expression project, we apply cell-type deconvolution algorithms to evaluate the immune landscape across the healthy human body. We discover that 49 of 189 infiltration-related phenotypes are associated with either age or sex (FDR < 0.1). Genetic analyses further show that 31 infiltration-related phenotypes have genome-wide significant associations (iQTLs) (P < 5.0 × 10), with a significant enrichment of same-tissue expression quantitative trait loci in suggested iQTLs (P < 10). Furthermore, we find an association between helper T cell content in thyroid tissue and a COMMD3/DNAJC1 regulatory variant (P = 7.5 × 10), which is associated with thyroiditis in other cohorts. Together, our results identify key factors influencing inter-individual variability of immune infiltration, to provide insights on potential therapeutic targets.