Nonhuman primates, and great apes in particular, possess a variety of cognitive abilities thought to underlie human brain and cognitive evolution, most notably, the manufacture and use of tools. In a relatively large sample (N = 226) of captive chimpanzees (Pan troglodytes) for whom pedigrees are well known, the overarching aim of the current study was to investigate the source of heritable variation in brain structure underlying tool use skills. Specifically, using source-based morphometry (SBM), a multivariate analysis of naturally occurring patterns of covariation in gray matter across the brain, we investigated (1) the genetic contributions to variation in SBM components, (2) sex and age effects for each component, and (3) phenotypic and genetic associations between SBM components and tool use skill. Results revealed important sex- and age-related differences across largely heritable SBM components and associations between structural covariation and tool use skill. Further, shared genetic mechanisms appear to account for a heritable link between variation in both the capacity to use tools and variation in morphology of the superior limb of the superior temporal sulcus and adjacent parietal cortex. Findings represent the first evidence of heritability of structural covariation in gray matter among nonhuman primates.