According to the 'use it or lose it' hypothesis, a lack of mentally challenging activities might exacerbate the loss of cognitive function. On this basis, retirement has been suggested to increase the risk of cognitive decline, but evidence from studies with long follow-up is lacking. We tested this hypothesis in a cohort of 3433 civil servants who participated in the Whitehall II Study, including repeated measurements of cognitive functioning up to 14 years before and 14 years after retirement. Piecewise models, centred at the year of retirement, were used to compare trajectories of verbal memory, abstract reasoning, phonemic verbal fluency, and semantic verbal fluency before and after retirement. We found that all domains of cognition declined over time. Declines in verbal memory were 38% faster after retirement compared to before, after taking account of age-related decline. In analyses stratified by employment grade, higher employment grade was protective against verbal memory decline while people were still working, but this 'protective effect' was lost when individuals retired, resulting in a similar rate of decline post-retirement across employment grades. We did not find a significant impact of retirement on the other cognitive domains. In conclusion, these findings are consistent with the hypothesis that retirement accelerates the decline in verbal memory function. This study points to the benefits of cognitively stimulating activities associated with employment that could benefit older people's memory.