Individuals with temporal lobe epilepsy (TLE) may have significant language deficits. Language capabilities may further decline following temporal lobe resections. The language network, comprised of dispersed grey matter regions interconnected with white matter fibres, may be atypical in those with TLE. This review explores the structural changes to the language network and the functional reorganisation of language abilities in TLE. We discuss the importance of detailed reporting of patient's characteristics, such as, left- and right-sided focal epilepsies as well as lesional and non-lesional pathological subtypes. These factors can affect the healthy functioning of grey and/or white matter. Dysfunction of white matter and displacement of grey matter function could impact each other's ability, in turn, producing an interactive effect on typical language organisation and function. Surgical intervention can result in impairment of function if the resection includes parts of this structure-function network that are critical to language. Further, impairment may occur if language function has been reorganized and is included in a resection. Conversely, resection of an epileptogenic zone may be associated with recovery of cortical function and thus improvement in language function. We explore the abnormality of functional regions in a clinically applicable framework and highlight the differences in the underlying language network. Avoidance of language decline following surgical intervention may depend on tailored resections to avoid critical areas of grey matter and their white matter connections. Further work is required to elucidate the plasticity of the language network in TLE and to identify sub-types of language representation, both of which will be useful in planning surgery to spare language function.