Recent studies have shown reduced physical activity at early stages of the COVID-19 pandemic. However, there is a lack of investigation on longitudinal changes in physical activity beyond lockdowns and stay-at-home orders. Moreover, it is unclear if there is heterogeneity in physical activity growth trajectories. This study aimed to explore longitudinal patterns of physical activity and factors associated with them. Data were from the UCL COVID-19 Social Study. The analytical sample consisted of 35,915 adults in England who were followed up for 22 weeks from 24th March to 23rd August 2020. Data were analysed using growth mixture models. Our analyses identified six classes of growth trajectories, including three stable classes showing little change over time (62.4% in total), two classes showing decreasing physical activity (28.6%), and one class showing increasing physical activity over time (9%). A range of factors were found to be associated the class membership of physical activity trajectories, such as age, gender, education, income, employment status, and health. There is substantial heterogeneity in longitudinal changes in physical activity during the COVID-19 pandemic. However, a substantial proportion of our sample showed persistent physical inactivity or decreasing physical activity. Given the well-established link between physical activity and health, persistent or increased physical inactivity is likely to have both immediate and long-term implications for people’s physical and mental health, as well as general wellbeing. More efforts are needed to promote physical activity during the pandemic and beyond.