Background Implementing the current guidelines for leisure-time physical activity (LTPA) provides significant health benefits, especially for middle-aged adults, but it is unclear whether LTPA also translates into cardiovascular health benefits among elderly people. Therefore, we aimed to assess the association of LTPA with the risks of cardiovascular disease (CVD), including coronary heart disease (CHD) and stroke, and all-cause mortality in an elderly population. Methods In this prospective cohort study, 32, 942 participants aged 60 years or older who participated in a health check-up programme in China between 2010 and 2018 were included. We evaluated the morbidity and mortality risks through the Cox regression model, competing risk model and restricted cubic spline model. Results During a median of 6.84 years of follow-up, there were 6, 857 elderly people with incident CVD; a total of 6, 324 deaths occurred due to all causes and 2, 060 deaths occurred due to CVD. Compared with the inactive group, reductions in CVD morbidity and mortality were observed, with hazard ratios (HRs) of 0.89 (95% CI: 0.83-0.96) and 0.81 (95% CI: 0.71-0.92) in the insufficiently active group, 0.86 (95% CI: 0.80-0.92) and 0.79 (95% CI: 0.69-0.90) in the sufficiently active group, and 0.79 (95% CI: 0.70-0.89) and 0.58 (95% CI: 0.45-0.76) in the highly active group, respectively; but no significant reductions were observed in the very highly active group, with HRs of 0.87 (95% CI: 0.71-1.06) and 0.99 (95% CI: 0.70-1.40), respectively. Compared with the inactive group, reductions in all-cause mortality were also observed, with a HR of 0.90 (95% CI: 0.84-0.97) in the insufficiently active group, 0.82 (95% CI: 0.77-0.89) in the sufficiently active group, 0.77 (95% CI: 0.67- 0.87) in the highly active group, and 0.80 (95% CI: 0.64-0.98) in the very highly active group. A restricted cubic spline diagram showed that there was an L-shaped association between LTPA and the risk of all-cause mortality but a U-shaped or reverse J-shaped relationship between LTPA and the risk of CVD morbidity and mortality, especially stroke. In addition, a subgroup analysis showed that elderly population who consistently performed LTPA for ten years or more had a lower risk of morbidity and mortality. Conclusions In an elderly population, even insufficient activity is associated with a decreased risk of all-cause mortality and CVD, and moderate levels of LTPA may be optimal for CVD prevention. In addition, elderly people who consistently perform LTPA over several years may experience greater health benefits.