Probably more of a question of how fit one really needs to be for LD riding. I'm a reasonably active person with overall fitness for my age. Perhaps it's more of a need for some vs others.
I believe this is too personal for anyone to answer. I've seen riders that looked totally out of shape and rode circles around those that looked as fit as one can be, there are so many factors that go into the notion of fitness and of course this silliness that we do to generalize it.
If I were to give advice as someone who works out 4 or more times a week and is reasonably fit, consider what hurts, gets tired or generally doesn't feel good and focus on a routine to improve that as a start.
Start small, maybe general light full body workouts, if you're overweight drop a few pounds through simple diet adjustments and see how you feel.
If you feel better and can ride longer/harder, then you're on a good path, if you feel worse then stop.
Never hurts to consult a doctor before changing or starting any routine and a few hours with a trainer to "learn" the proper way to exercise to avoid injury.
Mental fitness is more important than physical fitness.
I believe both are equally important for very different reasons and both benefit each other.