Those of us who are ex UK armed services may remember from many years ago that the term fitness referred to being "fit to fight for 24 hours".....
I would say that I am a long way from being where I was in terms of fit to fight physically, mentally is a different matter, but personal fitness to ride for 24 hours will certainly help anyone to remain alert and able to focus on the task in hand.
There are some folk who can certainly ride long distance and remain alert and able to focus, despite not being as fit and probably as young as we used to be, but it doesn't stop us!
Physical "fitness" is not a pre-requisite to go LD riding, but it will certainly put you in a better position to cope!
The main criteria is SAFE long distance motorcycling, knowing when to knock it off, when to stop and rest or when to simply STOP the ride.
You may get tired and suffer a micro sleep, a nod, a moment when your eyes are closed for more than just a blink, that is when you need to STOP, you may not waken up from the second nod.....