Muscle doesn't "turn into" fat, except in the sense that as the muscle atrophies, the protein comprising the muscle fibers is broken down and returns to circulation, where it's metabolized either to burn energy or stored as general body fat, but that doesn't happen locally (i.e. the muscle doesn't -literally- turn into fat -in place-).
You lose muscle cross-sectional area as it atrophies due to lack of use, AND you gain body fat as your calorie intake rises above your calorie expenditure.
There's a false-correlational element to the idea that "people who were fit when they were younger get fit easier when they're older." That is, people who were fit when they were younger are more likely to be genetically predisposed to being "fit" in general, which means that when they're older, they're still genetically predisposed to being fit, or at least having it easier when it comes to becoming fit again. It has nothing to do with their actual fitness levels when they were younger. If they happened to be complete slugs in their 20s, but had the genetic material to be superstar athletes, when they start working out in their 30s and 40s, they're going to figure out real quick that they had that potential.
By the same token, if you've always found fitness difficult to maintain, it probably will always be difficult for you. Personally, I put about 10 hours a week (lifting and running) just to perform at 'meh.'
You don't maintain things like muscle cross-sectional area, motor unit-fiber recruitment, and mitochondrial density for very long after you stop working out. Fortunately, research has shown that you really only need to work the muscle about once a week in order to simply maintain strength. Not sure about mitochondrial density for endurance, though.
One thing, though...if you are just starting to lift, run, or whatever...you're going to be HELLA SORE for 24-72 hours after the first few times you do it. Science can't really explain this (delayed-onset muscle soreness) yet, but it kind of stops happening after a few weeks. Don't be discouraged by it. You aren't broken. Your physical body is just lazy as hell and tries to avoid doing work at all costs, because evolutionarily it's adapted to famine, not overabundance.