The biggest mistake tabletop RPGs ever made was to switch from boxed sets to books. The second was to make those books fat fucking tomes of bullshit and blather. The third was to push that blather into the players' faces. Why? Because each one ended up contributing to a lack of clarity, and when an alternative came along that was crystal clear from the first glance as to what you did and how you did it, gamers bailed.
There are three parts to the clarity matter. The first is clarity of function, so that when you sit down to play you know what you're doing and how you do it. Ryan Dancey explained this as "core story", but it's better called "default play model". For D&D, the model is "You explore dungeons and fight monsters to get the treasure." All of the other enduring classics have something like that, which any Game Master can fall back on whenever things slow down. It's the "What the fuck do I do?" question. The push over time for clearly delineated mechanics, with equally clear delineation of interaction between them, stem from this.
The second function is clarity of form. Gamers don't like wasting time not playing, and they consistent demonstrate by behavior this preference by (a) not reading rules unless they need to and (b) not bothering with lore unless it impacts them immediately during play. Related is the stubborn resistance to changing established setting facts, which matters for those playing in settings they don't own. This is the "I am not doing homework, so fuck you." matter. (For all you folks sick of Pink Slime fantasy, and authors who become glorified fan-fic writers, this is your biggest contributing factor.)
Together, I call this "Cold and Stupid". No preparation, and no prior knowledge. Just sit down, get playing, and learn as you go. In the early days, this was explicitly advertised as the way to break new people into the hobby; you can see why getting away from it was a bad idea now. Yes, this means that part of the solution to ending the tailspin of tabletop RPG commercial viability does mean reducing both your lore and your rules down to the minimum required to actually get on with playing. You should be selling a boxed set with booklets, if you sell that as physical product at all; these days, a wiki and a blog will be plenty sufficient.
(Yes, this means that your business model has to change. That's for another post.)
There is one other aspect to consider: clarity of experience. Gamers want to know what they're getting out of their playtime, and they've demonstrated their hatred of that uncertainty over time with their feet and their wallets. They prefer games that tell them exactly what they're going to get for their time and money, and the enduring classics (see the default model above) endure because the players do have that clear expectation.
Take a look at the alternatives. You don't need to know a goddamn thing to get into Skyrim or Dark Souls. You do know what you're going to get. The asymmetric horror games? Crystal clear as to what you're supposed to do, how you do it, and even what boundary conditions enhance or restrict your options. You know exactly what to expect when you play, and you don't get swerved. You don't need to know the rules to Descent to play; only the guy running the game does, and the rules themselves are in a slim booklet. Lore? You don't need to know a goddamn thing to enjoy any Final Fantasy title, and being so informed doesn't do much (if anything) to enhance the experience. See where I'm going here?
Meanwhile, tabletop RPGs are dumping more and more homework on users with no payoff of substance for doing so. The complexity, actual and perceived, keeps growing and it's not helping. The clarity is getting lost on multiple levels, such that competing media that focuses (see yesterday's post) on a clear subset will bleed off players and they likely won't come back.
So, you need to tell your audience what they can expect from playing your tabletop RPG. They need to be able to sit down and just play it without ever looking at the rules and not worry about fucking it up or getting screwed. (Yes, this means being anti-MechPilot by design.) Clear expression of experience, function, and form. Do that, and you're on the right road.