The best practices in tabletop RPG design focus on making the most of liminality. (See the AD&D post on the sidebar.) This is why a game's rules and mechanics are best when they are few and easily applied to a wide variety of specific circumstances, making it really a rubric for the Game Master to issue rulings in a consistent manner. You want just enough substance to establish a context one can build upon, but stop well before your gameplay experience is more about working mechanics (Mech Piloting) than engaging the situation in terms of natural everyday language.
Take a look at the Basic D&D sets of Moldvay/Cook and Frank Mentzer (BECMI or Rules Cyclopedia) for the lighter end of practical, and AD&D's 1st edition for the heavier end- and then write your technical manual explaining how to use it in clean, simple language. You will see that this is what Wizards of the Coast aimed for with D&D's 5th edition. For all its faults, Palladium Books nailed the liminality aspect; it's part of the reason it's endured so long despite bad leadership and other chronic errors (such as how those books are organized).
What does this come down to in practice?
- Keep It Simple, Stupid: Your prospect should go from "What's this?" to playing their first game, with a character they created from scratch, in five minutes. Every last element in that process should be as simple and brick-to-face obvious as it gets, and the fundamental gameplay experience needs to be just as simple. You're selling virtual experiences, not storytelling bullshit (You will lose to competing media every single time if you try that.), so remember the wargaming roots of tabletop RPGs and focus on engagement with the situation at hand--upon "What Do You Do?"--and taking any urge to do otherwise out behind the woodshed to beat into a pulp and then drown to death in a vat of horse piss.
- Doing, Not Being: The game is about doing shit, not sitting around bullshitting about things that aren't doing shit while pretending to be someone else or engaging in writing room exercises while pretending to be playing a proper RPG. That's one of the elements playing into the continued domination of D&D; you're playing the game to seek adventure and recover treasure, becoming stronger if you succeed. In the course of pursuing objectives, you're going to do a lot of varied things; negotiate here, fight there, explore this place, develop that one, etc.- a swiftly-changing set of activities that otherwise require separate minigames to handle as competing media doesn't do liminality well (if at all), but tabletop RPGs do when made just right.
- Help the Game Master: The man running the game is the critical component. You need to give him all the help that you can so he can competently handle the liminality of the medium, and a lot of that can't be done with mechanics or rules; he needs confidence to make that happen, and confidence comes from acquiring familiarity with a thing. Take the time to talk directly to prospective GMs, and show them how it's done- another reason for you to engage via blog posts, podcasts, and videos/livestreams. Don't just give him a hammer; show him how to use it properly.
- Know Your Limits: In tabletop RPGs, the real limits aren't the rules or mechanics. It's the subject matter that limits the game; know the borders of your subject matter, and respect those limits. Don't sell someone Not Star Wars and then turn around to do Not Resident Evil- something a lot of people who really ought to know better somehow fail to comprehend, always to their detriment. On a related note, know your limits as a designer and publisher; if you can't do something properly, get help. You can work on improving your skillset after the immediate need has passed. (Related: Pay whom you hire well and treat them right. You not only get what you pay for, you are more likely to get more people wanting you to hire them down the road.)
Do this and you'll have something people will want to play, and content they will pay you for so they can play it without making a second hobby out of it. This means we're about to bring it all home. Tomorrow.