Before, I talked about role-playing in terms of mechanics contributing to a game's realism. However, there's more to that issue than simply making things "realistic", per se. The way a game presents itself affects how players think about it, and think about what's happening. For example, the institution of an armor system can affect how players see combat - either blows glancing off armor or the armor absorbing the blow, depending on whether armor is used as evasion or damage reduction. Health and "hit points" play a similar role.
The role-playing process (or how players understand it) can be broken down into a few key questions: "Who am I", "What am I doing", "What can I do", "What's going on", and so on. It's a translation of "the rules" to "the universe": the rules exist to determine what happens, but to really roleplay the characters must understand what's happening to the characters. The more things are thrown in the way of that, the less useful it is to determine how such an event would affect the characters.
This is the kind of stuff that can end up being intimidating to a person new to the hobby. It's one thing to say that they're taking on the role of a character and doing things, but it's another to introduce a huge number of rules and regulations about it. Improv acting is relatively easy, but doing so in an unfamiliar setting is hard, and "playing a game" at the same time is harder.
Character: "Who Am I?"
There are several different concepts of characters as they exist in RPGs, but there's two major concepts I think can be identified. The first of these is the "iconic" character, which is usually associated with a class-based system. In this setup, the class defines the character - the brave warrior, the cunning thief, the pious cleric, the scholarly wizard. That's not just for fantasy, either; every genre has a standard set of archetypes, whether it's western or pirates or spies or sci-fi. There's always some character concepts that people can quickly go "oh, right, he's an x" or "yeah, I remember, she's a y, like in that movie".
The more realistic setup is an "experience" setup, where a character's experience and background informs their skills. This is the system used by Warhammer Fantasy Roleplay and Traveller, for example. A character's background, home, and career affect their skills: a blacksmith would know how to mend armor and weapons, a sailor would know how to crew a ship, and so on. The buildup here is based on a logical understanding of the background: "This character has x experience, so they can do y". The issue is generally that this leaves less room for "fantastic" aspects like character abilities. These things are easily justified by archetypes, but don't always make sense within the realm of a logical human-like character.
Problems can arise when the former is used to try and connect to the latter. Archetypes are not all-encompassing, and neither, for that matter, is "character alignment" in D&D. They serve very limited narrative functions, because in a logical setup "this class gets x class abilities" does not make sense. Therefore, attempting to explain normal people's logical backgrounds as classes does not make sense either, because it's an abstraction for "heroic fantasy" purposes. An iconic character requires suspension of disbelief in favor of genre consistency. This makes it useful for quick identification, but less so for in-depth analysis. The same is true of alignment: it's easy to go "Chaotic Good = Robin Hood" but it's harder to actually look at the components of it under a stringent light.
The difference can be summed up thusly: One's easier to describe, and the other's built up logically but has less room for abstraction.
Iconic character description: "He's a barbarian - you know, like Conan. He's strong and tough and also has a bunch of related abilities like tracking, but he hates magic and doesn't like wearing armor. He can do berserker rage, like most barbarian types. He's chaotic good, so he hates authority but still wants to help people."
Experience-based character description: "Your character grew up in the wilds, so he knows how to hunt and find food. He's physically strong and has worked as a sellsword, so he's got some combat experience. The culture he comes from prefers direct combat to 'cowardice' and values personal freedoms above all, so that would affect his beliefs (but not wholly determine them)."
Interaction: "What Can I Do?"
If you asked a player (without setting up any rules or statistics) what they would do in a situation, how do you think they would respond? Say, for example, that you were in a room with three levers and a skeleton advancing on you. What would you say? Would I need to say what system it is for you to say something like "I pull one of the levers" or "I attack the skeleton"? What factors would you want to know more about to make your decision?
This is the realm of logic and meta-logic. The player's understanding of what they can do is going to be a combination of the tools they have available and the targets they can use them on. In an improvisational scenario, this is very clear cut: "I have x, I will attempt to use it on y". Games, on the other hand, up the ante by including a great deal more tools in the form of character abilities. Furthermore, they change the scenario such that using "real-life" logic no longer applies, because the world temporarily becomes turn-based, or ignores something that would simply make sense.
A new player should be given a very simple start, with a limited inventory, spell list, or whatever. That way, they can work out the application of those items to their situation, in the same way that they might do so in Zelda or an adventure game: "The way is dark, but I have a lantern", "There's a thing stuck on a high shelf, I'll knock it over with a push spell", and so on. It's a pretty simple concept: you want to give them things that connect to uses, but you don't want to give them so much stuff that they don't know where to start.
Combat is a bit more trouble. It's hard to represent combat "believably" because of the necessary abstractions. Still, some concepts are simple enough, depending on how they're presented: get hit with sword, lower health. The question becomes "how do I visualize what's going on", and "how do I deal with combat options other than 'I hit them with my sword'?" This is sort of a strange hurdle to overcome, because logical actions don't necessarily make sense in a turn-based setting: "The ogre strikes at you" "I block with my shield!" "You can't, it's not your turn", or something along those lines. In essence, the game should exist in a way that the universe being represented (in real time) is made as close to game logic as possible.
Setting: "Where Are We?"
The way a setting is represented is key in a tabletop RPG. While the GM is capable of describing things, the ability to draw upon existing imagery cannot be underestimated. Therefore, a GM should try to find a setting that the players are familiar with. If they're fans of Westerns, then they'll know all the classic Western concepts (the creaky old saloon, the hard-bitten prarie, the local sheriff's office) pretty well. If they're thrown into an unfamiliar setting, they won't have those automatic images to rely upon. The more players know about a setting, the more they can automatically fit into it. If they don't know about the setting, then they'll miss details and have to spend time asking about things their characters ought to know about.
Take Star Wars or Star Trek, for example. A person who doesn't know about the aliens in those series would require an explanation of them every time they came across a new species, and for reasons of brevity that explanation would be lacking. If a fan of the series saw an Ithorian or a Sullustan or an Andorian, they already know what that is. Existing imagery is drawn upon in a way that helps further their understanding of the setting. Fans of Star Wars know the distinctive noises of blasters, lightsabers, and TIE fighters. They're familiar concepts that are automatically drawn upon when they encounter one in the game.
A tabletop RPG can be a great way to introduce people to a concept by getting them involved and really helping them feel immersed in the setting. However, that feeling will be augmented if they have existing ideas to work with. Games work with almost no senses - everything has to be described, rather than shown directly. Most media works with two senses (sight and sound), and thus has an automatic advantage in terms of connecting to their audiences. Even the use of prose and narrative in books can help create a lasting image more easily than a beleaguered GM can.
In short, either find a setting players can relate to, or encourage players to look at media outside the game. If you're running a campy spy-fiction game, have them watch old James Bond movies. If you're running an action-packed modern game, have them watch Schwarzenegger and Stallone. Most gamers will have a thematic soundtrack to play during games for this very purpose. Reinforcing concepts of tone is as important an issue as anything else in the game when it comes to sensory connections.
Roleplay: "How Should I Act?"
The concept of roleplaying is pretty diverse. It can range from simply "driving" a character (i.e. using an avatar) to actually attempting to "be" a character (first-person dialog, making choices based on their motivations). This is going to connect to the previous three concepts, because those are going to influence how the player thinks about their character.
Just as there are archetypes for characters, there are also archetypes for players. Some players, for example, enjoy the social aspects of role-playing games, and thus will focus on that part. Some players may enjoy making mechanical decisions using in-universe logic, but will shy away from actually "acting" like their character. Some players may prefer the "game" aspects to the "role-playing" aspects.
Whatever their goals, a player's expectations should be established from the start. The important part is that they're interacting with the world in one way or another. A player who's shy of "talking like their character", for example, can easily convey similar concepts in third person ("My character does x", "my character says y"). This can convey the same information while being less "intimidating".
Players shouldn't feel obligated to play characters that they don't want to - and by that I don't mean in terms of classes and combat roles. A character's motivation should match the player's motivation: if a player is out to get gold and indulge in hedonism, they shouldn't be playing a paladin. If they want to do that, they can (that's roleplaying, after all), but it seems odd to create a divide between "the character's motivation" and "the player's motivation" when the player's motivation is what's actually reflected in the character.
In essence, things like "class" and "perspective" should be considered carefully rather than being prescriptivist or unrelated to the player's choices in the first place. Not every player is "good", and they shouldn't write down "good" on their character sheets if they don't want to play like that. It's perfectly reasonable to play a band of self-indulgent folks who want to get more money as long as there's no illusions about being noble heroes out to save the world.
When a new player is being introduced to a game, they tend to deal with things in very simple ways. This is because that's how hypothetical scenarios are generally treated: based on the use of real-life logic. It's up to the players whether the goal of a game is to create new logic or use existing logic, but by default most people are going to assume that a scenario is something like this:
"Say you're a Sheriff in some border town. Some punks just came into your local bar and they're tearing up the place. What do you do?" "Well, I'd kick their asses and throw them out." "Okay, let's go into the combat rules. What are you going to do first?" "I'll punch one in the face." and so on.
The rules in that scenario would exist to determine whether they managed to pull it off, and the scenario would hence develop from that point. The rules exist to fairly moderate events. However, in a rules-heavy situation, the expected answer would be something more like "I'll use my class ability to do something about them", which is less intuitive of an answer unless you are dealing with a situation where those abilities are understood. A person playing a Jedi, for example, would probably know about the hand-waving hypnosis thing. They would not necessarily know about abilities exclusive to the campaign.
So, To Sum Up:
1) Most players are going to go into a tabletop scenario with the understanding that it's like improv with rules.
2) Being able to draw upon existing concepts of logic and expectations, whether real or fictional, can greatly help the players to understand their role and choices.
3) Every aspect of a character's existence should be considered and related to the player's understanding of the game, and a tone should be established.
4) Basically, imagine yourself trying to play the game with a person who doesn't know the rules. Imagine explaining their class to them, the options available to them, the setting, their motivations, and so on. See how easy it is to translate their "uneducated" choices into the game's rules.