Aliens and non-humans.

One of the most iconic concepts of sci-fi and fantasy is the "not quite human" - the elves, the dwarves, the bumpy-faced Star Trek aliens, and so on. There's a bunch of reasons that these races or species are popular. They're able to basically do all the same things humans do involving "being a biped" and "holding stuff with opposable thumbs" and all that, but they're visually different in a few ways. In short, a humanoid has all the general possibilities of a human combined with an exotic aesthetic. But how do they develop beyond that? How do you properly transition them from "humans with one weird trait" to "a race in their own right"?

One of the first key issues is the issue of diversity. Humans are an amazingly diverse bunch, with different fashions, skin colors, body structures, and so on. Yes, most of their bodies are the same, but by and large humans have a lot of external and internal differences, ranging from biology to behavior to culture. In contrast, the role of a non-human is usually to represent one culture or stereotype. Without that trait, the race as a whole has no point; Vulcans exist to be logical, other than that they're not different enough from humanity to be worth exploring. The need for a recognizable or distinctive trait plagues almost every sci-fi race, most notably Star Wars. 

In fact, almost every race in Star Wars is defined by the first one that shows up in the series: All Rodians are bounty hunters (because of Greedo), all Hutts are gangsters (because of Jabba), all Sullustans are pilots (because of Nien Nunb, the little guy who rode with Lando in ROTJ), all Bothans are spies because many Bothan spies died to bring the rebellion the information about the second Death Star, and so on. To be fair, in terms of their actual introductions there was nothing to suggest this was the case. Greedo, for example, was just an alien who happened to be a bounty hunter. There was no talk about the rest of his species, or whether he was a fair example of one, any more than you'd assume that all humans are bartenders because the guy who ran the Mos Eisley Cantina was.

One interesting counter to this came in the form of "Star Wars Galaxies", the massively multiplayer online game set in the Star Wars universe. The fact that players could choose from a huge number of races to make characters meant that all of these races had to be given customization options, and in addition all of them were capable of choosing a variety of paths in life, from soldiers to architects to dancers to hunters. The design team also had to create clothing and weapons that would suit a wide variety of body shapes. The long and short of it is that SWG basically made these alien races into people - people with diverse appearances and different jobs and outlooks on life - because it was necessary for the player to actually behave like a person in the game. SWG was one of the rare games that actually seemed like a virtual world, rather than a theme park to mess around in. You could build or craft basically anything, from big things like vehicles and buildings down to little things like food, drinks, and clothing. By necessity alone, it made every Star Wars race into a diverse group with some innate advantages, rather than stereotypes with some minor differences.

In general, I felt that Star Trek reflected a similar concept. Yes, the alien races were kind of shoehorned into their major stereotype trait, but that tended to actually be kind of mixed. In one episode of The Original Series, the Klingons - who were still essentially space Mongols, rather than the bumpy-faced barbarians of The Next Generation and beyond - are trapped aboard the Enterprise and forced to fight with the crew for the amusement of a powerful being. The Klingons are aggressive towards the crew, but reveal a lot of humanizing characteristics of their own. They care for their fellows, and fear the Federation as murderers and rapists - the kind of things propaganda would depict. 

Most of the other races are the same way, with the potential exception (explained in-universe, at least) of the Vulcans. Each race has some major trait (Klingon brutality, Romulan treachery, Ferengi greed), but there's also enough difference between them for them to be regular people. One of the key differences between Star Wars and Star Trek is that in Star Trek all these different species are independent (with their own government), while in Star Wars the centralized "galactic government" means that each species can just sort of pigeonhole itself into a single role. In Star Trek, if you go to Romulan space, there are going to be Romulans doing all the jobs, because they're all Romulans there. In Star Wars, you'd either have all humans (if it's the Empire) or everyone sort of filling their racial niche. The one major issue I have with Star Trek is the fact that each race (humans included) is part of a single-species governmental organization. There's alliances between races, like the Federation or the Dominion, but in general it's kind of monolithic.

From this, there are two major rules that can be established:

1: To create a believable species, every reasonable need must be fulfilled. Therefore, there must be people available who can fill all those needs.

2: Unless there is some explanation for similarities, creatures vary depending on their environment and region. This is true for most animals on Earth, so why wouldn't it be true for sentient aliens?

It's easy to identify areas where these rules can clash with the "one key trait" system used by most SF/Fantasy. For example, the Predators (as in, the split-faced ones that fight xenomorphs) were originally introduced as one specific Predator who came to Earth to hunt humans. However, the fact that the Predator was iconic meant that as the universe behind it was developed, the entire culture turned out to be shaped around hunting. Despite this, they have incredibly advanced technology and capabilities. So who, exactly, builds all this? The comics, for example, show the non-hunting Predators as loincloth-clad invalids only useful for cooking dinner with the most basic tools. How, exactly, did they build spaceships? It worked when it was just one hunter, because then you can assume that their culture is like human culture, and this hunter is like a human hunter - the rest of his species does other stuff, but this one particular Predator likes to hunt lesser races for sport. When they all become hunters, the question of "who does all the other stuff" inexorably worms its way to the surface.

The TNG Klingons suffer a similar problem, albeit with a twist. They're brutal warriors favoring muscles, physical prowess, and melee combat - and they have built up a space empire with ships in space. Spacecraft are fairly technical, and while a strong warrior spirit would be important for morale it's not exactly the same. The whole "caveman/barbarian" concept, which can roughly be connected to orcs if you really felt like it, works in a fantasy setting (because a muscular enemy with a club is still a threat even if he's primitive), but doesn't make sense when both sides are firing lasers at each other across trillions of miles.

Fantasy, though, doesn't get off scot-free on this account. However, fantasy in general is based off of mythology - wait, let me amend that. Fantasy in general is based off of Lord of the Rings, which is based off of mythology, so it has that as an excuse. In Germanic mythology, dwarves were considered excellent craftsmen, and were often associated with hills and caves. This led to the LOTR interpretation, where they dug great mountain halls and enjoyed riches and metalworking. Every interpretation of dwarves after that basically used this model; "mining" in general is one of their key features, along with "short" and "bearded". Without mining and tunneling, there's really no point to having dwarves in a setting.

"Dwarf Fortress" managed to simultaneously use and avoid this concept by having dwarves be tunnel-centric, and psychologically connected to precious metals, but also making them have an entire economy and system of life. Even though dwarves like mining and beer and living underground, they also need food, shelter, and so on. Therefore, an entire economy is necessary to keep the fortress running. Every conceivable need is accounted for, but it's all done in a way that's thematic for the race as a whole.

Finally, there's the issue that's been kind of avoided thus far, and that is: what if an alien race isn't humanoid? I mean, frankly, making aliens "humans but with a weird face" is kind of a limited field, since every alien species is going to represent some facet of humanity, just like a racial stereotype. How would you build a truly alien species from the ground up?

To this, I present one of my favorite alien species: the Hivers, from the Sci-Fi RPG "Traveller". While most of Traveller's alien races are kind of boring (cat people, dog people, psychic humans), the Hivers actually represent a pretty detailed "alternate evolution". That is, a Hiver is to a termite what a Human is to an ape. Hiver culture and physiology reflects the necessity of their environments and development. Some of my favorite touches are things like how their culture built up from its roots - their cities are essentially termite mounds on a grander scale (albeit made of improved materials), and their relationship with the Snohl almost resembles the farming of ants and aphids. Furthermore, the fact that they retain affection for the Snohl despite them not being really useful in the same sense anymore is remarkably human - when was the last time you used a dog to hunt foxes, or a cat to keep out mice? 

Even their attachment to children is different - they don't care at all about their larva, but once a child has survived the wilderness and come back to the hive they are welcomed with open arms (or whatever). This is unthinkable to most humans, but it reflects a key biological difference that leads to a cultural one: humans have few children, and thus care for them, while hivers have thousands of children, but relatively few survivors. It's biologically sound, and different enough from humans to be distinct as a separate species rather than just a weird trait on its own.

The Star Control series had a few interesting aliens along these lines too, but they weren't as developed. In general, Star Control aliens were limited to a few encounters because they had to be described and developed quickly. Furthermore, there's really only one "model" per race, so when you encounter a Mycon you're going to get the Mycon conversation tree every time (and so on). This pretty much guarantees that each race is going to have a single major trait, because (a) they're all the same and (b) they need to be distinct from each other. Still, there were some interesting designs, like the crystalline Chmmr, the blobby Umgah, or the gaseous Slylandro. While they might have been simplistic, I also thought that they felt actually alien - as in, a life form that would evolve in an environment dissimilar to our own. They're identifiably different from humans, and should behave as such. You can't substitute them with a normal human, unlike many alien races, because they're so radically different that it's just not the same. 

As a side note, I'd like to state my annoyance with both Mass Effect and Dragon Age for simultaneously using and mocking the "all non-humans are the same" business. It's pretty clear that, in Mass Effect, all Turians are orderly, all Krogan are warriors, and all Salarians are super-intelligent (whether in espionage or research), but there are times in both series when a character who is otherwise a perfectly standard representation of his race tries to pull the "we're more complex than you thought" card. This is a really hypocritical message that creates a schizophrenic moral to the game: on the one hand, it's an indulgent Space Opera / fantasy setting that uses all the standard tropes and cliches. On the other hand, it wants to think it's deep, so it throws in a line about defying your horrible, backwards preconceptions about these brutish, militaristic thugs who unanimously love fighting. Mass Effect 2 was a step forward by showing aliens as regular citizens, doing things that were unquestionably "human" while still being affected by their cultures and biological makeups, but in general it's still pretty embarrassing that it thinks of itself as defying the genre.

So to sum up:

1: A given species should be able to acquire or produce all the things it would need to survive and prosper.

2: A given member of a species should be treated as an individual affected by different forces and experiences, just the same as any human.

3: Maybe once in a while try making an alien that actually seems alien?