NPC's are the populace of your game. They are what give the player a reason to fight and they are what give the player a purpose in the game. Without NPC's the story line would have to be developed through cut-scenes and this would detract from game-play. NPC's usually give the character advice, and of late have been providing opinions of other characters. Another use of NPC's is as secondary party. You can have NPC's accompany you to aid your cause. The NPC is an important, yet often overlooked quality in an RPG.
Monsters, are they the stupid NPC's?
Monsters are your general enemy. Nobody knows why you fight, but you do. Monsters are usually looked at as stupid NPC's, but they are often smarter. NPC's, in the majority of games that I have seen, tend to stand around and all they really do is talk. Monsters have AI that allows them to hunt the player down. So far, there has been disappointing Monster AI for RPG's. If you think about the quality of Bots as opponents in First Person Shooters (FPS) then you can ask yourself why you don't get that kind of quality in Tile Based, Isometric RPG's. On the whole, the rendering time for an RPG is greatly smaller than that of a FPS, but still the AI is better in those shooting games. One thing I think that should be used in monster AI is battle calls. If a monster sees you then they are likely to make a battle cry and charge you. This would alert those around it to the danger and the player would have a much more interesting battle. With the current system, it is far too easy to rope one creature in at a time for a nice even match up. Another thing, if a creature gets too injured, then it should make an attempt to escape. I have seen this done poorly in some games.
NPC's again: Why should I wait for you?
It has been a tried and true method of setting up a sequence of events to have NPC's scattered everywhere who tell you to "Go see <NPC X>" to get quests or advice. These NPC's are always in the same place and never move from them, at least not so as to make a difference. No NPC is going to be waiting around for a stranger. If the player expects any favours from NPC's then that player is also going to have to do favours for NPC's. This would allow the player to befriend NPC's and also become enemies with NPC's. The more an NPC trusts the player, the more likely that NPC will want to talk about certain things with that player. This could allow for a truly well known character to be an attraction for NPCs (well, if that character was well known for good reasons) and if the character was infamous, the NPCs might reject the character and ban them from town.
Furthering NPC's: Emotive response
NPC's have been saying the same old things for eternity. "Do this quest for me…" or "Do this one and I will give you…" but that is pretty much all that NPC's have been used for. In step with what was mentioned in "NPC's again: Why should I wait for you?" the player should have the ability to befriend or begrudge an NPC. This would allow for emotions in the game. To get a player attached to the game, a certain involvement in the game needs to be created. I think finding an NPC that the player feels comfortable with and attracted to can achieve this. By recording visits and topics that were discussed, the game can actually figure out whom the player has an interest in. Flirting was a method that was discussed on GameDev.Net to get the player to visit that NPC more. If the player keeps coming back to flirt with the same NPC then you can make that NPC ask the player to do different things. This would make a fine experiment whether or not it would ever be implemented in a game. If a player gains an affinity with an NPC, you can use that NPC to further the story by such methods as kidnapping or killing that NPC, therefore making the player exact out revenge.
Conversing with NPC's, "click here for this quest"
The method that is currently used to communicate with NPC's is all well and good, but it tends to give away the story too. The player would have more fun if they were actually required to remember exactly what quest they were on when they needed to ask about it. I am not talking about drastically changing this, I am merely stating that if you display a topic in a menu, it gives away the surprise. A simple modification where an NPC says, "What do you want to talk about?" and the player has to type into a subject box a keyword for what they want to talk about. This would allow a lot of versatility in subject matter, and would allow for expansion to the game without having to patch the conversations.
NPC speech and listening in on conversations
(Thanks to 'The Alchemist'). Not all NPC's are likely to want to speak to your character. This means that the Mayor of the town may be unlikely to talk to you about the dragon's lair because you are a stranger. This means that a new method of communication may be required. The idea that was put forward was that the Mayor may be unlikely to tell you about the Dragon's lair, but is likely to tell someone about it. This means that the player can overhear conversations and gather information like that. I am not sure how this would be implemented, but It certainly would add a new element into the game. There are also prospects for gossip behind people's backs, so you may find out what people REALLY think about your character.
NPC and monster Advancement, allowing your party to improve
Too many times I see the powermaxers getting stronger and the NPC's getting weaker. There needs to be a balance. NPC's should be able to gain levels as well, and monsters should be able to increase their effectiveness to be a worthier opponent to the player. There needs to be some increase for the monsters, to be able to keep even with the character and make the game more challenging. No more shall the monsters be simple to kill. There should be no genocide! Accompanying NPC's should also increase levels so that they become more effective in alliance with the player.
Learning 'goblinish', breaking the language barrier
(Thanks to Taharez and Staffan). Instead of just assuming that goblins and humans both speak the same language or both are unable to understand one another; you could have a different language specified for the goblins and allow the human player to learn it. This is quite a difficult task that is somewhere in the realm of Tolkien's efforts into creating his world. I think that this would really be useful if your game could get a following because you would have a similar novelty such as Star Trek's Klingon language. This could also be a quest for the player. The player comes across a goblin and the goblin and human start trying to communicate to each other. Your game may have lessons on different things and you may also be able to 'teach' the goblin your language (make it easier for the goblin to tell you what it means). Just a thought about bringing the player into the game really, but it is not just about goblins in particular. This language idea holds for all races.
Recruiting a party or being tutored on your journey
In games like Diablo II a teacher (Cain, the elder from the preceding original game) accompanied you, but you never travelled. By this I mean that your character did not trek all over the countryside accomplishing his/her goal. What I would like to see is for your character to be accompanied by a party. This was already part of the game, but I would like to also see a wider variety of characters following you. Tutors, fighters, builders – whatever suits your game. It would make quite a party scene if you could wander in with a retinue and attract some attention in town.
The bard is a member of the ancient times that was held in great esteem. One who could make music and sing of greatness is one that is welcomed wherever they go (provided they stick by the rules). As such, if the player become accompanied by a bard they are more likely themselves to be accepted in a town. People also could learn of the player's great deeds and so news could also travel to other towns of the great hero and of his/her deeds. The bard could also be a means of saving a game. Once the player reaches the town, the bard has saved that character's reputation as it was in that town at that particular time. This would be an interesting alternative to saved games and save spots as they are in current.
Imagine that you are in a battle that you cannot run from, and you are running low on health potions. What do you do? Do you take the simple teleportation/portal approach in your game to allow people to simply materialize in town to get the required help? This would go against the very essence of magic and would make it a common occurrence. It has been suggested that you could designate a member of your party to go to town to get the required help for you. This idea means that you can send any member of your party on an errand that you see fit and they will go and carry it out as best they can. This would require a fair bit of Artificial Intelligence, but would add to the game rather than detract from it as I see it. This is something that I think we can all look forward to in future CRPGs.
As Solthar suggested, you could make a world where there were no NPC's to interact with. Something that should first be mentioned though is that NPC's are almost non-existent in RPG's as it is. The problem stems from a lack of AI for NPC's and from scripted speech. Anyway, before I begin ranting about NPC's I should get back to the topic. A world without NPC's is quite a feasible idea, and I think it could be done quite well. A game that reminds me of an NPC-less world, though it could have been done better, is Myst. Myst did really well about telling its story without NPC's. It used books and notes to pass on information, and I think that such things could really make an intriguing game. If you could wander around an empty world and were required to figure out what happened to the inhabitants then this would make a truly interesting plot that I do not think has been visited in any RPG-like games. Another game that comes to mind when thinking about this sort of plot is Ecco the Dolphin, in which the seas of his home were stripped of all life and he was required to find out what happened to his family. This is an element that I do not think has been visited enough in games to date.
It may seem strange that I am campaigning for games without NPC's when I have so often supported NPC advancement, but the truth is that without NPC's you can have a non-interactive medium in which to tell the story, instead of a poorly done interactive (if that) one. It is because NPC's are not interactive that I say they should either be advanced or removed. If they stay as they are now then they will detract from the game, whereas if you can work a lack of NPC's into a story then it will enhance the gaming experience. It all comes back to consistency, immersion and reasoning.
(New) Multiplayer games and the NPC's responsibility
For multiplayer games, my stance on NPC's somewhat varies. Instead of NPC's as the smart ones with which the entire interaction takes place, you have stupid NPC's that fill up space and are there mainly to entertain the Players. These NPC's have to sacrifice everything, even their lives, to entertain the player. The way that I see it is that these NPC's are not terribly intensive in their AI cycles, and as such should not really be classed as NPC's. Seeing as they are there to please the players during times of inactivity (meaning, times of the day when the world is less populated by Players) they should keep the player entertained. This is why NPC's must freely give their lives to save the hours of boredom from creeping up on the player. Seeing as games are mainly there to occupy time for players, as long as they are happy in killing the NPC's, they are enjoying your game.
There are alternatives to just "turning a blind eye" to the killing. You could add in plot elements of outlawing them. This would not be a bad thing as such, as it could give the player a bit of a side-story from whatever they were currently seeking to do in your world. For example, by killing Shopkeeper Bob the player-character gets run out of town by the guards and you flee into the woods (because you have nowhere else to run really). In the woods you could join an outlaw band and this could lead to other adventures.
Don't punish the player for boredom; just give them an alternative. If the player is bored then it is your fault as a game designer not to have given them other opportunities to explore, or given them the knowledge that the opportunities exist. If they act out in boredom then you should give them something else to do. Boredom does not condone the killing of PC's in any way. Player killing is frowned upon no matter what.