tl;dr
Here's my take on the whole thing: roleplaying, balance, etc. are by themselves isolated events; you could have roleplaying without any game system at all, you could have a balanced/imbalanced game system (wargames), etc. However, we are discussing a game system that relies on the following elements:
1. Story
2. Individual characters who are part of the story, being played by humans
3. Preferably one human who is in charge of bringing the story to life
It's these three elements of the game that we have a whole lot of friction on
everything. Some players don't mind playing it simple or giving way to other players, others are just plum shy and don't want to be in the spotlight *at all*, some don't even care for the rules and just want to be immersed in the story, some just focus on the rules and -- at the extreme end -- eschew story, etc.
Here's my question though: what is the purpose of the genre in the first place?
Here's how I look at it: roleplaying games are a means to not just convey a story, but to create a story... the group's story. To date, only tabletop roleplaying games and live action roleplaying have been able to fully allow such an ability, as even computer RPGs can at best create only the DM's story, which is basically either single railroad or multiple railroad, but still railroad. Tabletop RPGs are the primary source of DM frustration and innovation, as all but the most iron-fisted of them would allow and encourage players to think out of the box and tackle the story in a way never before thought by the DM.
So how do you prevent the whole thing from breaking down into a mish-mash of chaos? Through the use of rules. Rules that are, for the DM, only guidelines, which means that no matter what system, the guy moderating the event will always be the guy with the last word.
Now, with the existence of rules, the DM now has some control over what players can and can't do in the game. However, in spite of the rules, there's always a risk of one player -- be it the most creative, the most rules-lawyer-y, the most munchkin, etc. -- being able to overshadow the other participants (even in a rules-free roleplaying session). This is where "balance" comes into play: the guy with the sword not capable of wielding magic the same way as a magician? Limit the magician and boost the guy with the sword! The guy with the dagger not capable of fighting at the same skill as the guy with the sword? Let him get to do stuff the guy with the sword can't hope to do!
Note that as a side effect of the introduction of rules, it becomes easier to discourage two things:
-> reckless behavior
-> creative action
And the sad thing is, these two things are often interchangeable. As a result, any time someone wants to do stuff, it's less of the vague "is it logical?" and more of the specific "is it in the rules?"
Spoiler:
Show
Not to intentionally diss the edition -- as just about every edition past D&D Basic and AD&D would be at fault here -- but 3.X had it worst: while the system was still more of a guide than a harness, the skill system (first introduced in 2E Oriental Adventures as non-weapon proficiencies, if my lore is accurate) basically not only had way too may non-combat skills, but those skills had to share skill points with combat skills. Not only that, but the entire system was effectively built on too much flexibility combined with too much micromanagement (IMHO). The bad thing about it was that creativity without rules was MUCH harder to pull off in 3.X than any other edition... but the interesting thing is that the entire system allowed for a different sort of creativity: the creativity you could associate more with "how do you recreate things using the rules?" than with "how do you recreate things in spite of the rules?".
Star Wars: Saga Edition was an attempt to try new things, taking stuff from D&D 3.5E and then some. Based on the lessons learned from that system, D&D 4E was born. Unfortunately, the whole change in format and stuff PLUS the bad marketing done by WotC damaged beyond belief the system that could have been what we see now as D&D Next.
[ Personally I think D&D 4E is decades ahead of its time, but again that's just me. ]
Back to the topic of balance vs. roleplaying: it's not so much balance vs. roleplaying as
rules vs. roleplaying. People generally don't want to do things "wrong", even in an environment where there really isn't any wrong answer. How often do you have players with low CHA doing the talking? How often do you have 8 STR Wizards leading the charge with mundane daggers and no magical buffs? A negligible statistic would do so, I believe. And that's where the rules and roleplaying experience friction: even if it makes more sense that, let's say a Warrior Guild member speak with members of the Warrior Guild, more often than not it's the guy with either high CHA or "Charm Person" who would do the speaking, simply because the rules (unintentionally) say that he's the better speaker, at least until the DM explicitly states that the Warrior Guild member has situational bonuses that make the Warrior Guild member at least as competent as the Face. Even if, story-wise, it makes great appeal for a wizard lost in his rage against the jester that killed his mentor to pick up a dagger and just punch him with it, it makes better sense rules-wise to use spells, because he's better at spells than at fighting with hand-to-hand attacks.
Vancian spells are the worst part of rules-vs.-roleplaying. Traditionally, it took a lot more effort by both the DM and the system to keep the whole thing in check, for two reasons:
1. Magic is, traditionally, a DM resource (how many times has magic
not been used as a sort of
deus ex machina in novels?)
2. Magic, due to its nature, openly defies DM ruling-based-on-logic (e.g. laws of physics being regularly broken by practitioners of Fly, Teleport, and the like)
In short: "balance" is an adjustment of the rules to allow everyone to contribute in a scenario in their own special way, instead of having everything solved by a couple of spells. Roleplaying has nothing to do with balance, although the leeway provided by the rules for roleplaying
does.
