As I've said before, a properly-implemented open gaming table or similar system can auto-solve the problem of encounter balance. If you provide a variety of encounters, and a choice for the players of which encounters to engage, the problem of encounter balance solves itself. Your players, if they have any capacity for learning, will fight whatever creatures they feel comfortable fighting. If they would prefer to fight ECL-2 foes and accrue experience slowly but risklessly, they can do that. If they would prefer to fight ECL+2 foes and accrue experience quickly but at great risk to themselves, they can do that, too. This is basically the same principle espoused in the Dungeon Master's Guide, except it involves less need for the GM to be good at reading his players. (It's probably inaccurate and offensive to say that D&D players are less likely to be good at social skills like reading people than your average person, so instead I'll just imply it. Though D&D players are certainly more likely to be good at such skills than, say, pure video gamers.)
So with that problem neatly and tidily solved, what other potential balance issues do we need to think about?
Well, the obvious: intra-party power balance. Consider a party consisting of a wizard played by a skilled player, a druid played by an unskilled player, a fighter played by a skilled player, and a monk played by an unskilled player. It is well accepted that wizards and druids are tier 1 while fighters and monks are tier 5. Which is to say, a well-designed, well-played wizard or druid will always be orders of magnitude more useful in a party than a well-designed, well-played fighter or monk.
Well, it has been my experience that low-optimization parties are better balanced than high-optimization parties. A poorly-designed, poorly-played wizard is probably worse than a poorly-designed, poorly-played fighter. Wizards have so many options that if you make poor spell choices, you can't contribute anything, while fighters can always hit things with sticks. The difference between a good fighter and a bad fighter is smaller than the difference between a fighter with a d20 that tends to roll well and a fighter with a d20 that tends to roll poorly. So that's one option: play only with newbies.
A high-optimization party, similarly, will usually have players who avoid tiers lower than 2 on principle, and will auto-balance. They will also break any encounter you throw at them and ask for more, so you want to avoid this situation unless you're running the aptly-named Tomb of Horrors.
So, returning to the good wizard / bad cleric / good fighter / bad monk example. Here, the cleric, fighter, and monk will all be roughly on par with one another, with the wizard the odd one out. The other three will always feel useless. The wizard may, depending on his player's personality, become tired of constantly being in the spotlight, or constantly needing to carry a disproportionate amount of weight. How do we fix this?
The tier system provides four suggestions:
#1: Alter point buy. The wizard has fewer ability points to spend than the monk. This helps to balance them a little, because the tier 1 classes tend to be strongly Single Ability Dependent and the tier 5 classes tend to be strongly Multiple Ability Dependent. But what if the monk multiclasses into druid for his second and subsequent levels? A one-level dip into monk is probably worth 16 extra points. You can solve this, but the solution will always be messy.
#2: Gestalt. Tier 1 and 2 can't gestalt, tier 3 and 4 may gestalt with tier 6, tier 5 and 6 may gestalt with other tier 5 and 6. This also gets weird when multiclassing comes into the picture. You get bizarre and undescribable things like (fighter 1 || knight 1) / cleric. It's possible, but messy.
#3: Ban the top few tiers. If you ban tier 1 and 2, great, you just banned 90% of the dedicated spellcasters. Hope you're in a low-magic setting!
The above suggestions also penalize the bad cleric just as much as they penalize the good wizard. We don't want a situation where that happens, the crappy cleric needs as much help as he can get. On the other hand, we also don't want to have a situation where we say "You're an optimizer, so you get penalized; You're incompetent, so you get rewarded", if for no other reason than because that's so subjective.
#4: Go through, find the reasons why the tiers 1 are so good, and nerf them. Find the reasons why the tier 5s are so bad, and improve them. Ban Polymorph. Ban Natural Spell. Give monks full BaB. Give fighters a feat every level. This? This is a butt-ton of work.
The above methods are all clunky yet potentially workable. But I like two alternate methods, one also clunky, one more subtle:
#A: You need to take a certain number of levels of low-tier classes before you can take a high-tier class. For example: you can start only with a tier 3, 4, or 5 class. You can't multiclass into tier 2 unless you're already at least third level, and you can't multiclass into tier 1 unless you're already at least sixth level.
You can, of course, apply this with more steps. For example: At first level, you must take a tier 5 or 6 character (so no dedicated spellcasters other than Healer). At level 2, you may take a tier 4, 5, or 6 class. At level 3, you may take a tier 3-6 class. At level 4, you may take a class of any tier except 1. At level 5, you may take any class.
Similarly, you could simply give high tiers a level adjustment. For example, assign Tier 1 LA+2 and Tier 2 LA+1. It might occur to one to give Tier 6 classes a negative level adjustment, though that's tantamount to saying, "taking a level in this class is exactly as good as not taking a level in anything at all" - even Warrior has full BaB and a good Fort save, so everybody can have +1 BaB and +2 Fort for free. This option does not, however, work with multiclassing (a Commoner 19/Wizard 1 would have to have much the same LA as a Wizard 19/Commoner 1, barring liberal application of weird math).
This works because, generally, the best way to improve a spellcasting character is to give him more levels in his spellcasting class. Conversely, the best way to nerf a spellcasting build is to include non-spellcaster levels. Effectively, this puts a spellcaster a certain number of levels behind where he might normally be at any given level. Getting Polymorph at level 7 is a complete game-breaker. Getting Polymorph at, say, level 13 (but still casting with a CL of 7, unless you take the Practiced Spellcaster feat) is much more in line with the sorts of things other characters can do at level 13. A certain kind of gish character might be inclined to take those non-spellcasting levels anyway, and is thereby not penalized any more than they're already penalizing themselves with their choice of build, which is probably fine.
#B: Drastically limit the kinds of items players can buy in stores, and distribute magic items with care. Be generous with things like magical weapons and armor, Belts of Ogre Strength, Monk's Belts, Potions of Bull's Strength, and other things the tier 5s can use better than the tier 1s. Be sparing with scrolls for the wizard or artificer to copy, metamagic rods, Headbands of Intellect, and other things the tier 1s can use better than the tier 5s. This depends on which particular classes you have in the party. If you've got a monk but no cleric or druid, go ahead and give them a Periapt of Wisdom. If you've got a swashbuckler or rogue and no wizard or archivist, go ahead with the Headbands of Intellect.
Similarly, tailor encounters to the strengths of your tier 5s. Golems are good for this, though don't go too overboard with them (the goal is to make the tier 5s as useful as the tier 1s, not to make the tier 1s completely useless forever). If there's a cleric, don't use too many undead, especially if there's also a rogue or a fighter with a +1 Keen Rapier.
There is a more general point here, though: when you make any alterations to or interpretations of any rules, try to make sure you're not making tier 1-2 classes more powerful and you're not making tier 4-5 classes weaker.
I myself have broken this rule of thumb on occasion (engineers make great wizards; the list of scrolls wizards can scribe is longer if you allow them to copy non-Sorcerer/Wizard scrolls; mongrelfolk wizards/sorcerers/druids can get water elementals as familiars/animal companions with two feats). But at least I haven't nerfed monks.