I just ran through a skill challenge in a play-by-post game I'm DMing. I picked the primary skills with consideration as to what would make sense to the challenge and with any eye toward its simplicity (of design, not of the PCs accomplishing the challenge), rather than what skills the PCs were good at. I picked a Complexity of 3, which I thought reflected the lowercase-c-complexity of the actual situation, i.e. convincing three ghosts that they are ghosts and need to pass back through a shadow crossing into the Shadowfell. I used moderate DCs for their level. I discouraged the use of Take 10 and Aid Another for what I think were valid reasons, and imposed another -2 penalty on all checks due to another event taking place. They went ahead and used skills they weren't trained in.
And they failed. They would have failed even without the -2 penalty or if the complexity had been 2 (though maybe not if both).
I'm still a bit shocked by this. I'm used to PCs powering easily through any problem they face with the loss of nothing more than some daily powers and some healing surges. I'm feeling like I was too hard on them. I didn't make it clear that they were in a skill challenge, for one thing, so perhaps they didn't know they would run out of chances. There are things I could have done differently. As the shock fades, though, I'm seeing that this is a key strength of having a framework for skill-based obstacles: taking the decision on success or failure out of the hands of the DM.
I don't mean that the dice should be the only determiner of where the story goes. What I mean is that sometimes defeat is the more interesting (or at least equally interesting) outcome to a scenario, but that it's hard for me to decide that my PCs merit defeat and to stick with that. You can see that even with the rules in place, I'm still second-guessing myself, so there's no way I could just railroad them into defeat even if it was far more interesting to me and the story. If I wasn't "required" to give them only three failures, things might have gone very differently.
For instance, the second roll of the challenge was a natural 20 for a total of 30 in Diplomacy. I know that there are no rules for special effects on a skill roll of 20, but I like to reward such a high roll. If this had not been a skill challenge, I would definitely have felt that such a high roll deserved at least a partial victory, if it didn't resolve the situation then and there. The skill challenge framework kept me grounded. It forced me to keep in mind that as high as that check was, it represented only a part of the solution to the scenario. In my reply to the player I did my best to respond to reward the player's roll with more hints about what was really going on with the ghosts, but the check remained only an eighth of the necessary effort.
Part of what makes all this work, is having a defeat condition that is interesting and keeps the game moving. If defeat meant that the PCs would not get what I considered a vital piece of information, or the death of an important NPC, or that they themselves would be killed, then I might not let the skill challenge framework overrule me and my story. As it was, I entered into the skill challenge what I think (and what I hope the PCs will come to think) were two interesting possibilities and very little attachment to either one.
I've witnessed games in which the DM/GM/Keeper/referee had a firmly vested stake in the PCs success at an activity. The PCs would fail roll upon roll and the referee would keep offering them chances until they finally succeeded. I could tell that this was because the scene was set up so that failure would grind the game to a halt or take it in a boring direction. The trick, I now realize, is to make sure (at least to one's self, if not also to the players) that there are no boring directions, that as long as they make an honest effort they might "lose" but that the game will still be engaging.
Whether or not you use the exact guidelines in the DMGs, if you need a way to feel okay about potentially handing a loss to your players use some sort of a skill challenge mechanic that limits the number of chances they have. Then, to ease any concern about trying to use the mechanics of the challenge to give the PCs a precise and tailored percentage chance of success, decide on victory/defeat conditions that are interesting and move the game along no matter what the outcome of the challenge.
Here's a link to the discussion in the community.wizards.com/skill_challenges_4... group in which I discuss the skill challenge in more detail: community.wizards.com/skill_challenges_4...