I apologize in advance if this question has already been anwsered on this forum before.
It's okay. There's already an army of serial killers to set you to a painful and bloody end. 
I have recently begun DMing for a small group of friends who were interested in playing D&D. I had some experience as a PC and they thought that I could do this since I am myself an aspiring writer. I quickly got pulled into the DMing thing and have found myself doing it pretty much every weekend.
Pulled into? Make sure it's something you want them to do. Nothing worse than DMing for something you would rather not do. Also, I'm going to second what Centauri said earlier. Do not treat DMing the same as writing. The same skills that creative writing requires can serve you well in DMing, but the process and execution are quite different.
The problem I have is with my PC's.
Big shocker.
I have six people altogether, one with a lot of experience, three with a bit of experience, and two with almost none at all. But three exact individuals make it really hard for me to run the campaign properly. To put it in a way that's easier to understand: They troll. A lot.
First, I want to ask if I have permission to use your title line. I like it a lot better than "turd in my punch-bowl." Second, which specific players are the trolls? A lot of times when players troll, they do it because they're bored. Sometimes, this happens with experienced players. Third, consider talking with these trolls out of game, and asking them what they want from your game.
They mostly don't do things that are a threat to them and their companions, but they ask questions like "Can we rape her/him?" when encountering NPC that I describe as attractive,
My solution to this isn't going to work for everyone, but I know exactly how I'd handle this. Let them rape the npc. Then, take the next 5-10 minutes, describing from start to finish, and in gloriously sickening detail, the act of the rape. Don't leave the slightest detail out. See, I like to make my game-world as real as possible in light of the setting.
Now, what you need to do after that, is describe the horrific suicide that the rape victim commits. Then make the family of the rape/suicide victim come after the players responsible. Or the rape victim grows new strength, decides she wants revenge on the people who raped her, and sets out to kill them in brutal and sick ways. See, they seem to be after humor, and like doing that kind of stuff so they can make jokes about it. They won't do it if they get blindsided by this uncomfortable pit in their stomach when they were expecting comedy. I'm also not afraid to run games so dark that they make A Song of Ice and Fire look like The Chronicles of Narnia.
All that said, what I and others have said about checking with the players about what they want can go a long way toward creating a game that everyone can enjoy. But my world reacts realistically to things. Including rape.
they refuse to do certain things just because they know those things are somehow related to the story, they purposely provoke certain events during battle so they can jest about it, etc.
Don't tell a story. Create a world. Provide oppotunities for them to make meaningful choices. Once they create characters they care about, they will have those characters behave accordingly in that world. If nothing else, having no real story will shock them into a different pattern of behavior, if they realize the change at all. I remember the first time I didnt' tell a story, but provided an opportunity for them to do as they wished in a given situation, and didn't have an obvious story. They pulled the session over for 10 minutes, asking me what they should do. I must have said, "Whatever you want," 40 times that night. It was beautiful.
I know they have to be penalized for it, but I don't know what sort of penalty they should get.
They don't. Don't penalize them at all. If they are actually making the game unenjoyable for other people, it needs to be handled out of game. This is where we circle back to talking to them about what they really want from the game.
