I'm not necessary just talking about the final boss. I'm talking about what gives the players a sense of challenge? A “villain” doesn't necessarily mean “boss”. It could be the setting (the campaign), the player's motivation, the opposition, status, ect. When your players can bend space and time with the powers of magic, what makes something truly a challenge?
Is it the price of failure?
Is it the an enemy that can wipe the floor with you?
Honestly, it's a little of A and B.
If you give your characters POWER, they will become bold and not fear the enemies. If you set the bar low, it's not a challenge. When enemies are just a source of EXP with no chance of failure- it gets dry and the silliness comes out. (“I wonder if I can kill them with only scrolls...” says the Fighter) Often times GMs will just say “Oh... I'll just increase the CR of the fights. That will make it harder.”
Yes... but no at the same time. It will make it HARDER but not MORE CHALLANGING. Toss in a new mechanic (“WTF! These enemies steal healing effects!”) or toss the players into a new sort of situation (“So... we are all ghosts now???? Fuck!”).
Imagine “challenge” as a dark room in a rickety old house and your players are little kids. If you give them electricity and light-switches, the house isn't so scary. Give them a failing candle and watch them jump at every little sound and imagined movement.
Sure! Toss a hellish fight or two at them! Make combat SCARY. Be a bit of a dickish GM! LET them go try to kill the orc's raiding the town. (Sure... they are 8CRs above them....) Make the enemy some indomitable force that looms over them. Have a player get killer. Let the develop a connection to an NPC and then watch him get run threw! That will DRIVE the players towards your plot hooks. Don't rush it!
The next part is the IMMAGINED fear they players feel. The Lovecraft mythos are full of this. When we talk about games like Call of Cthulhu the line always comes up, “It's not about winning. That's impossible- it's about how long you live and what fantastically awesome way you die in.”
Let them feel the despair.
Let them dance on that knife's edge.
Impose that stress.
Let them see their own mortality.
Everything should be an uphill climb while they drag their friend's body with them.
Because after all that.
After all that hell and brimstone.
After all that terrible, visceral fight for survival.... there is hope.
Let them take that final breath deep exhale when it's all said and done.
If they have a short, easy, little walk through the park- it won't amount to anything.
If they look back and see the unimaginable HELL they walked through, they will have a sense of PRIDE in their accomplishment.
From a literary perspective it's about the building action. The trials the heroes overcome are a measure of the triumph they will experience. If someone has to go next door to deliver some cookies- getting that “thanks” won't measure up to the trials of a little disgusting cripple who fought economic adversity, forces of a malevolent god, societal norms, and an abusive father figure to become a duke.
That being said, I think this is a fitting place to talk about the “small explosions” concept. In movies a big explosion is cool. It gets the “action movie” vibe up. However, studies show that small explosions invoke a lot more fear. They are more grounded and realistic. Unless you are going for a “suspension of disbelief” style game/movie/campaign (“high fantasy”), danger should appear REAL. Just look at games like FFX. If your character can summon Ifrit in a hellish blaze of fire and magma- why should a flan pose any real threat?
I touched on it earlier, but there are more things than combat to make your game more challanging. Here are a few:
Your characters don't have a lot of money. That suit of full-plate is just a pipe dream. A magic sword? WOW JESUS CHRIST! HE HAS A FUCKING +1 SWORD! O SHIT O SHIT O SHIT RUN! HE MUST BE A BADASS! When money is a concern, you can also involve social classes. Keep a close eye on rations. Hungry characters bite more plot hooks :D
Make your players bow when nobility passes. Make them have to give taxes to a tyrant. Describe a few social injustices. Let them see people starve. (Works great with the economic challenges!)
Is that really there? Are you IMAGINING the guards are talking about you? Did you have 56 or 57 gold? Why is my sword now a staff? Was it ever a sword? Mental challenges are fun.. but that's one for another day.
Really? WWII and the Inquisition are not enough of an inspiration?
That being said- make your games more challenging. I'm not saying harder. I'm say more challenging :D