We all know the economic crisis and blame it when we hear someone isn't as profitable as before. But people spend more time at home, as they find it hard to finance going out, and might actually spend more on their hobbies. Some gaming related products bacame more profitable with this trend, but roleplaying games as a whole lost some of the playerbase. And this isn't specific to D&D. It is about a trend common to RPGs.
The Big Picture
What caused this loss of playerbase? I would attribute it to 3 main reasons. There might be other reasons, these reasons might be not the most important ones, but lets see how it looks like. Some of it will be "common sense" and some of it will be unavoidable facts, but they can help to understand more about the picture.
New games, new hobbies, new fun
We all see how video games like Diablo, World of Warcraft, Star Wars: The Old Republic, etc. some would say lets learn from them. But roleplaying games lost customers to other hobbies as well. If someone loves to travel using low-cost airlines and plans a trip for each weekend instead of playing D&D at same time, should we copy low-cost airlines too?
People are different. In 60s young people had their best chances for fun either by listening to music, reading books, playing wargemes, enjoying sports or treks to wilderness. Later video, roleplaying games, etc. led to a more diverse market and stole some of their customers. Now video games, low-cost airlines and their trips, etc. take away more people. It is nature of life and nature of free markets.
A lot of younger people hate reading. It is hard to sell books to them. Look at your shelf with D&D products on it. what do you see? Books. People who hate reading is already intimidated. They won't buy D&D. They say killing monsters is easier to do in Diablo II. And no matter how you streamline your game to make it easier and simpler for them, they will see the rulebooks and they will know they hate reading and they will leave. As their numbers grow it will be harder and harder to sell RPGs.
Target audience / playing styles
I know a lot of you love tactical gaming. But I am sure if I would want tactical gaming, I would choose a tactical wargame or a tactical video game (even an RTS) would work well. Even if people love some tactical aspects in RPGs both around the gaming table and in video games, if people want tactical gaming they can get that without "roleplaying aspects". You as existing D&D players might remember some battles, your hard earned victories and want more about that. But advertising D&D with its tactical focus is dangerous at best.
If you want to make people buy RPGs you should build on the strengths of the genree you don't find in other kinds of games or rare in them. If you speak about depth, freedom, complexity, how you can use your creative mind, how it works with stories, some educated people will come. But if you say "tactical gameplay" people will ask: why would I choose it over a tactical wargame.
I won't say D&D 4E isn't D&D anymore because of tactical focus and simplicity. Simply because I know original D&D was called as a tactical wargame, and was just as simple. I would call going back to basics is "discarding all the advancement and evolution of the game". If you want to go back by 10-15 years and you advertise that it is what you get. And going back to old times was the advertised value of both 3E and 4E.
How WOTC (mostly) lost me as a customer
As you might see, I have a D&D Insider subscribtion now, but I don't own any other 4E product, and don't plan to get them. Tried them and they weren't my favorite. While I see how existing customers tend to discuss editions, and create theories how WoTC lost others, that is mostly speculation. I know how some former D&D players bash 4E for pursuing the same declared goal as 3E did and how they claim it isn't D&D anymore. Luckily I tend to give more specific reasons about how WOTC lost me, and I know my reasons well.
If I can't play the "D&D characters" I like, why would I play D&D?
D&D has a somewhat rich literature and it has / had somewhat rich background. It led to some ideas I have about characters. With 2nd edition and maybe Players Option, I felt I can make them happen. But then 3E came. And the game was changed. Most of my character ideas didn't work, people said "rules are here to maintain balance" and yet I seen some insane builds.
So it was an edition where I had to give up some characters because of game balance, yet I had no game balance. Was it a good deal? No it wasn't. And I seen the promised freedom didn't came with creative freedom, but it came with obviously broken designs and insane builds. I decided I wait, I pick up some products, see what happens, maybe with compromises it will work.
Yet, when I seen how WOTC said they plan to go back to 1E dungeon crawl style, how D&D is about killing monsters (yes that quote is from 3E time) I said, if I want hack and slash, I can sit in front of computer. If D&D focuses at "classic dungeon crawl", and I can't play characters inspired by D&D lore I won't play.
With advent of 3E lots of stuff changed. WOTC uniformized races by removing race / class restrictions. Sadly it led to strange things. When one race lacked talent for arcane before (but had a resistance to it) their relationship with more magic using races reflected it, and the history of the setting reflected this. When the rules changed the whole structure changed, and with this the game lost of a lot of depth.
Yet people celebrated this streamliving and the ability of "careless" approach of adventure design, where they didn't have to care about how the world works, what characters think about it... They only had to deal with feats and prestige classes.
This uniformity of races came with something extra: Multiclassing that didn't make sense. While people usually spend decades to become mages, at 3rd level it was possible to see a "Wizard / Cleric / Psionicist" multiclassed character still at his early 20s. It was about tactical advantages in builds, and not characters with personalities, background and history.
I said one thing: As long as D&D doesn't focus on creative freedom but gives tactical freedom to create such builds I won't find the game good. If friends say I should play with them I will consider it, but it won't be a game I consider important.
After all if I can't play my beloved D&D characters because these design changes, the changes to the world, when I don't have the creative freedom I enjoy, but I seen tactical builds and tactical challenges designed with them in mind (which isn't interesting) why should I play D&D? Why should I buy D&D?
This is how I decided I won't be a collector.
Of course with open source we seen many products with "even more ultimate powerfull classes and feats" and some of them was kinde incompatible. A good system is like a shared language, standardized with some quality assurance in design. Open Sourcing it destroyed this "shared language, shared basis" approach.
Then 4E came and...
And it celebrated even more streamlining, and with "points of light" concept even Diablo had more depth and creative freedom than most official modules, etc. And I had to say: If I want to kill monsters without seeing any depth to the game, and want a simple story with "Points of Light" I would load a shooter. Preferable on XBOX since PC games are complex and have depth. But I am not that kind of player.
Bought insider to have some basic acces to rules. Yes: You can't share rules online to show players in online groups how the game is good without subscribtion. An average player who would play in chat rooms or over skype first. etc. won't start with spending money, reading much stuff first, if a friend can't show him the game he would leave. Yet I paid, pay and tried.
But even if I said it isn't my style, I said, it is executed well. If WOTC sees the problem with sales, they couldn't point to bugs, or complexity (much less at essentials) so they might learn from it, and reduced lifecycle, reduced income will make them thing whats wrong... So I hoped for a D&D next. And it came.
What would make me return?
If instead of copying simplicity of yet another game (maybe yet another Blizzard game) and implement its hack and slash WOTC would see there are people around who love depth, complexity, creative freedom. Some of them also love to read, learn, etc. and are more mature players.
If they would love that even in many MMOs there are quite a lot of people who choose RP server first and try to roleplay in cities (as long as they don't see how it doesn't work). Please learn that there are people who would love to roleplay their characters. Even in an MMO, even in a CPRG. People who seeks immersion, a complex, lifelike world and a system that tries to follow the world somewhat accurately.
There are people who when they load up the "latest MMO" end up in a Cantina, with an in character and in depth discussion of Sith code and how some Sith can use the Light Side. People who organize such meeting, spend a night with roleplaying, when killing monsters would be possible just out of the door of the cantina, and it would be even more faster, easier and streamlined than what you have in any edition of D&D.
Because there are people who love immersion, and the complexity and attention to detail it involes. And yes, they also enjoy the freedom and try to play it in a "sandbox" way.
Sandbox games with immersion and popular even in video games industry. And RPGs are better at implementing a complex sandbox with plenty of creative freedom than any video game. If you advertise RPGs and D&D with it, you focus on it the game will be solid.
Even if people will use the said creative freedom to implement their tactical playing style or some munchkin way of gaming.
Can we have a good D&D?
Lots of people fear what they don't know and they fear RPGs. No matter what you do for some D&D will be evil. But if the game has depth, and we learn how to research, how to see things from a different angle, we put these skills in good use in our lives, we all win.
D&D should be a game you can advertise to parents and teachers, not a game about killing "monsters" (different intelligent beings) and taking their treasure (robbing the dead) and looting ancient ruins (graverobbing often).
It should be a hobby that makes us love reading, makes us understand probabilities at math, helps us with social skills, makes us more interested in history, that teaches us how to research and investigate (for an adventure).
I know on this forum this won't be popular. People who come here love the tactical streamlined easy D&D, they don't want to try something else. But if they would know Starcraft as their tactical fun, they wouldn't try D&D either.
If you want you can call this game AD&D. And you can also keep the simple one (even Essentials simple) as D&D and see what your players love. I know, you done it before, and advanced won. I know since you made things simple you lost customers. But if you love streamlining, maybe it makes sense to keep "advanced" and "essentials" lines separate. And basic / essentials can stay as is. But if you want to float (swim instead of sink) please consider releasing an advanced game next. And I hope we will get advanced games back not only in D&D, but in other RPGs too. After all many follows your example. If advanced works, maybe we will have advanced Star Wars, advanced Shadowrun, etc. again.