The strategy for everyone to find that median is the same, no matter who is running or who is being run. That strategy is experimentation ~ and those who won't employ it are bound to be weak or abusive DMs.
Unfortunately, most of us, the writer here included, were initially trained to think that an adventure design worked like writing a book. We write the whole book, every inch of it diagrammed and sorted, and only when the book is complete do we dare present it for our players to try out. Frankly, this is pretty stupid. We keep talking about how the game is supposed to be a collaboration between the DM and the players, but then we send the DM off to work in total isolation while we wait like an audience of critics waiting to tear pieces out of the DM's solo design.
We do this because this is how modules were designed. Modules have to be designed like this because they are products, in which all the collaboration takes place between the fabricators prior to handing them over to the manufacturer. There is NO collaboration between the maker of modules and the DM, and then no collaboration between the DM and the players. It is passed down the line with the message, "This is as good as it gets; if you don't like it, tough."
DMs then try to rewrite the module a little (still solo), to tailor it for the players, but we still don't have much recognition of the players tailoring the module for themselves. At best, we have this desperate ploy: a very old story describing what most DMs do when they're noobs and don't know better. It goes like this and it's familiar: the DM has nothing planned, or has something planned which doesn't seem very good. The players speculate about what might be up ahead; the DM hears this and thinks, "That's great! I'll do that!"
Is this what I mean by the players collaborating with the DM to make an adventure? No. But we'll come back to this.
We also pre-write adventures because we think this is the only way to make them complicated and detailed, as in what happens with room #4 is very important when the party gets to #7, which has the key they need for room #9, and so on. We can call this The Myst Method. The Myst Method sucks. DMs don't realize this, but they're very tiresome to play over and over and, in the end, the players will soon become savvy enough to recognize what's happening. The key in #4 might as well have a tag on it that says, "This is very important, test it out in every hole you come to going forward until it fits in something. Thank you, the Dungeon Master."
It's a bigger surprise if the key doesn't fit anything. Except that we're still going to spend an hour of game time trying the key a thousand times.
Finally, we pre-write adventures because we don't understand design. We think this is proper and correct. We think this is how stuff gets designed. It used to be, but now every company that still designs that way has been destroyed in the market place by competitors who employ what's called "agile design."
Let's not get into the particulars ~ there is plenty of stuff online to read if you're not familiar with the concept. You may realize the last post was about agile design. At present, I want to make a point about what agile design is supposed to accomplish.
Writing the module in one go is called "waterfall design" ~ the old way of doing things. Basically, the river goes along, goes along, goes along, then it falls off the cliff and there we are. All designers used to do this; there were no focus groups brought in to question the design, no testing of the product (and no requirement to test it) and sometimes, no assurance that the product would do what it's supposed to do. Movies and other artworks, for the most part, are still being made this way. That is why so many of them suck.
Agile design is what's called "iterative." This means that it is done frequently and repetitively, until what is desired is achieved. Agile was invented for the software industry, which absolutely died on the vine when attempts were made to design it in a waterfall fashion. Now, everything is released in bits in pieces, testing it and testing it, relying very heavily on the user to fix what's wrong and improve the experience. Video game designers release an alpha version of the game, sometimes updated once, then a beta version. The beta version is then repeatedly released as it is tweaked and fixed, relying on players to identify what must be fixed. Finally, all the problems have been supposedly identified and the game is released.
Games are much more complex now, with fewer bugs, because of agile design. Back in the day, when waterfall design was normal, programs were not only crippled by bugs (rather than a few annoying things, like now, it used to be impossible to use software without continuous work-arounds for bugs), there was nothing the company could do about it. There was no proper effort to account for what had to be changed; this meant that "fixing" a program meant bringing in a completely unfamiliar team, who would have to start on the code from scratch. It was easier to burn the code to the ground and start again.
When designing your adventure, you want to be agile about it. But this will immediately produce the response, "How am I supposed to keep stuff secret from my players if I'm sharing the design?"
Easy. Don't keep it secret. Have you not noticed how game are played now? Consider titles like Don't Starve, Oxygen Not Included and Subnautica. Knowing what the adventure IS won't change the doubt players have about the die, their ability to overcome known enemies or steadily explore vast realms. RPGs have vastly over-emphasized the importance of fog, as though this is the entire game, when in fact fog is a very small feature.
What is the best part of not knowing? It produces stress from uncertainty. But if you know you're equally matched with an enemy, does that reduce stress? If you know the enemy's intentions or the enemy's abilities, does that detract from the player's level of stress? No. It doesn't. Not in my experience (which I can prove with text from game campaigns, if necessary).
The only time knowledge removes stress is when it tells the players they are certain to win or that they are certain to lose. Once a player is convinced of either, the game is lost.
So come clean. Sit down and say to your NEW players in your NEW campaign, directly, that you're thinking of giving an encounter. Warn them that it is coming. Tell them, before it starts, before their characters see it, what the encounter is going to be. See how they react. Ask them how they are reacting. Test them. Then, after the encounter is run, find out how they felt about it. Ask them, was it too hard? Was it too easy? Don't take their word literally, as they will skew their responses with a desire for survival, but ask! You can't learn anything if you don't ask.
Tell your players that you're planning to put a small, deserted keep on the edge of the nearby forest. Don't wait for them to ask a bartender: just tell them, "You happen to know about this deserted keep." Ask them how big they want it to be. How much treasure. How many monsters they'd be willing to fight to get that much treasure. How many they think they'd have to fight. Make a joke that it would have to be a lot more and see how they respond. Tell them that if they can't handle that many monsters, maybe they're not tough enough for the treasure, and see how they respond. Assure them that there is a keep. Assure them that there is treasure there. What the hell. You think it's some big secret? You think they don't know you're going to put treasure everywhere they go to fight a monster?
Stop hacking off your left arm to spite your right. Stop doing this on your own. Get your players to tell you want they'd like to fight, give them a chance to do it, assess the effect and then start to adjust your plans with the knowledge you've gained. Obviously, you're always going to be the most creative one in the bunch. You have the least to lose and you're getting experience watching them. This isn't rocket science ~ but it is about being forthcoming. You're not going to make a good game by yourself, not yet. Someday, but not yet. You've got to learn first how to design before locking yourself in a room and designing.