Back when Kirk was space-dicking around the galaxy the balance of power was very different – there were so many powerful gods mucking about in the 1960s/2250s. Some of them were there for a specific narrative purpose too – they'd act as the deus ex machina that allowed the Enterprise crew to be pulled out of a dangerous situation through moral fortitude alone. To be fair to Trek, the premise very much allowed for gods to swoop in whenever and however.
'Arena' (the one with the Gorn) is the classic case: the tension of the episode is specifically about which captain will kill the other, stakes far higher than just "will Kirk escape". Narratives like that are great for drawing in audiences, but they're often somewhat contrived in setup, and don't offer any opportunity on their own for a meaningful message. Either Kirk kills the Gorn, and we declare victory because the guy we already knew won, or Kirk declares moral victory and doesn't kill the Gorn, leading to either his death or some unsatisfying stalemate. There's a tension between the narrative desire to focus the climax on the moral question and the audience's desire to have material stakes (read: "a punch-up"). What you need is for the moral victory to also cause a material victory, and an easy way to achieve that is to have a god swoop in to evaluate. In this case the twinkiest (I claim) of all known gods arrives to give Kirk his number and drive him home, because that's what awaits the morally virtuous who don't kill Gorns. Fuck yeah.
100 years later in TNG times, those gods have disappeared from the galaxy. TNG pushes on with hopeful morality plays anyway, attempting to craft situations where the right virtuous act at the right time somehow causes material benefit on its own. More than a few times the day is saved by a simple earnest speech. That works for comfort tele (and conservatives) but it rings false to me. Like, if you're feeling sceptical reading this, watch 'Measure of a Man' again and tell me you believe any of that would play out that way today, as if the ridicule of one man in a tiny court would do anything to prevent the abuse of androids outside that court. Hell, TNG itself put to bed any notion that Starfleet had learned anything with the episode 'The Offspring', which aired only a year later. For me, this kind of storytelling makes TNG a bad show.
DS9 had other ideas; in its universe, 'moral victory' and 'material victory' become entirely independent, and our heroes are in pursuit of material victory at almost any cost. You can ask if terrorism is bad, but DS9 is more interested in whether it worked as a tool for liberating an oppressed people, and it took for granted the idea that liberating an oppressed people is good. Whether terrorism is good is... well, your call. Ditto the other controversial actions of the main cast. This is what I think most people really mean when they say it's "darker" – it's not necessarily promoting a morality with which one would disagree, but said morality is explored in a way that takes you to unconventional or uncomfortable places. In truth, it's quite hard to deduce that 'terrorism always bad' unless you've already assumed terrorism always bad, and DS9 wants to explore that difficult space.
What about Starfleet? Hm, tough one. Starfleet has done some questionable stuff, and in this new framing it's time to start questioning. But DS9 has to at least pretend it exists in the same universe as TNG, so... they make up Section 31. Section 31 is a narrative device that allows Starfleet's morally dubious acts in DS9 to be separated from the existing 'Starfleet' concept – that of a shining beacon of moral decency. Starfleet can thus maintain its moral victory while Section 31 "acts in the shadows" to achieve material victory.
From what I can tell, DS9 seems at least tacitly on board with the idea that occasional murder or genocide for the greater good is justifiable. It doesn't go as far as to claim such an act is morally justifiable though, because, of course, Starfleet would never condone genocide. Section 31 will though. So, Starfleet isn't responsible for Section 31, even though Section 31 is part of Starfleet. This is... hot nonsense.
Two things interest me about Section 31. The first is that DS9's writers' room seems very aware that this is a cop-out, but still take advantage of it anyway. Bashir has a line about how many so people in Starfleet must be aware of Section 31, and Odo comments on the 'tidy little arrangement' of pretending Section 31 is a separate body when it obviously isn't. Like, they know this is a cop-out. That Odo line is a fourth wall break. But they draw the line there, at 'Section 31', and so never need to examine ideas like 'hey, why isn't anyone resigning from Starfleet about this? Isn't the whole thing corrupt af?'. Sections 1-30 of Starfleet sure get off light during the run of DS9, and I consider that a failing of the show.
The second is that, outside the universe, Section 31 can be used in exactly the same way. People are able to use 'Section 31' as a shorthand for 'the morally difficult stuff', or even 'the notion that moral and material victory are not the same'. The result is these weird fandom discussions where people talk about whether 'Section 31' is a good or bad concept and they get nowhere because they're really arguing whether Trek ruined itself for them when it applied the 'morally difficult stuff' to Starfleet. Others will talk about how Section 31 is a necessary add-on of any morally good organisation, as though they can't conceive of a universe where you can righteously succeed without having to park morality at the door sometimes, which, no? You can't both claim moral virtue and shirk it "when necessary", right? I hope not. I dread to think what kind of nonsense can be justified with a framework like that.
'Inquisition' dropped the ball and should've just made Sloan a regular Starfleet guy instead of coining 'Section 31', I guess would be the above in a sentence.
Reply from John Colagioia
This is going somewhere, I promise.
Years back, I saw a pitch for DC's Suicide Squad franchise (criminals recruited to work for a secret military project, where they either walk to their deaths or survive to receive a commutation of sentence) that had the clever idea of adding layers to the idea. The original version was just normal human specialists, but the famous version that got movies had costumed villains. This suggested that you actually have three teams in the book. One team is the supervillains-acting-like-superheroes, who are protagonists, but basically decoys in the grand scheme. They exist to attract attention away from a team of plain-clothed supervillains doing nefarious stuff on behalf of the government. But even they only exist for superheroes to "uncover" when they realize that the colorful team makes no sense; this team exists so that nobody digs deep enough to find the real organization of mundane criminals doing criminal things for the government.
I bring this up, because Section 31 feels a lot like a TNG concept. With the exception of Conspiracy, a.k.a. "we need to slaughter Starfleet's leadership, because they might be influenced by foreigners," the crew keeps running into terrible officers in Starfleet, or terrible civilians appointed and supported by Starfleet, and they're always dismissed as just bad apples. A significant percentage of Picard's save-the-day speeches that you mention are actually assertions that the system is terrific, and just needs better policing – presumably at the top, since everybody else is required to bow to authority – to keep the bad people from running things. It's always a personal failing of the leader, never a systemic failing of a culture of leadership that put that leader in charge of (for example) an anti-Romulan witch hunt.
In that sense, Section 31 makes perfect sense: Starfleet has created an almost cartoonishly amoral, unaccountable team, flashy enough to draw the attention of anybody worried about the Federation's ethics, and draw attention away from the fact that no amount of legal wrangling ever gives artificial people civil rights, the authorities gleefully destroy hundreds of careers in their hunts for "subversives" like the genetically altered and Romulans, the rogue crews that murder hundreds of civilians based on unsubstantiated rumors, and so forth.