(Don't confuse authority with legitimacy, which the supposed permissibility of creating and enforcing rules. A government might be legitimate but not authoritative.)
Most people believe some governments have authority over some people over many things. For instance, they think that they are obligated not to snort coke because the government said no.
It's one thing to say that you have a duty to pay a reasonable tax toward a genuine and needed public good or to drive near the speed limit. But what's odd is that people believe that governments have significant authority to override rights or to force us to permit them to commit injustice.
Why would anyone think that? After all, on their face, none of the major theories of authority seem to imply this. Consider:
1. Consent theory/social contract theory holds that government has authority over you because you make some sort of binding promise or hold some sort of contract with others to obey the government.
Problem: Promises to commit injustice or allow injustice (that you can easily stop) aren't binding, ever. If I promise to let Uncle Sam kill my kids, or to kidnap them and send them to kill other people's kids, when Uncle Sam shows up, I am obligated to intervene. My promise has no moral force, even if Uncle Sam gave me favors and benefits in exchange for the promise. The fact that I made such a promise simply shows I'm a rotten, vile person, or at best deeply confused about right and wrong.
Duties don't disappear by fiat or in because you accepted benefits from others. There isn't some special escape clause from your moral obligations that makes them vanish when you create a compact with a third party.
I can maybe make a binding promise or enter into a contract to allow Uncle Sam to hurt me in various ways, but I can't relinquish my obligations to respect others' rights, to give them what they deserve, or to act beneficently toward them, in virtue of making a promise. If I owe it to you to rescue you from Uncle Sam before I make any promises to him, then I continue to owe it to you to do so even if I promise not to help you.
2. Good Samaritan theory: Holds that government has authority over you because giving government such authority is necessary for you to discharge your obligations to help others and provide easy aid.
But how would deferring to injustice, permitting injustice, or aiding and abetting injustice help promote your obligations to help others and provide easy aid?
I mean, I can construct hypothetical cases where it would. Imagine that the zombie apocalypse virus is spreading. Thanos credibly offers to save the world, but only if you outlaw listening to Taylor Swift, whom he hates. He credibly can renege on saving the world if we don't obey the new laws forbidding anyone from listening to her. Swift. Fine. Take the deal.
But when people try to argue real life governments are like this, it seems implausible. If the government wants us to let it spy on or murder innocent people, it's bizarre to think Good Samaritanism would explain why, After all, being a Good Samaritan means, well, helping.
3. Fair Play: Says you have to do you fair share to maintain public goods and principles that everyone benefits from.
Problem: Like the last theory, it's unclear why this would require you to promote evil, bad, harmful, or unjust things.
To borrow an example from Michael Huemer: Suppose we're on a sinking boat. We can stop it from sinking only if people bail out water. Suppose Mike comes up with a fair and equitable scheme for dividing up the water-bailing duties. We should go along with it. But if Mike also adds, "In addition to bailing out water, I need all of you to self-flagellate, abstain from listening to Taylor Swift, and to report any Mexicans you see to me so I can attack them later," we aren't obligated to do that, too. We're only obligated to do our fair and equitable share in bailing out the water.
4. Raz's theory: Raz claims that the normal way to show one person has authority over the second is if the second person is more likely to comply with the reasons which apply to him if he follows the dictates of the first person than by trying to follow those reasons directly. For instance, suppose utilitarianism is true, and your friend Bob is much better at calculating consequences than you are. Bob is also honest and trustworthy. You should follow Bob's advice rather than decide for yourself.
Problem: It's hard to see how this would bind you to act wrongly. You don't have independent moral reasons to violate people's rights, tolerate right's violations, tolerate injustice, and so on. By complying with unjust laws or unjust political actions, you are in the first instance not complying with the moral reasons which apply to you, by instead thwarting or ignoring them.
Now, I realize that defenders of these theories often acknowledge some of these problems and try to overcome them. (I think they also realize they are not widely seen as having successfully done so.)