The short answer is as you note, cultural taboo and religion that extant societies haven't been libertarian enough to remove the laws about. But the law really isn't the important thing in this case.

The longer answer is that it's selection bias of stable societies sticking around. Monogamous mating makes for a stable society while polygamous mating, without strong mitigating factors, makes for an unstable society. The instability comes from the fact that humans have an approximately 50/50 sex ratio, so in polygamy a bunch of men won't have any serious mating prospects. They're typically young and when the situation becomes clear, extremely angry.

Historically the excess of young men have been killed in wars and fighting, or shunned/excommunicated from their home society into the larger world (Mormons and some Islamic states), while the de facto polygamy was also restricted to the extremely rich.

Today with stable monogamous relationships at arguably an all time low in western culture, but not primarily because of polygamy. Instead we have a sort of serial monogamy where the cliche is that many monogamous couples divorce, the man often remarries a younger woman while the older woman often single. The result is still a lot of sexually disenfranchised younger men. These single men are sedating themselves with porn and modern weapons grade home entertainment (largely video games), though that has been much less effective at avoiding societal instability.

Ultimately in any society where women have the nominal right to decide whom they marry or sleep with, it's up to women's collective choice of cultural norms whether we get a stable monogamous society, or an unstable polygamous society that self deletes. Unfortunately any significant minority of women can spoil it for the rest if they choose to share popular men, or to swear off the available men altogether. Because even if most women remain monogamous, we may still have enough disenfranchised men to cause significant problems.

This doesn't have to be accomplished through law, and as libertarians we view the law as a bad way to try. But it does have to be done somehow, so on the spectrum of bad things government does it's pretty low priority to legalize it. Aside from law, it's usually been some combination of women shaming other women if they aren't relatively chaste, low social status for sex workers and sluts, severe reputation or legal penalties for adultery etc.