It's completely contextual, and honestly it's probably difficult for a native speaker to fully explain when it would be ok, when it wouldn't, and when it would be borderline — you don't really think about the exact unwritten rules of something like that unless something sounds a bit jarring because it's broken those unwritten rules.

As a simple rule of thumb, I think in a lot of contexts "ladies" is safer and less likely to be infantilising (e.g. "the ladies in the accounting department said..."), or just not mentioning gender at all if it's irrelevant (e.g. "the accounting department said...")

To have a stab at explaining when it's ok, I think it's less likely to be an issue when the speaker themselves is a woman (particularly a younger woman), when there's not a power imbalance between the speaker and the person/people being referred to, or when you're talking about "girls and boys" in almost a tongue in cheek manner.

So for the women thing, it's quite common for young women to call each other "girls" either as a casual equivalent to "guys", to emphasise a sense of sisterhood (e.g. "girls' night out"), or because as a younger woman it sometimes feels a bit weird to call someone the same age and level of seniority as you a "woman" or "lady". (E.g. I'd tend to say "there's a girl in the office called [X] who I get on with" rather than "a woman" unless the person I was talking about was significantly older than me)

For the power imbalance thing, I think it runs along similar rules as to when it's ok to call someone a casual pet name like "love", "duck", "pet", etc, in terms of age, seniority, gender, social class, familiarity, etc. I would probably bristle at a middle class senior male manager that I didn't know well referring to me as a 'girl' in a way that I might not if it was coming from someone less senior than me, someone working class, or someone older who I had more of a mentor/mentee relationship with.

And for the tongue-in-cheek context, I have a work trip coming up that will likely involve us glamping (oh joy). The events coordinator can talk about the "girls' tent" and the "boys' tent" even though the people involved range from mid twenties to 50+ because it's just mildly silly to speak as if we're kids on a school trip. Or equally, if you were doing a quiz and there was a roughly equal gender split you could be like "girls vs boys?".