Why do non-football fans in the UK still view our fans as hooligans?

So I'm a rugby union supporter and have never seen any trouble at a match (including internationals, cup finals etc). We (like most sports other than football) do not have segregation in grounds. E.g. a high stakes European match where we were playing a vastly more experienced side at home. Only lost by a few points, so almost certainly would have won if the ref had given the two red cards which he should have (both were confirmed as reds in the subsequent disciplinary hearings). In a nearby bar afterwards, a handful of the opposing fans walked in and silence fell. You could see they looked a bit uncertain until we all started clapping and they were offered seats and pints. In contrast, the one football match I've attended was a beginning of season 3rd tier down (division one now, don't know what it was back then) fixture, not a local derby. I didn't really support either team, just went because it was before the rugby season had started and I was bored. In the home end, I saw such hatred from pretty much everyone present, including teenagers, that I was shocked. Totally different atmosphere from when I'd previously been to that ground for top tier, local derby rugby matches. It felt like it wouldn't take much at all for that venomous atmosphere to turn into outright hooliganism and I've never even thought about going to another football match after that.