This response may take the question more seriously than intended, but... here goes.

One theme lurking under the surface of TBL is socioeconomic status, and the PRESENTATION of wealth. People differ to wealth and look down on poverty. They assign virtue to wealth, "Mr Treehirn draws a lot of water in this town", and look down on poor people, "He looks like a fucking loser". Simply put, I believe the thugs would have treated millionaire Lebowski very differently. Although they would have still threatened and intimidated him, it feels very unlikely that they would have assaulted an (apparently!) rich man in a nice suit in a wealthy neighborhood. Walter was able to do that toward the end, but he knew TBL was a "phoney" millionaire, shedding the psychological protection afforded to the rich.