Certified first aider and fire marshal here.

With all due respect, they were right. I do understand why you stopped to grab your brother's reliever (more on that later), especially since you were 9 at the time and protecting your brother was more important than anything else. You did what you thought was best, and no one should've blame you for that.
However, fire drills are in place not just to see how quickly everyone can leave the building, but it's a test to fire marshals as well to teach them how long the "sweep" can take.
Fire consumes everything. Literally everything. If a fire burns hot enough, it can literally pulverize concrete or bricks. It doesn't take long for an average classroom to be engulfed in flames, and once flashover occures in around 3-5 minutes (depending on what burns, but there are many things in a classroom with low ignition temperature), the chance to survive is literally 0%.
So when the fire alarm sounds, the absolute best you can do is to leave through the nearest exit.
The two main reasons you shouldn't take anything with you (apart of what you can instantly grab as you get up, like phone, keys, or whatever is in front of you) is time and space. The more time you fiddle with stuff, the later you get out, and the bigger the bag you're trying to carry, the more space you take through a door. And if a 100+ people are trying to leave a building through a single door, you're gonna need as much space as you can have.
Making sure children (and tbf, adults too...) are understanding this is a huge responsibility. If you're responsible for dozens or even hundreds of people, you'll immediately understand how infuriating it can be if someone is randomly stopping to grab a few things, even if it's just a drill. In a fire, every second means life or death. You practice PEEP procedures to make sure no one dies if there's an actual fire, and trust me, even a drill can be ridiculously stressful, even if you're not rated on it.

Now about your brother's reliever: as first aiders (and I'm sure your scool used to have one too), our job is to help anyone who needs immediate treatment. An asthma attack, although a serious condition, and can actually kill someone, isn't gonna do that immediately, it takes time. To treat an asthma attack without the reliever is pretty straight forward; you sit them down up straight, and call 999. As soon as you tell the operator that you don't have the reliever, it's gonna be a Category-1 (life-threatening) call, and the ambulance will be there within 5-8 minutes. It is absolutely possible that the person will lose consciousness in the meantime, but the upside of that - especially with an asthma attack - is that it'll very likely stop the attack itself as their muscles will relax no matter what. But if even that doesn't help, we can perform CPR.
So while yes, absolutely understand why your first thought was your brother, I also understand your school's response too. You were their responsibility, along with your brother, and since the school knew about your brother's condition (since they were the one pulling you out of classes), they had to be prepared to him having an attack, even tho they weren't (and legally couldn't) trained to administer his reliever. Not even first aiders are allowed to do that. We can assist, we can hold the reliever, push it into their mouth, but we can't push the button, they have to do it.