Nah... that's not stupid. Now STUPID would be cutting a hole in the grill cloth and unscrewing four of the unwanted screws out through the FRONT of the grill cloth.
You did fine. Never hurts to ask.
Personally, I'm with RockerDuck. It's more trouble but much better to have 8 fasteners on that puppy. Never got why Leo cheaped out on this one feature. Four screws just always felt a wee bit inadequate to me, with regard to even distribution of torque on a thin metal frame speaker.