I guess what it really comes down to is a choice between a complaint here and there about small font sizes, or running the risk of having your site layout fall apart because someone inadvertently has their IE settings set to display text at Largest. If you find a way to deliver a site that is 100% compatible with every browser and resolution out there, you will make a mint
As I mentioned before, if you are really concerned about the minority, use a detection script that will deliver a different style sheet for each different browser/resolution.