I have the almost the same question. I plan to add variants for the first time. But, my situation is different only in that I will be adding only about 2 or 3 variants per product, for about 20 or 30 products in total. Currently my store has 500 products but will soon have 2000. Will those variants slow my cart down at all?
I think from this post that I will be fine:
http://forum.x-cart.com/showthread.php?t=58561
Mysticearth, Alex said ~10 was too many variants per product. 4 is ok. Perhaps you should specify how many products that you'll add, and how many variants per product. We both could then use the answer.