The stock CSS files and templates in X-Cart are HUGE and bloated. We typically rebuild them from scratch for each branch. Tons of unnecessary code and 4 templates when one would suffice. Also, there are classes in the main css file for things that can be turned on/off, so they load even if you have a feature turned off. So yes, there is definitely room for improvement if you clean up the default templates.
You can also use something like this to really speed things up:
http://www.maxcdn.com/features/integrations/x-cart/