There are two widgets that render the categories menu:
- \XLite\View\TopCategories
- \XLite\View\TopCategoriesSlidebar
I believe the one that you need is the first.
So, in your custom module, you should create a CSS file that adds your custom styles, then "decorate" the \XLite\View\TopCategories widget from your module and link it to the new CSS file via the getCSSFiles() method.
You can see an example in this article (but you should "decorate" \XLite\View\TopCategories class, not \XLite\View\AView as in the article):
http://kb.x-cart.com/display/XDD/Working+with+footer