[qualiteam]

You should alter:
skin/common_files/lib/jqueryui/jquery.ui.theme.css
OR
skin/%your_current_skin%/lib/jqueryui/jquery.ui.theme.css

Find in the code something like this and click the link to visually configure the theme:

/*
* jQuery UI CSS Framework
* Copyright (c) 2010 AUTHORS.txt (http://jqueryui.com/about)
* Dual licensed under the MIT (MIT-LICENSE.txt) and GPL (GPL-LICENSE.txt) licenses.
* To view and modify this theme, visit http://jqueryui.com/themeroller/?ffDefault=Verdana,%20Tahoma,%20Arial,%20sans-serif&fwDefault=bold&fsDefau...diusShadow=8px
*/

Once you've done save the theme upload to the same location renaming to X-Cart style.

Update: http://help.qtmsoft.com/index.php?title=X-Cart:Customizing_storefront#How_to_change_popup_di alog_colors_and_layout.3F