Out of the box XC loads colorbox only if it is used for detailed images.
Open skin/common_files/customer/service_head.tpl (or the same in your custom skin) and just before
Code:
{load_defer_code type="css"}
{load_defer_code type="js"}
add
Code:
{if $main eq "product"}
{load_defer file="lib/colorbox/colorbox.css" type="css"}
{load_defer file="lib/colorbox/jquery.colorbox-min.js" type="js"}
{/if}
Adjust paths if needed. jQuery should be already added so no need to call it again.