I don't have the answer to your question but the place to start is in skin1/customer/main/subcategories.tpl...
The line of code...
<td valign="top" rowspan="{count value=$subcategories print="Y"}"><img src="{if $current_category.icon_url}{$current_category.icon _url|amp}{else}{$xcart_web_dir}/image.php?id={$cat}&type=C{/if}" alt="{$current_category.category|escape}"{if $current_category.image_x} width="{$current_category.image_x}"{/if}{if $current_category.image_y} height="{$current_category.image_y}"{/if} /></td>
The width="{$current_category.image_x}" and height="{$current_category.image_y}" statements determine the size of the category icon.