I think this article on creating custom payment methods for X-Cart 5 may be useful:
http://kb.x-cart.com/display/XDD/Creating+payment+method
Collecting payment information from an Iframe box inside the checkout page is not supported in X-Cart 5 out of the box. You will have to code this logic yourself, unfortunately.