Hi,
1. What do you mean by "catching the flow of the checkout method" ? You want to add a payment method to the payment method selection interface of the checkout ? In that case, it would be best to create a new payment plugin for that. Or have two instances of the payment method with different settings.
2. The customer receipt (I believe that you mean by that the order creation notification) is sent by several things. First the save function of the file administrator/components/com_hikashop/classes/order.php calls the loadOrderNotification function in that same file, which uses the functions of the file mail.php in the same folder in order to load the email data from the database and from the files of the email in the folder media/com_hikashop/mail. Then, the email is sent by HikaShop via the function sendMail of the same mail.php file which then calls the Joomla library in order to send the email via your email server.
For the vendor email, it's almost the same.
3. The templates of the emails are stored in media/com_hikashop/mail.