Paypal payment notification refused:invalid response - please advise

-- HikaShop version -- : 3.1.1
-- Joomla version -- : 3.7.4
-- PHP version -- : 5.6.28
-- Error-message(debug-mod must be tuned on) -- : Paypal payment notification refused:invalid response

We have an ongoing issue with a Hikashop installation and would be grateful for your assistance. Payments are made via PayPal Pro Hosted (the payment page is hosted on PP’s own server) and using the Hikashop Paypal Website Payments Pro Hosted Payment plugin plugin.
Normally the a successful order confirmation is returned to the site without any problem. However, as of three weeks ago, we get the message from HS as below:
Subject: Paypal payment notification refused:invalid response
Hello,
A paypal notification was refused because the response from the paypal server was invalid
Check the documentation concerning this issue at www.hikashop.com/index.php?option=com_up...rror#invalidresponse
This notification was for the order JD17-B5D9A8 on the website shop.justducks.co.uk/ You can access the order details directly by clicking on the link below after logging in your back end:
shop.justducks.co.uk/administrator/index...sk=edit&order_id=598

Orders are marked as “Created”, not completed. The user can manually change the order to “Completed”, but this is a hassle and is not how it was working previously.
We have tried the following:
1. Updated both HS and Joomla to the latest stable versions
2. Applied the patch as outlined in a separate post
3. Changed PayPal account to UTF-8
4. Checked the payment logs – nothing appears out of place compared to successful transactions
Please advise on what steps should be taken to resolve this.
Many thanks




ADDITIONAL SERVER INFORMATION
PHP Built On Linux s166-62-85-241.secureserver.net 2.6.32-042stab094.7 #1 SMP Wed Oct 22 12:43:21 MSK 2014 x86_64
Database Version 5.6.35
Database Collation latin1_swedish_ci
Database Connection Collation utf8mb4_general_ci
PHP Version 5.6.28
Web Server Apache
WebServer to PHP Interface cgi-fcgi
Joomla! Version Joomla! 3.7.4 Stable [ Amani ] 25-July-2017 11:11 GMT
Joomla! Platform Version Joomla Platform 13.1.0 Stable [ Curiosity ] 24-Apr-2013 00:00 GMT

Have you checked the latest PayPal security policies?

There may be a UK special version, but this is the PayPal information from the USA announcements:

1) TLS 1.2 Upgrade

The most secure protocol for sharing information on the web today is Transport Layer Security (TLS) version 1.2. PayPal is enabling support for TLS 1.2 for all secure connections and in 2016 will start requiring its use. You will need to verify that your environment supports TLS 1.2 and if necessary make appropriate updates. PayPal is updating its services to require TLS v1.2 for all HTTPS connections in June of 2017. After that time, all TLS v1.0 and TLS v1.1 API connections will be refused.

Under PCI Compliance requirements, TLS 1.2 is becoming mandatory for ALL websites that accept credit cards by January 2018, so anyone experiencing problems won't just be seeing it on your site, it will begin to affect their entire online buying experience more and more over the next few months.

2) IPN Verification Postback to HTTPS

If you are using PayPal’s Instant Payment Notification (IPN) service, you will need to ensure that HTTPS is used when posting the message back to PayPal for verification. After June of 2017 HTTP postbacks will no longer be supported.

The full announcement by PayPal is here (many of these changes have already been done over the last year): devblog.paypal.com/upcoming-security-changes-notice/

I noticed your site isn't using SSL, so that may be the issue right there: cl.ly/2e230f1m1U1N

www.hikashop.com/forum/payment-methods/8...sactions.html#273747

Thanks for both suggestions.

PolishedGeek, we'll have a look at the SSL and see if this changes anything for us.

Jerome, that thread suggests hat this applies only to the standard PP plugin and not the Hikashop PayPal Website Payments Pro Hosted Payment plugin, so I'm hesitant to apply it.

I can confirm that we're running a different payment module, because the client has PayPal Pro Hopsted.

Separately, the site isn't SSL because the transactions are conducted on PayPal's servers, not ours - when the "complete purchase" button is pressed, the user is taken to the PP site to enter their payment information.

Any more ideas welcome!

Hello,

Please check that forum thread : www.hikashop.com/forum/payment-methods/8...ents-pro-hosted.html
Which HikaShop payment plugin are you using ?

Regards,

Jerome wrote: Hello,

Please check that forum thread : www.hikashop.com/forum/payment-methods/8...ents-pro-hosted.html
Which HikaShop payment plugin are you using ?

Regards,

Hi Jerome,

We're using the "Paypal Website Payments Pro Hosted Payment" plugin, so I don't think the thread you reference applies to us. But thanks for the suggestion.

As an update to this issue, we've been in contact with PayPal and they have confirmed that one of the transactions which was returned to the HS system with the original error message was, as far as PayPal is concerned, a fully valid transaction.

This (slightly expurgated) from their response to us:

Time Created
Jul 31, 2017 06:54:26 PDT
PayPal Account
JustDucks Ltd
Transaction ID
xx25M
Delivery Status
Sent
HTTP Response
200
Last IPN send Attempt
Jul 31, 2017 06:54:28 PDT
Destination URL
shop.justducks.co.uk/index.php?option=co...t&lang=en&Itemid=105
Number of Retries
0
Type
Transaction made
IPN Text
mc_gross=32.00&invoice=635&protection_eligibility=Ineligible&payer_id=SUJW85VBX62ZU&
tax=0.00&payment_date=06:54:12 Jul 31, 2017 PDT&payment_status=Completed&charset=UTF-8&
first_name=timo&mc_fee=1.45&notify_version=3.8&custom=&payer_status=verified&
business=xxxxx@xxx@xxx&quantity=1&
verify_sign=An7H3kH8.DY56ocyWs9pEuVQUM5wA2seUWPPNlgayLnPGgf-xEVmm2cE&
payer_email=xxxxx@xxx@xxx&txn_id=7Y523873A2521025M&payment_type=instant&
last_name=xxx&receiver_email=xxxxx@xxx@xxx&payment_fee=&
receiver_id=JYQVF7NJL87Q8&txn_type=web_accept&item_name=&mc_currency=GBP&
item_number=&residence_country=DE&handling_amount=0.00&transaction_subject=&
payment_gross=&shipping=0.00&ipn_track_id=9e387cf973036

Hi,

Here is a modified Paypal Website Payments Pro Hosted Payment plugin which includes the same modification that was implemented in the PayPal plugin:

Install it on your website and it should automatically work. There is no option to activate this time.

Thank you.

We'll give it a try and report back on results...

I'm afraid to say that the problem is still occurring - the first order placed after the update was applied failed with the same error message.

The order is now on the system as "Created" and not "confirmed".

Hi,

Here is a new version of the plugin. Please try with it.

If that doesn't help, then it means that the problem doesn't come from the same issue than the one on the other thread ( www.hikashop.com/forum/payment-methods/8...sactions.html#273747 ).
In that case, it means that the PayPal server is refusing the IPN some some reason. Could you ask the PayPal support what is the reason that their IPN server is refusing the IPN verification request sent by HikaShop's payment plugin for one of your "invalid response" payment ?

Thanks, we'll try this and report back to you when we have some results from live transactions.

In the meantime, PayPal is reporting that the IPNs for the transactions that have generated the error messages are OK. We provided them with several transaction IDs and they came back with the following:

"I can verify the IPN for the transaction you mention is fine"

The fuller details are in an earlier post on this ticket.

More news when we have it...

No good, I'm sorry to say: all the orders that have come into the system since the patch was applied are showing as "Created" and not "completed", so the problem still stands.

Hi,

If they say that everything is fine, then why is their server saying that the IPN is invalid ?
That invalid error comes from the PayPal server saying that the IPN request is invalid.

You could replace the line:
if(!$verified) {
to:
if(false) {
in the plugins/hikashoppayment/paypal/paypal.php file and that will deactivate the check with the PayPal server. However, that means that anyone would be able to call the payment notification URL to validate orders so I don't recommend that.
The best is that PayPal tells what is the problem with the IPN request been invalid.

Hi nicolas,

I'm not clear on your reponse. PayPal have been telling us that the IP{Ns are fine for the errored transactions. My understanding of hte issue was that the IPN was coming back from PP as OK, but that HS was not processing it correctly, leaving orders as "created" and not "confirmed". The error email quoted at the beginning of the thread is being generated by HS, not PP.

From PP techs: "I understand you are having issues with the IPNs. Actually when I see your logs I do not see any failed attempt to send you an IPN. All of them are marked as received by your server."
and
"I can verify the IPN for the transaction you mention is fine."

However, HS is not completing the order fully, sticking at "created". Are you saying that the IPN text I quoted from PP earlier in the thread indicates that the IPN is not OK? Apologies - this is beyond my pay grade!

As an aside, the shop is currently configured with the payment page being PP's own, hosted on their own servers. On that basis we'd not considered it necessary to set up https on the server as all the transactions were being processed off-site. However, is it possible that the revisions to IPN Verification Postback to HTTPS outlined by PP here ( www.paypal-knowledge.com/infocenter/inde...916&viewlocale=en_US ) might be at the root of the issue?

Hi,

When a payment is made, PayPal sends an IPN to HikaShop.
At that point HikaShop sends a request to PayPal with the parameter cmd=_notify-validate and all the parameters that PayPal sent to HikaShop in the IPN.
Then, PayPal responds to that request to says either "verified", meaning that the IPN comes from PayPal, or "invalid" meaning that the IPN doesn't come from PayPal or that there is a problem with how the parameters were sent to PayPal.
If PayPal responds "verified", then HikaShop changes the status of the order.
If PayPal responds "invalid", then HikaShop sends you that invalid response email notification.
Since you got that email notification, it means that PayPal refused the "notify-validate" request from HikaShop for the IPN.
So yes, the IPN is valid, we know that, but what we don't know is why PayPal says that the "notify-validate" request on the IPN data is invalid.

And yes, one of the reasons PayPal might refuse a validation of the IPN is if the server is not using HTTPS. But they should be able to tell you that in that case...

Ni nicolas,

This is the IPN text that we got back from PP for one of the problem transactions. From what I can see, it doesn't include the text you mention.

IPN Text

Hello,

That's not how the IPN is working.
developer.paypal.com/docs/classic/produc...ayment-notification/

1 - Paypal contact your website (on a URL you define) with some extra parameters.
2 - You call back Paypal giving it the same parameters
3 - Paypal configure that it's really him who made the call (point 1) and indicated that it's "verified", etc.

Regards,