Degradation to FPS Direct

Incident Report for Form3

Postmortem

At 09:10 UTC, we began to see a degradation of the FPS Direct service at our gateway provider level. This led to a small number of payment admissions being reversed and a small number of submissions failing incorrectly. We highlighted this issue to our gateway provider who were investigating the issue.

This incident was caused by an issue in our gateway providers infrastructure relating to their transformer component. This component manages the conversion of message types. Although 4 instances of this component were running, one of the instances began to see increased decline rates, this caused circa 20% of transactions being processed to be negatively impacted.

This instance was removed to allow the component to run on 3 instances and service levels resumed as normal at 19:12 UTC.

During the incident a further issue with the network components led to Form3 being placed into automatic Stand In at 14:45 UTC.
This was investigated and resolved at 19:12 UTC and we are confident of no further issues

Following a period of monitoring we posted a final update to our Status Page to resolve the incident.

Posted May 31, 2019 - 13:52 UTC

Resolved

Our FPS Direct gateway provider has recovered from the issue degrading payment processing and normal service has resumed since 20:12:49 BST. 

Two separate but related issues were identified both of which have now been resolved. We are working with our gateway provider to investigate the root cause to ensure mitigation measures are in place to prevent future occurrences.

Our incident response team will continue to monitor the service to ensure there is no ongoing impact. (edited)

Posted May 30, 2019 - 20:00 UTC

Update

We have confirmed all FPS Direct Customers have come out of Stand In and we are seeing payments flow through that were queued.

We continue to monitor the gateway provider issue

Posted May 30, 2019 - 19:32 UTC

Update

all customers remain in stand in. We continue to monitor and will provide a further update once available

Posted May 30, 2019 - 15:59 UTC

Identified

Please be advised that all Direct Customers have been placed into Stand-in by our Gateway Provider

Customers may see Inbound and Outbound payments being queued.

We will post an update at 16:45 unless there is further information to share before this time

Posted May 30, 2019 - 15:19 UTC

Investigating

We continue to work with our gateway provider on this degradation of our FPS Direct service. The issue has been identified, a fix is under investigation but we have no confirmed timeline to resolution.

Customers will continue to see some outbound payments fail incorrectly and some inbound transactions may be reversed.

We will post an update once we have further information to share, customers should continue to monitor their payment activity for impacted transactions

Posted May 30, 2019 - 14:01 UTC

Update

We continue to work with our gateway provider who have located the issue and continue to work on a fix.

Payments continue to be impacted as previously advised.

We currently do not have a time frame associated, we will post an update at 15:00 BST unless there is further information to share before this time

Posted May 30, 2019 - 11:57 UTC

Update

We are working with our gateway provider who has located the issue and working on a fix.

We continue to see that some outbound payments are failing and that some inbound payments are reversed

We currently do not have a time frame associated, we will post an update at 13:00 BST unless there is further information to share before this time

Posted May 30, 2019 - 10:53 UTC

Update

We are working closely with our gateway provider to resolve this issue as soon as possible.

Customers may see that some outbound payments are failing and that some inbound payments are reversed.

We will post an update at 11:50 BST unless there is further information to share before this time

Posted May 30, 2019 - 10:11 UTC

Identified

We are currently experiencing a degradation of our FPS Direct. Our team are investigating and will provide updates as the investigation progresses

Posted May 30, 2019 - 09:42 UTC

This incident affected: _Archived Components (FPS RT - Direct).