Intermittent Network Degradation to FPS Direct

Incident Report for Form3

Postmortem

From 17:17 - 00:12 BST the Form3 FPS gateway provider’s network provider suffered a major outage due to an internal infrastructure issue.

This resulted in a number of short periods of degradation during which submissions were delayed before reaching final state and admissions timed out and reversed.

Automated protection measures were engaged to mitigate the degradation before the system automatically recovered.

We were in communication with the gateway provider throughout and they have now confirmed the incident was resolved and mitigations are in place.

Form3 will manage submission of incident forms to the FPS scheme.

We continue to monitor the service for further issues and continue to work with our gateway provider to identify enhancements to the service to address the behaviour seen.

Posted Apr 28, 2020 - 11:14 UTC

Resolved

Following a period of monitoring with our gateway provider, we can confirm no further errors have been seen since approximately 00:20 BST.

Impacted customers will be contacted directly regarding any payment impact during this degradation and further details will be shared in the post mortem which will be shared later on today.

We continue to monitor service behaviour for any further degradation and work with our gateway provider to put in place mitigations to the issue seen.

Posted Apr 28, 2020 - 03:02 UTC

Update

Our Incident Response team has confirmed payments are now processing without further degradation.
We continue to monitor the situation and work with our gateway provider to ensure the service continues to operate as normal.

Posted Apr 27, 2020 - 21:43 UTC

Monitoring

We have been alerted to an issue with network connectivity at our FPS gateway provider.
Customers may see intermittent delays in payment submissions reaching a final state, while a small number of admissions may time out and result in a reversal.
We are working with our gateway provider to mitigate this issue and resume normal service.
We will provide further updates once we have received confirmation from the gateway provider the issue has been resolved and payments are being processed without further issue.

Posted Apr 27, 2020 - 20:21 UTC

This incident affected: _Archived Components (FPS RT - Direct).