#7493 - Admission Processing Degradation
Incident Report for Form3
Postmortem

Scheduled maintenance by our gateway provider, for FPS Direct Service, caused a service impacting issue between 19:17 – 19:42 BST where a small number of admissions did not reach Form3 and resulted in timeouts. Submissions were unaffected by this issue. 

Gateway provider investigations have isolated the cause to an issue with a single service instance which did not restart as expected. During this issue, traffic continued to route to the instance leading to some admissions being stopped.

Services were recovered once the gateway provider manually intervened and initiated the recovery process. Once the affected instance was restarted correctly, the service stabilised and no further impact was seen. Services continued to be monitored closely with no further disruption observed.  

To remediate any further occurrences of this issue the gateway provider has updated their internal maintenance processes which now mitigates services restarting incorrectly, as well as enhancing monitoring during restarts.

Posted Aug 10, 2020 - 16:46 UTC

Resolved
Our FPS Direct service has recovered and no further issues have been seen since 19:42 BST. 

From 19:17 - 19:42 BST, Form3 saw an increase in admission errors where payments did not reach Form3 from our gateway provider before being reversed. The gateway provider have confirmed the connection has stabilised and continue to investigate root cause.

We continue to monitor the service and will provide further details as part of the Post Mortem report.
Posted Aug 08, 2020 - 19:16 UTC
Identified
We have identified an issue on the FPS Direct service causing some admissions to fail without reaching Form3. We are working closely with our gateway provider to investigate and resolve this issue as soon as possible.

Submissions are unaffected.

We will post an update as the investigation develops.
Posted Aug 08, 2020 - 18:47 UTC
Investigating
Form3 have received an alert related to the FPS Direct Service.

Investigations are ongoing and an update will be provided when more information becomes available.
Posted Aug 08, 2020 - 18:34 UTC
This incident affected: _Archived Components (FPS RT - Direct).