#26140 - Platform Admission Latency
Incident Report for Form3
Postmortem

A 3rd party provider degradation resulted in an outage on admission processing across several services from 15:16 - 16:57 BST.

We observed errors in internal services connectivity during the impact period, preventing response on synchronous messages and blocking payment status updates.

Form3 are reviewing service enhancements to address the behaviour seen and mitigate future impact.

Payment impact per service is detailed below:

FPS RT - Direct:

Admissions timed out and reversed from 15:16 - 15:45 BST. Form3 invoked Stand-In for all impacted customers to mitigate payment failures from 15:45 - 17:03 BST. Submissions were unaffected.

Form3 engaged with relevant external parties in line with our incident management procedures to provide necessary details on the degradation.

SEPA Instant - Direct:

Admissions were accepted and notification held internally at Form3 until the incident was resolved and all impacted payments were notified to customers. Submissions were unaffected

FPS RT Indirect - LHV/ SEPA RT Indirect - LHV:

Both admissions and submissions were processed successfully and notification held internally at Form3 until the incident was resolved and all impacted payments were notified to customers.

SEPA Credit Transfer - Direct:

Admission files received in the last cycle of the day were processed successfully and notification held internally at Form3 until the incident was resolved and all impacted payments were notified to customers.

Posted Sep 29, 2021 - 15:27 UTC

Resolved
The issue affecting payment processing on the FPS RT - Direct; FPS RT - Indirect - LHV and SEPA Instant services has been fully resolved as of 17:40 BST.

We have confirmed, following a period of monitoring, that the services have returned to normal processing, impacted admissions have all reached their final state and no further degradation has been observed.

We are working closely with the 3rd Party to understand root cause and mitigations.

Further details will be provided in the postmortem in due course.
Posted Sep 28, 2021 - 16:42 UTC
Monitoring
A fix has been implemented and we are in a period of monitoring.

FPS RT - Direct: Stand-in was revoked at 17:03 BST and admissions are now processing as expected.

FPS RT - Indirect - LHV; FPS RT - Indirect - Starling and Sepa Instant - Direct: Impacted customers should now see that normal processing has resumed on the impacted services.

We will provide an update on resolution in due course.
Posted Sep 28, 2021 - 16:06 UTC
Identified
Root cause of the service degradation is known and we are working with our 3rd Party Provider to expedite resolution.

FPS RT - Direct: Stand-in has been invoked. Admissions are currently being qualified accepted and being queued.

FPS RT- Indirect - LHV and Sepa Instant - Direct: Admissions are being received but there will be latency in them reaching a final state.

Submissions are unaffected.

We will post an update at 17:00 BST, unless there is further information to share before this time.
Posted Sep 28, 2021 - 15:30 UTC
Update
We continue to investigate the degradation to the impacted services.

FPS RT - Direct: Stand-in has been invoked. Admissions are currently being qualified accepted and being queued within our infrastructure.

FPS RT- Indirect - LHV: Admissions are being received but there will be latency in them reaching a final state.

Sepa Instant - Direct: Admissions are being received but there will be latency in them reaching a final state.

Submissions are unaffected.

We will post an update at 16:30 unless there is further information to share before this time.
Posted Sep 28, 2021 - 14:51 UTC
Investigating
Form3 are investigating a performance degradation to FPS RT - Direct; FPS RT - Indirect - LHV and SEPA Instant services.

Admissions are currently impacted, submissions are unaffected.

Investigations are ongoing and an update will be provided as soon as more information becomes available.
Posted Sep 28, 2021 - 14:36 UTC
This incident affected: _Archived Components (FPS RT - Direct, FPS RT - Indirect - LHV, SEPA Instant - Direct).