Form3 Status - #29926 - Investigating potential degradation to the Form3 service

#29926 - Investigating potential degradation to the Form3 service

Incident Report for Form3

Postmortem

On 08 December between 20:14 - 22:50 UTC, Form3 experienced an intermittent service degradation which impacted customers' ability to send payments.

The following services were impacted during the incident window: FPS, SEPA Instant and SEPA CT submissions. Admissions were processed however, in some instances there was a delay in notifying customers of the final state.

Bacs and SEPA DD were not affected.

Root cause has been isolated to a specific piece of activity undertaken during a planned customer migration. Actions to recover service were taken at 22:50 UTC and after this time, no further impact was observed.

Mitigations are in place to avoid the risk of reoccurrence.

Following a reassessment of impact to service 9 December, this incident has been reclassified as a degradation and Status Page has been updated accordingly.

Posted Dec 09, 2021 - 18:16 UTC

Resolved

Recovery actions are underway and improvements are being observed.

A further update will be posted as the service stabilises.

Posted Dec 08, 2021 - 22:56 UTC

Update

We have isolated an issue on the Form3 service that is causing performance degradation.

Internal teams continue to investigate as high priority.

Further updates to follow as investigations progress.

Posted Dec 08, 2021 - 22:18 UTC

Update

We have isolated an issue on the Form3 service that is causing performance degradation.

Investigations continue as high priority.

We will post an update within the next 30 minutes.

Posted Dec 08, 2021 - 21:48 UTC

Identified

We have isolated an issue on the Form3 service that is causing performance degradation.

Our Incident Response team is actively working to resolve this as high priority.

We will post an update within the next 30 minutes.

Posted Dec 08, 2021 - 21:17 UTC

Update

We are continuing to investigate this issue.

Posted Dec 08, 2021 - 20:48 UTC

Investigating

At 20:21 UTC, Form3 were alerted to a potential performance degradation] to the Form3 service.

Investigations are ongoing and an update will be provided as soon as more information becomes available.

Posted Dec 08, 2021 - 20:44 UTC

This incident affected: _Archived Components (FPS RT - Indirect - Starling, FPS RT - Direct - 2, FPS RT - Indirect - LHV, SEPA Instant and SEPA Credit Transfer - Indirect - LHV, Bacs - Direct Participant, SEPA Instant - Direct, SEPA Credit Transfer - Direct, SEPA Direct Debit - Core / B2B).