#71775 - Potential degradation to the FPS service
Incident Report for Form3
Postmortem

On 10 December, Form3 observed a major outage on the Form3 - UK Instant service causing impact to both inbound and outbound payment processing between 16:39 UTC and 21:20 UTC.

Recovery actions restored payment processing on the platform at 21:20 UTC. After this time additional steps were taken to remediate any queued payments and to restore resiliency.

Root cause has been isolated to an internal network connectivity issue. Form3 are implementing preventative measures to avoid the risk of reoccurrence.

Posted Dec 12, 2022 - 21:49 UTC

Resolved
The issue affecting FPS service has been resolved.

We have confirmed, following a period of monitoring, that the service has returned to normal processing and no further degradation has been observed.

Further details will be provided in the postmortem in due course.
Posted Dec 11, 2022 - 00:49 UTC
Update
A fix has now been fully implemented and we are in a period of continued monitoring.
Normal processing has resumed on the FPS service and we will provide an update on resolution in due course.
Posted Dec 10, 2022 - 23:42 UTC
Update
A fix has been implemented and we continue to work with customers to restore services.
Impacted customers should now see that normal processing has started to resume on the FPS service for both inbound and outbound payment flows.
We will provide a further update at 23:30
Posted Dec 10, 2022 - 23:04 UTC
Update
A fix has been implemented and we continue to work with customers to restore services.
Impacted customers should now see that normal processing has started to resume on the FPS service for both inbound and outbound payment flows.
We will provide a further update at 23:00
Posted Dec 10, 2022 - 22:25 UTC
Update
A fix has been implemented and we continue to work with customers to restore services.
Incident response team continue to work on this with high priority.
Next update 22:30
Posted Dec 10, 2022 - 21:45 UTC
Update
We continue to monitor the initial fix and work with customers to restore service.
Incident response team continue to work on this with high priority.

We will provide a further update at 21:45
Posted Dec 10, 2022 - 21:03 UTC
Monitoring
An initial fix has been implemented and we starting to see improvements with payment processing.
Incident response team continue to work on this with high priority.
We will provide a further update at 21:00
Posted Dec 10, 2022 - 20:31 UTC
Update
We continue to work on the fix.
Both inbound and outbound payment flows remain impacted.
We will provide an update at 21:00 or as soon as more information becomes available.
Posted Dec 10, 2022 - 20:07 UTC
Update
We have identified the issue, and continue to work on the fix.
Form3 have manually logged off all FPS participants from Scheme.
We will provide an update at 20:00 or as soon as more information becomes available.
Posted Dec 10, 2022 - 19:20 UTC
Update
We are continuing to work on a fix for this issue.
Posted Dec 10, 2022 - 18:48 UTC
Update
We have identified the issue, and continue to work on the fix.
We will provide an update at 19:15 or as soon as more information becomes available.
Posted Dec 10, 2022 - 18:30 UTC
Update
We have identified the issue, and are working on the fix.
We will provide an update as soon as more information becomes available.

Next update: 18:30
Posted Dec 10, 2022 - 17:57 UTC
Identified
We have isolated an issue on the FPS service that is causing the outage.
Our Incident Response team is actively working to resolve this as high priority.
We will post an update at 18:00 unless there is further information to share before this time.
Posted Dec 10, 2022 - 17:35 UTC
Update
We are continuing to investigate this issue.
Posted Dec 10, 2022 - 17:31 UTC
Update
Form3 are investigating an outage to the FPS service which is impacting both inbound and outbound payments flows.
Investigations are ongoing and an update will be provided as soon as more information becomes available.
Posted Dec 10, 2022 - 17:09 UTC
Investigating
Form3 are investigating a performance degradation to the FPS service.

Investigations are ongoing and an update will be provided as soon as more information becomes available.
Posted Dec 10, 2022 - 16:52 UTC
This incident affected: Form3 - UK Instant (Direct (IRE), Direct (MQ), Direct (WS)) and _Archived Components (Indirect, Direct Non Settling).