Degradation to SEPA Instant
Incident Report for Form3 - Financial Cloud
Postmortem

We identified a misconfiguration in the network settings of our connectivity provider. This led to increased and fluctuating latency when conveying messages to and from RT1. The misconfiguration was only present in production, and as such had not been detected in test environments. The network configuration was changed at 02:41 UTC, and controlled tests in the production environment have shown consistent, low latency, in line with expectations.

We continued to monitor through the day to ensure the service was handling the new configuration without further issue. All tests showed successful processing without latency issues.

We continue to work with our provider to ensure full and complete remediation is in place to prevent this issue from impacting customers in the future.

Posted Dec 05, 2019 - 14:01 UTC
Resolved
Our SEPA Instant service has recovered and normal service has resumed. 

The degradation seen over the past 24 hours has been fully mitigated and customers should see admissions process without issue.

We will continue to monitor the service to ensure there is no ongoing impact.
Posted Dec 04, 2019 - 17:48 UTC
Update
FORM3 continued investigations over night with our network provider. Mitigation's have been identified and applied to the environment.

Monitoring will continue throughout the day and updates will be provided as the investigation continues.
Posted Dec 04, 2019 - 09:01 UTC
Update
We continue to investigate the intermittent impact on the SEPA Instant - Direct service with our network provider.

The service is operating with intermittent impact to admission processing resulting in payments being reversed.

We will post further updates as we continue to investigate the issue and implement a full remediation plan
Posted Dec 03, 2019 - 17:53 UTC
Update
We have not seen a recurrence of the issue since 13:22 UTC, we continue to monitor this issue and investigate further enhancement
Posted Dec 03, 2019 - 14:30 UTC
Update
We have identified a further occurence of the degradation intermittently impacting SEPA Instant admissions.

We are currently investigating and will update as the investigation progresses
Posted Dec 03, 2019 - 13:38 UTC
Monitoring
We are now undergoing a period of monitoring to confirm service health
Posted Dec 03, 2019 - 10:58 UTC
Investigating
We continue to investigate the degradation and are working with our network provider to determine the root cause of the behaviour seen.

There has been no further degradation since 07:55 UTC and a period of monitoring is in place to identify any further issues and confirm service health
Posted Dec 03, 2019 - 10:50 UTC
Identified
We have identified the issue causing a small number of admissions to fail between 07:42 - 07:55 UTC. We continue to investigate this issue for further mitigation

Since 07:55 UTC we have seen admissions process successfully

We will post an update once we have completed our investigation
Posted Dec 03, 2019 - 08:25 UTC
Investigating
We have identified an issue impacting SEPA Instant admissions

Customers may see admissions failing

Our engineers are working to return to normal service as soon as possible and we will provide an update as the investigation continues
Posted Dec 03, 2019 - 07:59 UTC
This incident affected: SEPA Instant - Direct.
← Current Status Powered by Statuspage