Degradation to SEPA Instant - Direct

Incident Report for Form3

Postmortem

We identified a misconfiguration in the network settings of our connectivity provider. This led to increased and fluctuating latency when conveying messages to and from RT1. The misconfiguration was only present in production, and as such had not been detected in test environments. The network configuration was changed at 02:41 UTC, and controlled tests in the production environment have shown consistent, low latency, in line with expectations.

We continued to monitor through the day to ensure the service was handling the new configuration without further issue. All tests showed successful processing without latency issues.

We continue to work with our provider to ensure full and complete remediation is in place to prevent this issue from impacting customers in the future.

Posted Dec 05, 2019 - 14:01 UTC

Resolved

Our SEPA Instant service has recovered and normal service has resumed. 

The degradation seen over the past 24 hours has been fully mitigated and customers should see admissions process without issue.

We will continue to monitor the service to ensure there is no ongoing impact.

Posted Dec 04, 2019 - 17:48 UTC

Update

FORM3 continued investigations over night with our network provider. Mitigation's have been identified and applied to the environment.

Monitoring will continue throughout the day and updates will be provided as the investigation continues.

Posted Dec 04, 2019 - 09:01 UTC

Update

We continue to investigate the intermittent impact on the SEPA Instant - Direct service with our network provider.

The service is operating with intermittent impact to admission processing resulting in payments being reversed.

We will post further updates as we continue to investigate the issue and implement a full remediation plan

Posted Dec 03, 2019 - 17:53 UTC

Update

We have not seen a recurrence of the issue since 13:22 UTC, we continue to monitor this issue and investigate further enhancement

Posted Dec 03, 2019 - 14:30 UTC

Update

We have identified a further occurence of the degradation intermittently impacting SEPA Instant admissions.

We are currently investigating and will update as the investigation progresses

Posted Dec 03, 2019 - 13:38 UTC

Monitoring

We are now undergoing a period of monitoring to confirm service health

Posted Dec 03, 2019 - 10:58 UTC

Investigating

We continue to investigate the degradation and are working with our network provider to determine the root cause of the behaviour seen.

There has been no further degradation since 07:55 UTC and a period of monitoring is in place to identify any further issues and confirm service health

Posted Dec 03, 2019 - 10:50 UTC

Identified

We have identified the issue causing a small number of admissions to fail between 07:42 - 07:55 UTC. We continue to investigate this issue for further mitigation

Since 07:55 UTC we have seen admissions process successfully

We will post an update once we have completed our investigation

Posted Dec 03, 2019 - 08:25 UTC

Investigating

We have identified an issue impacting SEPA Instant admissions

Customers may see admissions failing

Our engineers are working to return to normal service as soon as possible and we will provide an update as the investigation continues

Posted Dec 03, 2019 - 07:59 UTC

This incident affected: _Archived Components (SEPA Instant - Direct).