#7170 Form3 Platform Network Degradation

Incident Report for Form3

Postmortem

At 22:13 Form3 was alerted to an increased error rate associated to accessing the Form3 API from external sources.

Investigations established an issue with our network provider, Cloudflare, link to postmortem, issued by Cloudflare https://blog.cloudflare.com/cloudflare-outage-on-july-17-2020/

We experienced 17 minutes of impact and the impact seen is listed below

Customers may have been received error messages when accessing our API during the 17 minutes of unavailability
FPS RT - Indirect - Starling and FPS RT - Indirect - LHV received no admissions during the incident
FPS RT - Direct customers were placed into Stand in

Throughout the incident, we worked closely with our FPS direct and indirect partners to establish impact, other scheme connections that don't require DNS resolution were unaffected.

Customers may have been unable to reach the Form3 API via the internet, depending on their location and choice of DNS provider. Customers using AWS Privatelink or leased lines would have been able to connect to Form3 during the incident without disruption.

Alongside the existing mitigating actions available to us (should this issue have persisted) we will also continue to explore further fail over measures that could be implemented

Posted Jul 20, 2020 - 15:42 UTC

Resolved

Form3 were alerted to an issue at 22:13

We began investigations and located wider issues across internet and cloud providers, this issue caused our network provider to be impacted and in turn disrupted the normal receipt of payment messages into our environment.

The following 3 services were impacted for a period of 17 minutes.

FPS RT - Direct
FPS RT - Indirect - Starling
FPS RT - Indirect - LHV

During this impacted time Form3 received no Admissions or Submissions for the above services and each of our FPS RT - Direct customers were placed into Stand in. We have communicated with our gateway provider and informed FPS scheme.

Form3 will complete all relevant documentation and no Scheme Alert will be issued

The environment has been stable since 22:34, we will continue to monitor our environment

Posted Jul 17, 2020 - 22:28 UTC

Investigating

Form3 have received an alert related to the performance of the FORM3 service.

Investigations are ongoing and an update will be provided when more information becomes available.

Posted Jul 17, 2020 - 21:34 UTC

This incident affected: _Archived Components (FPS RT - Indirect - Starling, FPS RT - Direct, Account Management Services, Bacs - Direct Participant, Reports, SEPA Instant - Direct).