Degradation to FORM3 Platform

Incident Report for Form3

Postmortem

At 14:40 UTC services at Form3 experienced connectivity issues to the database, this caused an intermittent degradation on payment processing impacting the FPS Direct, FPS Indirect and SEPA Instant services.

As a result of this a small number of payment admissions timed out and and were cancelled.

Payment submissions processing also became delayed during this window, with some submissions taking up to 30 minutes before processing was completed. All payments are now fully processed and show a final state.

This system fully recovered at 14:48 UTC and processing of new payments was handled without further impact.

Form3 are working with our cloud provider to put in place remediating actions to enhance the logic to handle connectivity errors more gracefully and improve alerting for connectivity issues.

Posted Jan 23, 2020 - 15:57 UTC

Resolved

We have confirmed our platform has operated as normal as of 14:48 UTC.

Form3 experienced a service degradation from 14:40 - 14:48 UTC that caused all services to be intermittently unavailable causing some admissions to be timed out and a small number of submissions delayed in processing.

We have confirmed all payments have reached a final state in the api.

We are continuing to investigate the degradation with our cloud provider and will be in contact with all necessary parties to meet reporting requirements on behalf of our customers.

No further impact has been seen on the platform and our Incident response team continues to monitor for any further issues.

Posted Jan 21, 2020 - 16:40 UTC

Update

Form3 can confirm a degradation to the FORM3 Platform @ approx 14:40 for a period of approx 5 minutes.

Customers may have seen reversals and a delay to payment processing. This has now stabilised.

Form3 Engineers are continuing to investigate while continuing to actively monitor payment traffic.

Posted Jan 21, 2020 - 15:25 UTC

Investigating

Form3 are currently investigating an alert relating to the Platform

Our engineers are working to return to normal service as soon as possible and we will provide an update as the investigation continues

Posted Jan 21, 2020 - 14:53 UTC

This incident affected: _Archived Components (FPS RT - Indirect - Starling, FPS RT - Direct, Account Management Services, Bacs - Direct Participant, Reports, SEPA Instant - Direct).