Intermittent Degradation to Form3 API authentication

Incident Report for Form3

Postmortem

We use a 3rd party provider as part of our platform authentication. From 07:39 - 08:14 UTC we saw intermittent failure in these calls resulting in some requests time out but succeed on retry.

This was due to an internal issue at our provider degrading their ability to generate new tokens, we continue to follow up with our provider to ensure edge cases such as the root cause identified are fully mitigated.

Payment processing was unaffected by this degradation, only the initial authentication.

Posted Nov 13, 2019 - 09:00 UTC

Resolved

Following further monitoring we have confirmed our authentication service has recovered and normal service has resumed. 
We identified an issue in generating new tokens with our authentication provider from 07:40 - 08:14 UTC causing authentication calls to intermittently fail. Tokens are now being generated without issue and we continue to investigate further with our provider.

Posted Nov 12, 2019 - 08:45 UTC

Monitoring

We have identified and fixed the issue which was causing authentication to intermittently time out, we will continue to monitor for the next 15 minutes to ensure no reoccurrence.

We will keep you informed.

Posted Nov 12, 2019 - 08:26 UTC

Identified

We have identified an issue causing intermittent timeouts in authentication, our customers may experience time outs when fetching tokens, we recommend retrying while we continue to investigate the issue.

We will keep you updated with our progress

Posted Nov 12, 2019 - 07:59 UTC

This incident affected: _Archived Components (FPS RT - Indirect - Starling, FPS RT - Direct, Account Management Services, Bacs - Direct Participant, Reports).