Intermittent Degradation in Network Connectivity

Incident Report for Form3

Postmortem

Yesterday, during a routine platform deployment, there was a brief period of increased error rates between 15:10 and 15:25. This was due to the way our deployment code updates firewall rules.

At 15:30, the error rates ceased and an impact assessment confirmed that none of our services were affected by these errors.

At 22:52, we were alerted to a very brief degradation in our network connectivity. In spite of this being within operational expectations, as a precaution we decided to escalate our platform monitoring. We completed this extra monitoring phase at 10am this morning and found no further occurrences during or since.

We have reviewed the code and process for updating firewall rules to avoid future occurrences of this problem.

Posted Mar 28, 2019 - 14:30 UTC

Resolved

Continued monitoring confirms that the remediation was successful and the platform is operating as normal.

Posted Mar 28, 2019 - 10:09 UTC

Update

Due to the highly intermittent nature (low frequency of events) of this degradation we are continuing to monitor.

Posted Mar 28, 2019 - 07:20 UTC

Monitoring

We have identified the root cause of this network degradation. We will continue to monitor while we implement a fix.

Posted Mar 27, 2019 - 23:58 UTC

Investigating

We are current investigating an intermittent degradation in our network connectivity. No payments have been impacted at this time.

Posted Mar 27, 2019 - 23:34 UTC

This incident affected: _Archived Components (FPS RT - Indirect - Starling, FPS RT - Direct, Account Management Services, Bacs - Direct Participant).