Intermittent Degradation in Network Connectivity
Incident Report for Form3 - Financial Cloud
Postmortem

Yesterday, during a routine platform deployment, there was a brief period of increased error rates between 15:10 and 15:25. This was due to the way our deployment code updates firewall rules.

At 15:30, the error rates ceased and an impact assessment confirmed that none of our services were affected by these errors.

At 22:52, we were alerted to a very brief degradation in our network connectivity. In spite of this being within operational expectations, as a precaution we decided to escalate our platform monitoring. We completed this extra monitoring phase at 10am this morning and found no further occurrences during or since.

We have reviewed the code and process for updating firewall rules to avoid future occurrences of this problem.

Posted 4 months ago. Mar 28, 2019 - 14:30 UTC

Resolved
Continued monitoring confirms that the remediation was successful and the platform is operating as normal.
Posted 4 months ago. Mar 28, 2019 - 10:09 UTC
Update
Due to the highly intermittent nature (low frequency of events) of this degradation we are continuing to monitor.
Posted 4 months ago. Mar 28, 2019 - 07:20 UTC
Monitoring
We have identified the root cause of this network degradation. We will continue to monitor while we implement a fix.
Posted 4 months ago. Mar 27, 2019 - 23:58 UTC
Investigating
We are current investigating an intermittent degradation in our network connectivity. No payments have been impacted at this time.
Posted 4 months ago. Mar 27, 2019 - 23:34 UTC
This incident affected: Account Management Services, FPS RT - Indirect, FPS RT - Direct, and Bacs - Direct Participant.