Previous incidents

Due to the CrowdStrike issue, our team had to manually bring up each server. Service is restored. A maintenance window on the 21st maybe required to update an emergency restore we did to bring services up quickly.

The issue was identified and a fix was pushed. Users can now login.

We identified the issue as an issue connecting to a backend service that was required for the application to run. Out team was notified though with travel to a conference were not able to respond while in the air. We are adding additional monitoring to this server as it's the first error it has had in 9 months.

We identified the issue as a network event with AWS Vendor code not recovering well from the event. We've logged an internal ticket to investigate if we can recover automatically from this in the future.

Weekend updates to handle bot clicks created a condition where the email event loading service was active and stalled simultaneously. The service was restarted and the backlog has finished processing the data. Additional monitoring for the scenario was added to our internal Ops dashboard.