Intermittent message send failure and trouble connecting

Incident Report for Discord

Resolved

Things look calm, all alarms have resolved on our end. We are going to close this out and continue our march towards Python 3 tomorrow... thanks for your patience, and our apologies for the interruptions!

Posted Jul 10, 2019 - 22:19 PDT

Monitoring

We've deployed fixes to the relevant systems and are monitoring. At this point, we believe everything is resolved and will not recur.

Posted Jul 10, 2019 - 22:03 PDT

Update

We have rolled back a problematic change to our workers (machines that run periodic/on-demand tasks) that were getting duplicated and causing stress to our underlying databases. We are still working through some downstream problems this has caused in our search system.

Posted Jul 10, 2019 - 21:46 PDT

Identified

We are having some intermittent issues with sending messages and logging in. We know what the cause is and we're working to remediate the issue now.

Posted Jul 10, 2019 - 21:27 PDT