The secondary that previously failed has successfully snuggled up near a warm fire with a cozy blanket and some hot cocoa and is now adequately warm to serve traffic at nominal latency.
Posted 8 months ago. Oct 04, 2018 - 11:35 PDT
The secondary appears to be sufficiently warm now, and we are noticing error rates and latency drop to nominal levels.
Posted 8 months ago. Oct 04, 2018 - 11:31 PDT
We are aware of issues regarding higher than normal latency, and failed message sends. The team is aware of the root cause of the issue (a host error on one of our database secondaries caused it to reboot).
During this time, the secondary is "cold" as it loads data into memory. We are monitoring the instance as it warms up, and expect latency to fully recover in 5-15 minutes, and latency to continue to as the secondary gradually warms up.