On July 12th, 2023, our engineers received alerts that some sites were unavailable and immediately began looking into the issue. After discovering and addressing the root cause, they were able to restore access to the impacted sites within a few minutes.
A rapid increase in storage usage in a short period of time caused several databases to become unresponsive. This issue was exacerbated when automated systems were unable to increase the storage allocation fast enough to mitigate the issue. Engineers collaborated with our third party vendor to mitigate the issue and restore access to the impacted sites.
We are continuing to investigate the underlying cause of the rapid storage increase. We have also added additional alerting and monitoring in the impacted area.