On October 10th, 2023, we received alerts indicating an abnormal number of errors coming from Portals, and began investigating. We soon found the root of the issue and were able to address it. We continued to monitor Portals and search performance to ensure the issue did not return.
A backend service placed high load on search infrastructure, leading to degraded performance in Portals and Assets.
The backend service has been updated to prevent this type of issue from re-occurring. Additionally, updates have been made to search infrastructure to reduce the impact on Portals when issues occur.