On September 22, 2019 at 5:38 PM CDT (UTC-5) our engineers were alerted of increasing delay in search updates and began investigating. After investigation our engineering team increased resources to handle the extra load to mitigate the issue. On September 23, 2019 at 7:30 AM CDT our engineers were again alerted of an increasing delay in search updates and began investigating. After investigations our engineering team found that increased load on the analytics cluster caused a delay in displaying new assets and metadata within search results. Our engineering teams implemented a fix to address this issue, restore expected functionality and began programmatically updating all customer indexes.
Increased load on the analytics cluster caused a delay in displaying new assets and metadata within search results.
We are improving our alerting and identification processes around the analytics cluster so we are able to more quickly identify and mitigate the impact of this issue.