Widen Collective Search Issue.
Incident Report for Widen Collective
Postmortem

Overview

On September 22, 2019 at 5:38 PM CDT (UTC-5) our engineers were alerted of increasing delay in search updates and began investigating. After investigation our engineering team increased resources to handle the extra load to mitigate the issue. On September 23, 2019 at 7:30 AM CDT our engineers were again alerted of an increasing delay in search updates and began investigating. After investigations our engineering team found that increased load on the analytics cluster caused a delay in displaying new assets and metadata within search results. Our engineering teams implemented a fix to address this issue, restore expected functionality and began programmatically updating all customer indexes.

Root Cause

Increased load on the analytics cluster caused a delay in displaying new assets and metadata within search results.

What We’re Doing?

We are improving our alerting and identification processes around the analytics cluster so we are able to more quickly identify and mitigate the impact of this issue.

Posted 14 days ago. Oct 03, 2019 - 11:36 CDT

Resolved
We’ve resolved the issue that delays displaying new assets and metadata in search results. Contact support@widen.com if you continue to experience any ongoing issues.
Posted 14 days ago. Oct 03, 2019 - 11:34 CDT
Monitoring
We’ve resolved the recent reported issue and are monitoring to ensure stability.
Posted 24 days ago. Sep 23, 2019 - 14:59 CDT
Update
Our engineers have fixed the underlying issue and we're continuing work to resolve the delay in displaying new assets and metadata in search results.
Posted 24 days ago. Sep 23, 2019 - 12:14 CDT
Identified
We’ve identified the issue that is delaying metadata updates from displaying in search results, and we’re actively working on resolving it.
Posted 24 days ago. Sep 23, 2019 - 09:46 CDT
Investigating
We’re currently investigating an issue with a delay in displaying metadata updates in your search results. Rest assured, your changes have been added and will display in search results shortly.
Posted 24 days ago. Sep 23, 2019 - 08:11 CDT
This incident affected: API (REST v2), Assets & Templates, Portals, and Workflow.