Widen Collective Search Issue.
Incident Report for Widen Collective
Postmortem

Overview

On July 29, 2019, at 3:50 PM CDT (UTC-5) we were alerted of increased errors with our search cluster. After investigation our engineering teams found that increased memory consumption on a search cluster caused search queries to slow down or fail for a subset of customers. At 5:59 PM CDT (UTC-5) all search functionality was restored to all customers affected.

Root Cause

Increased memory consumption on a search cluster caused search queries to slow down or fail for a subset of customers.

What We’re Doing?

Our engineers are deploying a fix to mitigate this issue moving forward and we’re refactoring our current search cluster configuration to improve efficiency and reliability.

Posted 3 months ago. Jul 31, 2019 - 16:38 CDT

Resolved
We’ve resolved the issue affecting the availability of search on some sites. Contact support@widen.com if you continue to experience any ongoing issues.
Posted 3 months ago. Jul 29, 2019 - 20:47 CDT
Monitoring
We’ve resolved the recent reported search issue and are monitoring to ensure stability.
Posted 3 months ago. Jul 29, 2019 - 17:59 CDT
Update
We continue working to resolve the issue affecting the availability of search on some sites.
Posted 3 months ago. Jul 29, 2019 - 16:36 CDT
Identified
We’ve identified the issue affecting the availability of search on some sites, and we’re actively working on resolving it.
Posted 3 months ago. Jul 29, 2019 - 16:05 CDT
Investigating
We're aware of an issue affecting the availability of search on some sites and are actively investigating.
Posted 3 months ago. Jul 29, 2019 - 15:58 CDT
This incident affected: API (REST v2), Assets & Templates, and Portals.