On August 10th, 2023, we received reports that some customers were seeing one of a number of "Bad Gateway" error messages when attempting to load pages in DAM. In some cases refreshing the page would temporarily solve the issue. After investigation we found an issue with an update to an internal service. We released a fix for this internal service to mitigate issues users were seeing, and engineers continued to monitor the state of the service until they had confirmed the errors had subsided.
Multiple servers were generating an increased number of error messages. This meant that when attempting to load a page, if the request was handled by a server having issues, customers would see one of the error messages. Requests handled by other servers were not impacted.
To resolve the issue, we made multiple updates to our servers. In addition, we're making significant changes to the impacted service to reduce the frequency of this issue moving forward.