Xerox Workplace Cloud has been experiencing intermittent issues in the web services layer impacting multiple operations. This may manifest as authentication errors or job submission and release errors. We are actively monitoring, working with Microsoft, and have put mitigation efforts in place to lessen the impact while we work on the root cause and longer-term resolutions.
Update 11:00 AM EDT: Engineering identified a potential cause of the intermittent request failures and a patch was applied on the XWC web services nodes at 3 AM EDT. Services, since the patch was applied, have returned to normal. We continue to monitor the situation to assess the completeness of the patch.
Update September 20: Patch applied was successful in resolving the intermittent issue. The issue was a result of the software update on 9/11. Under production load this resulted in IIS holding the available IIS connections open and eventually exhausting all resources. This issue was not seen in lower environments during testing and validation. In addition to improvements in pre-production validation, additional logging and monitoring has been added across XWC to allow operations to proactively address such a recurrence of a non-responsive node going forward.
"Innovating how the world communicates, connects and works."