Performance Issue Identified | October 24, 2022
CivicPlus monitoring has detected a possible performance issue that may be affecting your CivicPlus solution. CivicPlus Systems Engineers are investigating this alert and a timeline to resolution and next steps will be provided as soon as available.
Comments
2 comments
-
Official comment
CivicPlus Incident Report: Service Disruption on 10/24/2022
Incident Description
At 3:20 PM CDT internal staff received reports that both internal and customer users were unable to sign in and/or load the CivicPlus Organization service. A team of CivicPlus product engineers began investigating the issue at 3:36 PM and was able to uncover the root cause of the issue.Cause
It was determined that a connection error with a system queuing service that exists to synchronize data between the CivicPlus Authentication and Organization services had become overloaded, causing both systems to become unresponsive. As a result, users were unable to authenticate or load the organization service until resolved.Resolution
CivicPlus engineers restarted the Authentication and Organization services and restored service for the affected customers around 4:13pm CDT on 10/24/2022.Preventative Measures
CivicPlus Engineers identified two performance enhancements to improve the resiliency of the queuing system in the event of a future overload. The team is presently working to include additional logging to more easily identify issues as they occur and to refactor the synchronization code itself to ensure a failure in Organizations does not impact Authentication in this same way moving forward.We believe these steps will adequately address the root cause of this incident and no further updates for this incident are planned.
-
CivicPlus Systems Engineers have applied a fix and have restored service to your CivicPlus Solution. An incident report will be posted here within 24 business hours.
0
Please sign in to leave a comment.
Available Templates
No Change Update
CivicPlus Systems Engineers continue to work on resolving this issue as quickly as possible. An update will be added to this post when it is available.Timeline Identified
CivicPlus Systems Engineers have identified the issue and expect to have a resolution deployed by ESTIMATED TIMELINE (if not a specific date, include business hours / business days clarification).Fix Implemented
CivicPlus Systems Engineers have applied a fix and have restored service to your CivicPlus Solution. An incident report will be posted here within 24 business hours.