Skip to main content
Attention customers within hurricane impacted areas:
Due to widespread impacts resulting from hurricanes, CivicPlus is committed to enhancing our resources to manage the anticipated increase in traffic and mitigate any potential disruptions. We will prioritize all assistance requests from affected areas to ensure we provide the highest level of support possible. Please be assured that we are here to assist you in any way we can during this challenging time. Your safety and satisfaction are our top priorities.

Web Central Status Page

Performance Issue Identified | April 29, 2023

Carl Bowen

CivicPlus monitoring has detected a possible performance issue that may be affecting your CivicPlus solution. CivicPlus Systems Engineers are investigating this alert and a timeline to resolution and next steps will be provided as soon as available.

0

Comments

4 comments

Date Votes
  • Carl Bowen

    CivicPlus Systems Engineers continue to work on resolving this issue as quickly as possible. An update will be added to this post when it is available.

    0
  • Carl Bowen

    CivicPlus Systems Engineers continue to work on resolving this issue as quickly as possible. An update will be added to this post when it is available.

    0
  • Carl Bowen

    Your CivicPlus Solution appears to be functioning normally again we are monitoring it closely. 

    0
  • Carl Bowen

    Shortly before 9:00AM CST the Release Engineering team in coordination Software Development started to apply SQL change scripts to client databases in an automated fashion. Running this script during the scheduled maintenance window was a prerequisite to address several client-reported issues. The change script had been tested for a single site without any issues or downtime. However, when the script was launched against all sites, it did cause an issue resulting in sites responding slowly or to stop responding.

    System Engineers and Datacenter Operations personal (SRE Team) were automatically alerted via our monitoring system. The SRE Team concluded that the issue was caused by this script and that there was no evidence of malicious activity.

    The script had additional actions that are normally removed from automated deployments. These additional actions were expensive operations that resulted in SQL servers having to execute many background tasks. These background database tasks resulted in performance issues. Restarting the SQL Service would resolve the issue temporarily.

    Sites experienced reoccurring random periods of slowness and various errors.  The rate at which these errors occurred had significantly slowed down by early afternoon. However, there were several sites that experienced these errors intermittently until around 8:30 PM CST.

    Processes are being reviewed to ensure more comprehensive review of SQL changes, testing, and execution.  These changes should help to identify elevated risk operations and identify ways to mitigate the scope and duration of any risks presented.

    We apologize for the inconvenience and strive to improve our processes and procedures to reduce the likelihood of a repeat of this incident.

    0

Available Templates

No Change Update

CivicPlus Systems Engineers continue to work on resolving this issue as quickly as possible. An update will be added to this post when it is available.

Timeline Identified

CivicPlus Systems Engineers have identified the issue and expect to have a resolution deployed by ESTIMATED TIMELINE (if not a specific date, include business hours / business days clarification).

Fix Implemented

CivicPlus Systems Engineers have applied a fix and have restored service to your CivicPlus Solution. An incident report will be posted here within 24 business hours.

Please sign in to leave a comment.