Skip to main content
Attention customers within hurricane impacted areas:
Due to widespread impacts resulting from hurricanes, CivicPlus is committed to enhancing our resources to manage the anticipated increase in traffic and mitigate any potential disruptions. We will prioritize all assistance requests from affected areas to ensure we provide the highest level of support possible. Please be assured that we are here to assist you in any way we can during this challenging time. Your safety and satisfaction are our top priorities.

Meetings Select Status Page

Performance Issue Identified | November 21, 2022

Raul

CivicPlus monitoring has detected a sporadic performance issue that may be affecting your CivicPlus solution. CivicPlus Systems Engineers are investigating this alert and a timeline to resolution and next steps will be provided as soon as available.

0

Comments

6 comments

Date Votes
  • Official comment
    DeAun Michael

    Incident Report 

    Incident Description
    Sporadic reports were received of CivicClerk V8 Admin sites not being responsive and hung when trying to load the page early last week.

    Cause
    Some CivicClerk V8 Admin sites were failing to connect to the SQL server that hosts our databases. These database connections were waiting for a response from the SQL server and eventually timed out and threw an error, causing a few clients to be unable to load their sites. The connection to the SQL server appeared to be failing from a Network Firewall Rule on the Azure Virtual Machine that hosts our SQL Server. This firewall rule has been used for months and was not recently added. We were not able to identify why the Azure Virtual Machine suddenly stopped processing this rule.

    Resolution
    This specific Network Firewall Rule was being used by our Azure DevOps pipeline agents to release SQL changes to our client databases. We resolved this issue by removing this Network Firewall Rule and the connections to the SQL Server started working immediately and CivicClerk sites began loading.

    Preventative Measures
    Since the incident, we have created a self-hosted Azure DevOps agent that is now used to release our SQL changes so that Network Firewall Rule is no longer needed.

  • Raul

    CivicPlus Systems Engineers continue to work on resolving this issue as quickly as possible. An update will be added to this post when it is available.

    0
  • Raul

    CivicPlus Systems Engineers have applied a fix and are continuing to monitor the reported issue to verify if the applied fix has restored services to your CivicPlus Solution. An update will be added to this post when it is available.

    0
  • Raul

    CivicPlus Systems Engineers continue to work on resolving this issue as quickly as possible. An update will be added to this post when it is available.

    1
  • Raul

    CivicPlus Systems Engineers have applied a fix and are continuing to monitor the reported issue to verify if the applied fix has restored services to your CivicPlus Solution. An update will be added to this post when it is available.

    0
  • Raul
    CivicPlus Systems Engineers have monitored the applied fix and service has been restored to your CivicPlus Solution. We are working with CivicPlus Systems Engineers to document an incident report. An update will be added to this post when it is available.
    0

Available Templates

No Change Update

CivicPlus Systems Engineers continue to work on resolving this issue as quickly as possible. An update will be added to this post when it is available.

Timeline Identified

CivicPlus Systems Engineers have identified the issue and expect to have a resolution deployed by ESTIMATED TIMELINE (if not a specific date, include business hours / business days clarification).

Fix Implemented

CivicPlus Systems Engineers have applied a fix and have restored service to your CivicPlus Solution. An incident report will be posted here within 24 business hours.

Please sign in to leave a comment.