Skip to main content
Attention customers within hurricane impacted areas:
Due to widespread impacts resulting from hurricanes, CivicPlus is committed to enhancing our resources to manage the anticipated increase in traffic and mitigate any potential disruptions. We will prioritize all assistance requests from affected areas to ensure we provide the highest level of support possible. Please be assured that we are here to assist you in any way we can during this challenging time. Your safety and satisfaction are our top priorities.

Web Central Status Page

Issue Identified | October 12 2022

Carl Bowen

CivicPlus monitoring has detected a possible issue that may be affecting your CivicPlus solution. CivicPlus Systems Engineers are investigating this alert and a timeline to resolution and next steps will be provided as soon as available.

0

Comments

4 comments

Date Votes
  • Official comment
    Alaina Coffee
    • Edited

    Root Cause Analysis for Outage on 10/12/2022

    At approximately 4AM CST, CivicPlus' 3rd Party Hosting Provider was alerted to an infrastructure disruption. Investigation into the disruption determined that the underlying infrastructure was experiencing connectivity issues with the storage. Initial investigation didn't show any obvious hardware failures. Engineers spent the next 3 hours diagnosing the issue. At approximately 7:00AM CST, engineers were able to identify the issue and work to restore services.

    This service disruption appears to be related to a software bug and the data center has opened a ticket with the vendor manufacture to diagnose the issue. The data center is also evaluating additional options to build additional resiliency and redundancy into the switch infrastructure. We will continue to work with the data center on any action items resulting from this outage.

    Technical Details:

    At approximately 4AM CST, CivicPlus' 3rd Party Hosting Provider was alerted to an infrastructure disruption. Investigation into the disruption revealed a partial Storage Area Network (SAN) connectivity issue to the Virtual Infrastructure. Initial investigation didn't show any obvious hardware failures and the SAN network design is N+1 throughout. Engineers spent the next 3 hours diagnosing the issue. At approximately 7AM CST, engineers identified that the core switching network was not passing traffic to the SAN. At approximately 7:15AM CST, engineers cleared the core switch cluster, and rebooted them one at time. Following the reboot, traffic resumed and was flowing as expected.

  • Carl Bowen

    The Issue has been identified and a fix is being worked on.

    0
  • Carl Bowen

    Fix Has been implemented the impacted Sites are recovering. 

    We are closely monitoring as things recover. 

    A Root cause analsys  is being initiated we will post that information here once it is complete

    0
  • Carl Bowen

    All sites are operating normally at this time.

    0

Available Templates

No Change Update

CivicPlus Systems Engineers continue to work on resolving this issue as quickly as possible. An update will be added to this post when it is available.

Timeline Identified

CivicPlus Systems Engineers have identified the issue and expect to have a resolution deployed by ESTIMATED TIMELINE (if not a specific date, include business hours / business days clarification).

Fix Implemented

CivicPlus Systems Engineers have applied a fix and have restored service to your CivicPlus Solution. An incident report will be posted here within 24 business hours.

Please sign in to leave a comment.