CivicClerk Service Disruption | 9/4/2018
Completed
CivicPlus Systems Engineers are investigating a service disruption affecting your CivicPlus solution CivicClerk. Our engineers are working with our hosting provider resolve this incident as quickly as possible following the following distribution from Azure. A status update will be added to this post by 8:00 AM CST.
Azure’s South Central Region US - Investigating Starting at 09:29 UTC on 04 Sep 2018 a subset of customers in South Central US may experience difficulties connecting to resources hosted in this region. At this time Engineers are investigating an issue with cooling in one part of the data center which caused a localized spike in temperature. Automated data center procedures to ensure data and hardware integrity went into effect when temperatures hit a specified threshold and critical hardware entered a structured power down process. The impact to the cooling system has been isolated and engineers are actively working to restore services. The next update will be provided as events warrant.
Comments
6 comments
-
Official comment
Microsoft Azure technicians have applied a majority fix and have restored service or services are beginning to see signs of recovery as mitigation efforts continue.
- Outage Details: Starting at 09:29 UTC on 04 Sep 2018, customers in South Central US may experience difficulties connecting to resources hosted in this region. Additionally, this issue impacts Azure Active Directory authentication, which may impact services in other regions.
- Root Cause: A severe weather event, including lightning strikes, occurred near one of the South Central US datacenters. This resulted in a power voltage increase that impacted cooling systems. Automated datacenter procedures to ensure data and hardware integrity went into effect and critical hardware entered a structured power down process.
- Further Review: Engineers have successfully restored power to the data center. Additionally, engineers have recovered a majority of the impacted network devices. While some services are starting to see signs of recovery, mitigation efforts are still ongoing.
-
CivicPlus engineers and hosting provider continue their investigation to identify and resolve this issue as quickly as possible. A status update will be added to this post by 8:30.
0 -
CivicPlus engineers and hosting provider continue their investigation to identify and resolve this issue as quickly as possible. A status update will be added to this post by 10:00.
0 -
Microsoft Azure, CivicClerk's hosting provider, is now in the process of restoring power to affected devices as part of the ongoing mitigation process. Some services may also be experiencing intermittent authentication issues due to downstream Azure Active Directory impact, and engineers are separately working on mitigation options for this also. A status update will be added to this post by 11:00 am.
0 -
CivicPlus engineers are working tirelessly with the hosting provider to resolve the current service disruption. A status update will be added to this post by 1:00 P.M.
0 -
CivicClerk has partially restored functionality. CivicPlus is currently testing and working with the hosting provider to ensure optimal performance. Once testing is completed a final post will be published. You can login to sites and navigate around Agendas and Events, but cannot open any associated files or attachments.
0
Please sign in to leave a comment.
Available Templates
No Change Update
CivicPlus Systems Engineers continue to work on resolving this issue as quickly as possible. An update will be added to this post when it is available.Timeline Identified
CivicPlus Systems Engineers have identified the issue and expect to have a resolution deployed by ESTIMATED TIMELINE (if not a specific date, include business hours / business days clarification).Fix Implemented
CivicPlus Systems Engineers have applied a fix and have restored service to your CivicPlus Solution. An incident report will be posted here within 24 business hours.