CivicEngage Central: September 12, 2019 - CivicMedia Service Disruption



  • Current Update
    Raul Winslow

    CivicPlus Systems Engineers and TikiLive have applied a fix and have restored service to your CivicPlus Solution. Currently, TikiLive has identified the issue as being a DNS Error and have made the appropriate changes to restore the service. However, this may take time due to DNS Propagation. An incident report will be posted here within 24 hours.

    Comment actions Permalink
  • Raul Winslow

    Incident Report:

    • Outage Duration: 8 hours

    • Root Cause: Server Side IP updates on TikiLIVE that lead to a DNS Routing issue with unexpected results. DNS routing issue failed to connect web-traffic to web-service hosting.

    • Further Review: TikiLIVE made the necessary changes to their DNS to mitigate the issue caused by the Server Side IP Updates. However, their record was negatively cached in our servers, as well as servers from Organizations who had previously used or visited TikiLIVE API Endpoints via CivicSend. CivicPlus Engineers have flush the DNS Cache on the Upstream server, as well as WebServer where sites are hosted, to clear out the negatively cached DNS Record. This caused the service to resume normal operation. DNS Redundancy through TikiLIVE Name Servers have been added, as well as added additional monitoring services to detect the specific routing issue in real time in the future and will send an alert to system admins should a similar event recur.

    Users should now be able to resume normal site usage.

    Comment actions Permalink

Available Templates

No Change Update

CivicPlus Systems Engineers continue to work on resolving this issue as quickly as possible. An update will be added to this post when it is available.

Timeline Identified

CivicPlus Systems Engineers have identified the issue and expect to have a resolution deployed by ESTIMATED TIMELINE (if not a specific date, include business hours / business days clarification).

Fix Implemented

CivicPlus Systems Engineers have applied a fix and have restored service to your CivicPlus Solution. An incident report will be posted here within 24 business hours.