CivicEngage: September 12, 2019 - CivicMedia Service Disruption

Completed
2

Updates

  • Current Update

    CivicPlus Systems Engineers and TikiLive have applied a fix and have restored service to your CivicPlus Solution. Currently, TikiLive has identified the issue as being a DNS Error and have made the appropriate changes to restore the service. However, this may take time due to DNS Propagation. An incident report will be posted here within 24 hours.

    Comment actions Permalink
  • Incident Report:

    • Outage Duration: 8 hours

    • Root Cause: Server Side IP updates on TikiLIVE that lead to a DNS Routing issue with unexpected results. DNS routing issue failed to connect web-traffic to web-service hosting.

    • Further Review: TikiLIVE made the necessary changes to their DNS to mitigate the issue caused by the Server Side IP Updates. However, their record was negatively cached in our servers, as well as servers from Organizations who had previously used or visited TikiLIVE API Endpoints via CivicSend. CivicPlus Engineers have flush the DNS Cache on the Upstream server, as well as WebServer where sites are hosted, to clear out the negatively cached DNS Record. This caused the service to resume normal operation. DNS Redundancy through TikiLIVE Name Servers have been added, as well as added additional monitoring services to detect the specific routing issue in real time in the future and will send an alert to system admins should a similar event recur.

    Users should now be able to resume normal site usage.

    Comment actions Permalink