Skip to main content

Performance Issue Identified | Auto-Billing | February 11, 2022

Completed

Comments

5 comments

  • Official comment
    Rachel Ebaugh

    Timeline Update

    CivicRec Systems Engineers have identified the issue and working through a release timeline. An update will be added to this post when the timeline is available.

  • Rachel Ebaugh

    No Change Update

    CivicRec Systems Engineers continue to work on resolving this issue as quickly as possible. An update will be added to this post when it is available.

    0
  • Rachel Ebaugh

    Timeline Update:

    CivicRec Systems Engineers have deployed a temporary fix for the auto-billing issue. We will monitor the progress over the weekend and expect a full resolution to be deployed Monday February 21st, 2022 if no issues are discovered.

    0
  • Rachel Ebaugh

    Fix Implemented 2/21/2022

    CivicRec System Engineers have deployed a permanent fix for the auto-billing issue. Auto-billing service has been restored to your CivicRec Solution. An incident report will be posted here within 24 business hours.

    1
  • Rachel Ebaugh

    CivicRec Auto-Billing Issue Incident Report 

    February 21, 2022 

    Incident Description  

    Several multi-tenant CivicRec customers who utilize auto-billing for credit card processing, were seeing an increase in declined credit cards. The auto-billing process was failing after about 4 minutes into the billing process. 

     

    Cause  

    As part of core stability work, CivicRec System Engineers recently refactored a large portion of the codebase to to an open-source project. When the codebase was first released, the feature on to reuse TCP sockets and multiplexing requests across a single socket was not turned on. At scale, these connections accumulated so quickly that they exceeded the hard limit imposed by the operating system causing the auto-billing process to fail at around 500,000 connections. 

    Resolution  

    A fix was identified and implemented on 02/22/2022 at 1:45pm central time. Use a singleton pattern to share a single instance (and any open tcp sockets) amongst all the callers of the library code. 

     

    0

Available Templates

No Change Update

CivicPlus Systems Engineers continue to work on resolving this issue as quickly as possible. An update will be added to this post when it is available.

Timeline Identified

CivicPlus Systems Engineers have identified the issue and expect to have a resolution deployed by ESTIMATED TIMELINE (if not a specific date, include business hours / business days clarification).

Fix Implemented

CivicPlus Systems Engineers have applied a fix and have restored service to your CivicPlus Solution. An incident report will be posted here within 24 business hours.

Please sign in to leave a comment.