Performance Issue Identified | Auto-Billing | February 11, 2022
Completed
CivicRec monitoring has detected a performance issue with auto-billing that may be affecting your CivicRec solution. CivicRec Systems Engineers are investigating this issue and a timeline to resolution and next steps will be provided as soon as available.
Comments
5 comments
-
Official comment
Timeline Update
CivicRec Systems Engineers have identified the issue and working through a release timeline. An update will be added to this post when the timeline is available.
-
Fix Implemented 2/21/2022
CivicRec System Engineers have deployed a permanent fix for the auto-billing issue. Auto-billing service has been restored to your CivicRec Solution. An incident report will be posted here within 24 business hours.
1 -
No Change Update
CivicRec Systems Engineers continue to work on resolving this issue as quickly as possible. An update will be added to this post when it is available.
0 -
Timeline Update:
CivicRec Systems Engineers have deployed a temporary fix for the auto-billing issue. We will monitor the progress over the weekend and expect a full resolution to be deployed Monday February 21st, 2022 if no issues are discovered.
0 -
CivicRec Auto-Billing Issue Incident Report
February 21, 2022
Incident Description
Several multi-tenant CivicRec customers who utilize auto-billing for credit card processing, were seeing an increase in declined credit cards. The auto-billing process was failing after about 4 minutes into the billing process.
Cause
As part of core stability work, CivicRec System Engineers recently refactored a large portion of the codebase to to an open-source project. When the codebase was first released, the feature on to reuse TCP sockets and multiplexing requests across a single socket was not turned on. At scale, these connections accumulated so quickly that they exceeded the hard limit imposed by the operating system causing the auto-billing process to fail at around 500,000 connections.
Resolution
A fix was identified and implemented on 02/22/2022 at 1:45pm central time. Use a singleton pattern to share a single instance (and any open tcp sockets) amongst all the callers of the library code.
0
Please sign in to leave a comment.
Available Templates
No Change Update
CivicPlus Systems Engineers continue to work on resolving this issue as quickly as possible. An update will be added to this post when it is available.Timeline Identified
CivicPlus Systems Engineers have identified the issue and expect to have a resolution deployed by ESTIMATED TIMELINE (if not a specific date, include business hours / business days clarification).Fix Implemented
CivicPlus Systems Engineers have applied a fix and have restored service to your CivicPlus Solution. An incident report will be posted here within 24 business hours.