Data Saving Issue
Valued customers, as many of you probably noticed, we had an issue last Friday that may have caused some disruption to your organization. We sincerely apologize for the trouble & inconvenience this caused. We always strive to put our customers first and promise to do better. With that in mind I’d like to provide some background information on what happened.
We are in the middle of a major initiative that will more than double our server capacity, upgrade our underlying database server technology, and increase our network capacity – all steps that will provide an improved environment for delivering ChamberMaster/MemberZone. Because we have over 3,100 customers, including in other countries, we can’t simply take our servers offline for a day to make all these changes. Our business requires making major IT upgrades while trying to minimize the disruption to our customers and keep you up and running.
Recently we’ve had more disruptions from these upgrades than we anticipated. At times we were a bit too aggressive at making changes that we thought were safe. We learned our lesson. And in a few cases we’ve simply had some bad luck and have run into bugs and problems with 3rd party software. In the end it is still our issue and we have learned from this and can’t make our problems your problems.
The following may be more technical information than you care about, but in the spirit of full transparency, we want to provide the full context. One of the upgrades we are making is to improve the redundancy in our underlying database technology, SQL Server. Besides upgrading to SQL Server 2017 we implemented a real-time failover feature called Availability Groups. On Friday we experienced some issues where the server inadvertently failed over to a read-only backup for approximately an hour. The software appeared to be working fine and would allow users to do most everything. However, data wasn’t getting saved to the database because it was in read-only mode. So approx. 2,000 customers could have data that didn’t get saved for a period of time on Friday, June 22. We are digging through log files to try to recover as much of this as feasible, but much of it is not likely to be recoverable. As we learn more we will contact the customers that were directly affected. Most of you probably noticed things weren’t saving and that we were experiencing issues. But some of your members may not have and could have been trying to use the system. We know that some credit card payments worked fine but didn’t get recorded into your system – we should have those recovered in the next day or two.
We are sincerely sorry for these issues; I know it is very frustrating when these things happen. As an online software provider we use as much online software as we can, and it is annoying when that software is down or isn’t working. Thank you for your understanding and support as we continue to improve to serve you better.
Scott Juranek, CEO of GrowthZone (provider of ChamberMaster and MemberZone)