IT directors and others responsible for technology aren’t usually given to exaggeration. In many areas of business, the phrase “major incident” is a dramatic way of moaning about whatever’s creating stress this week. But in an IT director’s lexicon, “major incident” means “having one of the worst days of my entire career.”
That’s what one Netfor client experienced on a recent Saturday evening. In addition to its headquarters staff, the company employs several thousand people at nearly a dozen production sites nationwide. And shortly before 8:00 p.m. that evening, all the technology operating within and connecting those locations just stopped working. The culprit was the Cryptolocker ransomware, which plunged the company into chaos.
Netfor had long provided Tier One after-hours support for the company, so the IT leader immediately called our Technical Account Manager (TAM) to alert her to the problem. The company’s support team was scrambling to restore operations, so Netfor needed to be ready to handle a flurry of support calls. They also asked Netfor to begin providing full 24/7 support by end of day Monday so their internal team could concentrate on recovery.
The company’s situation was desperate, to say the least, and Netfor was eager to help a valued client get back up to speed. The TAM reached out to Netfor’s top management team, who jumped on an early Sunday-morning call to craft a plan for transitioning to 24/7 support on a significantly faster-than-normal timeline. As the plan came together, the TAM collaborated with her client contact to gather specifics about where Netfor should focus its efforts. That was critical because we had been delivering support on several thousand devices, from solving problems with networked large equipment to walking employees through updates to their Outlook accounts.
Always There When You Need Us
We spent several hours talking with the company’s leadership team to get a better understanding of the pain they were experiencing. Because we had only been providing after-hours support, we weren’t entirely certain of what the daily volume of service would look like. To increase our flexibility, we encouraged some agents to work overtime and shifted other resources around to ensure we could provide sufficient support to the company without impacting the service levels our other clients count on. Ultimately, our role became helping the company bring the chaos under control by stepping in and helping their end-users.
While the Netfor team wasn’t directly involved in data recovery (aside from helping the company retrieve some files from locked-down servers), we assisted with the cleanup and ongoing support. As a result of the incident, the company stepped up its cybersecurity, and we guided end-users through the installation process. The company’s leaders saw how our step-by-step approach and knowledge base worked with every system.
Eradicating the ransomware and getting all the systems and networks back to normal took about two weeks. When the work was complete, the company’s leadership asked Netfor to continue the 24/7 support on a permanent basis. They had been impressed with our team’s adaptability and willing to provide enhanced support when they needed it most. They also recognized the inherent flexibility of our services and their ability to grow as the company’s requirements changed.
Most of all, they saw that when their pain was at its most intense, the Netfor team responded immediately to help them get through the crisis. By making their problem our problem, we helped them get back to business in less time and with less lost revenues. It’s why Netfor clients truly consider us more of a partner than a vendor.