All SPOCs and Technical Contacts:
July 22, 2024 UPDATE: All of the gateway servers have been restored. All that remains is to restore all the CWS/CMS workstations. Unfortunately, each workstation must be restored individually.
The CWDS Infrastructure team and IBM have developed a fix for the CWS/CMS Dedicated County workstations which will require the use of a bootable USB drive. The instructions are being drafted and will be sent out later this morning. Co-Existent Counties may use this same process to restore connection to their workstations.
Dedicated Counties: Follow the instructions, once received, to create a bootable USB drive. If unable to create the bootable USB drive, you can request a copy of it from the IBM Help Desk and it will be mailed to you using next‑day shipping. Then, use the bootable USB drive on each CWS/CMS workstation to remove the files which are preventing the CWS/CMS workstations from correctly connecting to CWS/CMS.
In addition, a WebEx conference call is being set up for counties to assist with how to create and use the bootable USB drives and to receive updates on the status of connected workstations in each Dedicated County. The meeting will focus on assisting Dedicated Counties, but Co-Existent Counties can join the call and listen to the instructions. We encourage all Dedicated County Technical Contacts to join the call. We apologize for the short turnaround. Here is the logon information for the meeting:
Time: 11:00 AM – 12:00 Noon
Join from the meeting link:
https://osi-ca.webex.com/osi-ca/j.php?MTID=m65ab2e8f5f7011527ab1839d6e489a63
Join by meeting number:
Meeting number (access code): 2541 692 4576
Meeting password: hYNamCuA823
Tap to join from a mobile device (attendees only)
+1-415-655-0001,,25416924576## US Toll
Join by phone
+1-415-655-0001 US Toll
An additional WebEx session will be set up to assist with troubleshooting with implementing the Microsoft fix: from 1:00 to 3:00 PM. Here is the WebEx information for the troubleshooting call:
Time: 1:00 PM – 3:00 PM
Join from the meeting link
https://osi-ca.webex.com/osi-ca/j.php?MTID=m7cc56df578e7dfbf450297629737f61d
Join by meeting number
Meeting number (access code): 2546 841 7948
Meeting password: DXt4DZmf744
Tap to join from a mobile device (attendees only)
+1-415-655-0001,,25468417948## US Toll
Join by phone
+1-415-655-0001 US Toll
____________________________________________________________________________________________
July 19, 2024 UPDATE: The CWDS Infrastructure team and IBM continue to work with the California Department of Technology (CDT) to resolve this issue. Multiple gateway servers have been restored and CWS/CMS transactions are being reported. However, there is no current estimate of when the system will be fully restored and all CWS/CMS servers and workstations are connected and up.
There are no reported issues with CARES-Live or CWS-CARES RFA connection issues.
Additional updates will be sent out as new information is received.
_________________________________________________________________________________
A major incident has affected many systems worldwide. Crowdstrike performed a code update on Thursday, July 18, 2024, which has caused Windows workstations and servers to crash. Impacted systems include CWS/CMS, CARES-Live, and Microsoft Windows. All Dedicated Counties are affected. All Co-Existent Counties which use Crowdstrike are also affected. Crowdstrike has implemented a fix, but actions are required to recover the affected systems.
The CWDS Infrastructure team and IBM are working with the California Department of Technology (CDT) to implement the fix. They started with the CWS/CMS application’s servers. Then they are working on the workstations.
Some servers and workstations use BitLocker encryption and require a recovery key. A solution for these encrypted items is being developed.
Additional updates will be sent out as new information is received.
Action Required:
CWS/CMS Users
If users experience any issues with CWS/CMS, please follow your county process for reporting issues, or you may contact the IBM Boulder Help Desk at 1-800-428-8268.
CARES-Live Users
Please follow your county process for reporting issues. CARES-Live County Administrators may contact the CARES Service Desk through the CWDS ServiceNow Customer Support Management (CSM) Portal. Alternate methods are phoning 1-855-292-3444 or via email at Servicedesk@cwds.ca.gov during normal business hours. You may also email after hours, and a Service Desk representative will contact you the next business day.
Major Incident Status – Multiple Services Unavailable - UPDATE
If unable to find a specific Bulletin, please contact Communications.