March 21, 2025
Communications
All SPOCs and Technical Contacts:
Effective March 27, 2025, CARES-Live 2.8.33 Release is Scheduled to Move into Production
The CARES-Live team has scheduled release activity that will interrupt connectivity to the CARES-Live application. The release activity will begin at 6:00 PM on Thursday, March 27, 2025, and conclude at 9:00 PM. During this release, the CARES-Live team will focus on application changes, and users will not be able to access the CARES-Live application during this scheduled outage.
Users will not be able to access the CARES-Live application during this scheduled outage.
CARES-Live 2.8.33:
- This release addresses security findings and Java upgrades and other infrastructure updates and security patches not visible to CARES-Live users.
Action Required
If users experience any issues with CARES-Live, please follow your county process for reporting issues. CARES-Live County Administrators may contact the CARES Service Desk through the CWDS ServiceNow Customer Support Management (CSM) Portal. Alternate methods are phoning 1-855-292-3444 or via email at Servicedesk@cwds.ca.gov during normal business hours. You may also email after hours and a Service Desk representative will contact you the next business day.
December 06, 2024
Communications
All SPOCs, Technical Contacts, and CAD Contacts:
December 6, 2024 UPDATE: The CWDS Office and IBM have scheduled an outage for the Business Objects (BO) server to update the branding on the Business Intelligence login screen. Following the update, the branding will return to the normal CWDS branding instead of the generic SAP screens. The scheduled BO maintenance outage will be on Tuesday, December 10, 2024, beginning at 4:00 PM and concluding at 4:30 PM.
BO will not be available during the scheduled outage.
The CWDS Office and IBM continue to work with the vendor on the issue regarding transferring reports to and from the Rich Client server. An additional update will be provided once the fix for that issue has been determined.
_________________________________________________________________________________
November 21, 2024 UPDATE: After working with the Business Objects vendor, IBM and the CWDS Infrastructure team are reporting that Business Objects is now up and operational, with a few limited issues / differences that remain to be resolved.
Please Note: The login screens will look a bit different. Generic SAP screens will be used temporarily and will be updated at a later date (please see screenshot below).
In addition, there appears to be an issue with transferring reports from Rich Client to the server and back. IBM and the Infrastructure team will work with the vendor as a fix for this specific continuing issue becomes available.
Screenshot of the generic login screen:
__________________________________________________________________________________________
November 20, 2024 UPDATE: Several issues have been identified after the installation of the Business Objects patch upgrade. Work is taking place now to resolve those issues and a communication will be shared once resolved.
_______________________________________________________________________
November 20, 2024 FINAL UPDATE: The installation of the Business Objects patch upgrade has completed. Business Objects is now accessible; however, users may experience degraded performance while the Business Objects server catches up on the scheduled jobs from this morning. The CWDS Infrastructure team and IBM are monitoring for potential issues.
NOTE: Web Intelligence Rich Client users may need to update their workstation installation of the client.
_______________________________________________________________________
November 20, 2024 UPDATE: This is a reminder that the Business Objects production environment is not available this morning, to install a patch update. A follow-up communication will be sent out once the environment has become available, estimated to occur at around 12:00 Noon.
________________________________________________________________________
UPDATE: We are applying a patch to address this Business Objects issue, which will be implemented in both the Training and Production Environments. These environments will be unavailable during the following patch periods:
- Training Environment - November 19th, 8:00 AM to 12:00 PM
- Production Environment - November 20th, 8:00 AM to 12:00 PM
Action Required
If users experience any issues please follow your county process for reporting issues or you may contact the IBM Boulder Help Desk at 1-800-428-8268.
_____________________________________________________________________________
An issue has been discovered with the current version of Business Objects used with the CWS/CMS application.
Issue Description:
Whenever a numeric field, such as government entity code, is used as a condition in a data provider (query), the query will run correctly the first time. Whenever the data provider is edited, the system will return a type mismatch error and require the user to remove and re-add the condition. The query will then run, as long as it is not edited again.
This only appears to be an issue with numeric fields; character fields, such as Client Name, do not experience this issue. Once the query is run, it operates normally as long as the user does not go back and edit the data provider. The user can refresh the data and save, as well as edit the report formatting, without issue.
Issue Resolution:
A patch level update will be required to resolve the issue. The CWDS Office and IBM are evaluating the next Business Objects patch to determine if it will resolve the issue, without generating any new issues. There is currently no estimated time of when the patch will be implemented. Additional updates will be sent out as new information is received.
Workaround:
When Business Objects informs the user of a type mismatch error after a numeric field is used as a condition in a data provider, and the data provider is edited, then the user must remove and re-enter the condition. The query will run, without issue, as long as the data provider is not edited again.
Action Required
Until the issue is corrected with the appropriate patch update, users will need to perform the workaround listed above, by removing and re-entering the condition when they receive a type mismatch error message.
October 14, 2024
Communications
All SPOCs and Technical Contacts:
The California Department of Technology (CDT) reported a major incident, impacting multiple applications and services, caused by a CDT network outage. CDT has investigated the issue and has restored the services. There have not been any reported issues with CWS/CMS, CARES-Live, or CWS-CARES applications. There were reports of issues with connecting to the CDT Virtual Private Network and possibly to Citrix Server Based Computing (SBC).
Action Required
CWS/CMS Users
If users experience any issues with CWS/CMS, or with using VPN or SBC, please follow your county process for reporting issues, or you may contact the IBM Boulder Help Desk at 1-800-428-8268.
CARES-Live Users
Please follow your county process for reporting issues. CARES-Live County Administrators may contact the CARES Service Desk through the CWDS ServiceNow Customer Support Management (CSM) Portal. Alternate methods are phoning 1-855-292-3444 or via email at Servicedesk@cwds.ca.gov during normal business hours. You may also email after hours, and a Service Desk representative will contact you the next business day.
August 02, 2024
Communications
All SPOCs and Technical Contacts:
August 2, 2024 UPDATE: The cause of yesterday’s CWS/CMS outage was found and fixed. A network interface located at the Gold Camp data center was faulty. It was repaired around 9:00 PM last night.
The issue has been resolved and connectivity to the CWS/CMS application is expected to be fully restored. If a county continues to experience an issue with connecting to CWS/CMS, please contact the IBM Help Desk at 800-428-8268 for assistance.
______________________________________________________________________
The CWDS Office has received multiple reports of an outage impacting the CWS/CMS application. The CWDS teams are working with the California Department of Technology (CDT) on a resolution. Additional updates will be sent out as new information is received.
Action Required
CWS/CMS
If users experience any issues with CWS/CMS, please follow your county process for reporting issues or you may contact the IBM Boulder Help Desk at 1-800-428-8268.
July 22, 2024
Communications
All SPOCs and Technical Contacts:
July 22, 2024 UPDATE: All of the gateway servers have been restored. All that remains is to restore all the CWS/CMS workstations. Unfortunately, each workstation must be restored individually.
The CWDS Infrastructure team and IBM have developed a fix for the CWS/CMS Dedicated County workstations which will require the use of a bootable USB drive. The instructions are being drafted and will be sent out later this morning. Co-Existent Counties may use this same process to restore connection to their workstations.
Dedicated Counties: Follow the instructions, once received, to create a bootable USB drive. If unable to create the bootable USB drive, you can request a copy of it from the IBM Help Desk and it will be mailed to you using next‑day shipping. Then, use the bootable USB drive on each CWS/CMS workstation to remove the files which are preventing the CWS/CMS workstations from correctly connecting to CWS/CMS.
In addition, a WebEx conference call is being set up for counties to assist with how to create and use the bootable USB drives and to receive updates on the status of connected workstations in each Dedicated County. The meeting will focus on assisting Dedicated Counties, but Co-Existent Counties can join the call and listen to the instructions. We encourage all Dedicated County Technical Contacts to join the call. We apologize for the short turnaround. Here is the logon information for the meeting:
Time: 11:00 AM – 12:00 Noon
Join from the meeting link:
https://osi-ca.webex.com/osi-ca/j.php?MTID=m65ab2e8f5f7011527ab1839d6e489a63
Join by meeting number:
Meeting number (access code): 2541 692 4576
Meeting password: hYNamCuA823
Tap to join from a mobile device (attendees only)
+1-415-655-0001,,25416924576## US Toll
Join by phone
+1-415-655-0001 US Toll
An additional WebEx session will be set up to assist with troubleshooting with implementing the Microsoft fix: from 1:00 to 3:00 PM. Here is the WebEx information for the troubleshooting call:
Time: 1:00 PM – 3:00 PM
Join from the meeting link
https://osi-ca.webex.com/osi-ca/j.php?MTID=m7cc56df578e7dfbf450297629737f61d
Join by meeting number
Meeting number (access code): 2546 841 7948
Meeting password: DXt4DZmf744
Tap to join from a mobile device (attendees only)
+1-415-655-0001,,25468417948## US Toll
Join by phone
+1-415-655-0001 US Toll
____________________________________________________________________________________________
July 19, 2024 UPDATE: The CWDS Infrastructure team and IBM continue to work with the California Department of Technology (CDT) to resolve this issue. Multiple gateway servers have been restored and CWS/CMS transactions are being reported. However, there is no current estimate of when the system will be fully restored and all CWS/CMS servers and workstations are connected and up.
There are no reported issues with CARES-Live or CWS-CARES RFA connection issues.
Additional updates will be sent out as new information is received.
_________________________________________________________________________________
A major incident has affected many systems worldwide. Crowdstrike performed a code update on Thursday, July 18, 2024, which has caused Windows workstations and servers to crash. Impacted systems include CWS/CMS, CARES-Live, and Microsoft Windows. All Dedicated Counties are affected. All Co-Existent Counties which use Crowdstrike are also affected. Crowdstrike has implemented a fix, but actions are required to recover the affected systems.
The CWDS Infrastructure team and IBM are working with the California Department of Technology (CDT) to implement the fix. They started with the CWS/CMS application’s servers. Then they are working on the workstations.
Some servers and workstations use BitLocker encryption and require a recovery key. A solution for these encrypted items is being developed.
Additional updates will be sent out as new information is received.
Action Required:
CWS/CMS Users
If users experience any issues with CWS/CMS, please follow your county process for reporting issues, or you may contact the IBM Boulder Help Desk at 1-800-428-8268.
CARES-Live Users
Please follow your county process for reporting issues. CARES-Live County Administrators may contact the CARES Service Desk through the CWDS ServiceNow Customer Support Management (CSM) Portal. Alternate methods are phoning 1-855-292-3444 or via email at Servicedesk@cwds.ca.gov during normal business hours. You may also email after hours, and a Service Desk representative will contact you the next business day.
July 19, 2024
Communications
All SPOCs and Technical Contacts:
July 19, 2024 UPDATE: Due to the CrowdStrike major incident, the CWS/CMS Release 9.4 deployment, originally scheduled for Saturday, July 20, 2024, has been postponed. The new release date is still to be determined.
The CARES-Live scheduled outage will also be postponed, to coincide with the new deployment date of the CWS/CMS Release 9.4. An update will be sent out once the new outage date is finalized.
_________________________________________________________________________________
In support of the CWS/CMS Release 9.4 deployment scheduled for Saturday, July 20, 2024, from 5:00 AM to 1:00 PM, the CARES-Live application will also not be available during this CWS/CMS scheduled deployment.
Action Required
If users experience any issues with CARES-Live outside of this scheduled outage, please follow your county process for reporting issues. CARES-Live County Administrators may contact the CARES Service Desk through the CWDS ServiceNow Customer Support Management (CSM) Portal. Alternate methods are phoning 1-855-292-3444 or via email at Servicedesk@cwds.ca.gov during normal business hours. You may also email after hours and a Service Desk representative will contact you the next business day.
July 16, 2024
Communications
All SPOCs and Technical Contacts:
In support of the CWS/CMS Release 9.4 deployment scheduled for Saturday, July 20, 2024, from 5:00 AM to 1:00 PM, the CARES-Live application will also not be available during this CWS/CMS scheduled deployment.
Action Required
If users experience any issues with CARES-Live outside of this scheduled outage, please follow your county process for reporting issues. CARES-Live County Administrators may contact the CARES Service Desk through the CWDS ServiceNow Customer Support Management (CSM) Portal. Alternate methods are phoning 1-855-292-3444 or via email at Servicedesk@cwds.ca.gov during normal business hours. You may also email after hours and a Service Desk representative will contact you the next business day.
February 05, 2024
Communications
All SPOCs and Technical Contacts:
In support of the CWS/CMS Release 9.3 deployment scheduled for Saturday, February 10, 2024, from 5:00 AM to 1:00 PM, the CARES-Live application will also not be available during this CWS/CMS scheduled deployment.
Action Required
If users experience any issues with CARES-Live outside of this scheduled outage, please follow your county process for reporting issues. CARES-Live County Administrators may contact the CARES Service Desk through the CWDS ServiceNow Customer Support Management (CSM) Portal. Alternate methods are phoning 1-855-292-3444 or via email at Servicedesk@cwds.ca.gov during normal business hours. You may also email after hours and a Service Desk representative will contact you the next business day.
January 22, 2024
Communications
All SPOCs and Technical Contacts in Amador, Del Norte, Lassen, Modoc, and Trinity Counties:
January 22, 2024 UPDATE: Verizon has reported that the fiber cut was repaired on Saturday, January 20, 2024, and that all services were available.
____________________________________________________________________________
The California Department of Technology (CDT) has reported a high-level outage impacting the Verizon network. This outage can cause a major impact to the CWS/CMS application, CARES-Live application, and other county services located on the impacted equipment.
CDT is working with Verizon to resolve the issue. There is currently no ETA for the resolution of this issue. Additional updates will be sent out as new information is received.
Action Required:
CWS/CMS
If users experience any issues with CWS/CMS, please follow your county process for reporting issues or you may contact the IBM Boulder Help Desk at 1-800-428-8268.
CARES-Live
If users experience any issues with CARES-Live, please follow your county process for reporting issues. CARES-Live County Administrators may contact the CARES Service Desk through the CWDS ServiceNow Customer Support Management (CSM) Portal. Alternate methods are phoning 1-855-292-3444 or via email at Servicedesk@cwds.ca.gov during normal business hours. You may also email after hours and a Service Desk representative will contact you the next business day.
