Latest updates
- Crowdstrike has announced they have a working cloud based solution. If you still have remote workers or computers with bitlocker and lost keys, this may be an effective solution. Please contact us ASAP if you would like to use this solution in your environment, by submitting a ticket to support@navisec.io
- Step by step self-remediation for Remote users
- Microsoft has released a USB tool today to help expedite resolving hosts still impacted by the CrowdStrike Falcon agent issue. Check this link.
- How to boot Windows Safe Mode (with or without command prompt)?
- How to receive BitLocker Recovery Key?
CrowdStrike has noted that customers are experiencing system crashes on Windows hosts related to a Falcon sensor content update. Below is their official technical alert with guidance to correct the issue. Please follow the steps below if you are experiencing any issues.
Summary
CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor.
Details
- Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor.
- Windows hosts which have not been impacted do not require any action as the problematic channel file has been reverted.
- Windows hosts which are brought online after 0527 UTC will also not be impacted
- This issue is not impacting Mac- or Linux-based hosts
- Channel file “C-00000291*.sys” with timestamp of 0527 UTC or later is the reverted (good) version.
- Channel file “C-00000291*.sys” with timestamp of 0409 UTC is the problematic version.
- Note: It is normal for multiple “C-00000291*.sys files to be present in the CrowdStrike directory – as long as one of the files in the folder has a timestamp of 0527 UTC or later, that will be the active content.
Current Action
- CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes.
- If hosts are still crashing and unable to stay online to receive the Channel File Changes, the workaround steps below can be used:
- Crowdstrike statement: We assure our customers that CrowdStrike is operating normally and this issue does not affect our Falcon platform systems. If your systems are operating normally, there is no impact to their protection if the Falcon sensor is installed. Falcon Complete and OverWatch services are not disrupted by this incident.
Workaround Steps for individual hosts:
- Reboot the host to give it an opportunity to download the reverted channel file. If the host crashes again, then:
- Boot Windows into Safe Mode or the Windows Recovery Environment
- Note: Putting the host on a wired network (as opposed to WiFi) and using Safe Mode with Networking can help remediation.
- Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory
- Navigate to the appropriate partition first (default is C:), and navigate to the crowdstrike directory:
- C:
- cd windows\system32\drivers\crowdstrike
- Note: On WinRE/WinPE, navigate to the Windows\System32\drivers\CrowdStrike directory of the OS volume
- Navigate to the appropriate partition first (default is C:), and navigate to the crowdstrike directory:
- Locate the file matching “C-00000291*.sys”, and delete it.
- Do not delete or change any other files or folders
- Cold Boot the host
- Shutdown the host.
- Start host from the off state.
Note: Bitlocker-encrypted hosts may require a recovery key. (how to receive a BitLocker Recovery Key)
Workaround Steps for public cloud or similar environment including virtual:
Option 1:
- Detach the operating system disk volume from the impacted virtual server
- Create a snapshot or backup of the disk volume before proceeding further as a precaution against unintended changes
- Attach/mount the volume to to a new virtual server
- Navigate to the %WINDIR%\\System32\drivers\CrowdStrike directory
- Locate the file matching “C-00000291*.sys”, and delete it.
- Detach the volume from the new virtual server
- Reattach the fixed volume to the impacted virtual server
Option 2:
• Roll back to a snapshot before 0409 UTC.
Workaround Steps for Azure via serial:
- Login to Azure console -> Go to Virtual Machines – Select the VM
- Upper left on console -> Click: “Connect” -> Click -> Connect -> Click “More ways to Connect” > Click : “Serial Console”
- Step 3 : Once SAC has loaded, type in ‘cd’ and press enter.
- type in ‘cd’ command
- type in : ch -si 1
- Press any key (space bar). Enter Administrator credentials
- Type the following:
- bcdedit /set {current) safeboot minimal
- bcdedit /set {current) safeboot network
- Restart VM
- Optional: How to confirm the boot state? Run command:
wmic COMPUTERSYSTEM GET BootupState
For additional information please see this Microsoft article (https://azure.status.microsoft/en-gb/status).
Workaround steps for hosts in a virtualized environment unable to access safe mode.
Inside of the selected hypervisor:
- Power off the host.
- Configure a 10-second boot delay on the host impacted.
- Launch a interactive console session of the host.
- Power on the host.
- Hit ‘Enter’ and repeatedly hit ‘F8’ to enter the recovery screen.
From there, you will be able to continue with the guidance below
NOTE: Additionally, Windows SafeMode with Command line can be accessed by doing the following steps.
- Choose Advanced options
- Choose Troubleshoot
- Choose Advanced options
- Choose Startup Settings
- Select Restart
- Select option 6 *
- When your computer starts in safe mode with command prompt, type:
cd \windows\system32\drivers\Crowdstrike
then type:
del C-00000291*.sys
restart your computer.
When your computer reboots, you should have functionality restored. If your computer has not froze or experienced a blue screen, no action is needed and it should not have a problem going forward.
* – if command prompt safe mode doesn’t work, try option 4 and boot safe mode normally and delete the file using Windows Explorer
NOTE: How to retrive BitLocker Recovery Key
- Choose Devices
- Select a dropdown
- View BitLocker Key
- Show recovery key
- Select Copy on the right
If you have Crowdstrike with NaviSec, our engineering team can provide you a list of potentially impacted hosts. If you’d like the list, please request it by submitting a ticket to support@navisec.io
If you are experiencing issues, please contact us at support@navisec.io
For the newest technical details, check this link.
