On July 19th, Crowdstrike performed a global update to their software. Performing this kind of an update is generally business as usual and does not cause issues. For example, many people have probably heard of “Patch Tuesday” when Microsoft issues their weekly patches/updates to Windows computers.
What was different about this, however, is that this update deployed by Crowdstrike had not been properly tested and resulted in a Windows “blue screen or death”, which means crash. The company released a fix fairly quickly, but one of the problems was that the original file that caused the issue had to be manually removed, significantly slowing the ability of companies to repair their systems. Most companies were ultimately able to deploy the fix and get back up and running.
There are ongoing questions about how this could have happened. Many companies, including Crowdstrike, use a process called “DevOps” to deploy their code/updates to live, production servers. Devops aka DevSecOps brings developers (who write the code), IT operations (who deploy the code) and cybersecurity teams (who secure the code) together to create a collaborative, automated and secure “pipeline” for pushing updates out to live production servers. Part of this process include quality assurance testing as well, that is designed to ensure code is fully tested so it functions correctly and as designed.
The issue is, what happened should not have been able to happen. There are many guardrails that are implemented as part of these processes to ensure that all code is tested before it goes into production. The fact that an update that caused such obvious and widespread computer crashes could have been deployed indicates either that either there was significant negligence/incompetence or there was something malicious at play. Crowdstrike has indicated that the issue came from a bug in a validation program that was supposed to test the new code before it was released. This coupled with other issues in their process (were not doing staggered releases, but routinely released the updates globally, and testing amount and quality were not adequate) created this massive IT outage. Of course, any company can make a mistake that can cause issues, but this was so obvious it is still hard to believe, in spite of the company indicating it was a number of process mistakes and failures, that there wasn’t malfeasance involved in some way. A test or “canary” deployment to even 10 machines would have discovered it. If it is malfeasance, the question of “why?” still remains.
Who is Crowdstrike?
Crowdstrike is a publicly traded American company, founded in 2011 and has headquarters in Austin, Texas. Crowdstrike is a major player in the cybersecurity industry providing endpoint detection and response (EDR) software products which are deployed on workstations and servers. EDR products are designed to identify and block malicious activity on those endpoints and provide alerts on this activity. Crowdstrike is the leader in this area and is utilized by most major banks as well as other multi-nationals, in addition to many smaller companies.
George Kurtz, Dmitri Alperovitch and Gregg Marston were co-founders of Crowdstrike. Many of the founders and some employees originally worked at McAfee, a large anti-virus vendor. Dmitri Alperovich, born in Moscow, was formerly Chief Technology Officer of Crowdstrike and has strong ties to the intelligence community, with connections to both the Department of Defense and the Department of Homeland Security. In addition, there are strong ties with the FBI; Crowdstrike hired Shawn Henry who was formerly an executive assistant director there. Other former FBI officials also work for the company.
Some other interesting data points and background:
The Center for Information Security (CIS) has an agreement with Crowdstrike to provide endpoint security for all CIS managed endpoints.
The Albert sensors, that are the intrusion detection system managed by CIS and deployed to the state and counties, run on CIS managed endpoints. It is a reasonable conclusion that Crowdstrike is running on the machines that are hosting the Albert sensors, because CIS has an agreement with Crowdstrike to provide EDR services.
EAC conformance/certification documentation indicates that Crowdstrike is not deployed to ES&S voting computers (Tabulators, ballot marking devices, EMS), but there are a number of questions:
- Do South Carolina and county workstations/servers use Crowdstrike and is data from those computers being sent to the CIS Security Operation Center for analysis and alerting?
- Epollbooks are not certified as part of the EAC certification process referenced in above, so is Crowdstrike running on the computers that are running the epollbook software or voter registration software? To wit, during the Crowdstrike outage, Arizona epollbooks and voter registration were taken offline.
Crowdstrike was the company tapped by the DNC and Perkins Coie to investigate a “breach” of the DNC server which they ultimately claimed was “hacked by the Russians.” This became part of the Russia collusion hoax related to Donald Trump. Subsequent investigations by non-leftist investigators have called their conclusions into serious question and the Russia collusion narrative has been debunked. The implications of this are concerning because Crowdstrike was at the center of this targeting of Donald Trump and potentially falsifying forensic reports and data to support this narrative. What else they would be willing to do is an open question, particularly in this crucial election year. The timing of this mishap is concerning and notable.
Security Implications of the Crowdstrike Outage for Elections
This outage has some very serious implications for elections in the United States.
- Availability and Disaster Recovery. Arizona was in early voting for a July 30th election and this outage took most of their voter registration and epollbook computers offline. The question to ask is whether or not the South Carolina state and counties have resiliency plans in place in the event this kind of outage were to happen during an election. If citizens can’t be checked in, or if lines are very long, then it will not be possible for them to vote and they will be disenfranchised. It is known that Republicans tend to vote on election day – in the 2022 election in Arizona, there were many technical glitches in largely Republican areas that occurred on that day. Given what just happened, a Crowdstrike like outage could heavily impact election day voting.
- Surveillance and 3rd party dependencies: Security professionals need visibility into systems and user activities in order to protect their companies. Crowdstrike software (and others like it) is installed on the workstations and servers to provide this visibility. It is highly privileged software that is constantly monitoring the endpoints including what is installed on them, and what users/software are doing. This software has the capability and permission to stop activities that it considers to be malicious. This means that any company using this software is trusting companies like Crowdstrike to not be malicious and to protect the company information. There are contracts in place and vendor security assessments are done to ensure the company is protected, but the fact remains that these companies have a great deal of power and visibility into what the company and its employees do. The potential impact on elections is:
- If South Carolina election officials are using computers that are running Crowdstrike then they are running software with that immense power and surveillance capabilities. What data could potentially be sent to the Center for Internet Security’s Security Operations Center?
- Albert Sensors, mentioned above, run on computers that are protected by Crowdstrike. Albert sensors collect network data and send it to the Center for Internet Security’s Secruity Operation Center, but an endpoint detection and response product will also send information on what is happening on the workstation/server itself – this data will also be going to the CIS Security Operations Center. It is not supposed to be sending any election related data there, but could this happen?
- Attack Detection – Albert sensors run on computers that have Crowdstrike installed. If those computers are taken offline, then that also takes down the intrusion detection capability so any ability to detect an attack is seriously impacted. This means the election infrastructure would be wide open to an attack and the ability to detect and respond to it significantly diminished.
Here are some questions to ask of your election officials in your area:
- What specific components of the ES&S election system utilize CrowdStrike? Our ES&S system has Windows servers which should utilize CrowdStrike. What about Epoll books which are connected to the internet? What about laptops or central county computers that house Electionware or the computer that reports results to SCYTL/Clarity (ENR) from the Electionware central county computer? Election IT professionals should check for the following: On Windows Operating System Crowdstrike files will be in C:\Windows\System32\drivers\CrowdStrike
- What specific data is being sent to the Center for Internet Security’s Security Operations Center and is this a uni-directional or two-way path for data transfer?
- Were the computers running the Albert sensors in the state/county environments impacted by the July 19th Crowdstrike outage? If so, has any forensic investigation been done to validate that there were no cyber-attacks during that time that could impact the state/county networks and potentially the election equipment?
- Please provide information on your emergency plan. At a minimum, poll workers should be trained on the use of paper poll books/rosters as a backup or redundant check in methodology. What would be better is if poll workers could be trained on the use of paper ballots and how to hand count them in the case of an even bigger outage. If there is not a plan in plan our group SC Safe Elections has developed one that would be a great way to mitigate this risk. See https://www.scsafeelections.org/the-gold-standard-for-elections/. We could train poll workers who will find it wasy and efficient.
- If Crowdstrike is in the environment, is it set for automatic updates? Are Microsoft updates set to be deployed automatically? If updates are not being tested by the county or state IT departments before deployment into their environments, then this type of outage could happen again.
Computers are infiltrated every day and our election systems do have some connectivity to the internet AND our electronic voting systems can also be penetrated via flash drives, cell phone “man in the middle attacks, wireless, and hidden modems. It is essential that our election officials take cybersecurity seriously. Our current system hasn’t even updated antivirus updates and patches for over 4 ½ years. Our state Legislative and Audit Council stated in their report of January 24 that cybersecurity had not been prioritized nor had funds that were provided by HAVA money been spent to upgrade security. If our SC State Elections board is serious about securing our vote, they need to take action and provide more transparency to our citizens. Not having an emergency backup plan would be a dereliction of duty. Our legislators also need to take this seriously and sound the alarm and partner with citizens to make the necessary changes in our law and our processes and procedures to enhance confidence in our system.
Special thanks to guest contributor cyber expert Julie Baker for her expertise and assistance with this post.