It has been almost two weeks since the City of Baltimore's network was shut down in response to a ransomware attack, and there is still no end in sight of the attack's influence. It may be several weeks before the city's services return to something similar to normally manual solutions put in place to handle some services now, but the city's water bill and other payment systems remain offline, as well as most of the city's email and much of the government's telephone systems.
The Ransomware attack came in the middle of a major transition at the town hall. Mayor Bernard C. "Jack" Young officially assumed office only days before the attack, after former Mayor Catherine Pugh, who is facing an ever-expanding corruption investigation, was deprived of. And some of the mayor's critical staff positions remained unfinished, Mayor's Chief Operating Officer, Sheryl Goldstein, is starting to work today.
To top it off, unlike the Atlanta city, which suffered from a joint ransomware attack in March of 2018 Baltimore has no insurance to cover the cost of a cyber attack. So the cost of cleaning up RobbinHood ransomware, which far exceeds the $ 70,000 that ransomware operators demand, will be borne entirely by Baltimore citizens.
It is not like the city not notified. Baltimore's information security officer warned about the need for such a policy during budget hearings last year. However, the final budget did not include funds for this policy, nor did it include funding for extended security training for city employees or other strategic investments that were part of the rapporteur's strategic plan for the city's information technology infrastructure.
This may take a while
In a statement to print May 17, Mayor Young said:
I cannot give you an exact timeline when all the systems are restored. Like any large company, we have thousands of systems and applications. Our focus is to get critical services back online, and do it in a way that ensures that we retain security as one of our top priorities throughout this process. You may see partial services starting to recover in a matter of weeks, while some of our more intricate systems may take months in the recovery process … we engaged senior business security experts on site 24-7 who work with us.
Some of the restoration work also requires us to build certain systems to ensure that when we restore business functions, we do so safely.
The city's officials have given some details on the extent of the attack, when the city cooperates with an FBI investigation. But it seems that ransomware was triggered on some systems in the early hours of May 7 when the email service was suddenly interrupted. The city's response to the attack has thwarted many city services or completely shut them down.
The attack was first reported by Baltimore's Department of Public Works, when the department's official Twitter account announced that the email access was cut off, and the reported phones and other systems were affected shortly thereafter. As it became clear what happened, the city's IT team's office stopped almost all of the city's non-emergency systems to prevent further spread of the attack. It's not clear how widespread the ransomware was within the network, but the city's email and IP-based phones were among the affected systems.
The city's officials have emphasized that contingency systems, such as the police and the fire department's network and the city's 911 system, were not affected. The 911 system suffers from a ransomware attack last year when some firewall settings were disabled during maintenance. But the Baltimore Police Department was dependent on the city's email servers, and surveillance cameras around the city have been affected by the network shutdown. Almost all other city departments also had interruptions.
Real estate purchases cannot be closed, although Mayor Young said a paper-based closure management solution would be put in place today. Water bills and other city fees (including parking tickets and quotes from the city speed camera and red network) cannot be paid. And many city workers have had to resort to using their own laptops without a connection to the city network, as well as personal email addresses and mobile phones, to get the job done. Other tasks are idle or have gone back to paper-based processes, the city was in the middle of trying to eliminate.
A Grateful Job
The mayor's office for information technology has struggled to regain its position in the past two years after a number of fired chief information officers – four consecutive CIOs were fired or forced to resign over a period of five years. Frank Johnson, who now has the titles of both CIO and the Chief Digital Officer for the city, was hired in November 2017 after leaving a position as regional vice president of sales to Intel. Johnson led the development of a digital strategy for the city that was intended to bring Baltimore's IT spending more in line with similar sized cities and change IT practices. According to a 2018 strategy document, Baltimore uses about half of which other cities are budget for IT, and the Office of Information Technology controls only about one percent of the total budget. Most of the IT expenses are part of other department's operational budgets.
Until the ransomware attack, the city's email was almost completely hosted and runs on Windows Server 2012 in the city's data center. Only the city's legal department had moved to a cloud-based postal platform. Now, the city's e-mail gateway has moved to a Microsoft hosted mail service, but it's not clear whether all emails will be migrated to the cloud or, if possible. While Mayor Young said the city had data backup, it's not clear how much backups were implemented. And Johnson didn't want to say if there was a disaster recovery plan in place to handle a ransomware attack.
Some of Baltimore's systems host elsewhere, including the city's primary website, which hosts Amazon Web Services and is run by a contractor. But the city almost lost that website last week and not because of ransomware: The contract for the operation of the site had expired and the city was offensive in its payments.
Tracking how and when malware entered the city's network is a significant task. The city has a huge attack surface, with 113 subdomains, about a quarter of which are internally hosted, and at least 256 public IP addresses (only eight of which are online, thanks to the network shutdown).
"We are engaged in leading business security experts on site 24-7, working with us, Young said." As part of our containment strategy, we have implemented enhanced monitoring tools throughout our network to gain additional visibility. As you can imagine, with about 7000 users, this time takes. "