10 tips for a successful incident response

In my previous story Gym bag saves university from ransomware, I explained the steps the Switch CERT Incident Response Team takes following a cyber attack. Based on my many years of experience, these ten top tips have proven their worth. After all, lots can go wrong during such incidents when we’re facing an emergency situation. If you can take a step back and hang on to your sense of humour, then that will help you deal with what’s happening and implement the tips below. Let’s get started.

1. An army marches on its stomach

The first very helpful tip that is often forgotten during major IT security incidents is to ensure that the team dealing with the crisis is fed. Everyone involved has their basic needs. These have to be met, even if the sky is falling in. At the end of the day, we automatically work less efficiently when we’re hungry, thirsty and stressed and can miss important things without realising it. So it’s perfectly legitimate to order 24 cheeseburgers and as many servings of fries at a Burger King drive-thru in the early hours of Sunday morning. You might even want to give a smile to the very surprised food server.

2. Y’all got any more of that whiteboard

Jot down, map out and plan things together. Even in the digital age, an analogue whiteboard, a blackboard or simply a large piece of paper can be very helpful in team discussions and strategic planning. Especially when you need to move fast, a beautiful PowerPoint presentation is the last thing you should waste time thinking about. Besides, there just isn’t time for that. ‘Quick and dirty’ is the best motto.

3. Brace yourself – the phone will ring

What is always underestimated is the interest of employees and the media in an IT security incident. The subject is very much a hot topic right now, and everyone will want to know what has happened. Phones will be hopping from calls and vibrating non-stop with messages from acquaintances. Sometimes, colleagues taking a quick cigarette break may want to talk and find out how things are going. If they try to wrangle their way into the situation room and take photos, then a certain level of suspicion is advisable at this point, if not before.

We shouldn’t underestimate the work of those on the frontline either – namely, the support team. They are the ones who have to field all the questions from employees and listen to their frustrations about restrictions or being temporarily prevented from doing their work. The crisis team can help in this by providing an emergency website and issuing public communications. It’s worth looking out for the members of your support team, as they have to absorb a lot.

4. Stress, stress everywhere

People deal with pressure and stress in different ways. For example, established hierarchies may often be turned upside down during incidents and an organic hierarchy may form. What does this mean for those involved? Essentially, they should embrace the chaos. People who usually work in the background but have a natural professional authority for those involved may automatically step up in such high-stress situations. Very often, these are not the people at the top of the organisational chart. Having the confidence of the team, they are able to get stuck into the chaos and provide stability. Incident response can be severely hampered if this natural process is halted. This means that an ‘ordinary’ member of the network team may suddenly end up leading all the parties involved during a crisis.

5. Nobody cares – get the help you need

Seek out the assistance you require. Security works best when different teams collaborate and exchange ideas. This may include the police, the NCSC, suppliers or even external security partners such as Switch CERT, who can help manage the respective incident. Working hand in hand, the various specialists can tackle the crisis together. If we try to deal with these things on our own and with misplaced confidence, the task can often seem impossible and the situation may drag on unnecessarily – a lot of helpful knowledge and information is wasted. Digitalisation has brought with it a great deal of convenience, but also greatly enlarged the area of attack. We know that attacks happen, that’s old news. What shouldn’t be forgotten is that even if we can avert an attack, it’s still a crime that should be punished. The idea that our digital media are our home is not yet very firmly established. If somebody tore down my fence without my permission to see if it could withstand the attack, I’d report it to the police.

6. Minimum Champion!

Who hasn’t heard of the 80/20 rule? When a security incident happens, our instinct is to follow up every lead so that we can precisely say what went wrong. And it’s always important to identify vulnerabilities before rebuilding. I’ve seen too many situations in which this effort was not taken, and the second breach followed immediately. However, data analysis and correlation take a lot of time. Some of the data needed to reconstruct the exact path of attack or infection may well be missing because certain logs were not recorded, short data retention periods were defined or new systems were not yet integrated into centralised log management. There has to be a balance between the Sherlock Holmes approach and the resumption of operation – less perfectionism, more practical relevance.

7. Ight Imma Head Out – Exercise!

‘Paper never refused ink’ goes an old yet very wise saying. Do you have an incident response playbook? Wonderful disaster recovery plans on paper sitting on some dusty shelf because nobody knows where they’re kept or that they even exist? How would you contact your team members if the office was unreachable?

These things are usually not rehearsed. And if you add the stress and pressure of an IT security incident, things can get very chaotic. That’s why what’s called for is practice, practice and practice. The concepts and plans may be fantastic. But if they’ve never been practised, they’re not going to work and incident response will fail at the smallest of hurdles. Do tabletop exercises with your teams. Talk through the what ifs and ask the uncomfortable questions.

8. No regrets – creativity

Checklists are all well and good, and serve as a useful guide. But they should not divert attention away from finding the best solution at the right time. Troubleshooting an IT security incident calls for creativity, because no two situations are the same. Established processes need to be scrutinised in order to find the best way forward. The attackers are always one step ahead of us. We have to be on a par with them to find effective fixes and everyone involved must be empowered to design and implement these unconventional solutions.

9. Run a marathon they said, It’ll be fun they said

Keep an eye on your team’s stamina level. Managing an IT security incident is a marathon, not a sprint. It will take weeks to resume normal operations. Many clean-up tasks and adjustments will follow based on the lessons learned. Especially in the most hectic of situations, people need time to de-stress in order to function at their best. Even those who want to work tirelessly without taking a break should be gently reminded to relax. After all, working non-stop for days that then turn into weeks would take anyone to the brink of burnout. Incident response processes should be feasible in the long term so that the marathon can be accomplished.

10. Are you sure about that?

Maintaining an overview of your IT infrastructure is quite the challenge. And the bigger and more complex the company, the greater it is. Old test servers that have been forgotten, admin/admin accounts that were not deleted – you can’t make an omelette without breaking eggs. Raising awareness of safe working practices at all levels – from end users to those responsible for technology – will itself prevent many incidents. This also ensures that there is a better overview of the company’s infrastructure and incidents can be dealt with more quickly. After all, you can only understand what has happened and figure out the next steps if you have a reasonable idea of how things stood.

Use the knowledge of the community

What have you found useful for a successful incident response? What have been your craziest experiences that still make you laugh today? In the spirit of tip no. 5, let’s exchange ideas and learn from each other so that we’re even better equipped to deal with future incidents.