So, think you know what IPS rules are enabled on your Firepower system, and do you feel comfortable with Cisco’s defaults and sleep well at night? This blog may just start keeping you up at night!
A couple weeks ago I was training/consulting at a very large school district outside of Columbus, Ohio. They have all 9300 NGFW’s to cover hundreds of thousands of students, so I immediately got to work on tuning these already configured systems, which a consulting company installed for them. They called me because their 10Gig links were saturated and they couldn’t find the problem, and also wanted additional training on their Firepower/FTD systems.
What I found upon arrival was the basic defaults of Balanced Security and Connectivity as their IPS policy, but worse, the consultants disabled the Security Intelligence (SI) completely saying it would bring down the network. That’s not true obviously, and the SI was a quick fix.
Question: Was their network secure because they had Cisco Firepower 6.3 code and the Cisco recommended IPS rules enabled on their systems? is your network secure by being configured this same way?
Let’s start by answering that question, and then I’ll circle back to how a Cisco FTD 9300 was brought to it’s knees (Never seen that before!), and how I solved this customers issues.
If you follow Cisco’s recommendations, your system will have both the IPS and NAP policies set to Balanced Security and Connectivity. Sounds reasonable, but will it protect you? Let’s take a look:
First, create a policy, and then choose to Create and Edit Policy:
Notice the Drop when Inline and the default Base Policy. When you’re first bringing up your system, the IPS policy needs to be tuned. I typically disable the Drop when Inline checkbox so the rules can be tweaked for the environment without dropping any traffic in the meantime. After a period of time (different for each network), I’ll enable the Drop when Inline and start dropping the bad traffic. However, let’s just concentrate on the Base Policy shown.
To understand what this does, start by going into your IPS police(s), scroll down to the Cisco base policy, then click on Rules:
Now open the Rule Content in the Rule accordion and scroll down to Rule Overhead as shown:
What the arrows show is which rules are enabled by default with each policy – click on each Overhead (Low, Medium, High, and Very High) to see which are enabled and disabled. I am not making this up, Cisco’s way of choosing which rules are enabled is only decided by Rule Overhead. They should just change the Overhead names from Low to Connectivity over Security, from Medium to Balanced Security and Connectivity, and High to Security Over Connectivity, and make it easier on us. (reality check here: it’s unlikely they use overhead only to make a rule determination, but there is no documentation to say otherwise [yet], so here we are).
For example, and to demonstrate what I am saying; if you choose the useless Connectivity over Security base policy, then only the Low Overhead rules are enabled as shown below, or about 500 rules, with all other 35,000+ rules disabled:
Next, the default of Balanced Security and Connectivity enables the Low and Medium Overhead rules, which again, is Cisco’s recommendation. Add up the Low and Medium Overhead rules, that is the amount of rules enabled. Period.
With Balanced Security and Connectivity, the High and Very High Overhead rules are all disabled:
Now understand that f you choose the Security over Connectivity policy, then the Low, Medium, and High rules are enabled, which gives us now 15,493 enabled rules (in this example).
However, none of the three policies I’ve mentioned would ever enable a Very High overhead rules, which means that 21,498 rules are disabled by default,.
If you double click on any of these disabled rules, you can see they are Very High overhead
So if you are using the default IPS policy, you’ll then ask yourself: “But what about this zero day attack that is occurring, and Cisco IPS rules just downloaded automatically; these important rules must be enabled by Cisco, so I am safe and secure, yes?” No, not unless all the rules imported are Low and Medium Overhead. The High and Very High Overhead rules are imported to your system, but they are disabled by default, even if the rules are all-so-important!
So, how many rules do you have available?
You can see the amount of rules on your system by going to the Rules option (click any of them) in any policy with no filter, which is 37,426 in this example:
By looking at the IPS policy summary, I have 10,514 rules enabled by default (the amount of low and medium overhead rules found on the systems). This means over 25,000 rules are disabled. Cisco’s reasoning for this default recommendation is that you need low latency. True, but my customer had $1.2M dollar NGFW’s that can easily handle tens of thousands of rules enabled! Even with 2110’s you can enable thousands more and be just fine.
You may be saying “Well Cisco knows what they are doing, and if they never enable a Very High Overhead rule, then neither shall I.”
Let’s go take a look at something. Shown here is a Rule update that came into my FMC (Rules are released twice a week – Tuesdays and Thursdays, so on Wednesday and Friday mornings you should be checking these!).
In this example, seven rules changed, and 35 rules were added. Seems that if Cisco added these, they might be important. So why are the first four rules here disabled? Because they are not important? No, they are very important, they are just Very High Overhead.
Is your network better off with Security over Connectivity as a base policy? Sure, but it can still be better. “Can I just forget about it then”, you ask? No, but you probably will, as most of my customers do. It’s better to go in twice a week and look for the new downloaded rules and enable the Very High overhead rules you think are needed for your network. But again, most of my customers only do this for a few weeks after I leave; just like when you go to the gym the first two weeks of January…you mean to go back to the gym, but hey, you’re busy!
Can My System Handle Security Over Connectivity (SoC)?
There is a caveat here. If you have xx40/50 (or the new 41×5’s!) or even 9300, enabling another 5-6000 rules going from Balanced policy to Security over Connectivity (SoC) should not cause you any issues, but maybe you just don’t feel comfortable changing from Cisco’s recommended settings because you know you’ll never login and check it.
Regardless, there may be a use-case for enabling the larger number of rules for a short time on any system without stressing or losing sleep, just to verify your data, then set it back to Balanced if you feel this will lower your stress level.
Understand that enabling a very large amount of rules for the long term may not be a good idea as you may see many of these rules ignored due to PPM (packet latency). This can be found in the Advanced tab of your ACP:
Since the PPM will make sure you don’t have latency when you enable more rules, why not just turn all the rules on? Because you will not get complete inspection, that’s the key. You think you’re getting better inspection but in fact, Snort is bypassing traffic to keep the latency low. Keep this in mind, especially when using lower power systems.
Important TIP! Lastly regarding enabling more rules:I always tell my customers that if you want to enable rules manually, that’s great! But, you also need a regular process, maybe once every 6 months, where you go back and disable the rules you don’t need anymore.
What about Firepower Recommendations?
So, if you use Firepower recommendations, first make sure your FTD box is North-South traffic (or don’t use this at all!) or Firepower will probably disable most of your rules. To match your SoC policy go to the Advanced configuration and move the slider bar to High, so that the Low, Medium and High Overhead rules will be enabled, because as usual, only the Low and Medium rules are enabled with Firepower.
Although you can still make your network more secure, this is still a lot better than the defaults.
But There is Still One More Setting: MAXIMUM DETECTION
I know this is a long post, and most of the people will never read this far, but if you did, you might as well keep going now!
The mostly unused Maximum Detection is an odd bird and must be tested on your network with the Drop while Inline disabled for a quite a while.
(UPDATE: This rule was update 5/2/19 and now has about 28k rules enabled, so I need to do a blog on just this policy soon)
This policy is the only one that enables some Very High Overhead rules; 5784 to be exact (as of this writing). However, it disables thousands of High Overhead rules that would normally be enabled with Security over Connectivity. I’ve never used this policy in production, nor do I know any customers that have, but again, you can use this, but just test it – and then please let me know what happens!
Lastly, what about my customer in Ohio and the 9300 that was brought to its knees? Yes, once we moved the IPS policy to a base of Security over Connectivity, we found the problems they were having quickly. The IPS events started triggering about 6000 hits a minute after we deployed, and the CPU on the 9300 spiked to over 80% as it was trying to process all the IPS events. That’s a LOT of high overhead events to do that to a 9300! The FTD 9300 was no longer passing data…wow, not only have I never seen this, I’ve never even heard of this!
After we were able to do some analysis on the new IPS events (which got to over 100,000 in just a few minutes!), I had to verify these weren’t false positives and then find the culprits. Drilling down, we found that their entire server farm was infested heavily with CnC’s, and only the High Overhead rules found these CnC’s. Those attacks were so well drilled into these servers, they each had thousands of connections, which would explain the bandwidth issue! We immediately started blacklisting these servers and the CPU percentage dropped to less than 10%, and the bandwidth saturation went from 80% to less than 20%!
The customer feared that they had these attacks on their systems for possibly years….and what found them? It wasn’t the Cisco default settings…
The customer had some serious issues on their servers to fix, but my job here was done.