It's been a while i started to think about to share my 2 years experience with technology but i was not quite convinced or in other words – i felt so disappointed even start to think about doing so because of all the issue with the technology itself.
So you can ask – why now?Well, couple of months ago i found "rant" article with a follow up …
… and i knew i had to add my "2 cents" since we are seeing lot of similarities (and not only that)
One funny thing – when first "FP rant" was released, one of the TACs contacted me directly asking if that's me 🙂
Well, let's start
To introduce myself, i'm network engineer currently working mostly with FP technology (FMC, FTD, FXOS etc.). Experienced with other NGFW/traditional enterprise vendors and products such as:
Checkpoint – both Splat/Gaia sensors, Provider-1, Security management
FortiNet – FortiGate sensors, FortiAnalyzer
Juniper – Netscreen sensors, NSM
In our case, we are talking about scalable dynamic data center / telco environment with huge amount of the changes within the platform itself.
What are we using
FMC4k appliances in HA for central management a database, some FP4k & FP9k appliances in logical FTD-HA as well as some 5512 ASA firewalls.
No virtual devices in place (not that's making any difference)
We started with 6.1.x version and now we are having three different environments running 6.2.3.x, 6.3.x and even 6.4 beta.
Honestly, it does not make that much of a difference speaking of SW version since there's minimal progress overall and it's all because of the – pardon my French – most idiotic FW design i ever seen in my life!!!
DESIGN, DESIGN, DESIGN!!!
This is a root cause of all evil and because of the design decisions made here, i was able to successfully develop a tourette syndrome after all those years working with the product.
I'm not quite sure what decisions were made during development, but if you think that using legacy ASA code along with FTD services without re-design is way to go, i can assure you that it's not .
This product has to be redesigned, otherwise i see no option how this product can be even close to enterprise NGFW solutions of other vendors.
Well, to rephrase – you can try, but we already know the results, and man … what a disaster it is
For sensor, you have so cold LINA engine which is nothing else but legacy ASA code (ACL filter) with magical FTD services on top of it and rabbitmq messaging between those. Cooked as unified FTD image.
Just imagine it as VM running on top of hypervisor provided by FXOS.
Speaking of VM on FXOS, new 6.3.0 version is finally allowing to run multiple FTD instances. You can share resources, interfaces and so on. Even routing between instances is possible.
I haven't had time to test this fully yet, but again – marketing slide is one thing, real experience is something different. From design perspective, each and every FTD is consuming 2 CPU resource for management plane. Those cannot be shared. Imagine running 12 FTD instances on top of FXOS. This is resulting in 24 CPU cores utilization only for management plane. Not possible to allocate those for data plane or share among instances. What a waste.
Usability is just a pain as well – you have different set of commands on each layer, some of those are not even ported correctly or not showing same results!
Basic troubleshooting commands such as "capture" are not the same on both LINA/FTD.
It's even worse on latest 6.3.0 version if compared to 6.2.3.x
Forget also about "grep" feature for "show route" in 6.3.0 FTD, it's gone. Why? Don't ask me …
Sure, you can still "workaround" some of these by switching between layers to get some results. You just need to develop a huge sense of humor and same level of understanding and forgiveness (luckily, i learned that already as a married man)
Then you have OPS perspective. Since beginning, we are unable to handover this technology to operation. This is causing overhead for me as network engineer and for my employer as well (believe me, i have better things to do than creating cases and trying to figure out how to solve and address daily issues)
FMC is not any better. First surprise is slowness. Second surprise is efficiency.
It run 2 fragile databases and we were able to corrupt both by non-graceful shutdown.
Amount of limitations is also alarming, it's not allowing you to route management using anything else but default route for IPv6, traffic logs can be shown but it's nightmare to export them (forget about more than 400k lines). Deployment times are joke (regardless what change is performed, you are wasting time cause FMC has to download whole FTD config and deploy changes back + validate on FTD/ASA).
More in section below
LIFE WITH FIREPOWER – EXPECTATIONS vs REALITY
It's quite challenging. As mentioned above, we consider design as main issue and almost every issue is caused by that fact.
When we first started, i couldn't understand how there can be a product on the market with no backup/restore functionality. Honestly, i still don't understand.
When reported, it has been even marked as "feature request" and not a bug. Yes, you read that correctly. To have backup/restore possibility for NGFW enterprise product, you need to file a feature request yourself. At the end, we manage to have it re-categorize as a bug but it caused a lot of pain. And sure, because of the design (sftunnel FMC to FTD) it's not even possible to restore if the device is removed from FMC. Forget it, it's gone. No ETA for fix! (6.4 affected as well)
Sure, you can use API's for that and we are doing so, but until 6.2.3.x those were useless! It's getting better tho.
But still, if we want to create our own backup/restore using APIs, due to our policy and design construct, it's taking ages (hours) to finish device configuration (FTD-HA – interface/zone/routing perspective)
Even, worth to mention – try to keep FTD and FMC version to be the same. We have experienced some APIs inconsistency in versions are not the same (you won't find this in guideline).
Speaking of FMC, this whole "FTD to FMC" procedure is 100% brainless. You need to remove FTD from FMC because of some issues? Sure, you can do so, but FMC will not preserve any configuration related to HA (forget about monitoring interfaces, secondary IPs and so), any routing/routing related configuration or bond of logical zones with interfaces. If your policy package is constructed in such way (as zone based FW) you will experience an outage – you have no chance to interrupt this action and FMC will just push configuration "as is".
Funny is that for version 6.2.0.x this has not been even mentioned in the guideline. We performed re-registration of FTD due to some issues and we caused an outage. Assigned TAC found reference since bug was in database already. I still remember – this "routing information removal" bug was having priority 6 – enhancement that time! (maybe it still has same prio, haven't checked that for long time).
Another critical bug affecting FMC-HA deployment resulted in deletion of "random" ACL lines from the devices to which configuration was deployed. It literally removed couple of lines causing an outage.
In 2018/2019 you would expect full IPv6 support as well. Reality is bit different. Version 6.3.0 is the first release supporting object search for IPv6 in FMC. Until now, FMC was unable to parse/handle ":" and there were no results. You cannot even tract objects within policy package ("where used" is not there yet).
Because of this and all other limitations of FMC, we are running 3rd party database with some excel sheets to backup and to search efficiently. I'm not even mentioning amount of the cases raised for FMC for past 2 years for all the limitations and missing features of the product.
FMC is unable to do the delta comparison, dry run, lock the policy, revert saved changes, search efficiently, track objects, create reports for audit purposes (due to known limitations of the whole engine, there's 400 000 line limit hardcoded for traffic log/intrusion log report)
And it is slow. Sometimes you have to withdraw changes you made to policy package cause it's saving it for 5+ minutes and it' easier for you to start over.
And if you want to laugh a lot, try to implement static IPv6 route. You simply cannot! Why? Apparently, there's a good explanation for that but we have not received it. Maybe they did not see use case for that 🙂
Same applies for capture feature on FTD, prior to version 6.2.3.x you could not specify IPv6 address to the filter. As a workaround, you have to capture whole interface (with no port type filter, otherwise you will get IPv4 only) and grep or use 3rd party software to work with.
Flexconfig feature to substitute missing or not implemented ASA-like commands is still in diapers. Prior to version 6.2.3 this even resulted in unsuccessful rollback in case you inserted set of commands with typos or with commands that were not taken by FTD fot whatever reason. Your configuration got wiped afterwards and FTD just crashed and went to the default state. Considering time of the deployment (8 mins for standard config and 19 for snort enabled platform) you can easily do the math. This has improved in 6.2.3 version but it's still fragile.
We experienced same for NAT.
Basically, if you are able to implement a change in FMC which is in direct conflict with FTD, FMC will try to deploy it, no matter what. This is resulting in rollback (and you should start praying to have a successful one).
Policy package construct is also weird. FTD is using same construct as for ASA. If you are creating a rule which consists of 1 source IP, 1 destination IP and 1 port, you are ending up nicely with 1 ACL rule. This rule is then redistributed to FMC.
Modern NGFW rule-base construct is fully aware of nested groups, nested objects and zone. Because of this simple combinatoric, rule expansion is actually rising "to infinity and beyond". In our case, we are facing situation using NGFW zone based firewall without zones because of this limitation (or call it a design). And for sure, this is well described in the guideline so we just have to keep our heads down and work with it somehow. When same configuration was deployed to other vendor SOHO box that's using object based construct, it went through successfully without any expansion issues. Sure, competitors has some limitations as well, but in this case our ACL construct consumed all memory available and crashed.
I almost forgot, if you are using the product, do me a favor. Try to ping any routed unreachable host with ping count of 2000 as an example or execute a trace route towards that IP address. Share the results 🙂
For Cisco, to buy Sourcefire was definitely a right move, as IPS/IDS this device works well for the current and former Sourcefire customers.
While trying to develop a strategy, we wanted to run some IDS-like solution first to see how many false positives are we getting and what's the impact on system resources. What we did was the scan of the platform using built-in Nmap solution, comparison with our 3rd party scanner and creating some host profiles.
First, Nmap installed on FMC is version 6.01 from 2012 is outdated (you can even easily check vulnerabilities for such version). Seems vendor is OK with it since this is a version installed on 6.3.0 version
Second, network discovery profiles are missing for whole variety of enterprise OS versions. Latest VBD database is also outdated. Recommendations are then not accurate and when we tried to feed FMC using tailored script and changed the value for OS, recommendations were still the same. Why would you even use recommendations if you cannot rely on it? How many false positives could we expect then?
Another open topic is a granularity of intrusion policies, or in other words – how many policies should/could we implement. Sure, there's no specific answer to that and it all just depends. Out of the record, customer are usually using single digit number of policies (at least per rumors we've heard)
I recommend you to check CCO live slides if you have an access to, there are full of recommendations, best practices and so on. Unfortunately, this is not giving us the answers for Nmap & discovery profile issues we are having.
UPGRADE, TEST, REPEAT
Like every other vendor, Cisco is providing major/minor/patches releases on some agreed basis. This is nice. Theoretically speaking of course.If i would get a penny for every promise regarding stability, usability etc. i could quit my job, move to some paradise island and drink mojitos all day long.
We are taking every single possibility to move this product further, we have two testing platforms for beta releases and/or for releases that are installed before released to our production environment and i'm getting tired of it.
Most of critical topics (and man, we have some!) were supposed to be targeted on version 6.2.3. Cisco committed to deliver stability. Stability that should help us to at least get rid of all inconsistencies while deploying or changing configuration. Amount of the TAC cases raised right after the upgrade is in direct conflict with this statement. Nothing has changed!
And version 6.3.0 is not any magical as well. They added backup/restore for FTD which is quite limited, in case your device is removed from FMC, you cannot restore it anyhow. End of story.
We don't mind the upgrades, we are even happy to see that there's a time line for major/minor releases in place and if the patch is necessary to be provided, it is provided mostly ad-hoc.
Unfortunately, if the design will not change we can do upgrades indefinitely. It simply doesn't work for us.
Long story short: working with this technology in dynamic environment is resulting in huge amount of cases, bugs, troubleshooting requests with no real progress. This is not something i made up, this is pure engineer/end user experience.
I personally feel like a beta tester of the product
WHAT ABOUT VENDOR?
TAC experience is great, these folks are doing their best to help us and address all issues we are having with the product. Advanced FP team is trying to solve or at least advise how to avoid such issues for the future.
On the other hand, the BU is for us nothing but a black box. We have very limited visibility once bugs/feature requests are raised.
I'm not in the position to judge, but if i'm waiting almost 2 years for critical design related bugs to be addressed, there has to be something wrong. Not even mentioning all feature requests we raised. You can try to guess what's the progress there …
I really want to be surprised but i have already lost most of the patience.
If you are asking who should go and buy this product, it's hard to say.
If you are in strong partnership with Cisco or if you are just replacing old ASA firewalls with new Firepowers, it might go bit better for you than for us. If your are not doing lot of intrusive/design or conceptual changes, you might even like the product. It is well suited for "set and forget" deployments or i see nice implementation for voice&video environments where only Cisco technology is used.
On the other hand, if you do not requite centralized management, i would stick with ASA. Maybe CDO (Cisco Defense Orchestrator) might be an answer for you, but we did not test it since our deployment requires "on-prem" solution and CDO is unable to provide that.
I believe that Cisco is pushing here, APIs are also improved and on version 6.4 beta it seems to be more or less stable.
If your environment is dynamic, and/or you have no preferred NGFW vendor for now and Firepower was in your scope as well, my recommendation is to avoid it. It's not worth at the moment and i believe that it will not be worth the money until it's completely redesigned to at least get close to other vendors.
TL;DR: I can only say the same as user laclobunu 6 months ago. Don't buy it, it's not worth your time, energy, mental health and money.
Read more: reddit.com