The Akamai origin disclosure non-vulnerability

This blog post is written by Laban Sköllermark, pentester and IT security consultant at Sentor, a part of Accenture Security.

In the middle of 2022 I worked on a web application pentest for a client who used Akamai to take care of all incoming traffic. This is the story of a problem I found that Akamai themselves didn’t see as a vulnerability. Also learn about the different security aspects of Content Delivery Networks in general.

Content Delivery Networks (CDN) and their security features 

Some CDN customers are mostly interested in the low-latency distribution of content around the world, but most CDN providers provide security benefits as well. This is a summary of the most common security features. 

Independent of whether the customer’s application is deployed in the customer’s own data center or in the cloud (somebody else’s computer), the CDN terminology is the origin server. In the case of “the cloud” the origin server is most probably many servers, but in those cases the origin server can be a domain name where the application is deployed. 

Distributed Denial-of-Service (DDoS) attack mitigation is a common selling point by CDN providers. The idea is that the origin server shall not receive any requests directly from end-users. Instead, all requests go through the CDN provider’s world-wide network of edge servers where cached content is served without any load on the origin server or its internet connection, and non-cacheable content, like personalized pages, are fetched by the CDN network from the origin server and then delivered to the end-user. In the case of an ongoing DDoS attack against the customer the CDN provider has multiple ways of detecting it and remediating the attack without overloading the origin server or denying requests from legitimate end-users. 

Web Application Firewall (WAF) is another common sales point for CDNs. The idea is that suspicious requests which are believed to include attacks on the underlying application shall be stopped by the WAF and thus, to some degree, protecting the application from malicious users. It could be common payloads for SQL injection, Cross-Site Scripting (XSS) etcetera. 

Rate limiting and bot mitigation is another useful security feature. If the protected application doesn’t have for instance login brute-force protection by itself, the CDN can be configured to limit the number of requests per IP address which is let through to the origin server. The solution of a CAPTCHA challenge can also be required. 

All the above security features are useless if the origin server’s location is known to attackers and the server is configured to serve requests from end-users. When protecting an application with a CDN it is however highly recommended to configure the origin server to not serve end-user requests but only serve requests coming from the CDN edge servers. By doing so, it is no longer possible to send malicious traffic without having it inspected by the WAF or sending brute-force login attempts without being rate limited by the CDN. A DDoS attack, however, can still be successful even if the origin server doesn’t process any requests. The firewall in or in front of the origin server might not handle the number of connection attempts the DDoS attack implies. The origin server’s internet connection can most probably not handle the traffic volume and requests coming from legitimate end-users are delayed or simply dropped. Modern DDoS attacks can peak at several hundred million requests per second or several terabits per second. If the Internet Service Provider (ISP) of the origin server does not have their own effective DDoS mitigation in place, non-cacheable pages of the target application will have a severe performance degradation. 

Vulnerability background 

On the 27^th of July 2022 I discovered that Akamai leaks information about the origin server’s location under some circumstances when testing a client’s website for HTTP request smuggling (HTTP desync) attacks. The client used AkamaiGhost / Akamai Global Host / Akamai Global Traffic Management (GTM). 

The bug seemed to occur only for some configurations of Akamai GTM. Our client sent traffic destined to some particular paths to separate origin servers than the site’s main origin server and the bug only manifested for those paths. For other Akamai customers it seems to work on the "root" path (/) as well (see below). 

It was believed that I had stumbled upon a security vulnerability (CWE-209: Generation of Error Message Containing Sensitive Information), since it allows for an attacker to circumvent Akamai and launch a DDoS attack directly on the origin server. I reported this to Akamai’s security team as an effort of coordinated disclosure. 

Reporting to vendor

The following description of the problem was sent to Akamai’s security team together with technical details (more on those later): 

[I] believe this is a security vulnerability since protecting Akamai customers' origin servers is a core business for Akamai. Disclosing the hostname and path of request to origin servers can at least bypass the DDoS protections of Akamai. If the client has not protected their origin servers with firewall rules allowing only traffic from Akamai edge servers, also rate-limiting and other protective services can be circumvented. That is the client's problem though, if they have forgotten or ignored to protect their origin servers, but it is probably quite common. 

Akamai’s response

Akamai investigated the issue and responded that they do not agree that it is a security vulnerability and told me that they leak information about an origin server’s location in several other ways already: 

We are aware that this information is publicly exposed, and we don’t consider this a vulnerability. Akamai (or any other proxy technology) cannot guarantee cloaking origin destinations beyond a weak attempt at security by obscurity. The origin information seen in the error response you triggered is, in fact, available through several other mechanisms we expose—all intentional, some necessary for the functioning of Ghost. We instead provide our customers with a set of capabilities such as Site Shield, Origin ACLs, and Ghost-to-Origin mTLS that provides tangible security guarantees through traffic filtering and cryptographic authentication, and therefore enforces the property that data flows on the intended Akamai to origin path. 

We regularly communicate the above point to our customers and we make no promises to hide origins contractually or otherwise. However, I do agree that the public perception may sometimes be contrary. I understand from your original email that you have a security engagement with one of our customers which you cannot disclose to us here. If you could please direct them to contact their designated Akamai support team, we will be happy to help them verify that they have the necessary defenses on this front, or otherwise we will work with them to implement the appropriate controls.

Technical details

I do not want to name our client since we keep information regarding who our clients are secret unless explicitly agreed otherwise. To not disclose, even to Akamai, which of their customers we were working for, I needed to replicate the finding on some other Akamai customer. 

I searched for known Akamai clients and found a famous Swedish brand. Their www.<REDACTED>.com domain pointed to www.<REDACTED>.com.edgekey.net, which showed that they were using Akamai. The below cURL request disclosed that their origin server was prod-<REDACTED>com.azurewebsites.net, which, unfortunately or deliberately, was open for everyone to access directly. Note that this configuration was against Akamai recommendations. 

The key to leaking an origin hostname (note that it is only possible on certain Akamai configurations), was to use an HTTP verb that was forbidden and to set the HTTP request header Transfer-Encoding to a certain value. By default, Akamai allows only the HTTP verbs GET, HEAD and OPTIONS but customers can configure to allow other verbs. By using a verb that was blocked by Akamai, the request was terminated already on the CDN layer and Akamai responded with the HTTP response status code 501 Not Implemented. One of the transfer encodings that triggered the leaking of the origin was gzip. 

HTML entities in the error message made it hard to interpret it in a console, but the tool html2text can be used to make it human readable:

Some values of Transfer-Encoding seemed to trigger the detailed error message which disclosed the origin server. Repeating the request with the value set to identity showed an error message that did not reveal the origin server: 

Human readable:

Suggested Action

This information is intended to aid security professionals with minimizing the impact of this vulnerability. SENTOR PROVIDES THE INFORMATION ON AN “AS-IS” BASIS WITHOUT REPRESENTATION OR WARRANTY AND ACCEPTS NO LIABILITY FOR ANY ACTION OR FAILURE TO ACT TAKEN IN RESPONSE TO THE INFORMATION CONTAINED OR REFERENCED IN THIS BLOG. 

The following recommendations are valid if you consider the origin server’s location to be sensitive information and you rely on the CDN to implement security controls, such as DDoS protection. 

If you use a CDN provider or similar for security purposes and not only to speed up your website or reduce the traffic to your origin server, make sure you protect the origin server from unwanted traffic. Consult your provider’s documentation and/or their support team what the possibilities are. Most probably, the traffic from the provider’s edge servers will come from a known list of IP addresses so you can block all other traffic in firewalls to protect against most of the simplest DDoS attacks on higher network layers, such as excessive amounts of (potentially expensive) HTTP requests. 

Most CDN providers can be configured to use a client TLS certificate in all connections to your origin server, thus terminating all connections from unwanted sources already in the handshake phase of TLS since they do not have a valid client certificate. 

Check if your provider can ensure that the location of your origin server is protected and not exposed in for instance error messages like above. 

Note that your origin server can also leak its location itself in e.g. error messages. If your server application sends emails to customers, such as via a password reset feature, the IP address or FQDN (Fully Qualified Domain Name) of the server could be exposed in email headers. 

If your application has an SSRF (Server-Side Request Forgery) vulnerability (or as a feature, like fetching user avatars from URLs) it will also leak the origin information if the requests are done from the origin server. 

If your website is a high-profile target and it is possible that the server location is somehow leaked, talk to your ISP (Internet Service Provider) to discuss DDoS protections. Do they have enough bandwidth to mitigate a large DDoS attack, and can they help protect your server and its connection to the ISP from the additional load during an attack? 

If your origin server has a hostname in the same domain as the public facing domain, remember that information in DNS is meant to be public and it can be hard to hide information. If the hostname is origin-www.example.com as some CDN providers have as an example, it is easy to guess it and look it up. If your domain name pointed directly to your origin server before you started using a CDN, be aware that there are search engines for historic DNS records. Also note that if your origin server is using a publicly trusted TLS certificate, it will be part of Certificate Transparency (CT) logs, searchable on for instance crt.sh. If the certificate is not a wildcard certificate it will tell where the server is located.

Credits

Thanks to members of the Swedish speaking Facebook group Säkerhetsbubblan for some of the tips on common server and application configurations that also leak the location of the origin server.

What about you? 

Do you want to find out if your application is vulnerable to this or other security problems? Let us test it! 

Do you want to help customers find these kinds of vulnerabilities? Join our team of pentesters in Sweden!