Elasticsearch Vulnerability
This was originally posted to engineering.blopboard.com which is now defunct.
On 2014-07-09, we received an e-mail with the subject line "Your Amazon EC2 Abuse Report."
Dear Amazon EC2 Customer,
We've received a report that your instance(s):
Instance Id: [redacted]
has been making Denial of Service attacks against remote hosts on the Internet; check the information provided below by the abuse reporter.
This is specifically forbidden in our User Agreement: http://aws.amazon.com/agreement/
Please immediately restrict the flow of traffic from your instances(s) to cease disruption to other networks and reply this email to send your reply of action to the original abuse reporter. This will activate a flag in our ticketing system, letting us know that you have acknowledged receipt of this email.
It's possible that your environment has been compromised by an external attacker. It remains your responsibility to ensure that your instances and all applications are secured. The link http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1233 provides some suggestions for securing your instances.
Case number: [redacted]
Additional abuse report information provided by original abuse reporter:
* Destination IPs:
* Destination Ports:
* Destination URLs:
* Abuse Time: Wed Jul 09 13:26:50 UTC 2014
* Log Extract:
<<<
It has come to our attention that Denial of Service (DoS) attacks were launched from your instance to IP [redacted] via UDP port(s) 8910. Please investigate your instance(s) and reply detailing the corrective measures you will be taking to address this activity.In the meantime, we have blocked outbound UDP 8910 on the instance(s) to prevent further abuse.
If you believe that you were compromised by an external attacker, the best recourse is to back up your data, migrate your applications to a new instance, and terminate the old one. Attempting to repair a compromised instance does not guarantee a successful cleanup in most cases. We recommend reviewing the following resources to ensure your EC2 environment is properly secured:
* Amazon EC2 Security Groups User Guide:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html* AWS Security Resources:
http://aws.amazon.com/security/security-resources/* AWS Security Best Practices:
https://media.amazonwebservices.com/AWS_Security_Best_Practices.pdfThanks in advance for your attention to this report.
How can I contact a member of the Amazon EC2 abuse team or abuse reporter?
Reply this email with the original subject line.
First, hats off to Amazon for alerting us to this, else we might not have known was going on. In short, the EC2 instance we were using to host Logstash and Elasticsearch had significantly elevated levels of outgoing UDP traffic, which was peculiar because we had no knowledge of anything on that box that would be using UDP. This traffic was apparently being used to perform denial-of-service attacks against other servers.
After poking around a bit using utilities like netstat and ps, we noticed that the elasticsearch user was running a process that identified as .Linux_time_y_2. In doing a filesystem search for this, we turned up a file in /tmp. A directory listing revealed more:
ubuntu@ip-10-153-163-198:~$ ls -la tmp
total 1264
drwxrwxr-x 8 ubuntu ubuntu 4096 Jul 9 18:18 .
drwxr-xr-x 6 ubuntu ubuntu 4096 Jul 9 18:28 ..
-rw-r--r-- 1 ubuntu ubuntu 4 Jun 7 00:45 12.txt
-rw-r--r-- 1 ubuntu ubuntu 13413 Jul 8 13:04 32.out
drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 9 18:18 hsperfdata_elasticsearch
drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 9 18:19 hsperfdata_root
drwxr-xr-x 2 ubuntu ubuntu 4096 May 7 19:30 hsperfdata_ubuntu
drwxrwxr-x 2 ubuntu ubuntu 4096 Jul 9 16:39 .ICE-unix
-rw-r--r-- 1 ubuntu ubuntu 90382 May 7 20:14 jffi2343492875620942134.tmp
-rw-r--r-- 1 ubuntu ubuntu 579 Jun 20 07:42 launch.sh
-rwxr-xr-x 1 ubuntu ubuntu 1128800 Jul 8 13:14 .Linux_time_y_2015
-rw-r--r-- 1 ubuntu ubuntu 7378 Jul 8 12:43 mempodipper.c
drwx------ 2 ubuntu ubuntu 4096 May 7 19:30 tmux-1000
drwxrwxr-x 2 ubuntu ubuntu 4096 Jul 9 16:39 .X11-unix
-rw-r--r-- 1 ubuntu ubuntu 657 Jul 5 21:41 zero.pl
The files .Linux_time_y_2015, launch.sh, mempodipper.c, and zero.pl all appeared to be of particular concern. We obtained an archive copy of these files for reference purposes and then removed them from the server after manually killing the running process. Attempting to attach the archive to an e-mail response for Amazon resulted in Gmail rejecting it for containing a virus, which seemed to confirm our suspicions.
Searching for references to the .Linux_time_y_2 process didn't turn up much except for comment #19 on this thread:
I also encountered the same thing, elasticsearch lead to remote execution vulnerability. A few days before being issued to inform service providers ddos attack, I still do not believe, because I have been using public and private key only, without password.
Further searching based on this information turned up a blog post describing an Elasticsearch issue very similar in nature to what the thread comment had described, which the Elasticsearch team confirmed fairly recently.
In addition to the configuration change suggested by the related security advisory to disable dynamic scripting, we found and followed instructions from this blog post to further secure our Elasticsearch installation. We also modified our firewall configuration to block all outgoing UDP traffic to prevent recurrences of the original issue.
We e-mailed Amazon with all of this information the same day we received initial abuse reports from them. Six days later, they responded by sending this:
Dear AWS Customer,
Elasticsearch (http://www.elasticsearch.org/) is a popular open source search server. We were recently made aware of two potential security issues with this software. While these are not issues with AWS, we wanted to notify our potentially affected customers so that they can take appropriate steps.
The first issue is an insecure default configuration for versions of this software earlier than 1.2, outlined here:
CVE-2014-3120 http://bouk.co/blog/elasticsearch-rce/
Attackers that take advantage of this insecure configuration can run arbitrary commands with the privileges of the Elasticsearch daemon.The second issue is a lack of access control, and applies to all versions of Elasticsearch. Anyone that can connect to the search port can query or alter any index on the server. These issues pose the greatest risk when an Elasticsearch server is open to the entire Internet and is running on the default port, 9200/tcp.
The most effective way to avoid any issues is to ensure that your search servers cannot be reached by every host on the Internet. You can use EC2 Security Groups to restrict access to 9200/tcp to only those hosts which should be querying your search index – more information on EC2 Security Groups can be found here:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.htmlIn addition, if you are running a version of Elasticsearch prior to 1.2, you should disable the dynamic script execution support in Elasticsearch – more about this can be found here:
http://bouk.co/blog/elasticsearch-rce/#how_to_secure_against_this_vulnerabilityThe following of your EC2 Security Groups appear to open 9200/tcp to all hosts on the Internet (0.0.0.0/0). If you are using Elasticsearch in production, we recommend that you audit your security groups and, if necessary, take appropriate steps to restrict access to your Elasticsearch servers.
Please let us know if you have any additional questions.
Sincerely,
The AWS Team
Hope this helps someone else. Thanks for reading!