SSH and SFTP Servers on OpenBSD 6.1 – Part 1

Share

Introduction

In this tutorial, we will continue to learn about OpenBSD 6 by setting up a SSH server alongside a SFTP server. As most of you may know, SSH is essential for remote administration. Nowadays, more and more network administration is done remotely and it is more important than ever to properly configure and secure outward facing services. Additionally, we often need to transfer files, either for patches, reuse configuration files or install additional applications, especially if your OpenBSD server is not Internet-accessible. In a later post of this tutorial, we will use a specific form of port knocking to hide the presence of our services to background scans to prevent brute-force and dictionary attacks. However for now, we will setup our SSH and SFTP servers. A companion video is available on YouTube.

Overview

While OpenBSD makes it easy to enable SSH and SFTP, we will do some additional preparation for increased robustness. First we will create the appropriate users and groups and then we will install each of these services within their own chroot directory to limit damage should ever these services get compromised. Once that done, we’ll enable the SSH daemon, including the internal SFTP service, and make sure they are accessible and working properly. Once done, we will customize and harden their configuration. For the purpose of this post, we will assume that you have a fresh install of OpenBSD 6.1 as if you just finished the steps listed in this previous tutorial, i.e. SSHD was NOT enabled during installation.

Creating the Chroot Directories

While optional, enclosing your services in chroot, especially the one exposed to external hosts, is an excellent security practice. Ideally, they should even be on their own partition. To keep the scope of this post on SSH and SFTP, we will simply create a new tree node and setup our chroot containers from there:

We will then install the basic required files from the OpenBSD 6.1 CD to create our chroot. Within the SSH chroot:
# mount -t cd9660 /dev/cd0a /mnt

And unarchive the base packages unto our chroot:

In any chroot, we need the /dev  directory to exist and populated with the required sub directory. Fortunately, OpenBSD makes it easy for us to do so using the MAKEDEV script:

Finally, we will link the shared libraries required by most programs with out chroot and enable our new sandbox:

The chroot directory doesn’t require any special action since it doesn’t need to execute any program. The only thing left to do now is to set the ownership and permissions:

Creating the users and groups

This step is optional if you intent to have this server only for your own personal use or a very few selected people. However if you expect your user base to grow and have different permissions, having groups makes it so much easier to manage your server while ensuring tighter controls on permissions. We’ll create 2 groups: ssh-users and ftp-users. Within each of these groups, we will have a single user. We’ll create ssh-user for accessing the SSH service and ftp-user for accessing the SFTP service by using the useradd and groupadd commands:

While adding groups requires no explanation, it is important to note that ftp-user requires some specific parameters. Namely, the shell is set to /sbin/nologin. FTP are not required to use a shell and are limited to the FTP server, which is actually integrated into the SSH daemon.

Setting up SSHD

In many cases, sshd will have been enabled during installation and automatically starts at boot. If you are unsure, you can quickly verify if the sshd is running:

If there’s no output, then the sshd is not started. Let’s quickly check that SSH works by first enabling and starting the service, and then testing it out. To enable the service, use the rcctl command:

Then start the daemon to actively listen for SSH connections:

And confirm that the SSH daemon is listening by using netstat:

And now, lets try to connect to our SSH service using the IPv6 loopback interface using  # ssh root@::1 . After entering your password, you should be greeted in your new shell.

Without much work, you already have a solid SSH server available within minutes with OpenBSD. However, let’s configure it further to our needs. We’ll stop the server for now:

First configuration items we will change is to disable remote root login and allow users part of the ssh-users group to access SSH. All configuration for the SSH and SFTP services are done thru the /etc/ssh/sshd_config file. Before editing it, you may want to make a backup to your /root directory. We will edit using the vi editor:

The first change we will make is to prevent the root account from using SSH. We’ll do this by setting the PermitRootLogin option to no. If the line is commented, uncommented it by deleting the ‘#‘:

Next we’ll specify SSH to use our specially created chroot sandbox for any SSH connection by adding or setting the following property:

And the last thing for now, we’ll allow only users part of the ssh-users and ftp-users groups to use SSH by setting this line in our configuration file:

Save you configuration and let’s restart our SSH server using rcctl to confirm these basic settings:

First, we should not be allow to login as root anymore. If you try to login as root, you should get denied access:

Excellent. Now let’s log as ssh-user and confirm we are in our chroot sandbox:

Setting up the SFTP Server

Now that we have a basic SSH server running, we’ll configure the basic SFTP server as well. The SFTP server is provided through the SSH daemon and as such, is configured via the sshd_config file as well. This is because SFTP is actually the FTP protocol tunneled through the SSH protocol. As such, the sshd includes a FTP server within itself. To enable SFTP, we simply need to tell sshd to

And ask sshd to use the internal-sftp subsystem by modifying the Subsystem sftp option to Subsystem sftp internal-sftp. Save the file again and restart the SSH server:

And let’s test our SFTP server.

You may run into a few issues here if something is misconfigured.  To help you diagnosed the problem, consult the /var/log/authlog. For example, you encounter this rather cryptic error message when trying to connect to your ftp server:  "Received message too long 1416128883" . This is often caused by the server producing an unexpected output message. In the context of OpenBSD. This may be caused by the banner send by default which is contained in /etc/motd, in which case you need to specify  Banner none in sections relating to the SFTP server.

You may also get disconnected as soon as you attempt to reach your SFTP server, in which case you will get an error message stating:

In which, you can check the log file to troubleshoot your problem

In this case, OpenBSD is complaining about the permissions set on the SFTP chroot directory. If they are too lax, sshd will simply refuse to allow connections to it. As such, make sure you have the appropriate directory permissions.

At this point, you have a server with SSH and SFTP enabled. You can be stop here if this configuration fills you needs, otherwise we can still customize furthermore the servers.

Additional Settings

For one thing, you may want to disable IPv4 if it’s not needed and allow only IPv6. In your sshd_config file, we will do so by setting the AddressFamily to inet6:

Another useful customization depending on your line of work is to change the SSH port. In this case, we will change the port to 443, which is usually reserved for HTTPS connections. The reason for that is that if I need to reach back to my server from a network with connection restrictions, local connections to remote hosts via port 443 is usually allows. In other words, if a firewall is blocking outbound connections to port 22, establishing as SSH connections via port 443 will usually be allowed, unless there are protocol restrictions in place. In any case, we can change the port with:

Depending on your level of paranoia, we can also tighten some controls relating to connectivity to mitigate brute-force or dictionary attacks:

The MaxStartups property is interesting and warrant further details. From the man page:

it specifies the maximum number of concurrent unauthenticated connections to the SSH daemon. Additional connections will be dropped until authentication succeeds or the LoginGraceTime expires for a connection.

It allows for unauthenticated connections to be denied at random in order to mitigate noisy scanning or DDoS from the Internet. In the example above, we specified the value “5:60:10“, which means that if 5 unauthenticated connections are alive, further unauthenticated connections will be refused with a 60% probability. If 10 unauthenticated connections are established, all further attempts will be denied.

And to further ensure additional security controls, confirm that the following parameters are commented so that the defaults value be used:

Authentication Modes

Another feature that you may want to customize based on your needs is how you or your users connect to the server. Three modes are usually considered to do so:

  • PubkeyAuthentication: requires your client to provide a public key in order to connect to your SSH client. If you have a limited number of users which connects from the same location, this is probably the best option. However if you intent to connect to your SSH server from multiple hosts, you would have to bring your public key with you.
  • PasswordAuthentication and ChallengeResponseAuthentication are very similar in practice. PasswordAuthentication request the client to provide a password via the SSH connection while the ChallengeResponseAuthentication can ask the client one or more question via a TTY. However in most cases, ChallengeResponseAuthentication is configured to ask a password and the only real difference is that the requesting client must type the password rather than providing it via the command line.

For example, the following command would not work with PasswordAuthentication set to “no” and ChallengeResponseAuthentication  set to “yes“:

That being said, there are ways to provide SSH password via the command line, such as using the sshpass package. There is therefore little difference between them. In this example, we configure these parameters as follow:

Hashing Known Hosts Files

When a client connects, the SSHD will store information about the client in the known_host file, which is located in ~/.ssh/. This file will contain the hostname of the client, its IP address and its key. This information is stored in plain text. An additional step to make the life of an intruder harder should your server get compromised is to obfuscated the data in this file be hashing its contents.

The listing above shows the contents of the file before hashing it. To tell SSHD to hash newly added data of the known_hosts file, we will add the following the HashKnownHosts line  in ~/.ssh/config.

From now on, all data added will be hashed. Should you need to hash data already residing in this file, use the command below:

We now have a very solid SSH server. You still have to remain vigilant about new vulnerabilities that may pop up for SSHD or one of its component. In the second part of this series, we’ll cloak our SSH server using some form of port knocking. For now, let’s just tweak our SFTP server slightly. Before that, let’s restart our server.

Final Touch on the SFTP server

We previously set our FTP chroot readonly, but we might want to upload some files to it. If we try it right now, we’ll get the following error message:

We’ll finish this tutorial by adding an upload directory and make it writable to the users of the ftp-users group:

Conclusion

We’ll conclude this part of the tutorial for now. In this post, we detailed how to enable a SSH server on OpenBSD 6.1. We also enabled SFTP and securely configure each service to increase robustness. That being said, nothing is impossible and vulnerabilities may remain: keys can be stolen or confiscated, misconfiguration of other services may be present or malicious internal users may still abuse the system. If you’re a network admin, make sure logging is enabled and more importantly, that logs are analyze either via software or if you have time, manually. In the next part of this tutorial, we will enable a form of port knocking to hide our SSH service to scanning from external hosts. By doing so, we will prevent detection by roaming threats and prevent or at least greatly limit effectiveness of brute force and dictionary attacks against our server.

References

Additional Readings

Srizbi is back

Share

Update: The new Estonian company that hosted the command & control server, Starline Web Services, was shut down. The domain name chase continues !

The Srizbi botnet is back online after being shut down by the closure of the criminal hosting company McColo Corp two weeks ago. Srizbi’s command and controls servers, now moved to an Estonian hosting provider, took back control of the botnet[1] in the last days.

The Srizbi Botnet

The Srizbi botnet is mostly a spam generating botnet. According to security firm FireEye, there are 50 variants of the bot, which controls altogether around 500 000 zombies across the world[2]. The most virulent forms of Srizbi are said to control around 50 000 bots.

The Srizbi botnet had a backup procedure in case its C&C servers went down, that is why it got back online very fast. Included in the bot, is a procedure that generates domain names[3] and tries to contact it to see if the C&C is available. Therefore the owners, knowing the random-generating domain name algorithm of the botnet, only had to register one or more of the domain names that will be generated by the bots and install their new control and command server on a machine registered a valid domain name. That is enough for bots to download a new version, pointing to a new address for the botnet. To explain it using pseudo-code, it would look something like this:

More information can be found about the random name generation algorithm at FireEye[4]. Interesting enough, the algorithm is based on date to generate a new set of possible domains names by period. FireEye had successfully discovered this function after McColo closed, but due to financial constraint, they could not register all the domain names that the bot generated. That would have implied to register more than 450 domains each week…

We have registered a couple hundred domains,” Fengmin Gong, chief security content officer at FireEye Inc., “but we made the decision that we cannot afford to spend so much money to keep registering so many [domain] names.[5]

Communications intercepted between a Srizbi bot and its Command and Control Server
Communications intercepted between a Srizbi bot and its Command and Control Server

According to the Symantec Srizbi webpage[6], the worm creates windbg48.sys and another randomly named .SYS file in the %SYSTEM% folder. It then registers the wingdbg48.sys as a driver by inserting the hidden HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\windbg48 key into the Windows’ Registry. Srizbi hides those keys by running in Kernel mode and hooking the ZwOpenKey and ZwEnumerateKey kernel functions among others. It might also try to block access to the registry. A tool is available in order to access the registry anyway.

It will also hide its files by hooking the NTFS file system driver. As if it was not enough, it will also modify the TCP/IP network drivers to bypass Firewalls and Intrusion Detection systems. It will also work in Safe Mode.

For those who wish to go deeper, Windows has two levels of execution: user mode and kernel mode. Usually applications run in user mode, which protects the kernel from applications so they won’t mess up the system. Kernel mode is a privilege mode where services and drivers have access to system resources such as the processor but also the memory… Hooking kernel functions is done by redirecting calls made to the kernel to a custom function. There are a couple of ways to do that in kernel mode, and one of them is to alter the System Service Descriptor Table, which is a table that maps every kernel function to an address in memory. By modifying this table to the address of your custom function, you could hook the kernel. This however would be easily detected by any anti-virus.

Another way is to insert an unconditional jump instruction into the kernel function by modifying the function directly in memory. The advantage of this method is that it’s much harder to detect, and can reproduce the same functionality of the hooked function. This is called inline function hooking.

This why this Trojan can also work in Safe Mode. I don’t know if this particular Trojan uses inline function hooking, but rootkits that uses this kind of hooking are quite hard and dangerous to remove.

Return of Srizbi

When McColo Corp. closed two weeks ago following and investigation by the Washington Post’s Security Fix, it made the news across the Internet as this hosting company was considered responsible for around 75 percent of all the spam sent across the web. Although many rejoiced, including me, at the sudden drop of spam as soon as McColo was turn off[7], everyone knew it was only temporary before the cyber criminals would found another hosting company.

Few knew that this random domain name generating routine was coded to connect to another C&C server though. As soon as it came back online, the first command it received was for a Russian spam campaign. By generating domain names such as yrytdyip.com, auaopagr.com, qpqduqud.com or ydywryfu.com, it was unthinkable for FireEye to register every possibility generated by Srizbi. It is becoming harder and harder to fight botnets on a technical basic. Fortunately, the economic fight could maybe put an end to spam, as mentioned in this Ars Technica article:

“… it suggests that spammers may be extremely sensitive to costs-more so than was previously believed. Even a small increase in the cost of sending an e-mail, they postulate, could have significant ramifications for the botnet industry, and might slow the rate at which it grows or put some spam operations out of business altogether.[8]

The Rustock, Cutwail and Asprox botnets are also making a come back[9], provoking a new surge in spam in the last few days, but not quite yet at the same level of the pre-McColo era.

See also:

Windows Rootkits of 2005, Part One“, James Butler, Sherri Sparks, Security Focus, November 4, 2005, http://www.securityfocus.com/infocus/1850, (accessed on November 27, 2008)

Fallback C&C channels“, Alex Lanstein, Atif Mushtaq, Julia Wolf, and Todd Rosenberry, FireEye, November 16, 2008,  http://blog.fireeye.com/research/2008/11/fallback-cc-channels-part-deux.html#more (accessed on November 27, 2008)


[1] “Massive botnet returns from the dead, starts spamming”, Gregg Keizer, ComputerWorld, November 26, 2008, http://www.computerworld.com/action/article.do?command=viewArticleBasic&articleId=9121678 (accessed on November 27, 2008)

[2] “Srizbi Botnet Re-Emerges Despite Security Firm’s Efforts”, Brian Krebs, Washington Post – Security Fix, November 26, 2008, http://voices.washingtonpost.com/securityfix/2008/11/srizbi_botnet_re-emerges_despi.html?hpid=news-col-blogs (accessed on November 27, 2008)

[3] “Technical details of Srizbi’s domain generation algorithm”, Julia Wolf, November 25, 2008, http://blog.fireeye.com/research/2008/11/technical-details-of-srizbis-domain-generation-algorithm.html (accessed on November 27, 2008)

[4] Ibid.

[5] “Massive botnet returns from the dead, starts spamming”, Gregg Keizer, ComputerWorld, November 26, 2008, http://www.computerworld.com/action/article.do?command=viewArticleBasic&articleId=9121678 (accessed on November 27, 2008)

[6] “Trojan.Srizbi”, Kaoru Hayashi, Symantec, July 23, 2007, http://www.symantec.com/security_response/writeup.jsp?docid=2007-062007-0946-99&tabid=2 (accessed on November 27, 2008)

[7] “Spam plummets after Calif. hosting service shuttered”, Gregg Keizer, ComputerWorld Security, http://www.computerworld.com/action/article.do?command=viewArticleBasic&articleId=9119963 (accessed on November 27, 2008)

[8] “Study: Storm botnet brought in daily profits of up to $9,500”, Joel Hruska, Ars Technica, November 10, 2008, http://arstechnica.com/news.ars/post/20081110-study-storm-botnet-brought-in-daily-profits-of-up-to-9500.html (accessed on November 27, 2008)

[9] “Srizbi botnet active again”, Jeremy Kirk, November 27, 2008, http://www.itworldcanada.com/a/Departmental-and-End-User-Computing/7167ba6c-1cd2-4c54-9338-95a63bea47fa.html (accessed on November 27, 2008)