Ftp

From freebsd.xn--wesstrm-f1a.se

Jump to: navigation, search

Contents

Overview

Now that you have a working web server you need a simple way to transfer your web contents to it and most people are probably used to ftp for that task. FreeBSD comes with an ftp server built-in and you just have to activate it.

Warning: Ftp has two modes of transferring a file; BINARY and ASCII. BINARY mode will transfer any file unaltered while ASCII mode is meant for converting new lines in text files between systems with different representation of a new line. In Windows a new line is represented by two bytes; carriage return + line feed (CR+LF), while in Unix a new line is only one byte; line feed (LF). If you transfer a binary file in ASCII mode between Unix and Windows, the file will have any occurrence of the source system's representation of a line feed converted to the destination system's representation and the file will be corrupt because there are almost always random occurrences of CR+LF and/or LF in a binary file.
Corruption can also occur when you actually want to use ASCII mode for a text file and the transfer is interrupted and you then want to resume it. The only way for the source system to know how much data was correctly transferred is to request a directory listing from the destination and look at the byte count. But there is no mechanism in ftp to know how many new line conversions have been made so far so the reported byte count will be interpreted from the perspective of the source system and the resume will begin at the wrong position and the file will be corrupt at the resume point.

Instructions

Access ftp from internal LAN

Making the ftp server accessible from the LAN only is a very simple procedure. The ftp server is very small and can be started by Inetd when requested, just like we start the pop3 server, and everything is already prepared for it. Edit /etc/inetd.conf and find the following line:

#ftp    stream  tcp     nowait  root    /usr/libexec/ftpd       ftpd -l

Uncomment it and save the file.

ftp    stream  tcp     nowait  root    /usr/libexec/ftpd       ftpd -l

Note that there is an almost identical line below for IPv6 (tcp6) but leave it alone. It's only used if you want to access the ftp server with IPv6.

Now you could restart Inetd to make it read the new configuration but there is another way to make it read its configuration without having to stop it. Many services can be forced to reread their configuration without restarting it by sending it the HUP signal.

# killall -HUP inetd

You should now be able to ftp to your router from your LAN and you logon with the normal user accounts you have added in FreeBSD.

Access ftp from the Internet

Allowing ftp access from the Internet complicates things. Ftp is a very old protocol and there are both security issues and technical issues that must be addressed.

Security

Remember that there is no encryption support in FreeBSD's ftp server so all logons are made in plain text. Anyone who is able to eavesdrop on your Internet connection can see your password. Preferably you should establish an ssh tunnel to the router and ftp through it but this is after all a hobby project so let's not make it too complicated right now. However, an ftp server is probably the second best target for a hacker after root shell access and you will have plenty of hacker attempts as long as you run the ftp server on its standard port 21. The first thing to do is therefore to move it somewhere else. This is security by obscurity but it's the best you can do for now. Open /etc/services in your editor and find the ftp tcp line and make a copy of it (don't mix it up with the udp version right below it):

ftp              21/tcp    #File Transfer [Control]

Select a new port on random between 1024 and 49151 and make sure it isn't already defined in this file. In this example I'll use 12345. Paste the old ftp line in its new place and change the service name to ftp2 instead and adjust the port number. Then save the file and exit.

ftp2            12345/tcp  #File Transfer [hidden]

Open /etc/inetd.conf again and restore the comment sign first on the ftp line that you removed previously. Then make a copy of that line, paste it below and change it like this (of course removing the comment on this line):

ftp2    stream  tcp     nowait  root    /usr/libexec/ftpd       ftpd -ll

ftp2 will make sure it won't interfere with ftp and yes, those are double lower case 'L's at the end. They will increase the level of logging so you better can monitor the use of your ftp server. After you reload Inetd's configuration, the ftp server should now respond on port 12345 instead of port 21.

# killall -HUP inetd

Active and passive mode

To better understand the firewall rules you need to add to make this work, I first have to get a little technical and explain how ftp communication works and why it sucks so immensely.

Under normal circumstance the client opens a random local outgoing port and connects to port 21 on the ftp server (this will be 12345 in my example). This is the control connection and when it's established it's used to send all commands to the server, like requests for directory listings and requests for file transfers. The actual data transfer, be it a directory listing or a file, isn't carried out here but will be performed on a totally separate and parallel connection.

In active mode the client opens another random local port, sends this port number and its local IP address to the server and then tells the server over the control connection to connect to it on this secondary port and transfer the file. This will be problematic because most clients today are behind firewalls and the firewall has no way of knowing that an internal client has opened a local port and expects the firewall to suddenly let the external ftp server initiate a connection through the firewall to that port. Remember that no data has been sent from the client on this port so no state has been created in the firewall to allow traffic back in. An even bigger problem is that most clients probably are behind a NAT device and use internal 192.168.x.x addresses. This is the IP address the ftp server is sent and expected to connect to and the server has of course no chance in hell to do that since these IP addresses aren't routable on the Internet at all. The data you requested will disappear in cyberspace and never come close to your computer. You probably realize now that the ftp protocol was invented long before there was NAT and firewalls.

In passive mode the client can work around these limitations. The control connection is established in the same way as in active mode but the data connection procedure is now reversed. In passive mode the server opens a random listening port, sends the client this port number and its IP address over the control connection and tells the client to connect to the server and transfer the file. This eliminates the problem for the client since outgoing traffic usually is allowed through the firewall. Instead it's up to the ftp server administrator to solve the firewall and NAT problems which in this example means you! :-)

Firewall rules

NAT won't be a problem for your ftp server since it runs directly on the router and uses its public IP address when a connection is made from the Internet. You only have to deal with the firewall rules and you need a rule allowing incoming connections on port 12345 but also a rule allowing incoming connections on all those random data ports the ftp server can listen to. By default in FreeBSD these incoming data ports can be anyone from 49152 to 65535. Opening up more than 16,000 ports in the firewall is of course unacceptable. Any misconfigured service, that accidentally binds to the external interface and a port in that range, would be completely unprotected. If you really need external ftp access to your router you can't eliminate this risk completely but you can at least narrow the port range considerably. That however could theoretically lead to a situation where the ftp server runs out of available ports. The port range above is called "Dynamic and/or Private Ports" in FreeBSD and its boundaries are controlled by two kernel variables. The ftp server reads these variables when it starts. To look at their current values you can type the following:

# sysctl net.inet.ip.portrange.hifirst
net.inet.ip.portrange.hifirst: 49152
# sysctl net.inet.ip.portrange.hilast
net.inet.ip.portrange.hilast: 65535

Keep the upper boundary and only change the lower value, reserving 256 ports:

# sysctl -w net.inet.ip.portrange.hifirst=65280
net.inet.ip.portrange.hifirst: 49152 -> 65280

Make this new value permanent by adding it to /etc/sysctl.conf:

# echo net.inet.ip.portrange.hifirst=65280 >> /etc/sysctl.conf
Note: Every file you transfer and every directory listing you perform will use one unique random port in this port range. When the transfer has finished the port will be closed but it can't be reused until 90 seconds later. If you transfer hundreds of very small files, which isn't unreasonable when it comes to html files, you'll probably run out of available ports. Other applications might use this port range too and increase the probability that you run out of them. You'll have to increase the port range in that case, adjusting the lower port boundary to a lower value.

Then add the following inbound rules last in the section called # Incoming to router. Their exact placement doesn't actually matter right now since there are no other rules there monitoring the same ports. In general though, you want to keep high priority, low bandwidth rules in the beginning of each section and default priority, high bandwidth rules at the end. The placement of the friends ftp rules before the default ftp rules are important though in case you want to allow ftp access to some of your friends in the future. You want that traffic to go into q_p2p instead of q_def so it must be matched first by the rule set.

pass in quick on $ext_if inet proto tcp from <friends> to ($ext_if) port ftp2 flags S/SA synproxy state queue (q_p2p, q_p2)
pass in quick on $ext_if inet proto tcp from any to ($ext_if) port ftp2 flags S/SA synproxy state queue (q_def, q_p2)
pass in quick on $ext_if inet proto tcp from <friends> to ($ext_if) port 65280:65535 flags S/SA synproxy state queue (q_p2p, q_p2)
pass in quick on $ext_if inet proto tcp from any to ($ext_if) port 65280:65535 flags S/SA synproxy state queue (q_def, q_p2)

Don't forget to reload the firewall rules.

# pfctl -nf /etc/pf.conf
# pfctl -F rules ; pfctl -f /etc/pf.conf

Creating a public ftp account

There are many configuration files controlling the ftp server's behaviour and you can read all about them in the manual page. You can get acquainted with a couple of them by creating a general ftp storage area with a single user account that will only be allowed to access your router through ftp. First add a new user like you did with the web user but make sure you select nologin as the shell this time.

# adduser
Username: upload
Full name: upload
Uid (Leave empty for default): 
Login group [upload]: 
Login group is upload. Invite upload into other groups? []: 
Login class [default]: 
Shell (sh csh tcsh nologin) [sh]: nologin
Home directory [/home/upload]: 
Use password-based authentication? [yes]: 
Use an empty password? (yes/no) [no]: 
Use a random password? (yes/no) [no]: 
Enter password: 
Enter password again: 
Lock out the account after creation? [no]: 

Verify that the information is correct and accept the new user.

Username   : upload
Password   : *****
Full Name  : upload
Uid        : 1004
Class      : 
Groups     : upload 
Home       : /home/upload
Shell      : /usr/sbin/nologin
Locked     : no
OK? (yes/no): yes
adduser: INFO: Successfully added (upload) to the user database.
Add another user? (yes/no): no
Goodbye!

The nologin shell used here is a special script that shows the user a message and immediately logs him off if he tries to logon with ssh. You should also lock (chroot) the user to his home folder so he can't poke around among the system files. A regular user don't even have read access to the most sensitive files in the system but there are still plenty of files he can access so for this purpose it's better to lock him down in his home folder which then will act as the general storage area. In principle you could've chosen a completely different folder for this but /home is as good as any. To chroot this user you add the userid to /etc/ftpchroot.

# echo upload >> /etc/ftpchroot

For the user to be allowed ftp access he must have a valid shell as listed in /etc/shells. For some reason nologin isn't listed here by default so you'll have to add it.

# echo /usr/sbin/nologin >> /etc/shells

Don't forget to add upload to /etc/mail/aliases and recompile that database.

# cd /etc/mail
# echo "upload: pp" >> aliases
# make

Now you can try to logon with upload through ssh and make sure you don't get in. Then a test through ftp and make sure you can't get outside your home folder. While you're there you can delete the hidden dot-files. They're not needed for ftp access.

Note: About resuming interrupted ftp transfers. Although the server supports resuming both on download and upload, it's not without it's complications. If a client gets interrupted while downloading from the server, it knows exactly how many bytes has been received so it's easy to tell the server to continue at an exact byte offset. But if the interruption occurs during upload the client doesn't know exactly how much data has been transferred correctly. The only way to find out is to list the folder contents and look at the file size. But this can be misleading if the client is a Windows machine and the files are in ASCII format for example. In Windows a new line occupies two bytes while they only need one byte in Unix. This conversion is done automatically by the server during transfer but a resume under these conditions will result in a corrupt file. For ftp transfers, checksum (MD5) files are vital as well as splitting large files in several smaller ones. In case something goes wrong you only need to upload one of the smaller parts again instead of the complete file. Many Windows based ftp clients also have an option to say if the remote server is a Unix host or not. This option has to be correctly set or some ftp commands will fail to work.

Summary

  • Kernel variables can be changed in /etc/sysctl.conf.
  • All ftp activity will be logged in /var/log/xferlog

Unresolved issues

  • I'll never feel comfortable with opening all those ports. Trying the user directive in the pf rules gets complicated since it's Inetd listening to the ftp port initially and it runs as user root. I would also have to write specific outgoing rules with the user directive to capture the active connections. Would a proxy of some sort be a better option? One that could integrate with pf smoothly if it even exists. Pointers are welcome.
  • Once again I lack the knowledge of setting up an ftp server that supports encryption. There are a few in the Ports Collection so a nice guide for this would be appreciated.

References


Next guide: MySQL
Personal tools