{ ^_^ } sinustrom Solving life, one problem at a time!

Router NFS performance benchmark

2017-01-21
Author: Zoltan Puskas
Categories: linux benchmark
Tags:

While exploring options for a low power shared network storage for home use, I’ve considered using components already available to me: a TP-Link N750 router and an USB HDD as a lightweight NAS. I’ve decided to run some performance benchmarks on a test setup to assess the viability of it for everyday use.

Considerations

The OpenWRT wiki mentions that one should not expect much performance from such systems as it can be seen from measurements done on a RouterStationPro:

  • basic setup did around 7MB/s writing and 10MB/s reading speeds
  • tuned setup did around 15.3MB/s writing and 17.4MB/s reading speeds

Above bandwidths mean that potentially this setup could be enough for only simple use cases like:

  • streaming music
  • playing video
  • handling medium sized files (photos, documents)
  • regular incremental backups

Note the emphasis is on incremental storage and general data access. Managing photos, editing video or doing anything more IO intensive would still have to be done locally on the laptop.

Setup

Network storage system

Device: TP-Link N750 (WDR4300 v1.7) router

  • CPU: Atheros AR9344@560MHz
  • RAM: 128MB
  • Flash 8MB
  • 2x USB 2.0 ports
  • Network 4x 1Gbps Ethernet + 1x 1Gbps WAN
  • Wireless: 5GHz@450Mbps, 2.4GHz@300Mbps
  • Seagate Momentus (ST9500421AS) 500GB 2.5” HDD in an Inatec USB3 enclosure

OS: OpenWrt Chaos Calmer 15.05.1

Client system

Device: Thinkpad X230

  • CPU: Intel i5-3230M@3.2GHz
  • RAM: 16GB DDR3
  • Disk: Samsung 850 Pro 256GB
  • Network: 1Gbps Ethernet (connected with a CAT-6E cable)
  • Wireless: 2.4Ghz@300Mbps

OS: Gentoo (~amd64)

Configuring the router

First the USB disk for storing content needed to be prepared. I formatted it as a single partition using the ext4 file system.

# fdisk /dev/sdb
n
p
w
q
# mkfs.ext4 /dev/sdb1
# e2label /dev/sdb1 NFS

Now the router had to be set up to make the disk available over the network. For this I logged into router over SSH and then commenced to install USB essential modules.

# opgk install kmod-usb-core kmod-usb2

After that I installed USB storage drivers. Note that some packages will be pulled in automatically, namely kmod-scsi-core, kmod-crypto-hash, and kmod-lib-crc16.

# opkg install kmod-usb-storage block-mount kmod-fs-ext4

At this point the drive should show up after plugging it into the router.

# ls /dev/sd*
/dev/sda    /dev/sda1

To use the drive it needs to be mounted and a publicly read-writeable directory created.

# mkdir -p /mnt/storage
# mount /dev/sda1 /mnt/storage
# mkdir /mnt/storage/pub
# chmod 777 /mnt/storage/pub

To make it accessible via network I chose to use NFS as it’s almost universally supported by all Linux distributions, Windows 8 and above, and MacOS X too. To have NFS service additional packages needed to be installed.

# opkg install nfs-kernel-server

The above command will also pull in kmod-fs-nfs-common, kmod-fs-exportfs, kmod-fs-nfsd, libuuid, libblkid, libwrap, librpc, portmap, kmod-dnsresolver, and kmod-fs-nfs.

Finally the exact share had to be configured and the NFS server started.

# vi /etc/exports
/mnt/storage/pub    192.168.1.2(rw,sync,no_subtree_check)
# /etc/init.d/portmap start
# /etc/init.d/nfsd start
# /etc/init.d/portmap enable
# /etc/init.d/nfsd enable
# cat /proc/fs/nfsd/max_block_size
16384
# cat /proc/fs/nfsd/threads
8

To change the block size for the NFS server the service has to be stopped and the new value written to the mounted nfsd proc tree before starting the NFS service again.

# /etc/init.d/nfsd stop
# mount -t nfsd nfsd /proc/fs/nfsd
# echo 32768 > /proc/fs/nfsd/max_block_size
# /etc/init.d/nfsd start
# cat /proc/fs/nfsd/max_block_size
32768

Configuring the client

I needed to enable NFS support in the kernel first:

File systems  --->
  [*] Network File Systems  ---> 
    --- Network File Systems 
    <M>   NFS client support
    <M>     NFS client support for NFS version 2
    <M>     NFS client support for NFS version 3
    [*]       NFS client support for the NFSv3 ACL protocol extension
    <M>     NFS client support for NFS version 4
    [ ]     Provide swap over NFS support
    [*]   NFS client support for NFSv4.1
    [*]     NFS client support for NFSv4.2

After recompiling the kernel I was ready to mount the router’s NFS share. To make testing more convenient I’ve added a line to /etc/fstab

# cat /etc/fstab | grep nfs
192.168.1.1:/mnt/storage/pub    /mnt/nfs    nfs noauto,rw,sync,users,rsize=16384,wsize=16384 0 0

and started the NFS client before mounting the NFS share.

# /etc/init.d/nfsclient start
# mount /mnt/nfs

Measurements

Input data

In order to make my tests representative of the most generic use cases I’ve prepared a sample data set on my local host. Since viewing photos is similar to reading documents, as they are about the same size, I did not do a separate test for documents access specifically.

$ du -d1 --block-size=1
782979072       ./linux-4.9
928673792       ./Movies
436101120       ./Music
151928832       ./Pictures
2299682816      .
$ find Pictures -type f -printf '%s\n' | awk '{s+=$0} END {printf "Count: %u\nAverage size: %.2f\n", NR, s/NR}'
Count: 52
Average size: 2919623.21
$ find Music -type f -printf '%s\n' | awk '{s+=$0} END {printf "Count: %u\nAverage size: %.2f\n", NR, s/NR}'
Count: 11
Average size: 39644094.91
$ find Movies -type f -printf '%s\n' | awk '{s+=$0} END {printf "Count: %u\nAverage size: %.2f\n", NR, s/NR}'
Count: 1
Average size: 928670754.00
$ find linux-4.9 -type f -printf '%s\n' | awk '{s+=$0} END {printf "Count: %u\nAverage size: %.2f\n", NR, s/NR}'
Count: 56206
Average size: 11567.46

Where Pictures contained a set of 6MP photos in JPG format, Movies had the Big Buck Bunny 1080p test video, Music contained a grabbed copy of an album from one of my CD-s in FLAC format, and linux-4.9.0 was a copy of a kernel source directory.

Method

The following operations were benchmarked:

  • copy test data from client to the empty networked storage using rsync
  • copy test data from networked storage to empty directory on client using rsync
  • stream video by playing it with vlc
  • stream audio by playing it with clementine
  • view photos with gwenview

Above measurements were made with different NFS options (only mount options that have changed are mentioned here):

NFS options in /etc/exports Client options in /etc/fstab
sync sync,rsize=16384,wsize=16384
async sync,rsize=16384,wsize=16384
sync sync,rsize=32768,wsize=32768
async sync,rsize=32768,wsize=32768

For details on these options see man pages nfs(5), exports(5) and mount(8).

Between the async and sync tests the remote storage was wiped and the router restarted. Setting async on the client side made no sense, as the client can easily cache all the content in memory and return with success immediately, while writing data in the background and thus providing us with non realistic numbers on the network speeds.

To get transfer statistics I’ve used rsync with the following flags:

rsync --stats -a /src/path /dst/path

To measure CPU usage on the router I’ve used the top command as well as the load graphs provided by the LuCI interface. No load measurements were made on the client side as the router is the clear bottleneck from the point of view of this setup.

To always force all IO to happen, before every measurement I’ve dropped all the caches both on the client machine and the router using the following command:

# echo 3 > /proc/sys/vm/drop_caches

On the client side the data was stored in tmpfs. Additionally I made sure there is no other network traffic going through the router to get the best case performance numbers out of the setup.

Finally when streaming I only evaluated whether the audio and video files play flawlessly, without interruption, while for photos I tried whether viewing them as a slide show works reasonably well.

Results

Transfer speeds

Data transfer rates for server/client settings with block size of 16384.

Data Upload [bytes/s] Download [bytes/s]
sync/sync async/sync sync/sync async/sync
Movies 6 051 450 6 131 337 7 401 575 7 343 065
Music 5 934 596 6 016 452 7 330 972 7 209 799
Pictures 5 730 622 5 730 622 6 749 400 7 063 325
linux-4.9 378 511 763 424 2 645 593 2 745 426

Data transfer rates for server/client settings with block size of 32768.

Data Upload [bytes/s] Download [bytes/s]
sync/sync async/sync sync/sync async/sync
Movies 7 582 838 7 708 694 7 521 439 7 582 838
Music 7 585 962 7 585 962 7 585 962 7 456 288
Pictures 6 749 399 7 063 324 7 407 878 7 063 325
linux-4.9 396 629 767 001 2 745 426 2 722 595

Router CPU load

The following table shows typical CPU load obtained from top on the router under different scenarios.

NFS Transfer CPU load [%]
operation a/sync usr sys nic idle io irq sirq
Large files to NFS sync 0 43 0 0 26 0 28
Large files to NFS async 0 46 0 1 26 0 24
Small files to NFS sync 0 34 0 3 44 0 17
Small files to NFS async 0 45 0 12 25 0 15
Large files to client sync 0 59 0 0 11 0 28
Large files to client async 1 58 0 0 10 0 29
Small files to client sync 0 47 0 12 21 0 18
Small files to client async 0 42 0 14 17 0 24
Multimedia streaming sync 0 7 0 83 5 0 2

I’ve also selected a few load graphs, which are a representative set for the CPU load on the router during the data transfers.

Router idle RAM usage with NFS running Router idling Copying 'Movies' to NFS (sync) Copying 'Movies' to NFS (async) Copying 'Movies' to client (sync) Copying 'Movies' to client (async) Streaming from 'Movies' (sync) Copying 'linux-4.9' to NFS (sync) Copying 'linux-4.9' to NFS (async) Copying 'linux-4.9' to client (sync) Copying 'linux-4.9' to client (async) Copying 'Music' to NFS (sync) Copying 'Pictures' to NFS (sync)

Usability

Streaming music or HD video was not an issue and worked smoothly. Given the bandwidth requirements for these (around 1.5Mbit/s or for FLAC and 12.5Mbit/s for MPEG-4 + AC-3) such a setup can easily serve multiple of these streams.

Looking at pictures was very usable in a slide show, however with higher resolution pictures there could be a noticeable lag, unless the viewer application caches images ahead of time.

Finally streaming a HD video while uploading an incremental backup also works flawlessly, though the upload bandwidth is reduced.

Conclusions

Performancewise we can see that the router is only capable of limited transfer speeds. Even though being on the slow side for a local network, they are still comparable with average broadband speeds (60/60 Mbit/s for up/down link respectively). This means such a setup could be still be very valuable on slower, or data capped Internet connections, or even as a self maintained cloud storage.

In async mode significant speedup was only gained during the upload of many small files. Increasing the NFS server block size to 32k improves the speeds across the board however, bringing up and download speeds practically to equal. At this point the transfer speeds seemed to reach their maximum potential.

Naturally there was some memory pressure on the router when doing large copies, but it was not the primary limiting factor. It was the rather limited the IO capabilities of the SoC in the router. This is confirmed by the CPU, RAM usage numbers. More RAM would of course mean more FS cache and thus better burst performance, but a better CPU with the same amount of memory would yield faster sustained performance. USB 2.0 ports were clearly in High Speed mode, but we were far from hitting the limits, and thus were not a limiting factor either.

Overall I found that the performance is acceptable for the average workflow outlined at the beginning of this benchmark and this setup could likely serve a few clients at the same time. Using such a setup as a large background storage complements really well the laptop that has much smaller, but very fast SSD based disk.

The only significant drawback of this solution is that we have no hope of applying block or FS level encryption to the NFS drive while maintaining network performance due to the router’s weak CPU.


Content