Connecting a Topfield PVR to a Raspberry Pi

I have a Topfield 5800 PVR (personal video recorder). It has a USB socket that can be used to transfer software and recordings to or from the PVR. Connecting this USB socket to my Raspberry Pi bare bones computer (that's on all the time) allows me to do a few useful things:

• Download up to 2 weeks of EPG data to supplement the 1 week that's broadcast over the air
• Update PVR software by FTP from my main computer
• Copy radio recordings from the PVR and convert them to mp3 format for listening to when out and about
• Backup various settings from the PVR

Preliminaries

The Raspbian operating system includes the "gphoto2 digital camera library" - a package that interfaces to a wide range of digital cameras. Unfortunately it thinks it can also connect to the Topfield PVR, so it has a "udev" rule to take ownership of it. Disabling this udev rule is quite easy, but will need to be done every time libgphoto2 is updated.

Firstly, connect the PVR to the Pi, turn it on and then use lsusb to check the "idVendor" and "idProduct" of the PVR:

jim@firefly ~ $ lsusb
Bus 001 Device 002: ID 0424:9512 Standard Microsystems Corp. 
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp. 
Bus 001 Device 004: ID 1a40:0201 Terminus Technology Inc. FE 2.1 7-port Hub
Bus 001 Device 005: ID 1941:8021 Dream Link WH1080 Weather Station / USB Missile Launcher
Bus 001 Device 007: ID 148f:5370 Ralink Technology, Corp. RT5370 Wireless Adapter
Bus 001 Device 008: ID 059f:0351 LaCie, Ltd 
Bus 001 Device 092: ID 11db:1000 Topfield Co., Ltd. PVR

This shows the idVendor to be 11db and the idProduct to be 1000. Now we can find the udev "rules" file that contains the idVendor value:

jim@firefly ~ $ grep 11db /lib/udev/rules.d/* /etc/udev/rules.d/*
/lib/udev/rules.d/60-libgphoto2-2.rules:ATTRS{idVendor}=="11db", ATTRS{idProduct}=="1000", ENV{ID_GPHOTO2}="1", ENV{GPHOTO2_DRIVER}="proprietary", MODE="0664", GROUP="plugdev"

This shows the rule to be in /lib/udev/rules.d/60-libgphoto2-2.rules, so edit this file (sudo will be needed) and comment out the line by inserting a # character at the beginning.

Now we can create a new group, add our user to that group and then create a new udev rule to assign the PVR to that group:

sudo addgroup --system toppy
sudo adduser jim toppy
sudo echo 'ATTRS{idVendor}=="11db", ATTRS{idProduct}=="1000", GROUP="toppy"' >/etc/udev/rules.d/10-topfield-pvr.rules
sudo udevadm control --reload
sudo udevadm trigger

The USB device corresponding to the PVR (e.g. /dev/bus/usb/001/092) should now belong to the "toppy" group.

Install software

There are two programs used to communicate with the PVR - "puppy" and "ftpd-topfield". These can be downloaded from http://birdman.dynalias.org/cgi-bin/tdl.cgi/armv5tel/puppy-new_dev_scan-static and http://birdman.dynalias.org/cgi-bin/tdl.cgi/armv5tel/ftpd-new_dev_scan-static. I renamed the downloaded files (removing the "-new_dev_scan-static" bit) and put them in /home/jim/bin.

A quick test is to use puppy to get the PVR's disc size and free space:

jim@firefly ~ $ /home/jim/bin/puppy -c size
Total  976749568 kiB  953857 MiB  931 GiB
Free   386535424 kiB  377476 MiB  368 GiB

Testing ftpd-topfield is a little bit more complicated. On the Pi, start the ftpd-topfield daemon in debug mode:

sudo /home/jim/bin/ftpd-topfield -D -P 2021 -d

Then on another machine FTP to port 2021 and get a directory listing:

jim@brains:~$ ftp jim@firefly 2021
Connected to firefly.tracy.island.
220 firefly FTP server (Topfield ftpd 0.7.7p) ready.
331 Guest login ok, type your name as password.
Password:
230 Guest login ok, access restrictions apply.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> dir
229 Entering Extended Passive Mode (|||33252|)
150 Opening ASCII mode data connection for 'file list'.
drwxr-xr-x   1 none     none                0 Jan  1  2003 DataFiles
drwxr-xr-x   1 none     none                0 Jan  1  2003 MP3
drwxr-xr-x   1 none     none                0 Jan  1  2003 ProgramFiles
drwxr-xr-x   1 none     none                0 Jan  1  1970 firmware
drwxr-xr-x   1 none     none                0 Jan  1  1970 turbo
226 Transfer complete.
ftp> bye
221 Goodbye.

You should see some debug output on the Pi when you do this. Finally kill the ftpd-topfield program with <Ctrl>-C.

Using xinetd to start ftpd-topfield

We can't leave ftpd-topfield running all the time, as each time the PVR switches on it gets a new USB device number. The solution is to install the "xinetd" package which can start ftpd-topfield in response to an incoming FTP connection.

Firstly install xinetd:

sudo apt-get install xinetd

Then create a file /etc/xinetd.d/ftp-topfield with the following contents:

service ftpd-topfield
{
        type            = UNLISTED
        socket_type     = stream
        protocol        = tcp
        port            = 2021
        server          = /home/jim/bin/ftpd-topfield
        server_args     = --port=2021 --logging
        wait            = no
        user            = root
        disable         = no
        instances       = 1
}

and restart xinetd:

sudo /etc/init.d/xinetd restart

Now you should be able to FTP from the another machine again. Monitoring processes on the Pi (with ps ax) should show an ftpd-topfield process running when the FTP connection is active.

Running a script when the PVR switches on

The final step in automating transfer of  files to and from the PVR is to run a script each time it turns on. This requires a simple modification to the udev rule created above, adding the name of a script to run:

jim@firefly ~ $ cat /etc/udev/rules.d/10-topfield-pvr.rules 
ATTRS{idVendor}=="11db", ATTRS{idProduct}=="1000", GROUP="toppy", RUN+="/home/jim/scripts/toppy-add.sh"

The script "/home/jim/scripts/toppy-add.sh" is run as root by udev, so if the user isn't my normal user it uses at to restart itself in the background. This allows udev to finish while the script is still running - vital to avoid blocking udev.

The script creates a lock file (to ensure only one instance of it runs at any one time) and calls my Python scripts that use FTP to do all the things I want:

jim@firefly ~ $ cat /home/jim/scripts/toppy-add.sh
#!/bin/sh
#
# toppy-add script
#
# run by hotplug when Toppy switches on

# check lock file
lock_file=/var/lock/toppy-add.lock
if [ -f $lock_file ]; then
  exit
  fi

# udev runs as root, so restart in background as user jim
if [ "$USER" != "jim" ]; then
  echo "sudo -u jim /home/jim/scripts/toppy-add.sh" | at now
  exit 0
  fi

touch $lock_file

# allow Toppy to settle and load TAPs
sleep 60

log=/tmp/log-toppy-add
rm -f $log

python /home/jim/scripts/imagetoppy.py >>$log 2>&1
python /home/jim/scripts/mei2toppy.py >>$log 2>&1
python /home/jim/scripts/toppy2pod.py >>$log 2>&1

# mail the log file
if [ -s $log ]; then
  /home/jim/scripts/email-log.sh $log "toppy-add log"
  fi

rm -f $lock_file

My first Raspberry Pi - day 4

Add normal user account

On other computers in my home network I have the user name "jim". For convenient logging into the Pi it's useful to have an account with the same name. This is easily created as follows (when logged in as root):

adduser jim
mkdir /home/jim/.ssh
chmod go-rx /home/jim/.ssh
cp /home/pi/.ssh/authorized_keys /home/jim/.ssh
chown -R jim:jim /home/jim/.ssh

Now I can login from my main computer with a simple ssh firefly command.

It's convenient to allow this account to run commands as root using sudo. This requires editing the /etc/sudoers file, which can only be done via the visudo command. I duplicated the last line in the file (for user "pi") and changed the user name to "jim".

Samba setup

Although most of the machines on my home network are running Linux of one sort or another it's still convenient to use samba to share files across the network. Enabling samba on the Pi is fairly straightforward, although there are too many configuration options, as in all samba setups. Firstly, install the samba packages:

sudo apt-get install samba samba-common-bin

Now edit the /etc/samba/smb.conf file to suit your network. I changed the following lines:

[global]
   workgroup = HOME
   interfaces = 127.0.0.0/8 eth0 wlan0
   security = user
   domain logons = no
   load printers = no
   domain master = no
[homes]
   read only = no

I also commented out the [printers] section.

After editing the configuration, the samba server needs to be restarted and a password for my user name created:

sudo service samba restart
sudo smbpasswd -a jim

Command aliases

There are some variations of standard Linux commands that I expect to find wherever I log in. These are easily added to all users' accounts by editing /etc/profile and adding the following lines:

alias df='df -h'
alias la='ls -ao'
alias ll='ls -l'

Weather station setup

I have a weather station with a USB interface which I've been using with another computer for several years to collect weather data and upload it to various web sites. The Pi is an ideal platform for doing this sort of thing as it uses little enough power that I don't mind it being on all the time. I've written some software called pywws that is well suited to small computers.

The first step is to create a suitable directory and download the pywws software:

mkdir weather
cd weather
git clone https://github.com/jim-easterbrook/pywws.git

Next connect up the weather station and make sure it's detected by running lsusb:

jim@firefly ~/weather $ lsusb
Bus 001 Device 002: ID 0424:9512 Standard Microsystems Corp. 
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp. 
Bus 001 Device 004: ID 1a40:0201 Terminus Technology Inc. FE 2.1 7-port Hub
Bus 001 Device 005: ID 148f:5370 Ralink Technology, Corp. RT5370 Wireless Adapter
Bus 001 Device 006: ID 058f:6390 Alcor Micro Corp. USB 2.0-IDE bridge
Bus 001 Device 007: ID 1941:8021 Dream Link WH1080 Weather Station / USB Missile Launcher

Before we can read anything from the station we need to install a Python USB library. There is a choice of library to use with pywws, but in general "pyusb" is the best one to use. Different Linux distributions use different names for this package, so the safest thing to do is to use apt-cache to search for it:

jim@firefly ~/weather $ apt-cache search usb | grep -i python
python-ftdi - Python module to control and program the FTDI USB controller
python-imobiledevice - Library for communicating with iPhone and iPod Touch devices
python-obexftp - Python binding to the object exchange file transfer library
python-ow - Dallas 1-wire support: Python bindings
python-usb - USB interface for Python
python-usbtc08 - Python wrapper for libusbtc08

This shows that python-usb is the one we want, so install it with sudo apt-get install python-usb.

Now we can test the weather station link:

jim@firefly ~/weather $ cd pywws/
jim@firefly ~/weather/pywws $ sudo python -m pywws.TestWeatherStation -v
12:04:06:pywws.WeatherStation.CUSBDrive:using pywws.device_pyusb
0000 55 aa ff ff ff ff ff ff ff ff ff ff ff ff ff ff 05 20 01 51 11 00 00 00 81 00 00 e5 07 00 d0 f6
0020 17 0c 17 0c 00 00 00 00 00 00 00 13 10 11 12 01 41 23 c8 00 32 80 47 2d 2c 01 2c 81 5e 01 1e 80
0040 a0 00 c8 80 a0 28 80 25 a0 28 80 25 03 36 00 05 6b 00 00 0a 00 f4 01 18 00 00 00 00 00 00 00 00
0060 00 00 55 1c 63 0a 2f 01 59 00 a2 01 59 80 a2 01 59 80 fb 00 a1 80 77 54 00 00 fe ff 00 00 ce 01
0080 0c 02 d0 ff d3 ff 5a 24 d2 24 dc 17 00 12 07 17 23 23 10 03 07 22 18 10 08 11 08 30 11 03 07 12
00a0 36 08 07 24 17 17 13 01 18 23 28 12 07 16 14 21 12 02 11 06 57 12 07 16 14 21 12 02 11 06 57 12
00c0 08 19 08 46 13 10 11 10 55 13 07 14 16 39 11 09 13 17 19 11 08 21 16 53 11 09 13 17 19 12 07 16
00e0 14 21 10 02 22 11 06 11 11 06 13 12 11 11 06 13 12 11 11 10 11 38 11 11 10 11 38 10 02 22 14 43

Note the use of sudo - the normal user "jim" currently doesn't have permission to access the weather station USB device. This requires creating a "udev" rule to give access to the station to a particular group of users, and adding "jim" to that group:

sudo addgroup --system weatherstation
sudo adduser jim weatherstation
sudo vi /etc/udev/rules.d/39-weather-station.rules

The udev rule file should have these three lines:

ACTION!="add|change", GOTO="weatherstation_end"
SUBSYSTEM=="usb", ATTRS{idVendor}=="1941", ATTRS{idProduct}=="8021", GROUP="weatherstation"
LABEL="weatherstation_end"

After rebooting the normal user "jim" can run TestWeatherStation.

The rest of the setup follows the pywws documentation. I installed the following packages:

sudo apt-get install gnuplot
sudo apt-get install python-oauth2
sudo apt-get install gettext python-sphinx

The version of python-tweepy installed by apt-get is out of date (it doesn't work with the current Twitter API) so I installed a more recent version using pip:

sudo apt-get install python-pip
sudo pip install tweepy

My first Raspberry Pi - day 3

External disc drive setup

I need to attach a decent size disc drive to my Pi to store radio recordings from my PVR. It will also allow me to create files, and use swap space, without worrying about wearing out the SD card. Given that I'm using a disc drive, it makes sense to put all the Pi's software on it, relegating the SD card to a "bootloader".

I found a useful guide to running a Pi from an external disc drive on raspberrypihobbyist.blogspot.co.uk. I've done things a bit differently, of course...

Firstly, I created four partitions: 2 of 8GB for "system" and "spare", 2GB swap and the remainder of the disc (about 212GB) for "home". Then I created and enabled the swap space, then formatted the other partitions:

fdisk /dev/sda
mkswap /dev/sda3
swapon /dev/sda3
mkfs.ext4 /dev/sda1
mkfs.ext4 /dev/sda2
mkfs.ext4 /dev/sda4

The next stage is to copy everything needed from the SD card, and create other directories and mount points:

mkdir /mnt/system
mount /dev/sda1 /mnt/system
cd /mnt/system
rsync -av /{bin,etc,lib,opt,root,sbin,selinux,srv,usr,var} .
mkdir boot dev home media mnt proc run spare sys tmp

mkdir /mnt/home
mount /dev/sda4 /mnt/home
cd /mnt/home
rsync -av /home/* .

Finally, edit /boot/cmdline.txt and /mnt/system/etc/fstab:

/boot/cmdline.txt:
dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/sda1 rootfstype=ext4 elevator=deadline rootwait

/mnt/system/etc/fstab:
proc            /proc           proc    defaults          0       0
/dev/mmcblk0p1  /boot           vfat    defaults,ro       0       2
/dev/sda1       /               ext4    defaults,noatime  0       1
/dev/sda2       /spare          ext4    defaults,noatime  0       1
/dev/sda3       none            swap    sw                0       0
/dev/sda4       /home           ext4    defaults,noatime  0       1

On reboot, check that everything's mounted as expected:

root@firefly:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs          7.9G  2.0G  5.6G  27% /
/dev/root       7.9G  2.0G  5.6G  27% /
devtmpfs        235M     0  235M   0% /dev
tmpfs            49M  264K   49M   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           507M     0  507M   0% /run/shm
/dev/mmcblk0p1   56M   19M   38M  33% /boot
/dev/sda2       7.9G  146M  7.4G   2% /spare
/dev/sda4       212G  190M  201G   1% /home
root@firefly:~# free -h
             total       used       free     shared    buffers     cached
Mem:          485M        59M       426M         0B       6.2M        27M
-/+ buffers/cache:        25M       460M
Swap:         2.0G         0B       2.0G
root@firefly:~# 

Make /tmp a ramdisc

For optimum performance it's nice to have /tmp in memory instead of being written to disc. With a large swap partition I needn't worry about running out of memory. If /tmp begins to fill up its least used files will be swapped out, leaving the active ones in memory.

All this requires is one extra line in /etc/fstab:

tmpfs           /tmp            tmpfs   size=256M         0       0

My first Raspberry Pi - day 2

Reminder: this is not a tutorial on how to set up a Raspberry Pi. I'll be going a long way "off piste" compared to the usual way of doing things. This illustrates how flexible and customisable Linux can be, but when I get into difficulty there's no support to help solve problems.

Memory usage

I was slightly surprised when I first ran the 'free -h' command to see that the Pi has swap enabled by default. Using swap on flash memory (such as an SD card) is a seriously bad idea, given the limited number of write cycles flash memory allows. I can only assume the normal Pi user is likely to run out of physical memory, making swap essential. I'll be adding an external disc drive later, and putting some swap area on that, but until then I've disabled the swap as follows:

sudo dphys-swapfile swapoff
sudo dphys-swapfile uninstall
sudo update-rc.d dphys-swapfile remove

Wi-fi connection

My wired computer network only extends as far as my "office" - elsewhere in the house I need wi-fi so I've bought a USB wi-fi dongle to use with the Pi. The dongle was recognised immediately I plugged it in to the USB hub, as shown by the output of 'lsusb'. I found several conflicting guides to setting up wi-fi on a "headless" Pi. I eventually got it working with the following /etc/network/interfaces file:

auto lo

iface lo inet loopback
iface eth0 inet dhcp

allow-hotplug wlan0
auto wlan0
iface wlan0 inet dhcp
wpa-ssid xxxxxxx
wpa-key-mgmt WPA-PSK
wpa-group TKIP CCMP
wpa-psk xxxxxxxxxxxxxxxxx

Lastly I changed the hostname used by the wired interface so I don't have duplicate hosts on my network if both interfaces are connected. This can't be set within /etc/network/interfaces so I had to edit /etc/dhcp/dhclient.conf instead, adding the following lines:

interface "eth0" {
  send host-name = "firefly-hw";
}

It would be nice to get the "firefly" part of the name from gethostname() but I couldn't find a way to do this.

My first Raspberry Pi - day 1

I'm a bit late to jump on the bandwagon, but I've finally got round to buying myself a Raspberry Pi bare bones computer. There are many web pages out there with good tutorials on how to set up a Pi. This is not one of them. It's just a brief account of what I've done with mine.

As I plan on attaching lots of USB peripherals (web cam, PVR, weather station, disc drive, wi-fi dongle, ...) I decided to buy a good quality USB hub with a fairly beefy power supply (3A). This is sufficient to power the Pi, so no additional supply is needed. The only disadvantage is that one of the USB ports is occupied just powering the Pi. Also there are two USB cables between hub and Pi - one for data and one for power - which is a bit of a waste.

Initial setup

I regard myself as reasonably proficient with Linux, so I decided to skip the normal "NOOBS" software and jump straight to a "headless" setup of raspbian. I downloaded the image (version "2013-09-25-wheezy") from http://www.raspberrypi.org/downloads and flashed it to a 4GB SDHC card. I put the card in the Pi, connected an Ethernet cable, powered on and within a few tens of seconds an entry for "raspberrypi" appeared in my ADSL modem router's DHCP table.

From a terminal window on my main machine I was able to ssh to pi@raspberrypi and login with the password raspberry. As prompted, I then ran 'sudo raspi-config' and selected the following options:
1 Expand Filesystem (to use all the memory card)
2 Change User Password (for now - I'll enable secure passwordless login later)
4 Internationalisation Options => I2 Change Timezone (set to Europe/London)
8 Advanced Options => A2 Hostname (all my computers are named after machines or characters from Gerry Anderson's "Thunderbirds" - the Pi is now called firefly)
8 Advanced Options => A3 Memory Split (set minimum GPU memory - I don't expect to use the GPU at all)

Passwordless login

There are many tutorials on the web about setting up passwordless login with ssh. As I already have this set up on my other machines I simply had to copy my key to the Pi by running 'ssh-copy-id -i ~/.ssh/id_dsa.pub pi@firefly' on my main computer. Now I could ssh to pi@firefly without a password.

Copying the key to the 'root' account enables passwordless login as root as well:

sudo mkdir /root/.ssh
sudo chmod go-rx /root/.ssh
sudo cp ~/.ssh/authorized_keys /root/.ssh/

Finally we can disable password login. On the Pi, edit the ssh configuration file ('sudo vi /etc/ssh/sshd_config') and change the line '#PasswordAuthentication yes' to 'PasswordAuthentication no', then restart the ssh daemon: 'sudo /etc/init.d/ssh restart'. Test by trying to ssh to a non-existent account: 'ssh fred@firefly' now gets the response 'Permission denied (publickey)'.

This is a bit more security than most people need, but I intend to allow ssh connections to my Pi from the Internet (so I can check up on things when away from home) so it's vital to frustrate the thousands of repeated attempts to login as root that any exposed machine will receive.

If a world famous search engine wanted some good publicity...

...they could help owners of copyright in images track down their "orphaned" works.

A hot news story in the UK in recent days has been the passing of a law to allow photographs to be published without the copyright owner's consent, provided a "diligent search" has been undertaken to find the owner. See http://www.bbc.co.uk/news/technology-22337406

Many photographers are careful to embed "metadata" within their photographs that explicitly declares the ownership of the image, but many online publishers (e.g. Facebook and the BBC) strip all such metadata from the "user generated content" they publish. This is probably excusable, since it's the only practical way to avoid the risk of obscene or libellous material being secreted in the metadata. By the way, if you're looking for an easy to use program to add metadata to your own photographs, I recommend Photini (what I wrote).

Apparently the new law makes it the photographer's responsibility to discover re-publication of his or her images and then to apply for payment from a central fund. The only problem is, how do you find out if one of your images has been re-published? I recently did a Google image search for my name, and found two instances of photographs I'd uploaded to Flickr being used without my permission. In one case the image was credited to me, in the other they'd included my name in their image file name. (In both cases, had they asked, I'd have said yes.)

With Google's huge database of Internet published images, and a bit of image processing software, they could make it much easier to find instances of an image being re-published elsewhere on the web where the perpetrator hasn't made it quite so easy to spot the theft. They already detect web pages with similar text, to reduce their pagerank on the assumption it's copied boilerplate text, so why not find very similar images in the same way? This would also help those performing a "diligent search" to find the true author of an image, rather than simply allowing them to assume it's an "orphan work".