Thursday 24 May 2018

Setting up a Raspberry Pi Zero W to run pywws

I recently bought my first Raspberry Pi Zero W, a tiny little computer just 3 inches long and 1½ inches wide (in its case). I intend eventually to get it working with pywws (some weather station software I wrote a while ago) and I'll be adding to this blog post as I go along, to keep it all in one place.

Getting started


As I'll be using the Pi in "headless" mode (no monitor or keyboard in normal operation) I decided to start as I mean to go on. This also means I don't yet need adapters for the mini HDMI and OTG USB ports on the Pi Zero.

I downloaded and copied the current version of Raspbian, called "stretch", and copied it to a 16 GB micro SD card. An 8 GB card would have been sufficient but the larger card only cost a few pence extra. Note that this is the full version of Raspbian (i.e. not "lite") and is not the NOOBS version.

Booting the Pi directly into headless mode turned out to be surprisingly easy. The instructions in this forum post were all I needed, and worked perfectly despite the negative comments on the post. I inserted the micro SD card, in an adapter, into my openSUSE Linux PC and used the "open with file manager" prompt to mount both partitions. The df command showed them to be mounted as /run/media/jim/boot and /run/media/jim/rootfs.

Having changed directory to the /run/media/jim/boot partition all I had to do was create an empty file called ssh (with the command sudo touch ssh) then use a text editor (run via sudo) to create a file wpa_supplicant.conf with the following contents:
country=GB
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1

network={
    ssid="my_wifi_ssid"
    scan_ssid=1
    psk="my_wifi_password"
    key_mgmt=WPA-PSK
}

After safely unmounting the SD card partitions I removed it from the PC, put it in the Pi, and turned the power on. About a minute later the hostname raspberrypi showed up in the DHCP server's list of hosts and I was able to ssh into it (the initial password is raspberry):
jim@brains:~$ ssh pi@raspberrypi
The authenticity of host 'raspberrypi (192.168.1.176)' can't be established.
ECDSA key fingerprint is SHA256:bw10eHPZ3debE04S+5a8xEYSDHT9AmZk8MJWxfNTapQ.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'raspberrypi,192.168.1.176' (ECDSA) to the list of known hosts.
pi@raspberrypi's password: 
Linux raspberrypi 4.14.34+ #1110 Mon Apr 16 14:51:42 BST 2018 armv6l

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Wed Apr 18 01:25:27 2018

SSH is enabled and the default password for the 'pi' user has not been changed.
This is a security risk - please login as the 'pi' user and type 'passwd' to set a new password.

pi@raspberrypi:~ $ passwd
Changing password for pi.
(current) UNIX password: 
Enter new UNIX password: 
Retype new UNIX password: 
passwd: password updated successfully
pi@raspberrypi:~ $

Creating a user


I could continue to use the default pi user name, but I find it more convenient to have the same user name on all my computers. As I'll be using NFS to share storage with some of those computers it's also useful if the user has the same UID number:
pi@raspberrypi:~ $ sudo adduser --uid 1026 jim
Adding user `jim' ...
Adding new group `jim' (1026) ...
Adding new user `jim' (1026) with group `jim' ...
Creating home directory `/home/jim' ...
Copying files from `/etc/skel' ...
Enter new UNIX password: 
Retype new UNIX password: 
passwd: password updated successfully
Changing the user information for jim
Enter the new value, or press ENTER for the default
        Full Name []: Jim Easterbrook
        Room Number []: 
        Work Phone []: 
        Home Phone []: 
        Other []: 
Is the information correct? [Y/n] y
pi@raspberrypi:~ $ sudo visudo -f /etc/sudoers.d/010_pi-nopasswd
pi@raspberrypi:~ $
The final visudo command is to add the new user to the list of users permitted to do things as root.

Logging in without a password


It may sound strange but this is a way to improve security when using ssh. Passwords can perhaps be guessed, but a 2048 bit RSA key is harder to crack. I already have a key, so I just needed to copy the public key to my new account on the Pi:
jim@brains:~$ ssh-copy-id -i ~/.ssh/id_rsa.pub raspberrypi
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/jim/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
jim@raspberrypi's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'raspberrypi'"
and check to make sure that only the key(s) you wanted were added.

jim@brains:~$ ssh raspberrypi
Linux raspberrypi 4.14.34+ #1110 Mon Apr 16 14:51:42 BST 2018 armv6l

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Fri May 25 07:13:41 2018 from 192.168.1.162
jim@raspberrypi:~ $
No password was needed to login to the Pi, as expected.

The next step is to disable ssh with passwords entirely. Edit the sshd_config file and restart the ssh daemon:
jim@raspberrypi: $ sudo vi /etc/ssh/sshd_config 
jim@raspberrypi: $ sudo service ssh restart
There is just one line to edit, to disable password logins uncomment the PasswordAuthentication line and set it to no:
# To disable tunneled clear text passwords, change to no here!
PasswordAuthentication no
#PermitEmptyPasswords no
Test by attempting to login as another user:
jim@brains:~$ ssh pi@raspberrypi
Permission denied (publickey).
jim@brains:~$

Completing initial setup


I used sudo raspi-config to set the host name (under network options), set the locale (en_GB.UTF-8) and timezone (Europe/London) under localisation options, and set it to boot to console rather than desktop (under boot options).

To minimise SD card "wear" I moved some directories to RAM by adding the following to /etc/fstab and then rebooting:
tmpfs  /tmp      tmpfs  defaults,noatime,nosuid,size=100m  0 0
tmpfs  /var/log  tmpfs  defaults,noatime,nosuid,mode=0755,size=100m  0 0
tmpfs  /run      tmpfs  defaults,noatime,nosuid,mode=0755,size=2m  0 0
(At first the Pi failed to boot, so I moved the SD card to my main PC, mounted rootfs and deleted everything in the directories that have been moved to tmpfs.)

I also disabled swap (with sudo apt-get remove dphys-swapfile) and turned off file system journaling.

I added a few of my favourite command aliases by creating a file /etc/profile.d/aliases.sh containing the following:
alias df='df -h'
alias la='ls -ao'
alias ll='ls -l'

To save a small amount of power consumption I turned off video output by adding the following to /etc/rc.local, before the exit 0 line:
# turn off video output signal
/usr/bin/tvservice -off

File sharing with NFS


It's very convenient to be able to copy files from one machine to another with NFS. The autofs service can be used to mount NFS shares when needed. I started by exporting the Pi's /home file system so it can be mounted on other machines. This requires installing the NFS server package:
jim@gordon:~ $ sudo apt-get -o Acquire::ForceIPv4=true update
jim@gordon:~ $ sudo apt-get -o Acquire::ForceIPv4=true install nfs-kernel-server
The -o Acquire::ForceIPv4=true option stops apt-get using IPv6, as I don't yet have an IPv6 internet connection.

I then added the following line to /etc/exports:
/home   192.168.1.0/24(rw,root_squash,sync,no_subtree_check)
This should allow any machine on my local network to read and write files in /home on the Pi, once the NFS server is restarted:
sudo service nfs-kernel-server restart

To mount other machines on the Pi requires autofs to be installed:
jim@gordon:~ $ sudo apt-get -o Acquire::ForceIPv4=true install autofs
I then edited /etc/auto.master and added the following line:
/auto   /etc/auto.tracy.island  --timeout=30
The file /etc/auto.tracy.island lists the NFS shares I want access to:
brains  -bg,intr                brains:/home
zero-x  -fstype=nfs4,bg,intr    zero-x:/volume1
brains is my main PC and zero-x is my Synology NAS box. (Fans of 1960s children's television should know what inspired my local network naming scheme.)

After restarting autofs I can access files on other machines:
jim@gordon:~ $ sudo service autofs restart
jim@gordon:~ $ ll /auto/brains/
total 32
drwxr-xr-x  7 1024 users  4096 May 19 09:42 admin
drwxr-xr-x 94 jim  users  4096 May 25 11:18 jim
drwx------  2 root root  16384 Aug 21  2011 lost+found
drwxr-xr-x 13 2001 users  4096 May 22  2016 sarah
drwxr-xr-x 19 pi   users  4096 Sep  9  2017 test
jim@gordon:~ $ ll /auto/zero-x
total 208
drwxrwxrwx 113 root root   4096 Mar  6 18:15 audio
drwxrwxrwx   6 root root   4096 May 18 15:12 brains
drwxrwxrwx   5 root root   4096 May 10 12:19 DVD
drwxrwxrwx   6 root root   4096 May 10 14:22 firefly
drwxrwxrwx   6 root root   4096 May  9 08:59 kyrano
drwxrwxrwx  29 root root   4096 May 15 14:07 photos
drwxrwxrwx   7 root root 131072 May 17 12:59 pvr
drwxrwxrwx   5 root root  12288 May 12 08:41 toppy
drwxrwxrwx  11 root root   4096 May 20 09:45 video
jim@gordon:~ $

Scheduling backup


Experience has taught me the value of backing up computers. The Raspberry Pi is no exception, and as I have a NAS that's always on backing up couldn't be easier. I created a "scripts" directory and added two files. /home/jim/scripts/backup_to_zero-x.sh does the actual backup:
#!/bin/sh

log=/var/log/log-backup

dest="jim@zero-x::gordon"
export RSYNC_PASSWORD=xxxxxx

rsync -av \
        /etc $dest >$log 2>&1
rsync -av \
        --exclude=".cache" \
        --exclude="cache" \
        /home $dest >>$log 2>&1

# mail the log file
if [ -s $log ]; then
  /home/jim/scripts/email-log.py $log "backup log"
  fi
/home/jim/scripts/email-log.py emails the backup log to an email server running on the NAS:
#!/usr/bin/env python

from email.utils import formatdate
import platform
from smtplib import SMTP
import sys

def main():
    file = sys.argv[1]
    subject = sys.argv[2]
    with open(file, 'r') as f:
        contents = f.read()
    if not contents:
        return
    sender = platform.node() + '@tracy.island'
    date = formatdate()
    msg = '''Subject: {subject}
From: {sender}
Date: {date}

{file} contents follows
===START===
'''.format(date=date, file=file, sender=sender, subject=subject)
    msg += contents
    msg += '''
====END====
'''
    smtp = SMTP(host='zero-x', port=25, timeout=20)
    smtp.sendmail(sender, 'jim@zero-x.tracy.island', msg)

if __name__ == '__main__':
    main()
I set up a root user "cron" job to run the backup script daily at around 4 am.

Installation of pywws


You'll probably want to install pywws from PyPI, but I use the development version from GitHub:
jim@gordon:~ $ mkdir weather
jim@gordon:~ $ cd weather/
jim@gordon:~/weather $ git clone https://github.com/jim-easterbrook/pywws.git
Cloning into 'pywws'...
remote: Counting objects: 9190, done.
remote: Compressing objects: 100% (24/24), done.
remote: Total 9190 (delta 10), reused 15 (delta 4), pack-reused 9162
Receiving objects: 100% (9190/9190), 4.55 MiB | 724.00 KiB/s, done.
Resolving deltas: 100% (5557/5557), done.
jim@gordon:~/weather $ cd pywws
jim@gordon:~/weather/pywws $ python3 setup.py build
...
jim@gordon:~/weather/pywws $ sudo python3 setup.py install
...
jim@gordon:~/weather/pywws $
The install process was hanging whilst fetching dependencies from https://pypi.python.org/ until I disabled IPv6 entirely by adding net.ipv6.conf.all.disable_ipv6 = 1 to /etc/sysctl.conf and rebooting.

The next step is to install the required Python USB library:
jim@gordon:~/weather/pywws $ sudo apt-get install python3-libusb1
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  python3-libusb1
0 upgraded, 1 newly installed, 0 to remove and 38 not upgraded.
Need to get 35.1 kB of archives.
After this operation, 189 kB of additional disk space will be used.
Get:1 http://raspbian.mirror.uk.sargasso.net/raspbian stretch/main armhf python3-libusb1 all 1.6.3-1 [35.1 kB]
Fetched 35.1 kB in 0s (67.1 kB/s)    
Selecting previously unselected package python3-libusb1.
(Reading database ... 124808 files and directories currently installed.)
Preparing to unpack .../python3-libusb1_1.6.3-1_all.deb ...
Unpacking python3-libusb1 (1.6.3-1) ...
Setting up python3-libusb1 (1.6.3-1) ...
jim@gordon:~/weather/pywws $
Finally I can test that the core pywws software is installed correctly:
jim@gordon:~/weather/pywws $ pywws-version -v
18.5.0
build: 1541
commit: d505b50
Python: 3.5.3 (default, Jan 19 2017, 14:11:04) 
[GCC 6.3.0 20170124]
USB:    pywws.device_libusb1
examples:
   /usr/local/lib/python3.5/dist-packages/pywws-18.5.0-py3.5.egg/pywws/examples
docs:
   http://jim-easterbrook.github.com/pywws/
jim@gordon:~/weather/pywws $ pywws-testweatherstation 
12:29:22:pywws.logger:pywws version 18.5.0, build 1541 (d505b50)
Traceback (most recent call last):
  File "/usr/local/bin/pywws-testweatherstation", line 11, in 
    load_entry_point('pywws==18.5.0', 'console_scripts', 'pywws-testweatherstation')()
  File "/usr/local/lib/python3.5/dist-packages/pywws-18.5.0-py3.5.egg/pywws/testweatherstation.py", line 117, in main
    ws = pywws.weatherstation.WeatherStation()
  File "/usr/local/lib/python3.5/dist-packages/pywws-18.5.0-py3.5.egg/pywws/weatherstation.py", line 462, in __init__
    self.cusb = CUSBDrive()
  File "/usr/local/lib/python3.5/dist-packages/pywws-18.5.0-py3.5.egg/pywws/weatherstation.py", line 316, in __init__
    self.dev = USBDevice(0x1941, 0x8021)
  File "/usr/local/lib/python3.5/dist-packages/pywws-18.5.0-py3.5.egg/pywws/device_libusb1.py", line 78, in __init__
    raise IOError("Weather station device not found")
OSError: Weather station device not found
jim@gordon:~/weather/pywws $

Connecting the weather station


Given the limitations of the "OTG" USB port on the Pi Zero I was pleasantly surprised that when I connected my test weather station console (an old one whose outside sensors have failed) via a standard USB A to micro USB lead (as used to connect a phone to a computer) it powered up normally. It also works with pywws:
jim@gordon:~/weather/pywws $ sudo pywws-testweatherstation 
12:32:56:pywws.logger:pywws version 18.5.0, build 1541 (d505b50)
0000 55 aa ff ff ff ff ff ff ff ff ff ff ff ff ff ff 03 20 01 41 11 00 00 00 81 7f 00 01 00 00 00 01 
0020 94 27 87 27 00 00 00 00 00 00 00 07 01 01 16 55 41 23 c8 00 00 00 46 2d 2c 01 64 80 c8 00 00 00 
0040 64 00 64 80 a0 28 80 25 a0 28 80 25 03 36 00 05 6b 00 00 0a 00 f4 01 12 00 00 00 00 00 00 00 00 
0060 00 00 5a 0a 63 0a 41 01 56 00 dc 01 08 81 dc 01 c5 81 68 01 75 81 95 28 d3 25 24 29 5a 25 fd 02 
0080 04 03 f4 ff fd ff 85 ff 91 ff 6c 09 00 14 10 19 06 29 12 02 01 19 32 11 09 09 05 18 12 03 28 13 
00a0 00 13 07 19 18 28 07 01 01 12 00 13 09 24 13 02 13 09 24 13 33 13 09 24 13 02 12 07 28 12 50 13 
00c0 09 24 13 02 13 10 14 16 18 12 02 07 19 00 16 03 28 04 45 13 01 04 10 28 15 01 30 05 22 12 03 10 
00e0 22 02 07 01 02 00 02 12 07 28 11 59 13 03 06 06 43 12 04 13 00 04 12 04 13 00 04 12 07 31 03 34 
jim@gordon:~/weather/pywws $
I used sudo as the USB device is "owned" by root by default.

The USB ownership is best dealt with by adding a "udev" rule. Create a file /etc/udev/rules.d/39-weather-station.rules containing the following:
ACTION!="add|change", GOTO="weatherstation_end"
SUBSYSTEM=="usb", ATTRS{idVendor}=="1941", ATTRS{idProduct}=="8021", OWNER="jim"
LABEL="weatherstation_end"
(Replace jim with your user name.)

Restart the udev system (sudo service udev restart) and unplug then replug the weather station. You should now have access to the weather station without using sudo:
jim@gordon:~/weather/pywws $ ll /dev/bus/usb/001/
total 0
crw-rw-r-- 1 root root 189, 0 May 25 12:20 001
crw-rw-r-- 1 jim  root 189, 2 May 25 13:03 003
jim@gordon:~/weather/pywws $ pywws-testweatherstation 
13:04:14:pywws.logger:pywws version 18.5.0, build 1541 (d505b50)
0000 55 aa ff ff ff ff ff ff ff ff ff ff ff ff ff ff 03 20 01 41 11 00 00 00 81 7f 00 01 00 00 00 01 
0020 94 27 88 27 00 00 00 00 00 00 00 07 01 01 12 31 41 23 c8 00 00 00 46 2d 2c 01 64 80 c8 00 00 00 
0040 64 00 64 80 a0 28 80 25 a0 28 80 25 03 36 00 05 6b 00 00 0a 00 f4 01 12 00 00 00 00 00 00 00 00 
0060 00 00 5a 0a 63 0a 41 01 56 00 dc 01 08 81 dc 01 c5 81 68 01 75 81 95 28 d3 25 24 29 5a 25 fd 02 
0080 04 03 f4 ff fd ff 85 ff 91 ff 6c 09 00 14 10 19 06 29 12 02 01 19 32 11 09 09 05 18 12 03 28 13 
00a0 00 13 07 19 18 28 07 01 01 12 00 13 09 24 13 02 13 09 24 13 33 13 09 24 13 02 12 07 28 12 50 13 
00c0 09 24 13 02 13 10 14 16 18 12 02 07 19 00 16 03 28 04 45 13 01 04 10 28 15 01 30 05 22 12 03 10 
00e0 22 02 07 01 02 00 02 12 07 28 11 59 13 03 06 06 43 12 04 13 00 04 12 04 13 00 04 12 07 31 03 34 
jim@gordon:~/weather/pywws $
As soon as I tried "live" logging, with the command pywws-testweatherstation -l -vv, I got USB timeout errors as reported by other pywws users on the pywws mail list.

A bit of experimenting with longer timeouts and measuring times to read data from the station showed the problem only occurs after pywws has paused USB activity for 5 seconds or more. I had a hunch the the Pi might be switching USB modes during this apparent inactivity, so I added the line dtoverlay=dwc2,dr_mode=host to /boot/config.txt to fix the USB port in "host" mode. This appears to have cured the timeout problem.

Final steps


To finish off I installed all the pywws dependencies I need, then copied the pywws data directory from the Pi 2 I was using. That machine has a disc drive so I haven't had to worry about SD card wear. I made a few changes to weather.ini to minimise writes to the card:
[config]
frequent writes = False

[paths]
local_files = /tmp/pywws/results
work = /tmp/pywws
I'm also using /tmp/pywws for the logfile. This avoids writing to the SD card, but does mean that the log is cleared every time the Pi is rebooted. I don't expect to have to do this too often.

The Raspberry Pi Zero is so small I'm tempted to stick it on the back of the weather station console, with a short USB lead (with right angle plugs?) to make a neat installation. However I suspect the wi-fi may interfere with the station's reception of data from the outside sensors.

No comments: