Python one-liner: return latest version

Today I need to find the latest version of some software. A normal ASCII sort (or even a numeric one), would be unreliable given the use of semantic version numbers. Instead I came up with this simple Python command that uses distutils LooseVersion to sort in the correct order:

python -c "import sys;from distutils.version import LooseVersion;print(sorted(sys.stdin.read().split(),key=LooseVersion)[-1])"

Simply pipe the input into stdin and Python will emit the latest version from the list of words or lines. For example:

$ echo octocat-0.0.2 octocat-10.0.3 octocat-1.2.0a octocat-6.1.0alpha | python -c "import sys;from distutils.version import LooseVersion;print(sorted(sys.stdin.read().split(),key=LooseVersion)[-1])"

returns

octocat-10.0.3

Using the Raspberry Pi sense hat to show the AuroraWatch UK status

Image © Barry Osborne 2017. Used with permission.

This post uses the Raspberry Pi sense hat to display the AuroraWatch UK alert status. It assumes you already have Raspbian installed on a Raspberry Pi, networking configured and enabled, and the sense hat fitted to the Pi.

The current status level is easily obtained from the AuroraWatch UK API with a custom python module. First, ensure you have the necessary python modules installed from the Raspbian software archive:

sudo apt-get update
sudo apt-get install python-lxml python-requests python-six

Then install two custom python modules. We will install them into the user profile, so run these commands as the user pi:

pip install --user --no-deps atomiccreate
pip install --user --no-deps aurorawatchuk

Finally, download the python script which fetches the status and displays it on the sense hat (I used bit.ly to shorten the original GitHub URL):

wget https://bit.ly/awuk-status-on-sense-hat -O awuk_sense_hat.py

The AuroraWatch UK status can then be displayed on the sense hat with this simple command:

python awuk_sense_hat.py

The program will monitor the current status and set the LED colours to show the status level (green, yellow, amber or red). For more information about the AuroraWatch UK status level meaning and interpretation see http://aurorawatch.lancs.ac.uk/alerts/. If you don't have a sense hat (I don't) it is even possible to try this with the sense hat emulator.

Credits

The geomagnetic activity status is courtesy of AuroraWatch UK and uses data from the SAMNET and AuroraWatchNet magnetometers.See also the AuroraWatch UK conditions of use for the API.

Thanks to Barry Osborne for testing the software with a real Raspberry Pi sense hat, and for providing the photograph.

AuroraWatch UK camera update #2

A Python module for the AuroraWatch UK API

The first goal in the journey to create an automated all-sky camera system for AuroraWatch UK was to interface to the camera using Python. If you missed update #1 you can read it at here.

One of the requirements of the camera software is to be able to change between different recording settings, for instance, in response to solar elevation and the AuroraWatch UK status level. Computing solar elevation is easily acheived with the astral module. For the AuroraWatch UK status level I began with fetching the status XML document but soon realised a much better approach was to write a Python module. The module automatically fetches the various XML documents, parses them and caches the results, both to memory and to disk.

I won't repeat the instructions for installing and using the module - that information is already given in an ipython notebook that you can view at https://github.com/stevemarple/python-aurorawatchuk/blob/master/aurorawatchuk/examples/aurorawatchuk_api.ipynb. 

AuroraWatch UK camera update #1

ZWO ASI 174MC colour USB 3.0 camera. Image from ZWO website.

Recently AuroraWatch UK purchased a ZWO ASI174MC colour USB 3.0 camera to (hopefully) record images of the aurora. The aim is to build an all-sky imager. To minimise the cost the camera will be connected to a Raspberry Pi single-board computer and be located outside in a waterproof enclosure. This particular camera was chosen for several reasons:

• No mechanical shutter to wear out. 
• The large pixel size (5.86um) should give good sensitivity. 
• The sensor has an IR cut-off filter fitted (filters cannot be used with the fish-eye lens). 
• The manufacturer provides support to access the camera from Linux. 

At present the Linux support from ZWO is limited to a software development kit (SDK) which provides a C language library and header file. This is sufficient for a Raspberry Pi to be able to interface with the camera. However my preferred approach is to write the image capture and control software in Python since that will simplify development and access to the AuroraWatch UK API. I therefore decided to write a Python wrapper for the C library. This is not something I have done before and after a little experimentation with ctypes, Cython and CFFI and I settled on using ctypes. You can find the source code for my Python wrapper on Github. It's also available on pypi so you can install it simply by running "pip install zwoasi".

I've written a short demo program to illustrate how to access the camera from Python. The camera can operate in two modes. The first, snapshot mode, takes a single exposure. The second, video mode,  continually takes exposures which can be downloaded from the camera as and when required. At first glance either mode would appear suitable for the purpose of taking a single exposure once a minute. My main concern is how to set the exposure correctly for an automated system which will run in a variety of different lighting levels (twilight, nighttime, and possibly even daytime). Fortunately this particularly camera provides an auto-exposure mode which can adjust the exposure time and gain settings according to the ambient light levels. It does this by analysing the image histogram after each exposure. The documentation doesn't explain the auto-exposure mode but I suspected that the camera would have to be operated in video mode for this to work. A quick test proved this to be correct. How well the auto-exposure algorithm works at night has yet to be discovered; since it is an astronomy camera I'm optimistic that it will be suitable and save me from having to implement an exposure control algorithm.

Sample image from ASI174MC and Fukinon FE185C057HA-1 2/3" fisheye lens.

The complete control system is intended to timestamp and watermark images, change the recording cadence according to AuroraWatch UK status and solar elevation, record temperature and humidity, and control heating and fans.

FTDI all-in-one serial converter, programmer, switch

FTDI all-in-one PCB layout.

There are a multitude of FTDI converters and breakout boards available but none of them exactly fitted my needs so this is my version tuned to suit my requirements. I'm publishing the design files in the hope they are useful for others. Like the Sparkfun FTDI Basic Breakout and the Adafruit FTDI Friend it uses the FTDI FT232R but there are a number of differences, for instance it has a USB type A connector to plug directly into a USB host port.

Serial converter

The FTDI AIO provides a serial adaptor with the standard 6 pin FTDI pinout (TX, RX, CTS, DTR, VCC, and GND). A jumper block enables the VCC pin to be switched between +3.3V and +5V. A separate jumper enables the logic levels of the TX, RX, DTR and CTS signals to be set for +3.3V or +5V operation. The Sparkfun FTDI Basic Breakout can also be switched between +3.3V and +5V operation but does not allow for independent selection of VCC and the logic levels. This means when the Sparkfun FTDI Basic is set for +3.3V operation it's not compatible with the standard FTDI cables, which always supply +5V to the VCC pin. Whether that's a good thing or a bad thing depends on your needs; the FTDI AIO gives you both options.

AVR ISP programmer

Programming AVR microcontrollers, even with just a small bootloader, can be painfully slow and normally I'd use the AVR Dragon. There are situations however where it would be useful to have a low-cost USB programmer. A bit-banging AVR programmer is easily made and Adafruit has an excellent tutorial which explains how to do this.The FTDI all-in-one makes this convenient by providing the standard AVR ISP header. The defacto AVR standard is that the VCC pin is set to the same voltage as the logic-level signals. Unfortunately this practice isn't followed by +3.3V Arduino boards which always set VCC to +5V. For the widest compatibility the AVR ISP VCC pin can be set to either +3.3V or +5V with the logic levels of the !RESET, MISO, MOSI and SCK signals independently configured to either  +3.3V or +5V.

On-board +3.3V voltage regulator

One of the features of the FT232R is that it has a built-in +3.3V regulator. This was used by older Arduinos to provide a +3.3V supply. However the current capability is limited to just 50mA which in many cases is not sufficient to power a microcontroller and all of its peripherals. The FTDI all-in-one can include an on-board voltage regulator with 250mA current output which should be sufficient to power an Arduino and several shields.

Auxiliary switch and LED

One of the key features I wanted was the ability to add an auxiliary switch and LED to the Raspberry Pi. Although the Pi has GPIO the AuroraWatchNet magnetometers normally have a radio fitted to this connector, which prevents it being easily used for other purposes. I worked around this by wiring a switch and LED to a Sparkfun FTDI Basic Breakout but I wanted a single PCB which plugged directly into one of the Pi's USB sockets.

Why would you want a switch? There are probably many uses. Mine is that I run one of the AuroraWatchNet magnetometers and I wanted a way to conveniently signal when I am going to cut the grass, or make some other magnetic disturbance in the garden.The data logging software, which runs on a Raspberry Pi, checks for the existence of a file which signals data quality might be bad. When the file is present data is logged to a separate set of files which aren't used by the AuroraWatch UK alerts system. While it's no bother for me to create the file by logging into the Pi I needed something simpler for other members of the family, so I wanted to wire up a switch. A separate daemon monitors the serial port to detect if the switch is pressed, if so the LED is turned on and the semaphore file created. If the switch is pressed again the file is removed and the LED turned off. The LED actually tracks the state of the file being present so it is possible to override the monitor process by creating or removing the semaphore file by other means.

Breakout board

All of the GPIO pins from the FTDI FT232R are conveniently broken out to pads spaced at 0.1" pitch. Also included are pads for the filtered +5V supply obtained from USB and +3.3V from the on-board regulator (if fitted, otherwise from the FT232R's built-in regulator).

Why use the FT232R?

Given the recent controversy regarding FTDI and Windows drivers you might wonder why I am using the FT232R instead of some other device. I'd probably prefer to use the Microchip MCP2200 USB UART converter for this application as it works well with my Calunium Arduino clone and it is available in a SOIC package which is easier to work with at home. Unfortunately its not possible to read the CTS line with the USB CDC driver which rules it out.

Open-source hardware

The Eagle PCB design files are available on Github and are licensed under the Creative Commons Attribution-ShareAlike 3.0 (CC BY-SA 3.0) license.

The PCB layout is not yet tested, although I did breadboard and test all of the functionality. I'll be sending the design off for manufacture in the next few days, it's likely I'll have a few boards spare.

Update: the first boards have been assembled and tested; all features work as planned. The bit-banging AVR ISP programmer is even slower than I feared, taking about 10 minutes to read the 4Kb EEPROM on the Atmel ATmega1284P on one of my Calunium boards.

Why Kp is a poor indicator for auroral alerts

What is Kp?

Kp is a an index to represent the planetary geomagnetic activity. The name originates from "planetarische Kennziffer" (German for planetary index). It has 10 main levels, numbered from 0 (no geomagnetic activity) to 9 (extreme geomagnetic activity). The intermediate ranges are subdivided with a + or - suffix, making 28 levels in total (0, 0+, 1-, 1, 1+, 2-, 2, 2+, ... 8, 8+, 9-, 9).

The Kp index is calculated for three-hourly intervals, beginning at UT midnight. To calculate Kp the daily "solar-quiet" (Sq) variation is removed from the measurements of magnetic field strength. Then the difference between the largest and smallest values is computed. By looking up the difference in a conversion table the local K index can be found. Stations at different latitudes (or more properly magnetic latitudes) have different conversion tables.

There is a pronounced daily variation in the K index at a single station, with intervals close to local midnight being substantially more disturbed in comparison with those centered on local noon; see http://www-app3.gfz-potsdam.de/kp_index/conv_tab.html. For this reason the planetary Kp index is computed from magnetic variations recorded by 13 magnetometers located around the world.

Why Kp is a poor indicator for auroral alerts

The Kp index only indicates geomagnetic activity within a 3 hour interval, which is too long to be useful for auroral alerts. Geomagnetic conditions can have recovered to a calm state long before the next Kp index is computed. This is particularly true for aurora caused by a substorm, as the entire substorm cycle (including the growth, expansion and recovery phases) is typically 2 to 4 hours duration; see http://www.ann-geophys.net/29/2031/2011/angeo-29-2031-2011.html.

It must also be noted that the official Kp values are not available in real-time. Not only must the data from the 13 stations be collected the quiet-day curves for each of the magnetometers must be calculated for that calendar month, a process which cannot begin until the month ends. Therefore any references to Kp for the current month will be to unofficial estimates. The most reliable estimate of Kp is probably obtained from NOAA, which is derived from a worldwide distribution of magnetometers, see http://www.swpc.noaa.gov/products/planetary-k-index.

In summary: Kp is not available in real-time. The 3 hour interval means current auroral activity may be much lower than an estimated Kp value suggests.

Estimated Kp can differ significantly from the local K index

As noted above, the pronounced daily variation of the K index means that there can be considerable difference between the average planetary geomagnetic activity (Kp) and that observed locally (K). The plot below shows the difference between the local K index measured by the British Geological Survey magnetometer at Eskdalemuir and Kp.

Difference between K index measured at Eskdalemuir and Kp.

The plot shows that in general there is good agreement but at certain times the K index can be higher or lower than Kp.

In summary: global measurements of Kp are not always valid indications of  regional geomagnetic activity.

What measurements should be used?

One of the better measurements is using the K index as long as it is from a nearby magnetometer or one on a similar geomagnetic longitude. It is still limited by the 3 hour resolution of the K index so it will not always be accurate for real-time measurements.

An alternative measurement is to consider the rate of change of the magnetic field, commonly referred to by its mathematical notation dB/dt ("D B by D T"). This measurement is of particular interest to operators of pipelines and long power distribution networks since it indicates the levels of geomagnetically-induced currents which might be expected. British Geological Survey publish real-time dB/dt measurements. These measurements require magnetically quiet sites as human interference is likely to cause sudden spikes with a high dB/dt value, limiting its usefulness to observatory measurements.

AuroraWatch UK publishes a real-time auroral activity measurement. It is the H component deviation, which is the difference between the current H component magnetic field strength and the expected value for the time of day taken from the "quiet-day curve". Unlike Kp this measurement is computed hourly and so can indicate a return to low activity values sooner than Kp. An accurately fitted quiet day curve is required to make the deviation measurement.

Data credits

Eskdalemuir K indices were downloaded from the British Geological Survey, https://www.bgs.ac.uk/discoverymetadata/13480122.html. Kp data was downloaded from GFZ-Potsdam, ftp://ftp.gfz-potsdam.de/pub/home/obs/kp-ap/tab/.

Calunium v2.1a

I've made a few minor improvements to the Calunium design, partly to satisfy compatibility between Calunium and Arduino shields and partly to fix a few minor hardware errors.

Disconnect VCC from the ISP header

The ISP header is a 6 pin male header which can be used to program the microcontroller, such as to upload the bootloader. On Calunium the VCC pin of this header is wired to the same supply voltage used by the microcontroller, ie +3.3V or +5V depending upon the operating volatge selected. This makes sense, especially when used when programmers such as Atmel's AVR Dragon which use the pin to set the HIGH voltage level on the SCK and MOSI control lines. Unfortunately it seems that on Arduino devices (such as the Ethernet shield) this pin is always wired to +5V. This is true even for the latest hardware versions which should be aware of the operating voltage via the IOREF pin. To avoid the possibility of connecting the +3.3V and +5V supplies I added a jumper. For normal operation the shunt can be omitted. When using an ISP header programmer that needs to know the operating voltage a shunt can be added.

Pull-down resistor on D13 FET

Calunium uses a field-effect transistor (FET) to switch on the D13 LED. This is to avoid load on the D13 SCK line when it is used for SPI communications. I learnt this trick from Freetronics who do something similar with their range of clones. In normal use, when D13 is an output, everything works fine. However, when it is an input, such as during RESET state or bootloader programming, the FET gate can float to potential such that the LED is switched on. A 1 megaohm pull down resistor connected to D13 is sufficient to prevent this happening without loading the SCK line. It also helps prevent damage by electrostatic discharge. The latest Calunium design now has room to add this 1 megaohm resistor. On older boards an 0805 surface mount resistor can be fitted between the gate and source of the 2N7000 FET, or alternatively between the D13 and adjacent ground pin.

Auto-reset behaviour corrected for MCP2200 USB interface

I recently added the option of fitting the MCP2200 USB-serial interface. Despite suggestions to the contrary this useful USB  device can automatically reset the Arduino when the device is opened. This auto-reset behaviour as it is known is used to activate the bootloader when uploading sketches. The intention was that auto-reset could be disabled by removing a shunt from the AUTO RESET jumper. Unfortunately an error in the schematic meant that auto-reset could only be disabled when using the FTDI interface, not the MCP2200 option. The latest version corrects this error.

Silkscreen label corrected

On Calunium v2.1 a jumper was added to select FTDI or USB power. Due to a rotation of the component the correct position for the shunt is opposite to that indicated by the silkscreen label. The latest version corrects the text.

Open source

The Eagle PCB design files for Calunium are available on Github and are licensed under the Creative Commons Attribution-ShareAlike 3.0 Unported License. Files to allow Calunium to be used with the Arduino 1.0 IDE are available on  Github and are licensed under the Gnu General Public License v2.