Using the Raspberry Pi sense hat to show the AuroraWatch UK status

Image © Barry Osborne 2017. Used with permission.

This post uses the Raspberry Pi sense hat to display the AuroraWatch UK alert status. It assumes you already have Raspbian installed on a Raspberry Pi, networking configured and enabled, and the sense hat fitted to the Pi.

The current status level is easily obtained from the AuroraWatch UK API with a custom python module. First, ensure you have the necessary python modules installed from the Raspbian software archive:

sudo apt-get update
sudo apt-get install python-lxml python-requests python-six

Then install two custom python modules. We will install them into the user profile, so run these commands as the user pi:

pip install --user --no-deps atomiccreate
pip install --user --no-deps aurorawatchuk

Finally, download the python script which fetches the status and displays it on the sense hat (I used bit.ly to shorten the original GitHub URL):

wget https://bit.ly/awuk-status-on-sense-hat -O awuk_sense_hat.py

The AuroraWatch UK status can then be displayed on the sense hat with this simple command:

python awuk_sense_hat.py

The program will monitor the current status and set the LED colours to show the status level (green, yellow, amber or red). For more information about the AuroraWatch UK status level meaning and interpretation see http://aurorawatch.lancs.ac.uk/alerts/. If you don't have a sense hat (I don't) it is even possible to try this with the sense hat emulator.

Credits

The geomagnetic activity status is courtesy of AuroraWatch UK and uses data from the SAMNET and AuroraWatchNet magnetometers.See also the AuroraWatch UK conditions of use for the API.

Thanks to Barry Osborne for testing the software with a real Raspberry Pi sense hat, and for providing the photograph.

Power over ethernet (PoE) magnetometer and cloud detector

Combined magnetometer and cloud detector.

To improve the performance and stability of the AuroraWatchNet magnetometers I recently began experimenting with a power over ethernet (PoE) version. With restrictions on power consumption lifted considerable performance improvements are possible. As a result I have developed the magnetometer hardware specifically to support a power over ethernet version. Another instrument I've been developing is a cloud detector. This too should benefit from a power over ethernet version. One problem I encountered was with dew settling on the sensor but fitting a heater is incompatible with battery-powered operation. Since both the magnetometer and cloud detector use almost the same hardware I decided to design an Arduino-compatible 'shield' that could be used to support both systems.

Combined magnetometer and cloud detector hardware

Printed circuit boards for the sensor shield (left), the IR or humidity sensor (top right)
and fluxgate magnetometer (bottom right). Click for annotated version.

The complete system requires six circuit boards (five for the wireless version). The first is the microcontroller board, I use my Calunium Arduino clone. I hope that in future it will be possible to use an off-the-shelf Arduino Mega2560 instead but the current firmware relies on Calunium's real-time clock to generate the hardware interrupts which control the sampling interval. There is also a sensor shield, the Arduino Ethernet shield (omitted on the battery-powered wireless version), and one board for each of the sensors (fluxgate magnetometer, IR temperature and humidity).

The sensor shield is based on the existing design and retains the option of battery-powered operation with radio communication. The magnetometer sensor and analogue-to-digital converter must be powered at 5V, which requires a level-shifting circuit when the microcontroller is powered from 3.3V, which is the case when operating from batteries. For power over ethernet use the microcontroller must also operate at 5V for compatibility with the Arduino Ethernet shield so the level shifting is not required. It is kept however as it provides buffering between the two circuit boards; 1.5m is a considerable distance for an I2C bus. The cloud detector uses a non-contact infra-red temperature sensor operating at 3.3V so a level-shifting circuit is required for PoE operation where the microcontroller is connected to 5V. (I've ignored the fact that a 5V version of the sensor exists since it isn't readily available in the UK). The sensor shield allows a humidity sensor to be connected so that estimates of the clear sky and cloudy sky temperatures can be made. As before, an on-board LM61 temperature sensor monitors the system temperature. The new sensor shield also adds a header to fit an Embedded Adventures lightning sensor module. I don't have one of these at the moment so I can't be certain it will work and there is no software support for it in the existing firmware. Fitting it at the same time as the cloud detector sensors will require long break-away headers to be soldered to the bottom of the lightning sensor module.

Fluxgate magnetometer sensor mounted on its sensor PCB.
Click for annotated version.

The fluxgate sensor is fitted on its own PCB which contains the analogue to digital converter. For PoE operation a linear voltage regulator is used to convert from 9V to the 5V supply it requires. For battery operation a MAX619 DC-DC charge pump boosts the battery voltage to 5V. Almost all of the temperature variation can be removed by placing the PCB at the bottom of a 1m length of soil pipe. The pipe is buried to a depth of 0.85m with its axis vertical. In the magnetometer-only system the microcontroller, sensor shield and ethernet shield (or batteries for the wireless version) are fitted onto a wooden frame to hold them into the top part of the tube. Positioning the rest of the system away from the fluxgate sensor helps to avoid unwanted effects from any ferro-magnetic components (such as the batteries), it also aids access and enables wireless data transmission.

Honeywell HIH6131 humidity sensor mounte on its sensor PCB.
The same PCB design is also used for the MLX90614 IR temperature sensor.
Click for annotated version.

The enclosure design used for the magnetometer isn't suitable for the cloud detector so the prototype cloud detector used a standard IP65 rated box, with the IR temperature sensor pointing upwards to view the sky. The humidity sensor was fitted above a breather hole in the bottom of the box. This concept will continue to be used for the cloud detector; the IR temperature sensor and humidity sensor are fitted to separate PCBs in the top and bottom of the box. To minimise costs the IR temperature and humidity sensors use the same PCB design. The mechanical design of the cloud detector part is something I'd like to improve upon, particularly to reduce the number of PCBs used. However the differing sensor requirements may prevent this.

For a combined system I plan to use the soil pipe to house the fluxgate sensor but locate the rest of the electronics in a separate box following the design of the prototype cloud detector.

Does it work?

I've not yet deployed a system outside but testing indicates the new printed circuit boards work as intended when used in power over ethernet mode. Battery-powered operation on this new version has not yet been tested.

Design files

The design files (hardware, firmware and software) are open source and can all be downloaded from the Github repository. A PDF version of the user manual describing how to construct and operate the magnetometer can be downloaded from http://aurorawatch.lancs.ac.uk/manual.pdf. At the time of writing only instructions to build the original FLC100 shield are included. Instructions to build the combined sensor shield described above will be added in due course.

Performance comparison of power over ethernet (PoE) and battery magnetometers

Introduction

I've been working to improve the performance and stability of the AuroraWatchNet magnetometers. It's apparent that both measurement noise and stability are considerably improved when the sensor is powered continually. Unfortunately with the sensor powered continually the batteries last for only a few weeks. I have therefore been testing a power over ethernet (PoE) version of the magnetometer. The hardware is essentially the same but the addition of the Arduino Ethernet shield requires the microcontroller to operate from 5V. Operating voltage is easily changed on my Calunium microcontroller board. With the power restrictions lifted the sampling interval can also be reduced. The test system has been operating reliably for almost two months, sampling every 5 seconds. With some minor configuration changes sampling every second is possible although I am not convinced the trade-off in measurement noise is worthwhile.

Performance comparison

Below is a plot comparing one hour of data from the new power over ethernet system with two of the existing battery-powered wireless models already in operation. The PoE system is on the same site as LAN1 but located nearer parked cars. I chose this period because it was free of man-made disturbances. I adjusted the baselines so that the plots overlap.Only the H component of the magnetic field is shown.

Click for larger version.

The graph shows that the power over ethernet version has much smaller measurement errors; it can probably operate with ~0.1nT accuracy compared to ~10nT for the battery-powered version. It's such an improvement that I found the measurement accuracy was being limited by the available resolution of the analogue-to-digital converter. The battery-powered versions derive the sample value from the median of 15 samples (taken as a burst over 4 seconds) to reduce noise. To improve the resolution for the PoE version it operates by taking the mean of 15 samples. Further improvement to the resolution may be possible by taking advantage of the programmable gain amplifier that is built into the analogue-to-digital converter.

Let's see how it compares against some observatory-grade measurements. The plot below shows the same interval but this time uses data from the British Geological Survey magnetometers at Eskdalemuir and Hartland. I obtained the data from the SAMNET data archive at Lancaster University, where the data has already been converted into HEZ magnetic coordinates. AuroraWatchNet also operates with HEZ magnetic coordinates, although usually the E and Z sensors are not present. As before, the baselines have been adjusted so that the plots overlap.

Click for larger version.

The similarity between the different traces is striking. Some differences are to be expected since Eskdalemuir is approximately 140km north of Lancaster and Hartland is 350km SSW of Lancaster. This interval is interesting because it shows Pi2 pulsations, starting at about 01:15 and ending around 01:30. The period of the pulsations are about 90 to 120 seconds.

Conclusions

For a home-built citizen-science magnetometer which probably costs 25 times less than its observatory grade cousin I'm very happy with its performance. The detection of Pi2 pulsations means a low-cost magnetometer can now notify of substorm onset, not just the arrival of geomagnetic storms. A network of such devices has interesting possibilities for the study of magnetic field line resonances.

You might wonder why anyone would want to buy an observatory grade instrument, there are good reasons. At present I am relying on the sensor manufacturer's calibration, whilst an observatory grade instrument would be supplied with an official calibration certificate. The observatory instrument would also have better long-term baseline stability, lower temperature variation and higher cadence. Calibration is an issue I hope to tackle at a later date. For space-weather monitoring only short-term variations are of interest and
the better performance provided by an observatory system may not be needed.

Data credits

The Sub-Auroral Magnetometer Network data (SAMNET) is operated by the Space Plasma Environment and Radio Science (SPEARS) group, Department of Physics, Lancaster University. Hartland and Eskdalemuir data is provided courtesy of the British Geological Survey.

AuroraWatchNet LAN1 magnetometer data by Steve Marple is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Based on a work at http://aurorawatch.lancs.ac.uk/data/aurorawatchnet/lan1/.

AuroraWatchNet LAN3 magnetometer data by Department of Physics, Lancaster University, UK is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Based on a work at http://aurorawatch.lancs.ac.uk/data/aurorawatchnet/lan3/.

Magnetometer progress report: link indicator LED

I recently added a new feature to the magnetometer remote sensor unit firmware. To help indicate when it is within radio communication range of the base unit an LED is turned on at the start of a message communication. When the sensor unit receives the acknowledgement from the base unit of successful message receipt the LED is switched off. In normal operation the LED should blink briefly every 30 seconds, after each sampling interval. If the LED remains on it indicates a problem with the radio link. For the initial installation the sampling time can be reduced to 5 seconds for to obtain faster feedback of whether communication is successful or not.

Improvements for battery-powered operation

Indicating link errors by using an LED is convenient during installation but the power wasted after installation is not compatible with battery-powered operation. To save power the LED is used only the first 15 minutes of operation. Only user-initiated reset actions (power on or reset switch pressed) cause the LED to be used, resets from by the watchdog timer or brown-out detector are ignored. The cause of the reset is detected by examining the microcontroller's status register (MCUSR).

Communication timeout feature added

I've also added a timeout which detects when communication has been lost for an extended period. The microcontroller system is rebooted in the hope that the error is recoverable. Loss of communication, along with low battery status, can also be sensed and reported by running the latest version of the data plotting software on the Raspberry Pi. If either error is detected a message can be sent via email, Twitter or Facebook.

Three-axis sensor unit for AuroraWatchNet

Three-axis sensor unit. The RJ-45 connector is for power and I2C data signals, not Ethernet.

AuroraWatchNet is a network of magnetometers for auroral alerts and citizen science. Whilst the  magnetometer is intended to function with one single-axis FLC100 sensor from Stefan-Mayer Instruments I have designed the system to support 3-axis operation by mounting 3 sensor boards at 90° to each other using just standard right-angle PCB connectors. This approach avoids the requirement for external mounting hardware which would be expensive to manufacture.

The power supply (a charge-pump DC-DC converter), temperature sensor and RJ-45 connector are fitted to just one circuit board. It is possible to use only one analogue-to-digital converter (ADC) but given their low cost it was presumed they would be fitted to all boards to enable simultaneous sampling of all 3 axes. The MCP3424 converter was chosen because eight options are available for the I2C address, thus jumpers can be fitted to select a unique address for every ADC on the I2C bus. This converter also features 4 input channels. All magnetometer outputs and the temperature sensor are connected to each ADC so that the spare channels can provide some redundancy in case of an ADC failure. The SYNC connectors of all sensors are linked to synchronise their excitation frequencies.

Several single-axis units have been built and one is in active use by AuroraWatch UK. The photograph above shows the first three-axis unit to be built. Initial testing indicates it functions as intended but the performance of 3 simultaneously-powered sensors in close proximity is yet to be established.

The importance of brown-out detection

The batteries expired in one of the AuroraWatchNet test magnetometers this week. I was expecting that to happen since the magnetometer wasn't operating in its low-power mode and the battery voltage from the two 1.5V alkaline cells had been hovering around 2V for some time. The Atmel ATmega1284P microcontroller is actually powered at 3.3V using a boost power supply. What I didn't expect was for the system to not start after replacing the battery. I could tell xboot was working since the LED flashed 3 times as usual after the reset button was pressed. After that the LED stayed on but nothing else happened, not even the start-up debug messages from the serial port. I guessed the flash program memory had somehow got corrupted.

I used the AVR Dragon to extract the contents of the flash memory (avrdude -P usb -c dragon_jtag -p atmega1284p -U flash:r:flash.img:r) and compared it against the saved raw image file using the linux split and cmp commands. Sure enough, three pages of flash memory were set to 0xFF, probably from some erroneous page clear operation. Brown-out detection was disabled by the fuse settings to save power. Big mistake! What I think happened was as the voltage dipped below its valid range the microcontroller started executing random commands, including 3 SPM page clears, one of which was the first page of flash which holds the interrupt vector table. Valid execution was doomed.

What I should have done was to enable brown-out detection in the fuses, 2.7V seems the most appropriate setting for an 8MHz clock and 3.3V operation. To save power during sleeping the brown-out detection can be disabled with a call to sleep_bod_disable(). Lesson learned!

Magnetometer progress report

About AuroraWatchNet

I am developing a simple, low-cost, battery-powered magnetometer for auroral alerts and citizen science. It is intended to be wireless to allow easy installation, a unique feature as far as I am aware. The magnetometer uses my own Calunium micro-controller board, which is an Arduino clone based on the ATmega1284P micrcontroller. Data is sent via a radio link to a Raspberry Pi base station which will then forward the data to AuroraWatch UK. I am hoping to deploy a network of such magnetometers in 2013 to enable AuroraWatch UK users to contribute data to improve the service. If you want to track the development then follow @aurorawatchnet on Twitter. 

The first AuroraWatchNet magnetometer is currently running in my garden. The environment is not ideal as I can see disturbances from cars and even the garage door being opened but it is convenient for testing and representative of what to expect from user-contributed data. I have been comparing the results from our SAMNET magnetometers with the AuroraWatchNet prototypes and generally the results are very encouraging given the difference in cost between the two systems. Now I can plot the AuroraWatchNet magnetometer data in the standard AuroraWatch UK style, using the same Octave code. Below is a comparison of today's plots. 

AuroraWatch UK

The data is taken from SAMNET's Lancaster magnetometer. The X axis is time and the Y axis shows the magnetic field strength in the direction of magnetic North (H component) in nanotesla (nT) relative to an unrecorded baseline. This exact plot appeared on AuroraWatch UK. We would normally expect cleaner data but our Lancaster site has recently been experiencing some interference.

SAMNET data from Lancaster, as used by AuroraWatch UK

AuroraWatchNet test magnetometers

The plots below correspond to the same time interval as the plot above. The first plot is from my garden magnetometer.

AuroraWatchNet data from near Lancaster.

The Y axis again records magnetic field strength but this time it is the absolute field. This magnetometer is operating in power-saving mode, turning off the sensor when not needed. This slightly increases the noise level but I hope will give one year of operation from two D cells.

Another AuroraWatchNet magnetometer has been deployed almost adjacent to the SAMNET magnetometer we already use for AuroraWatch. In this system the magnetometer sensor is continually powered, resulting in reduced noise levels. The alignment with magnetic North is only approximate. 

AuroraWatchNet data from the Lancaster University field station.

Conclusion

The AuroraWatchNet magnetometers are sufficiently sensitive and stable to make useful contributions for auroral detection and citizen science. My experience with the garden magnetometer suggests that in many cases performance will be limited by the noise at the site, rather than the magnetometer.

RFM12B shield for Raspberry Pi: Preliminary design

Raspberry Pi interface to the RFM12B radio module, with integrated ISP programmer. The cut-out is to enable the PCB to fit inside the Multicomp Raspberry Pi case.

Current status

Earlier in the year I posted a design for a wireless gateway for the Raspberry Pi. It featured interfaces for the Hope RFM12 and XBee (or Ciseco XRF) radio modules. Also included was an in-circuit serial programming (ISP) programming interface for Atmel microcontrollers. I later refined the ISP interface to include some protection for the Raspberry Pi by adding a buffer and FET switch. I have not yet tried accessing the RFM12B directly from the Pi but other people have had problems, mainly it seems because of the time-critical interrupt handling. I'm sure this problem will eventually be solved once the Eve Alpha boards become widely available. Martin Harizanov solved this problem by adding an Atmel ATtiny microcontroller to bridge between the UART interface on the Pi and the SPI interface on the RFM12B. I bought a bare PCB that did this from the Open Energy Monitor shop. I tested the RFM12Pi Raspberry Pi Expansion board communicating with one of my Calunium v2 development boards, everything worked. Martin chose the ATtiny84 microcontroller which has just 512 bytes of RAM. I would like to have a solution which more closely mimics the Ciseco XRF module that I am currently using, which has a 240 byte buffer. One reason for needing a larger buffer is that the radio protocol I am using supports over the air firmware updates, with the new firmware sent in 128 byte blocks. If I am to implement something with 240 byte transmit and receive buffers I will need a microcontroller with more than 512 bytes of RAM.

Preliminary design

Based on my previous work and inspired by Martin Harizanov's implementation I have designed a RFM12B shield for the Raspberry Pi with integrated ISP interface. The microcontroller on this board is the popular ATmega328P, as used in the Arduino. Including the ISP interface has considerably increased the size of the PCB but this is a price I am willing to pay. The board is intended to be used as part of the AuroraWatchNet magnetometer network and the ability to remotely update the firmware will be very useful, especially since I can already perform over-the-air firmware upgrades on the battery-powered Calunium node. Surface-mount passive components could be used to reduce the PCB size but this would be contrary to my goal of making the AuroraWatchNet magnetometers easy to build for novice constructors.

The 74LVC244 buffer has two 4-bit line drivers, allowing two independent ISP interfaces to be included. The first is for the on-board ATmega328P microcontroller, the second is wired to a standard Atmel ISP header to allow an external microcontroller to be programmed. It means that the ATmega1284P used in the remote Calunium of the AuroraWatchNet magnetometers can be programmed initially, or recovered in the case of an over-the-air firmware upgrade mistake.

I hope to have some circuit boards made in January 2013 so if you spot any mistakes please let me know as soon as possible!

Open source

The Eagle PCB design files for the RFM12B/ISP shield for Raspberry Pi are available on Github and are licensed under the Creative Commons Attribution-ShareAlike 3.0 Unported License.

RN-XV+SD shield

RN-XV+SD shield. Click on the image for an annotated version.

In my quest to make a low-power magnetometer for AuroraWatch UK there are two additional periphals needed: a Wifi connection for real-time data transfer and an SD card for configuration and data logging. I wasn't happy with any of the existing XBee or SD shields. Many of them use resistor-divider networks which prevent dual 3.3V/5V operation (very useful for development), and those which do use logic-level shifters seem to ignore the problem of mixing 3.3V devices with 5V ones.

In response to this need I designed my own shield. I investigated several Wifi modules, but the main contenders were the XBee wifi module and the similar RN-XV module made by Roving Networks. Although I like the idea of the SPI interface that the XBee module offers in the end I chose the RN-XV, partly because it was cheaper but also for its better availability. Both modules use the same pseudo-standard XBee footprint so to some extent either could be used, although the circuitry is designed for the UART interface and GPIOs of the RN-XV. I also included the circuitry for the PNI MicroMag3 module. This is outside the main shield area so as to be located away from the metal casing of the SD card and away from the Wifi module, which could cause interference. To test for temperature effects an LM61 temperature sensor can be used. Unlike the optional one on Calunium this is powered from a logic output for ultra-low power consumption. The shield is designed for Calunium, but will work with full functionality on the Arduino Mega2560. It will also operate on a standard Arduino but flow control and some of the GPIO features of the RN-XV are not available.

Fitting everything onto a 80 × 100mm PCB (the largest size possible in the free version of Cadsoft Eagle) was difficult so I chose to use surface-mount components. The many jumpers are included to make the board as flexible as possible. I sent the Gerber files to Iteadstudio for manufacture. For maximum yield I used very conservative 0.012" trace widths and 0.012" minimum separation, with 0.024" ground-pour isolation. The magnetometer circuitry is easily removed from the schematic and board design if it is not needed; on the manufactured boards the magnetometer section can be removed with a guillotine without affecting the rest of the circuitry. The cut-out is carefully arranged to avoid obstructing the reset button and the USB and JTAG connectors on Calunium.

RN-XV+SD shield on Calunium.

Eagle PCB design files available under the Creative Commons Attribution-ShareAlike 3.0 Unported (CC BY-SA 3.0) licence.

Test magnetometer results

Magnetometer electronics in their temporary housing.

On Friday I deployed my magnetometer to the same site used by SAMNET for the Lancaster magnetometer. I knew that several coronal mass ejections (CMEs) were due to impact the Earth's magnetosphere. The predictions were correct, on Saturday AuroraWatch UK issued an amber alert. Below is a comparison of the data from my test magnetometer and the SAMNET magnetometer.

Test magnetometer (top-panel) and the D component from SAMNET's Lancaster magnetometer (bottom panel).

Sensor temperature for the same period.

The results are encouraging even though at present the test magnetometer isn't calibrated, and the orientation is only approximately aligned with the magnetic east component ("D"). The baseline differences can be ignored as SAMNET does not make full-field measurements, just variations from a fixed baseline. I haven't measured it yet but the test sensor has a large temperature coefficient; if the FGM-1 behaves in the same way as the FGM-3 then I expect it to be around 25nT/°C. The graph shows the temperature over the same period varied from 16.5°C to 30°C.

Clearly some kind of temperature stabilisation is going to be needed. One option is to bury the sensor. Another is active temperature control, but care will be needed to ensure that DC currents for the heater do not affect the magnetic field being measured!

A simple magnetometer for detecting auroral activity

I've been playing with the idea of creating a simple magnetometer which can be used for detecting times when the aurora might be visible. AuroraWatch UK already provides a service for this, with email and Twitter alerts. The AuroraWatch web pages also show how to build a simple pop-bottle magnetometer, and a compass detector, but I wanted something which could easily output digital data. Now I've found a suitable sensor, I've started building a prototype using the open-source Arduino micro-controller platform.The recent Howduino workshop at Lancaster was very helpful in finding (and fixing) a few bugs in the DHCP support.

 

A short test at home didn't reveal much, except that cars cause far too much disturbance to the magnetic field. The next test will be at a more remote location but first I need to extend the sensor lead and add some buffering and protection circuitry. It's unlikely I'll have a network connection for time keeping so using NTP won't be possible so I'll probably add a real-time clock.