Magnetometer progress report: link indicator LED

I recently added a new feature to the magnetometer remote sensor unit firmware. To help indicate when it is within radio communication range of the base unit an LED is turned on at the start of a message communication. When the sensor unit receives the acknowledgement from the base unit of successful message receipt the LED is switched off. In normal operation the LED should blink briefly every 30 seconds, after each sampling interval. If the LED remains on it indicates a problem with the radio link. For the initial installation the sampling time can be reduced to 5 seconds for to obtain faster feedback of whether communication is successful or not.

Improvements for battery-powered operation

Indicating link errors by using an LED is convenient during installation but the power wasted after installation is not compatible with battery-powered operation. To save power the LED is used only the first 15 minutes of operation. Only user-initiated reset actions (power on or reset switch pressed) cause the LED to be used, resets from by the watchdog timer or brown-out detector are ignored. The cause of the reset is detected by examining the microcontroller's status register (MCUSR).

Communication timeout feature added

I've also added a timeout which detects when communication has been lost for an extended period. The microcontroller system is rebooted in the hope that the error is recoverable. Loss of communication, along with low battery status, can also be sensed and reported by running the latest version of the data plotting software on the Raspberry Pi. If either error is detected a message can be sent via email, Twitter or Facebook.

Cloud detector: a review of progress so far

The hardware

My cloud detector has been running outside for over 5 months now. Overall I'm very pleased with how it works. The battery-powered hardware is based on the AuroraWatchNet magnetometer design, which uses my own Calunium microcontroller development board. The remote sensor board and fluxgate sensor are omitted. I've added a Melexis MLX90614 non-contact infra-red thermometer to measure the sky temperature. Clear skies should give low temperatures whilst clouds are expected to have warmer temperatures, although still colder than the ambient temperature. The MLX90614 also outputs the sensor temperature, which should be close to the ambient temperature. I also added a Honeywell HIH-6131-021 humidity sensor which has an I2C interface. Do not confuse with the similar sounding version which has an SPI interface! The HIH-6130 also provides and ambient temperature measurement. To avoid direct contact with water the humidity sensor is placed at the bottom of the enclosure with a hole underneath to expose it to air. This hole also functions as a breather hole and ensures that the internal pressure matches atmospheric pressure. Since this hole was drilled there have not been any more incidents of water ingress.

The Calunium board is running a firmware is a modified version of the magnetometer firmware. It communicates with the Raspberry Pi data logger using an 868MHz radio link, with the same signed communication protocol used by AuroraWatchNet. This means I can use the data recording software from AuroraWatchNet. It also inherits the signed over-the-air firmware update capability. In principle it should be possible to combine both magnetometer and cloud detector functions in one unit.

I started off using the 90°  field-of-view version of the non-contact IR thermometer but later switched to theMLX90614ESF-DCH-000-TU-ND variant which features a 12° field of view. Its greater height meant it was easier to fit into the cable grommet housing. However I think the FOV is too narrow and I plan to switch back to the original sensor when I have time.

Data processing

The data processing and plotting now uses the auroraplot library for Python, which I developed to process the AuroraWatchNet magnetometer data. On the graphs I plot the sky temperature measured by the non-contact IR thermometer and the ambient temperature measured by the humidity sensor. For monitoring purposes I also plot the sensor temperature of the IR thermometer (marked as "detector temperature"); this should be similar to the ambient temperature but its exposed position makes it more likely to undergo solar heating and radiative cooling. For good measure I also plot the system temperature, which is measured by an LM61 temperature sensor connected to the ATmega1284P's analogue to digital converter. This measurement is noisy but useful to check the system doesn't overheat on sunny summer days.

The plan is to estimate the upper and lower bounds that I would expect for the sky temperature, and from that derive an estimate of the cloud cover. I initially expected that with complete cloud cover the sky temperature would match that the lifted condensation level, which I estimate using the ambient temperature and relative humidity. I soon saw temperatures higher than the LCL temperature. Research literature indicates that the clouds can act as a mirror at long IR wavelengths and thus the expected temperature should include an effect of ground temperature too. On the plots this is shown as the effective cloud temperature. The clear sky temperature is derived from results found in research literature but none of the equations tried so far have been a great match. Other researchers have fitted parameters for their specific location (including altitude) by comparison with visual measurements. I have yet to try this but daily measurements are made at the nearby Hazelrigg Weather Station. The graphs are too cluttered for production use but help me to understand what is happening.

Example plots

Below are a selection of plots. You can see the entire archive (from 2013-07-14) at http://aurorawatch.lancs.ac.uk/testing/cloudwatch/test2/.

Cloud detector data for 2013-09-03. The asterisk (*) indicates derived parameters (ie not directly measured).

The figure above include most of the effects that can be identified. After midnight there is thick cloud, which later clears for short periods. At around 0300 UT I think it must have rained, the sky temperature is almost identical to the detector temperature and the variability is much reduced. Whilst the sensor is wet no sensible conclusions about cloud cover can be drawn. At about 0700 UT the sensor clears of water and the sky temperature falls indicating clearer skies. The rest of the day is dry, with heavy cloud cover until 1400 UT. At around 1700 UT the skies are completely clear. The clear sky estimate is too low for this time. At 2000 UT the clouds return, with some clear patches.

Cloud detector data for 2013-09-28 showing the effect of clear skies.

The figure above shows an almost completely cloud-free day. The higher variability in the sky temperature measurements between 1200-1600 UT corresponds to similar effects in the humidity data (shown below); both are direct measurements made by different sensors.

Humidity data for 2013-09-28. Notice how the higher variability occurs at the same times as the sky temperature measurements.

Dew

During summer operation dew has not been a problem but now that the nights are colder I have noticed effects which I believe are due to the formation of dew on the sensor. In the first plot below the sky temperature apparently rises from around 2230 UT. The rise is smooth and during this period the ambient temperature is falling, which overnight is often a sign of clear skies. After midnight (second plot) the sky temperature increases slightly before falling sharply around 0630 UT. Sunrise on this day was 0622 UT.

This pair of plots is believed to show dew formation on the sensor.

Conclusions

I know from the informal comparisons with visual cloud cover that I regularly make the cloud detector does generally work very well. It does not function during wet periods. As anticipated dew is becoming a problem during the colder nights and a dew heater will be required for reliable winter operation. Future development will be to add a heater, which will need a wired connection to the cloud detector. Once a wired connection is made the radio link appears superfluous so I plan to investigate the options for power-over-ethernet. The Arduino ethernet shield is one possibility, although its compatibility with 3.3V operation has not been established.

Auroraplot: data processing software for AuroraWatchNet

With the dispatch of the AuroraWatch schools' magnetometers imminent I have implemented a  Python  toolkit to process the data. The numpy and matplotlib modules are used extensively. The toolkit provides an API to load data and perform various processing actions on it, such as plotting data. The concept is influenced by my previous Multi-instrument Analysis toolbox for Matlab. In addition to the loading and processing of magnetic field data auroraplot allows other data types to be added later.

Loading data

Data can be loaded with arbitrary start and end times very simply:

md = ap.load_data('AURORAWATCHNET', 'LAN1', 'MagData',
                  np.datetime64('2013-09-20T12:00:00+0000'),
                  np.datetime64('2013-09-21T12:00:00+0000'))


In this case the selected portion of data crosses midnight and two data files must be loaded, concatenated and trimmed to get the desired time range. This is performed automatically by auroraplot, the user need not be concerned with the format of the files or where they are located. It is even possible for the files to be downloaded on-the-fly using FTP or HTTP transfer protocols.

load_data returns an object (of type MagData) to the user containing the actual magnetic field data and various other metadata, such as a timestamp for each sample and the data units. Each object can store more multiple data channels but all data points must share the same timestamps, be of the same type and share the same units. Therefore it is not possible to store operating temperatures (units °C) in an object holding magnetic field strength (units tesla). The operating temperature data can be accessed as:

td = ap.load_data('AURORAWATCHNET', 'LAN1', 'TemperatureData',
                  np.datetime64('2013-09-20T12:00:00+0000'),
                  np.datetime64('2013-09-21T12:00:00+0000'))


Battery voltage (data type VoltageData) can be accessed in a similar same way.

Plotting data

High-level plot functions enable the data be be plotted very simply, for the magnetic field data loaded previously

md.plot()


will produce a matplotlib figure with a title and the axes labelled with the correct units. Temperature and voltage data are plotted in the same way.

I have created some tools to make working with numpy's datetime64 and timedelta64 objects more convenient, including rounding functions (round, ceil and floor) which round to an interval. They are useful for finding the start of an hour, or the end of a day. I have also created Locator and Formatter classes to sensibly label time axes using datetime64 times and timedelta64 intervals. Tick marks are located on the nearest second, minute, hour, day, month or year boundary (or multiple thereof) depending on the time interval being displayed. Thanks to matplotlib's structure the labels are automatically regenerated with the most appropriate time units when a plot is zoomed.

Other operations

Quiet-day curves

Other operations include the generation of quiet-day curves. These are the curves from which we measure geomagnetic activity and are of critical importance for AuroraWatch UK. There are are not flat but have a daily variation caused by the equatorial electrojet. The empirical algorithm selects the days (typically 5) with the least geomagnetic activity. A truncated Fourier series is used to guarantee that the quiet-day curves are cyclic, with the start and end points having the same magnitude and slope. This is essential otherwise our rolling plots would show up the discontinuities in the QDC at midnight, and would falsely cause step changes in the geomagnetic activity. An example QDC is shown below.

Quiet-day curve for magnetometer at Lancaster ,UK. This is derived from recorded data
and clearly shows the Sq current system caused by the equatorial electrojet.

From this we can see that even on a geomagnetically quiet day we would expect a 30nT variation in field strength seen by the magnetometer. The AuroraWatch threshold for minor geomagnetic activity is 50nT so this shows the importance of using a quiet-day curve instead of a flat line when calculating geomagnetic activity.

Stack plots

Stack plots (also called magnetograms) are a convenient representation for magnetic field data from a set of magnetometers separated in latitude.Data from the northernmost instruments is placed at the top and that from the southernmost at the bottom. An example stackplot is shown below:

Stackplot showing data from two Lancaster stations and from Ormskirk.

The magnetometer at Ormskirk is operated by the Met Office as part of a test. The stackplots will be more interesting as the network grows.

Open source

The source code is available under a BSD-type license from Github.You will need python, along with the numpy (version 1.7), matplotlib and scipy python modules.auroraplot has been tested under Debian Linux (64 bit version) and Raspbian on the Raspberry Pi.

Securing your SSH server against brute force password attacks

If you run an SSH server open to the whole internet you'll soon notice many attempts to break into your system using random username and password combinations. This 'noise' in the log files is annoying and might mean you miss an important message. If you are unlucky or use a common username and password your system might be compromised by one of these brute force attacks.

One method to drastically reduce the number of cracking attempts is to restrict access to the SSH server by IP address. This can done be done either by editing the SSH config file or by using configuring iptables. However this method requires valid users to have static IP addresses, something which most home users don't have. In many cases public key authentication is a better choice.

Use public key authentication

A better method to stop attackers from using random username and passwords is to switch off password authentication entirely and require public key authentication instead. Attackers will not be able to try random username/password combinations and will not create 'noise' in the log files. Valid users will still be able to connect.

To describe how this can be set up assume we wish to protect a computer with a static IP address called server. We wish to access it from another system called laptop, (which may be using a dynamic IP address). The account on server that we will access is dave. We'll assume that the SSH configuration file is /etc/ssh/sshd_config. Adjust these names to suit your setup.

Firstly, on laptop create a public/private key pair.

ssh-keygen -t dsa

A passphrase is not required although it is a good idea. On server make sure a ~/.ssh directory exists with the correct permissions:

mkdir ~dave/.ssh
chmod go= ~dave/.ssh

Using your favourite editor (emacs of course) paste the contents of the public key into the ~dave/.ssh/authorized_keys file on server. Putty users may find the instructions at http://www.howtoforge.com/ssh_key_based_logins_putty helpful. Check that you can log into server without requiring your password. If you are going to configure server remotely ensure you can log in without needing your password, otherwise you will lock yourself out! If you set a passphrase you will need to type that.

Configure the server

The key requirement for the server configuration is that by default password authentication is turned off. Find the line in /etc/ssh/sshd_config which sets password authentication. Make sure it is set to

PasswordAuthentication no

If necessary remove any leading # character. Whilst editting /etc/ssh/sshd_config its a good idea to disable empty passwords and root logins, ensure the following lines are set

PermitEmptyPasswords no
PermitRootLogin no

If you need empty passwords or root logins from specific hosts this can be overridden later.

Enable password authentication for trusted hosts

Once password authentication is disabled users on a multi-user system will have a problems copying their keys to grant them access! Exceptions can be made for trusted static IP addresses so that users can copy their public key. Assume we trust all IP addresses in the subnet 192.168.1.0/24. Passwords can then be enabled by simply adding the following lines to the end of /etc/ssh/sshd_config

Match Address 192.168.1.0/24
PasswordAuthentication yes

Enable root logins

Suppose a root login using password authentication is required for one host only (192.168.1.123). Add the following lines to /etc/ssh/sshd_config

Match Address 192.168.1.123
PasswordAuthentication yes
PermitRootLogin yes

Summary

These simple steps keep your SSH server accessible from everywhere but greatly reduce the likelihood of a successful brute force attack. The Match keyword requires openssh version 4.4 or later.

Cloud detector: first data

Annotated plot for 2013-05-30.

The cloud detector has been operating outside for a few days now. The plot above is an example of the data I am getting, annotations added manually! The ambient temperature is the temperature of the sensor. In reality this may be a little warmer than ambient when it is exposed to direct sunlight. The sky temperature is the object temperature from the sensor. It is an average over its field of view, and in the absence of clouds it is also averaged in height. From the plot a number of features can be identified:

Clear sky
With no clouds to reflect back thermal radiation the sky temperature for a clear sky is very low, typically < -10°C. This results in a large difference (> 20°C) between ambient and sky temperatures.


Overcast sky
With the presence of clouds most thermal radiation is not lost to space but reflected back to Earth and the sensor. The difference between ambient and sky temperatures is small (< 10°C). With a uniform cloud coverage the variation in the difference is small.


Cloudy sky
A sky with partial cloud cover has an intermediate difference in ambient and sky temperatures. The cumulus clouds resulted in a large variation of the temperature difference, caused by the more sensitive central viewing area observing alternately clear sky and cloud. Complete coverage by a thin layer of cloud is expected to give an intermediate temperature difference but low variability.


Wet sensor
When the sensor is wet with rain the sky temperature is measured incorrectly since the water is not transparent to long-wave infra-red radiation and matches the ambient temperature. As wet weather implies cloud this is not a major problem although I have concerns over the time taken for the sensor to dry off.


Dew
The formation of dew is observed in the evening as the ambient temperature falls. The difference between ambient and sky temperature is reduced to less than 10°C.

Future work

Early operation shows that dew is a problem and can indicate cloud when the sky is actually clear. I have considered creative solutions such as painting the top of the box white and the bottom black in order to minimise heat lost to the sky and maximise heat absorbed from the ground. I expect this will only delay the onset when dew becomes a problem, not eliminate it. The only viable solution is to heat the sensor slightly (~2°C) above ambient, which means moving away from battery-powered operation. If a cable is used for power then wireless communications makes little sense, power-over-ethernet seems the way forward. A heated sensor will also reduce the drying time after rainfall.

I still have some concerns regarding waterproofing around the sensor. I have ordered the MLX90614ESF-DCH-000-TU-ND which features a 12° field of view. This narrower view is obtained by fitting a refractive germanium lens above the sensor. The TO-39 package has been modified to be taller than standard to accommodate the lens. The additional height should mean that the top of the sensor is above the cable gland. Water should not collect so easily and it should be possible to use silicone sealant without accidentally coating the sensor window.

Previously I tried pointing the sensor downwards and using a metal surface as a mirror to view the sky. The experiment was not successful and I think this was in part due to the wide field of  view of the sensor (90°) preventing a clear upward view without also observing the ground or enclosure. It will be worthwhile repeating the experiment with the narrow field of view sensor.

I only have basic plotting routines at present. The goal is to produce a cloud cover index, varying between 0 indicating no cloud and 1 indicating complete cloud cover. The index will be used as the basis for sending alerts.

Cloud detector progress

Microcontroller and radio communications

To minimise the time and effort required to test the cloud detector concept I am following the approach used for the AuroraWatchNet magnetometer system and reusing as much hardware from it as possible. The MLX90614 non-contact infra-red temperature sensor is located outside. To minimise the infrastructure requirements the sensor unit is battery-powered and transmits its data over a bi-directional radio link. The sensor is controlled by one of my Calunium v2 microcontroller development boards. I'm using a pair of RFM12B radio modules operating at 433MHz to emulate a transparent serial connection. One is fitted on  the Calunium board and the other radio module is connected to a Raspberry Pi via my RFM12B shield. The Pi records the data and is responsible for uploading it for general access. I am also reusing most of the AuroraWatchNet firmware, which means I already have signed data communications and the capability to deploy over-the-air firmware updates - quite a good starting position for a new project!

Waterproofing the sensor

The biggest challenge I expect to face with this project is waterproofing the sensor. Although it is hermetically sealed there is no easy way to deploy it outside. As explained in a previous post, only a few materials are transparent to long-wave infra-red emissions and the better ones are toxic, water-soluble or expensive. I am therefore trying to avoid covering the sensor window and instead keep the electrical contacts dry. My first attempt was to drill a hole in the box and attach the sensor with silicone sealant. The mechanical fixing lacked strength and I wasn't convinced I had completely sealed around the sensor. To make matters worse I ended up with sealant on the sensor window which had to be cleaned off before it set. Another approach was required.

The new approach is to fit the sensor inside a cable gland. Before doing so I had to attach wires to the sensor connectors and insulate with heat-shrink sleeving. The result is shown below.

The next step is to fit the sensor in the cable gland before fitting the cable gland to the box.

The other end of the wires can then be soldered. For the prototype the wires are soldered to a small piece of stripboard. The stripboard has two pull-up resistors for the I2C bus and a decoupling capacitor. There is also an IDC connector to link the board to the unused JTAG interface. This was the easiest way to connect to a stock Calunium v2 board. For the lowest power operation the sensor is powered from logic-level output and I am using software I2C so all connections can be grounded when the sensor is not in use. The real-time clock on Calunium is connected the the hardware I2C interface on the microntroller and can be operated independently of the sensor.

The box and cable gland are rated IP65. Hopefully the result will be waterproof  

Results

The sensor is now deployed outside and reporting back to the Raspberry Pi base station. The transmitted packets are logged by the Raspberry Pi but as yet I don't have a convenient way to extract the ambient and object temperatures for analysis. That is the next task.

An open-access cloud-detection network

Introduction

Living in the North-West of England I find that my attempts to view the aurora or astronomical events such as meteor showers, lunar eclipses and comets are often prevented by cloud. On these occasions I wonder if it is worthwhile driving somewhere to find some clear sky. But where? If only I could look at a map to see where it was clear now. Publicly-available satellite images don't appear to have the resolution needed for this purpose. Maybe I should go to bed instead and wait for an alert when the skies have cleared? Could the question of where and when to find clear skies be answered by an open-access cloud detection network operated by citizen-science volunteers?

Requirements

I've together a draft list of requirements to construct a useful network:

• Open-access. While my primary concern is with viewing the aurora I can see there are many other uses of such a network. Astronomy and obviously meteorology. Open-access enables other observers to make use of the data for their own purposes.
• Near real-time. The weather here can quickly change. Ideally measurements should be taken every few minutes and be made available without unnecessary delay.
• Use open-source hardware (OSHW). More specifically, an OSHW design should exist which allows users to contribute readings with the lowest possible cost.
• Large geographical coverage. 
• Night time operation required, day time beneficial.

The most difficult requirement to satisfy is the coverage. With ground-based sensors the viewing area might be as small as 2km diameter (cloud-base 1000m, optical viewing angle ±45°); at this size complete coverage of Great Britain would require over 57000 installations. A useful network does not require complete coverage, for auroral and astronomical purposes good results could be achieved by positioning sensors in a few viewing locations selected for low light pollution.

Remotely-sensing clouds

Perhaps the simplest method to remotely-sense cloud cover is to measure the 'sky' temperature with a non-contact infra-red (IR) thermometer. In the absence of cloud cover this is very low (say -12 to -20°C for Spring in the UK) but with cloud cover the temperature measured is warmer (0 to 4°C). These temperatures vary over the course of the year but by looking at the difference between the sky and ambient ground-level temperatures an estimate of cloud cover is possible. Some of these thermopile sensors have a digital output for interfacing directly with a microcontroller.

A camera and image-recognition software is another approach but it is more complex to develop, install and operate. It is not clear how such a system would perform at night. In order to install the maximum number of sensors I plan to concentrate on cheaper methods.

Design concept

I believe there are two key factors which limit the number of sensors in the network: cost and ease of installation. To minimise the infrastructure I plan to begin with a battery-powered unit which communicates with a Raspberry Pi (or other computer) via a low-cost, licence-free radio module. This is the same concept used by the AuroraWatchNet magnetometer sensors.

Potential problems

Precipitation (rain, snow, hail) and dew on the sensor will prevent accurate sky temperature measurements, most likely indicating cloud when it may in fact be clear. A heater to prevent dew forming is an option but the temperature gradients may cause the thermopile to give incorrect measurements. The power consumption needed is incompatible with battery operation. It may be possible to point the sensor downwards, to avoid precipitation and dew, and use a reflective metal surface to observe the sky. Tests shows the sky temperature readings are partially compromised by this approach.

Sensor

I have found a number of sensors which may be suitable:

• Melexis MLX90614. This sensor contains thermopiles for sensing both the object and ambient temperature. An SMBus interface enables direct connection to a microcontroller. It is packaged in a TO-39 metal can. The standard 3.3V operation, 90° field of view (FOV) variant is readily available in single quantities from UK hobby electronics stores for around £15 each. Other variants exist for 5V operation and with different fields of view (10°, 35°, and a dual 70° FOV). The minimum object temperature is -70°C. The only irritation is that the sleep mode requirements are not compatible with low-power operation on a I2C bus.
• Melexis MLX90615. Similar to the MLX90614 but the sleep mode requirements are compatible with low-power operation on an I2C bus. Cost is similar to MLX90614 but no UK supplier exists. Package is a TO-46 metal can. Limited to 3.3V operation and a single FOV (80° or 100° variants). The minimum object temperature is -40°C.
• Texas Instruments TMP006. Small thermopile with I2C interface for direct connection to a microcontroller. Ball-grid array package. Readily available from UK suppliers. Cost is around £3 each for single quantities.
• Thermometrics ZTP-135SR. Thermopile with analogue output in a TO-46 package, with a thermistor for ambient temperature measurement. Being an analogue output additional processing is required. Around £10 from Farnell.
• Devantech TPA81. Linear array of 8 thermopiles. Cost is around £65.
• Panasonic AMG88 Grid-EYE. An 8×8 imaging sensor in a surface-mount package. The minimum object temperature is -40°C. Available from Digikey for around $40.

For the prototype system I plan to use the Melexis MLX90614 sensor. It should be possible to work around the problems with low-power operation by using software I2C and powering the sensor from a logic output, allowing power to be completely removed when not in use.

I excluded the TMP006 because the BGA package is hard to work with; not only is it difficult to solder it will require some kind of enclosure or window to protect it from moisture. Only a few materials are transparent to long-wave infra-red emissions and the better ones are toxic, water-soluble or expensive. The other sensors may be suitable but are more expensive. The TPA-81 linear array and Grid-EYE imaging sensor could provide spatial discrimination and thus distinguish between a thin layer of complete cloud cover and denser clouds with open areas of sky.

Initial results

Initial tests with the Melexis MLX90614 sensor confirm the concept is viable. I have not noticed any ill-effects from the sun within the field of view, suggesting that daytime operation is possible. Further work is required to package the system into a usable prototype which can be deployed outside permanently. Full-time operation will indicate the frequency of measurement errors from dew and precipitation.

Other work

It would be wrong to suggest I am the first to make measurements of cloud cover in this way, useful links are listed below. I am not aware of any existing cloud detection networks.

http://bildr.org/2011/02/mlx90614-arduino/

Useful code for interfacing the MLX90614 IR thermometer with an Arduino. This code works with my Calunium Arduino clone.

http://www.kwos.org/clouds_detector.htm

Good background information.

http://kcotar.org/arduino-weather-station-1/
A weather station which uses the Melexis MLX90614 sensor for cloud detection.

http://www.noao.edu/staff/gillespie/projects/cloud-detector.html
Similar in concept but using a Peltier device (in reverse) to measure the difference between ambient ground temperature and sky temperature.