Category Archives: Developing other sensors

I am developing an entire family of open source loggers for use in caves and underwater environments.

How to Configure I2C Sensors with Arduino

I’ve spent the last year in the ‘uncanny valley’ of the Arduino. That’s the point where you understand the tutorials at Arduino.cc, but still don’t get much from the material on gitHub because trained programmers would never stoop to using the wire.h library when they could just roll their own in native C++ using the avr-g compiler.  The problem with establishing sensor communication at the level of the TWI peripheral inside the AVR is that there are so many fiddling details to keep track of that it quickly overruns the 7±2 things this average human can hold in his head at one time: Computers aren’t the only things that crash after a buffer overflow!  So this post is meant to be a chunking exercise for beginner-intermediate level people like myself who want to get a new sensor working using the standard IDE.  I’ve tried to distill it all down to things that I run into frequently, but there’s still a lot of material here:  So pour yourself a cuppa before diving in...

The great strength of I2C is that you can put so many sensors on the same four wires. But for units with several pre-made modules connected you might have to remove a few smd resistors from the breakouts, or the pull-up on the bus might become too aggressive. Most of the time I just leave them all on, so I can extend the wire length, or crank up the bus clock…

REGISTERS are simply memory locations inside an I2C device. The summary of how many registers there are in a given sensor, and what they control or contain is called a register map. Most of the information on the sensor’s datasheet is about explaining how each register functions, and they can be quite a slog to read through because the information is rarely presented in an intuitive way.

To give you a sense of what I mean by that: take a look at page 14 of the manufacturers datasheet for the ADXL345 accelerometer:

A document only a hardware engineer could love…

Then take a look at the interactive register map for that sensor over at the i2cdevlib site:

Even if you’ve never worked with registers before, jrowberg’s visual grid layout makes it easy to see how the sensor’s memory is divided into sections, which are doing different things.

There are many kinds of registers but for this introduction I am going to group them into three general types: Control, Data and Status registers, and provide brief examples of code that you can use to work with each of them. The functions named with the i2c_ prefix should be generic enough to work with most I2C sensors, but I’ll also be referring to a few specific cases to show how you might need to modify those basic functions.

1) Control Registers

Most sensors change how they operate based on the values stored in control registers. Think of control registers as banks of On/Off switches, which you turn on by setting a bit to 1 and turn off by setting that bit to 0.  I2C chip-based sensors often have a dozen or more operational settings for things like bit-depth, sampling speed, noise reduction, etc., so you usually need to set bits in several different control registers before you can actually take a reading. And sometimes there are “special chip functions” that perform some kind of post processing on those sensor readings that would be hard to replicate on the Arduino. These can add an extra layer of control settings to take care of when you initialize the sensor.

Arduino’s wire library can only transfer 8-bit bytes over the I2C bus, so that’s the smallest amount of information you can write into a register memory location at one time. This can potentially change eight of those control switches simultaneously and, for parameters that are controlled by more than one bit, sometimes it’s actually required that you set them in one register-writing operation.  Most people use byte variables for the sensor’s bus and register memory addresses, but once you’ve figured out the pattern you need to set up in control register switch-bits, it helps to write that information as a long form binary number (eg. 0b00001111) so you can see the on/off states when you read through your code. 

Writing a byte to an sensor’s control register can be done with four basic steps:

Wire.beginTransmission(deviceAddress);  // Attention sensor @ deviceAddress!
Wire.write(registerAddress);   // command byte to target the register location
Wire.write(dataByte);                           // new data to put into that memory register
Wire.endTransmission();

The I2C deviceAddress is set by the manufacturer but some can be modified from their defaults by connecting solder pads on the breakout board.  Since the bus address of a given sensor IC can vary from one module to the next I keep Rob Tillaart’s bus scanner handy to find them, and more importantly to discover when two sensors are fighting with each other by trying to use the same address on the bus.  The registerAddress moves a pointer inside the chip to the memory location you specified. You can think of this pointer as a read/write head and once that pointer is aiming at a specific register, the next byte you send along the wires will over-write the data that was previously stored there.

The startup default values for a given control register are often a string of zeros because all the chip functions being controlled by that register are turned off. Unfortunately this means you’ll find lots of poorly commented code examples out there where people simply write zero into a control register without explaining which of the eight different functions they were aiming for because seven of those were still at their default zero-values anyway.

Reading data from a sensors memory register(s) requires two phases:

Wire.beginTransmission(deviceAddress);    // get the sensors attention 
Wire.write(registerAddress);    // move your memory pointer to registerAddress
Wire.endTransmission();           // completes the ‘move memory pointer’ transaction

Wire.requestFrom(deviceAddress, 2); // send me the data from 2 registers
firstRegisterByte = Wire.read();             // byte from registerAddress
secondRegisterByte = Wire.read();       // byte from registerAddress +1

The first phase tells the I2C slave device which register that we want to read but the wire.library buffers everything behind the scenes and does not actually send anything over the wires until it gets the Wire.endTransmission(); command.  The second phase is the data reading process and you can request as many bytes as you want with the second parameter in Wire.requestFrom. The memory location pointer inside the sensor will automatically increment forward from the initial memory register address for each new byte it sends.

These simple patterns are at the heart of every I2C transaction, and since they are used so frequently, they generally get bundled into functions:


byte i2c_readRegisterByte (uint8_t deviceAddress, uint8_t registerAddress{
byte registerData;
Wire.beginTransmission(deviceAddress);              // set sensor target
Wire.write(registerAddress);                                     // set memory pointer
Wire.endTransmission();
// delete this comment – it was only needed for blog layout.   
Wire.requestFrom( deviceAddress,  1);     // request one byte
resisterData = Wire.read(); 
// you could add more data reads here if you request more than one byte
return registerData;           // the returned byte from this function is the content from registerAddress
}
// delete this comment – it was only needed to maintain blog layout
byte i2c_writeRegisterByte (uint8_t deviceAddress, uint8_t registerAddress, uint8_t newRegisterByte
 {
byte result;
Wire.beginTransmission(deviceAddress);
Wire.write(registerAddress);  
Wire.write(newRegisterByte); 
result = Wire.endTransmission(); // Wire.endTransmission(); returns 0 if write operation was successful
// delete this comment – it was only needed for blog layout.
//delay(5);  // optional:  some sensors need time to write the new data, but most do not. Check Datasheet.
if(result > 0)  
{ Serial.print(F(“FAIL in I2C register write! Error code: “));Serial.println(result); }
// delete this comment – it was only needed for blog layout. 
return result;    // the returned value from this function could be tested as shown above
//it’s a good idea to check the return from Wire.endTransmission() the first time you write to a sensor 
//if the first test is okay (result is 0), then I2C sensor coms are working and you don’t have to do extra tests

//NOTE: copy/pasting code from blogs/web pages is almost guaranteed to give you stray/302 errors because
//of hidden shift-space characters that layout editors insert. Look at the line your compiler identifies as
//faulty, delete all the spaces and/or retype it slowly and carefully ensuring you enter only ASCII characters.


Those two simple functions will let you control the majority of the I2C sensors on the market, provided you can figure out the correct pattern of bits to send from the datasheet. A common strategy for keeping track of complicated multi-bit combinations for control registers is to use #define statements at the beginning of your program, which replace the human readable labels with the actual numbers at compile time.

For example the ADXL345 can range from 3 samples per second to 1600 samples per second, depending on four bits in the ADXL345_BW_RATE register. A set of define statements to represent those bit combinations might look like:

byte ADXL345_Address=0x53;     // the sensors i2c bus address (as a hex number)
byte ADXL345_BW_RATE=0x2c;    // the memory register address
#define ADXL345_BW_1600  0b00001111
#define ADXL345_BW_800    0b00001110
#define ADXL345_BW_400    0b00001101
#define ADXL345_BW_200    0b00001100
#define ADXL345_BW_100    0b00001011
#define ADXL345_BW_50      0b00001010
#define ADXL345_BW_25      0b00001001
#define ADXL345_BW_12      0b00001000
#define ADXL345_BW_6        0b00000111
#define ADXL345_BW_3        0b00000110
etc…. Note that all of these combinations assume normal power mode (bit4=0)

So a command to set the sampling rate to 50 Hz could be written as:

i2c_writeRegisterByte(ADXL345_Address, ADXL345_BW_RATE, ADXL345_BW_50);

 The cool thing about using defines is that they do not use any ram memory like byte variables would. And you can usually find code examples on gitHub where someone has transcribed the entire register address list into a set of defines, which you can simply copy and paste into your own code. This saves you a great deal of time, though there’s always the chance they made a transcription error somewhere. Also note that typical ‘c’ language examples would express those numbers as hex “0x0F” instead of “0b00001111”.

Writing a whole byte to a register is pretty straightforward, but it gets more complicated when you need to change only one of the bit-switches inside a control register. Then the standard approach is to first read out the register’s current settings, do some bit-math on that byte to affect only bit(s) you want to change, and then write that altered byte back into register’s memory location.

But bit-math syntax is one of those “devils in the details” that makes relatively simple code unreadable by beginners. The bit operators you absolutely must be familiar with to understand sensor scripts you find on the web are: the bitwise OR operator [|] , the bitwise AND operator [&], the left shift [<<] and the right shift [>>] operators.  Fortunately there is an excellent explanation of how they work over at the Arduino playground, with a set of bit-math recipes in the quick reference section that let you reach into a byte of data and affect one bit at a time.  Be sure to parenthesize everything when using bitwise operators because the order of operations can be counter-intuitive, and don’t worry if you have to look up the combinations every time because most people forget those details once they have their code working. I know I do. 

Two particularly useful procedures:

x &= ~(1 << n);   // AND inverse (~) forces nth bit of x to be 0. All other bits left alone
x |= (1 << n);       // OR forces nth bit of x to be 1.  All other bits left alone

And these let us add a third function to the standard set which will turn on or turn off one single bit switch in a sensors control register:

byte i2c_setRegisterBit ( uint8_t deviceAddress,  uint8_t registerAddress,  uint8_t bitPosition, bool state )  { 
 byte registerByte, result;
registerByte = i2c_readRegisterByte ( deviceAddress,  registerAddress ); // load the current register byte
// delete this comment – it was only needed to maintain blog layout
if (state) {   // when state = 1
  registerByte |= (1 << bitPosition);   //bitPosition of registerByte now = 1
//or use bitSet(registerByte, bitPosition); 
  }  
else {           // when state = 0
   registerByte &= ~(1 << bitPosition);   // bitPosition now = 0
//or use bitClear(registerByte, bitPosition); 
  }
// now we load that altered byte back into the register we got it from:
result = i2c_writeRegisterByte ( deviceAddress,  registerAddress,  registerByte );
return result;   // result =0 if the byte was successfully written to the register


The ADXL345 accelerometer supports low power modes that use about 1/3 less power than the ‘standard’ operating modes.  The sensor is not turned off, but the bandwidth is reduced somewhat, so the sensor responds more slowly to things like tap inputs.
An example which sets the single bit enabling this low power mode might look like:

i2c_setRegisterBit( ADXL345_ADDRESS,  ADXL345_BW_RATE,  5,  );

Many I2C sensors have power saving features like that which rarely get utilized. Note that bit position numbering starts with 0 and counts from the left OR the right hand side depending on the sensor manufacturer. 

Some devices have control registers that are 16-bits wide. These get treated as a pair of 8-bit bytes that are read from or written to sequentially, but you only have to specify the device & register address once at the beginning of the process because the sensors internal memory pointer gets incremented automatically.

This adds an extra wire.write step to the basic register writing operation:

Wire.beginTransmission(deviceAddress);
Wire.write(registerAddress);
Wire.write(MSB_registerData);    // Send the “upper” or most significant bits
Wire.write(LSB_registerData);     // Send the “lower” or least significant bits
Wire.endTransmission();

The MCP9808 is a common temperature sensor that uses 16-bit control registers.  Setting “bit 8” of the CONFIG register to 1 puts the sensor into shut down mode between readings and setting that bit to 0 starts the sensor up again. (yes, that’s opposite to the usual on/off pattern…)  The 8-bit limitation of the I2C bus forces us to retrieve the register in two separate bytes, so bit 8 of the 16 bits described in the datasheet ends up in the zero bit position of MSB. 

A custom function shutting down the MCP9808 might look like this:  

#define MCP9808_i2cAddress          0x18    // defines in setup are an alternative to using variables
#define MCP9808_REG_CONFIG   0x01    // the compiler swaps the text-name for the # at compile time
// delete this comment – it was only needed to maintain blog layout
void mcp9808shutdown()      //since we used defines, we did not pass any byte variables into the function

 byte MSB, LSB;
 Wire.beginTransmission(MCP9808_i2cAddress);
 Wire.write(MCP9808_REG_CONFIG);
 Wire.endTransmission();
// delete this comment – it was only needed to maintain blog layout
 Wire.requestFrom(MCP9808_i2cAddress, 2); //request the two bytes
 MSB = Wire.read();       // upper 8 bits described in data sheet as 15-8
 LSB = Wire.read();        // lower 8 bits described as 7-0 in the datasheet
// delete this comment – it was only needed to maintain blog layout
 MSB |= (1 << 0); // bitmath forces MSB bit0 (which is ‘bit8’ in the datasheet) to value one
 // using MSB &= ~(1 << 0); here would start the sensor up again by forcing the bit to zero
// delete this comment – it was only needed to maintain blog layout
 Wire.beginTransmission(MCP9808_I2cAddress);  // now write those bytes back into the register
 Wire.write(MCP9808_REG_CONFIG);
 Wire.write(MSB);                          // the one we modified
 Wire.write(LSB);                           // unchanged
 Wire.endTransmission();
}


This ‘position x becomes position y’ translation is common stumbling block for beginners working with multi-byte registers – especially when you add reverse order position numbering into the mix.  But there’s another gotcha with 
control registers that’s even more frustrating if you don’t catch it on your first pass through the datasheet:  Sometimes there are special “write protection” registers that have to be set before you can change any of the other control registers, and these have to be changed back to their “protecting” state before those new settings take effect. You might not get any error messages, but nothing will work the way it should until you get the protection bits disabled and re-enabled in the right sequence. Fortunately less than 20% of the sensors I’ve worked with have this  feature.

Another thing to watch out for are old code examples on the web that appear to be using integer variables to store device and memory register locations, with statements like Wire.send((int)(eepromaddress >> 8));  The I2C wire library only sends bytes/uint8_ts, but people got away with this (int) cast  because it was being corrected behind the scenes by the library, which re-cast the value into a byte at compile time.  The (byte) data type on Arduino is interchangeable with the (uint8_t) variables you see in most C++ coding tutorials. 

2) Data registers

Unlike a control registers bank-of-switches, I think of data output registers as containers holding numbers which just happen to be stored in binary form. Since eight bits can only hold decimal system values from 0 to 255 you usually have to “re-assemble” larger sensor output values from bytes stored in consecutive memory locations. For sensors like the ADXL345 you can concatenate the two 8-bit bytes into one 16-bit integer variable by shifting the MSB left by 8 positions and merging in the LSB with a bitwise OR :

Wire.beginTransmission(deviceAddressByte);  // the pointer setting transaction
Wire.write(registerAddressByte);
Wire.endTransmission();

Wire.requestFrom(deviceAddressByte,2);       // request two bytes
LSB = Wire.read();                                                // byte from registerAddressByte
MSB = Wire.read();                                              // byte from registerAddressByte +1
int combined = (int)MSB;             // MSB now in rightmost 8 bits of combined int
combined = combined<<8;          // shift those bits to the left by 8 positions
combined |= LSB;     // logical OR keeps upper bits intact and fills in rightmost 8 bits

Those steps are usually written in one single line as:

int combined = (((int)MSB) << 8) | LSB;

There are several other ways to combine bytes and some sensors send the MSB first – so you have to check the register map in the datasheet to know the order of the bytes that arrive from the output registers when you request multiple.

Now if you are thinking that looked too easy – you’re right! Most hobby market I2C sensors only have a 12-bit ADC, and since memory is a limited resource there are often status register bits mixed in with the data held in the MSB. Since these bits are not part of the sensor reading, you need to &-mask them away before you combine the MSB & LSB. It gets trickier when the sensor output can be a positive or a negative number because signed and unsigned integers are distinguished from each other by a special “sign” indicator bit, which can accidentally be turned into a “number” bit by bit shifting. (see: ‘sign extension’ in that bit math tutorial )

The temperature data output register in the MCP9808 is a good example of both of these issues:

Bits 15-13 (which become the top 3 bits of the upperByte in the code below) are status indicator flags identifying when high & low temp. alarm thresholds have been crossed. Bit 12 is a sign bit (0 for +ve temperature or 1 or -ve temps). The remaining bits 11-8 (=bits 3-0 of the upperByte) are the most significant 4-bits of the 12-bit integer representing the temperature.

So a sensor-specific approach to reading the temp. from an MCP9808 might look like this:

int TEMP_Raw;
float TEMP_degC; 

// spacer comment for blog layout
Wire.beginTransmission(0x18);    // with mcp9808 bus address written in hex
Wire.write(0x05);                             // and the temperature output register
Wire.endTransmission(); 
Wire.requestFrom(0x18, 2); 
byte UpperByte = Wire.read();          // and sometimes the MSB is called the “highByte” 
byte LowerByte = Wire.read();          // sometimes called the “lowByte” 
// spacer comment for blog layout
UpperByte = UpperByte & 0b00011111;  // Mask away the three flag bits
//easier to read when the mask is written in binary instead of hex
// spacer comment for blog layout
//now we use a mask in a slightly different way to check the value of the sign bit:
if ((UpperByte & 0b00010000) == 0b00010000)  {          // if sign bit =1 then temp < 0°C
UpperByte = UpperByte & 0b00001111;                             // mask away the SIGN bit
TEMP_Raw = (((int)UpperByte) << 8) | LowerByte;    // combine the MSB & LSB
TEMP_Raw-= 256;   // convert to negative value: note suggested datasheet calculation has an error!
 }
else  // temp > 0°C  then the sign bit = 0  – so no need to mask it away
 {
TEMP_Raw= (((int)UpperByte) << 8) | LowerByte;
 }
// spacer comment for blog layout
TEMP_degC =TEMP_Raw*0.0625;


Typically a data output register will continue to hold the last sensor reading until it is refilled with the next one. If your sensor takes a long time to generate this new reading (30-250 ms is typical, while some can take up to a second) and you read the registers before the new data is ready, you can end up loading the previous sensor reading by mistake. That’s where status registers come to the rescue.

3) Status registers

These tell you if if a specified type of event has occurred and I think of these registers as a set of YES/NO answers to eight different questions. The most commonly used status register is data ready [usually labeled DRDY] which sets a bit to 1=true when a new sensor reading is available to be read from the related output registers. Another common status register is one that becomes true if a sensor reading has passed some sort of threshold (like a low temperature alert, or a falling/tilt-angle warning).

A function to check the true=1/false=0 state of a single DRDY bit inside an 8-bit status register might look like this: 

bool i2c_getRegisterBit (uint8_t  deviceAddress, uint8_t  registerAddress, uint8_t  bitPosition) {     
byte registerByte;
registerByte = i2c_readRegisterByte(deviceAddress, registerAddress);
 return ((registerByte >> bitPosition) & 0b00000001);  // or use (bitRead(registerByte, bitPosition))
 }
// delete this comment – it was only needed to maintain blog layout
 //  You could use i2c_getRegisterBit to check the DRDY status bit with a do-while loop
//  and only move on to reading the sensor’s data output registers after the DRDY bit changes to 1
// delete this comment – it was only needed to maintain blog layout
bool dataReady=0;
do {
dataReady= i2c_getRegisterBit(deviceAddress, statusRegAddress, DRDYbitPosition);  
} while ( dataReady==0 );        // processor gets cycled back through this loop until DRDY=1


Holding the processor captive in a status-bit-reading loop is very easy to do, but it is usually avoided unless you are trying to capture a series of sensor readings quickly.  Most status register bits can be mapped to physical alarm output lines on the sensor module, and these can be used to trigger a hardware interrupt pin (D2 & D3) on the Arduino.  This lets you to setup an interrupt service routine (ISR) which grabs that new reading even faster than a bit reading loop. And since hardware interrupts can be used wake a sleeping processor, the interrupt method also lets you put your data logger to sleep until something actually happens. 

The only drawback to the ISR method is that the sequence of register settings you need to follow to create hardware alarms is another multi-step process to add to your sensor initialization code.  The conceptual pattern is usually something like:

  1. Disable the sensor’s global interrupt control bit (if there is one)
  2. Enable the sensors triggering function   (eg:  a max. temperature alert)
  3. Load register(s) with the parameter value for that trigger (eg:  52.5°C)
  4. Turn on the status register that listens to that triggering function
  5. Map that status register bit to a hardware output line
  6. Re-enable the global interrupt control bit

This LSM303 combined accelerometer / magnetometer sensor has two alarm outputs in addition to DRDY. So you could map the Accelerometers DRDY signal to int1, and the Magnetometers DRDY to DRDY.  Just to make life interesting with this sensor, the 3-axis output data  registers are arranged in a different order  on the magnetometer than  they are on the accleerometer. This is typical for multi-sensor chips, which you handle like separate sensors even if they come in the same package – you can even put one to sleep mode while the other one is taking a reading.

Sensors can have many different status monitoring functions, but they usually have only one or two hardware alarm lines.  So the status register -> hardware output mapping (step 5) listed above sometimes involves its own sequence of register settings.  As example, the ADXL345 reads acceleration on three axes, and it has double-tap detection functions for each x,y,z direction. But the Arduino only has two incoming hardware interrupt lines. So generally speaking, you would map all three of those tap-detect status registers to the same outgoing alarm line on the sensor module, and then have the program figure out which axis actually triggered the alarm by reading the status registers later on. High & Low temperature sensor alerts are often mapped in a similar fashion because many breakouts only have one outgoing line: especially if the DRDY status register has been permanently connected to the only other physical alarm line.

A conceptual twist here is that most of the time, the hardware output actually moves the line LOW when the alarm is triggered, even if the status bit it’s mapped from is true=1=high when the actual event occurs. No matter what the status bit->alarm pattern is, any of the four possible interrupt triggers: HIGH, LOW, RISING & FALLING can be used to wake a sleeping 328p processor (though the datasheet states differently).  

Another thing to watch out for on the Arduino side is setting your ISR to respond to HIGH/LOW levels rather than RISING/FALLING edges: Level based interrupts will keep triggering as long as that line is HIGH/LOW. This could cause a sketch to run extremely slowly until the interrupt handler is disabled in your program.  The thing that makes this choice somewhat tricky is that the most common type of sensor failure I see is one where the alarm stays on permanently.  If you set your interrupt to respond to LOW,  and the sensors starts self-triggering your event counters get pushed up to ridiculously large numbers – so it’s very easy to spot that failure in the data, and by the fact that the logger is usually kept awake till the batteries run dry.  If your ISR responds to FALLING, your counts go to almost zero in the same situation, and depending on the phenomenon you are recording it could be very easy to miss that a sensor problem has developed.  Even old analog reed-switched based sensors can suffer from this type of issue, as its not uncommon for something like a wind sensor to stop spinning right where the magnet is holding the reed-switch closed. 

For more information, there’s an excellent guide to interrupt handling over at the Gammon Forum. Probably the most important thing to keep in mind about using interrupts is that by default all interrupts are disabled once you are inside an interrupt subroutine so that the ISR can’t interrupt itself and create an infinite-recursion situation that over-runs the memory.  But the I2C bus relies on interrupts to function, along with timers and other important things.  So don’t try to change a sensor register while inside the ISR,  just set a volatile flag variable and deal with resetting registers later in the main loop.  The general rule of thumb is: “get in & get out ” as fast as possible, and I rarely have a sensor triggered ISR longer than this:

void  INT1pinD3_triggered()  {   INT1_Flag = true;   }

though sometimes I’ll also detachInterrupt(interrupt#) inside the ISR, to make sure it only fires once for things like button de-bouncing. 

Status registers are usually latched, and have to be reset by the I2C master after they are triggered. DRDY registers are cleared by reading information from the data registers they are associated with.  Most other status registers are cleared by reading the register’s memory location, which also turns off the hardware alarm signals that were mapped from them.  This is different from control registers which always have to be explicitly over-written to with new information to change them. If you are waking up a sleeping data logger based on something like a high temperature alert, you usually read the status registers to clear those alarms before enabling interrupts and putting your logger into a power-down state. Threshold based alarms allow interesting things like burst logging.

In Summary:

A good register map, and the four generic functions I’ve described here

  1. i2c_readRegisterByte
  2. i2c_writeRegisterByte
  3. i2c_setRegisterBit
  4. i2c_getRegisterBit

Should be enough to get a typical I2C sensor running, and you can easily tweak those functions to make custom versions for reading 16-bit registers and/or to mask the cruft out of data pulled from mixed registers.

After testing an I2C sensor combination, I pot them in epoxy. Detailed instructions here.

Initializing an I2C sensor is a multi-step process and the correct order of operations is often poorly explained in the data sheet because they are usually written “in reverse”.  Instead of a straightforward list saying “To get a reading from this sensor, do (1),(2),(3),(4), etc.” you find descriptions of the control register bits saying “before you set bit x in this register you must set bit y in this other control register”. When you look up that other control register you find that it too contains a sentence at the end saying “before you set bit y in this control register you must set bit z in this other control register”. So you have to work your way through the document, tracing all those links back until you find the things you were supposed to do first.  Finding the “prime control bit” can be such a time consuming process that it’s not unusual for people who figure out the sequence to wrap it all up into a sensor library so they never have to look at that damn datasheet ever again.

But if you use those libraries, keep in mind that they are probably going to configure your sensor to run at the highest possible bit-depth & data rate, unnecessarily burning away power in applications like data logging which might only need one reading every fifteen minutes.  So the majority of off-the-shelf sensor libraries should be seen as partial solutions, and you don’t really know what else your sensor is capable of until you read through the datasheet yourself.  As an example there are IMU’s out there that will do Euler angle calculations if you simply turn on those functions with the right control register. But libraries for those chips sometimes enable the bare minimum data output functionality, and then do computational handstands to accomplish those gnarly (long) calculations on the Arduino’s modest µC.

In addition there can be useful sensor functions hidden in plain sight, because the datasheet tells you how to turn them on & off, but gives you no clue when to do so. An example here would be humidity sensors like the HTU21D which has an on-chip heating element to help the sensor recover from long periods of condensation, but no status alert that would let you do this automatically. You could just run the heater once a day, but there is also no indication how long the sensor would last if you did that – just some vague references to “functionality diagnosis”. But then some manufacturers (Freescale and Sensirion come to mind…) commit more than just sins-of-omission, breaking away with non-standard I2C bus implementations to lock in customers. The logic there is that if you have to buy the one great sensor that only they make, it’s easier to buy the other four sensors for your device from them as well, rather than juggling low-level protocol conflicts. 

Another challenge when you are working with a new sensor is that Arduino’s C++ environment is not the same as vanilla C in some important ways. So many of the tutorials you find will describe methods that won’t work on an Arduino. Even when the code does compile, there are a number of different “styles” that are functionally identical when they pop out the other side of the compiler, so I’m still trying to wrap my head around the syntax that turns arrays into pointers when they get passed into functionsThat’s why I didn’t mention I2C eeproms in a post about memory registers: almost every multi-byte read/write example out there for EEprom’s uses array/pointer/reference thingies. If you absolutely have to read a series of sensor output registers into an array with a loop, my advice is to just make it a global until you really know what you are doing. And don’t try to store numbers in a char array, because the “temporary promotion” of int8_t’s to 16-bit during some operations can bung up the calculations.

But now it’s time to bring this thing to a close. While I’m still thinking about stuff I wish I’d known earlier, it occurs that a good follow-on to this post would be one about techniques for post-processing sensor data.  There are plenty of useful methods like Paul Badger’s digital smooth, and other code tricks like wrapping those functions in #ifdef #endif statements so those routines only get compiled when a sensor that actually needs them is connected to your logger.

That will have to wait for another day so for now I’ll just sign off with some links. Except for that last ranty bit, I’ve tried to stay out of the I2C handshaking weeds, because when you are up to your neck in bit banging, it’s easy to forget you were trying to measure the water level in a swamp.  But if that’s your thing, there’s some more advanced I2C code examples over at the Gammon Forum, an in depth reference to the Wire library at the Arduino playground , and some troubleshooting tips over at Hackaday

Addendum 2017-11-04

I wonder how many other sensors I could use this with? And if my pin-toggled oversampling method works on the ATtiny, this might provide better resolution than some commercial sensors; though I guess that would depend on how much data I could squeeze into 512 bytes of SRAM…

Somehow I always seem to run into a bunch of related material the day after I post something to this blog: There’s a cool little project over at Quad Me Up using ATtiny85 to turn an analog light sensor into an I2C slave device.  AN4418 from Maxim explains how to use I/O extenders to connect a compact-flash (CF) cards to the I2C interface, which is something I never thought I’d see. And then theres AN10658 from NXP with a method for sending I2C-bus signals over 100m. My own tests with the I2C sensors just hanging off the Arduino only reached about 20m.

Addendum 2017-11-05

Koepel over at the Arduino forum pointed out that the IDE supports some handy macros like bitSet(), bitClear(), and bitRead() that could replace the bit math & masking functions I described here. There’s also word(h , l) to combine two bytes, or highByte() and lowByte() to divide 16-bit variables into 8-bit two pieces. Those were new to me, so I thought I should list them here in case people run into sensor scripts using them.

I’ve also just found out that there are a small number of sensors there that require a ‘false’ modifier to be used at the end of an I2C transaction:  Wire.endTransmission(false);   This is called a repeated start, and the I2C master does not release the bus between writing the register-address and reading data with Wire.requestFrom();   The sensor responds to the I2C address with an acknowledge at the begin of the I2C transaction, and to each databyte that is written to the sensor, so the error code returned by endTransmission can still be used because it is a test if the I2C address was acknowledged by the sensor.

And there was another I2C quirk mentioned at the Gammon Forum:

“You can’t rely on the slave necessarily sending the number of bytes you requested. If you request 10 bytes, Wire.requestFrom() will return 10, even if only 5 have been sent. For the slave to terminate the communication early (ie. after sending less than the requested number of bytes) it would have to be able to raise a “stop condition”. Only the master can do that, as the master controls the I2C clock. Thus, if the slave stops sending before the requested number of bytes have been sent, the pull-up resistors pull SDA high, and the master simply receives one or more 0xFF bytes as the response…It does not help to use Wire.available() because that too will return the number of bytes requested.”

Addendum 2017-11-08

On my page about the DS3231 rtc I describe how to power that I2C chip from a digital pin during bus communication. That trick only works because the chip was designed to gracefully fail over to a backup coin-cell power supply. With other I2C sensors a leakage current might flow into the sensor through the pullup resistors, so you would have to power the bus pullups with the same digital pin to avoid this. And since the internal pullup resistors are enabled by default in the Wire library, you have to disable I2C you could pin power that I2C device.  Also don’t try to de-power a whole module with decoupling capacitors through a digital output pin as that creates big current spikes and really needs proper switching with a PNP transistor of p-channel FET.  99.99% of the time its better to simply find a sensor with a really low sleep current sleep state that you can enter by setting a control register. The best sensors are ones that automatically go into these low current standby-states whenever they detect no traffic on the I2C bus: then you don’t have to do anything.

Another thing I discovered while working with that RTC was that it had a Wire.begin() call hidden in the library, but I was already starting the I2C bus normally during setup. So without knowing it the I2C hardware was being initialized a second time. As the I2C peripheral registers are set to the same value as in the first Wire.begin() call nothing bad happened. However I can see where it might get’s problematic if you call Wire.begin() accidentally because it was buried inside some sensor library while you were running a data transfer,  and the hardware is re-set to an idle state. 

Addendum 2017-11-09

Most of us are familiar with trying out different libraries to drive a sensor, but I’d be remiss if I didn’t mention that there are also some alternatives to Wire library for I2C. The one that gets the best reviews is the I2c Master Library developed by Wayne Truchsess at DSSCircuits. This lib has a faster throughput and a significantly smaller code size: the original Wire library adds about 796 bytes to your sketch when included whereas Wayne’s I2C library only adds 140 bytes. And it has built-in commands that replicate all of the functions I described in this post. For 16 bit registers Wayne points out:

“Technically when sending bytes to a slave device there is no difference between data and an address. In other words let’s say you have a three byte address and three bytes of data. You could use the write(address, regaddress, *data) by making the first byte of your multibyte address equal to regaddress and then combine the rest of the address and data together into *data.”

and that’s equally true with the wire library. The memory savings alone would be worth exploring, but perhaps I2C Master library’s most compelling feature is a ‘TimeOut’ parameter for all bus coms, which could keep your logger from getting stuck in a while-loop if one of your sensors goes AWOL, though I wonder if it still has the 0xFF problem mentioned above, if the sensor sends less than you requested?

Addendum 2017-11-10

I thought using an ATtiny to convert an analog sensor into an I2C device was a neat trick. But it seems that Andreas Spiess has taken the idea to a new level with three HC-SR04’saccessible through on a single AT.  His youTube video #42 with three Ultrasonic Sensors for Arduino walks through the process, with a vocal track that leaves you thinking Werner Herzog has broken into maker videos…

Addendum 2017-11-13

The IDE compiler has an annoying quirk when it runs into Wire.requestFrom in those I2C register routines because the compiler throws up warning messages whenever it feels it has to resolve an ‘ambiguous’ call:  (click to enlarge)

 Turns out that requestFrom has two different implementations, one that can take int arguments, and one that takes uint8_t arguments. If you put in something which has no type like a number (or something you declared with a #define) the compiler has to decide which implementation to use. In the case shown above it chose to use the (int,int) flavor even though device address was specified as uint_8 at the start of the function. 

Anyway, to make those warnings disappear, simply cast the two parameters in Wire.requestFrom to either (uint_8) or (int):

And all those compiler warnings will disappear.

 

 

 

Measuring Electrical Conductivity with Arduino: An Overview

This post is a summary of my background research into electrical conductivity (EC) to serve as a backdrop for my own humble attempts at this interesting measurement challenge. I’m sure there are many other approaches that I’ve yet to discover, and if you know of one please leave a comment so that we can pass that knowledge on to others – Ed.

Obligatory blog-post monkey shot.

Pete & Trish doing profiles with a YSI EXO.

The conduction of current through a water solution is primarily dependent on the concentration of dissolved ionic substances such as salt. Since most fresh water derives from relatively clean rainfall, variations in EC provide a way to track the chemical and hydrological processes the water has been subjected to over time. High amounts of dissolved substances (usually referred to as salinity) can prevent the use of waters for irrigation and drinking, so one could argue that conductivity ranks as the most important inorganic water quality parameter.

A huge number of resources are dedicated to measuring EC and rather than re-hashing all that material, I thought I’d start with links to a few good background reads:

Conductivity, Salinity & Total Dissolved Solids
-discusses the older TDS measurements in part per million (ppm) which makes assumptions about the charge carriers that don’t reflect real world environments.  The conversion factor from EC (which is the thing you actually measure) to TDS changes for different dissolved solids, so instruments from different manufacturers often give you different TDS readings for the same solution, because the companies made different assumptions about what’s in your water.  Because of this confusion, straight EC measurements in siemens have been adopted as the standard by the international scientific community. One siemens is equal to the reciprocal of one ohm (S = 1/Ω)  and is also sometimes also referred to as the mho (℧) in older literature.

Conductivity Theory & Practice
-a white paper that covers basic probe designs, and mentions some non intuitive things like geometry/field effect errors.

Conductivity Sensing at PublicLab.org
-many groups at PublicLab.com have been working on different types of conductivity sensors and their overview page is another excellent introduction to DIY approaches. In fact it’s so good that I will be referring to several of those projects in this post.

Aqueous conductivity is commonly expressed in millisiemens/cm (mS/cm) and natural waters range from 0.05-1.5 mS/cm for freshwater lakes & streams up to about 55mS/cm for sea water. Water up to 3 mS/cm can be consumed, though most drinking/tap water is below 0.8 mS/cm.  Many of the Cave Pearl Project’s installations are in coastal areas where tidally driven haloclines require our instruments to cover that entire “natural waters” range.  Groundwater can vary even more, with measurements being complicated by organic acids and/or significant amounts of dissolved limestone.  Salt water is chemically aggressive and water hydrolyzes above 0.4v, so the probes for high-conductivity environments are usually made of resistant materials such as platinum, titanium, gold-plated nickel or graphite, making them somewhat expensive.

Ways to Measure Conductivity:

There are so many different approaches to measuring EC that it’s taken me a while to digest it all into some working categories.  I expect to build at least one prototype for each of these methods just to see if I can make it work.

Density Based Methods

Refractometers and density based hydrometers are used by aquarium hobbyists. Better quality acoustic doppler flow sensors can also calculate density based on the speed of sound through the water and infer salinity from that. Given then number of acoustic anemometer projects out there, I’m surprised someone has not already adapted the method for underwater applications, though this may be due to the timing limits of the affordable transducers.

Resistance Based methods:

a) Use submerged probes as part of a resistor divider / bridge :
This common approach measures the resistance between two probes using some type of voltage divider. Resistance =1/conductance, which allows you to derive conductivity with your cell K constant since conductivity=(conductance * length)/(area).  AC oscillators are tacked on to reduce electrode polarization, and this forces you to add even more electronics on the output side to convert the signal back to DC for reading. The resistance between the probes changes by several orders of magnitude in environmental waters so different probe surface areas & divider resistors are usually required to cover a significant conductivity range. Above 50% sea water, the resistance between the probes doesn’t change very much, so this method tends to get used more frequently for fresh water environments.

b) Change the pulse frequency of a 555 timer:
You can use resistance between the electrodes as part of an RC relaxation oscillator and then measure the 555’s square wave output frequency to determine the resistance.

This circuit from Thomas Allen’s site provides isolation, uses AC measurement, and the output is a frequency varies from about 42 Hz with the probes in air to > 8000 Hz depending on conductivity.

Circuits and instructions can also be found at at PublicLab.org and there are many good tutorial videos describing 555 based EC sensors on YouTube.  At this point I’ve run into so many projects using this chip that I’d  be willing to bet every environmental sensor I’ve ever heard of could be cobbled together from a few op-amps and a low voltage 555 timer. There are several frequency counting libraries available to help you get started, and if you are ready to sink your teeth into some code, Nick Gammon has produced the some elegant solutions for pulse/frequency timing. Note there are some duty cycle issues

c) Time the discharge of a capacitor through the solution:
Jim Conner’s describes this method in his youTube video at
EC Probes – How they work, and how to build one


A circuit like this might be easy to implement on an Arduino if you can put the internal 1.1v reference onto the comparator that’s also built-in to the 328.  Microcontrollers count time with far better resolution than you get from their ADC’s, but that doesn’t mean there aren’t other issues to deal with. Given that you can try this method with practically no extra circuitry, I will definitely be prototyping a few of these.  Like the 555 based circuits, it will be interesting to see if the method bumps into timing & interrupt handling limits (100 kHz?) when you use it with seawater.

Capacitance based approaches:

You often see capacitance used for liquid level sensors and soil moisture probes, and some of these could be adapted for EC.  To me, the raindrop detection pcb’s you see on eBay have always looked like prime candidates for re-purposing as capacitive sensors.

The circuit described for the Chirp Moisture Sensor uses a fixed resistor and a non-contact probe to create a low pass filter whose cutoff frequency changes with capacitance,  which is affected by the electrolytes in solution. This filters an 1-8MHz square wave and the output voltage is accumulated other side of simple diode peak detector circuit for reading. 

You can also vary 555 timer output frequency by changing the capacitor in the tank circuit, or create more complicated oscillator circuits. No matter which cap-based method you use, the supporting electronics have to be located near the sensor – because just about any length of wire will add enough stray capacitance to throw off your measurements. Another thing to keep in mind is that common ceramic capacitors have some of the worst thermal coefficients and aging effects imaginable. Plastic film capacitors using Polyphenylene Sulfide (PPS ±1.5%) or Polypropylene (CBB or PP ±2.5%) are much better for sensor circuits like this, and having a digital capacitance meter on hand is probably a good idea too.

DIY capacitor plate sensors are going to be small values (picofarads ?) and the resulting RC time constants make these methods more suitable for fresh waters, as this leverages the relatively large solution resistance to give you more accurate interval timing.

Potentiometric (4 electrode) Methods

Four-electrode cells uses two “driver” pins to place an electric field across two other “reading” pins that lie between them: 

This paper describes a DIY 4-probe sensor that was used for soil moisture sensing, and you will find quite a few articles using potentiometric methods over at IEEE and Sensors

Nokia/Apple audio jacks came to mind as soon as I saw this diagram, and they might be available with gold plating.  4-electrode methods often measure the voltage between the read pins, which is divided by the exciter pin current to determine the solution’s impedance = 1/conductance.  To obtain the conductivity, the conductance is multiplied by the cell constant of the inner poles. Tracking the pin current lets you compensate for fouling on the plates, and the method can cover a wide range of conductivity. Like inductive methods, this approach tends to work better as the concentration increases. 

Inductive Methods

The conductivity measurement is made by passing an AC current through a toroidal drive coil, which induces a current in the solution. This induced solution current, in turn, induces a current in a second coil, called the pick-up toroid. The amount of current induced in the pick-up toroid is proportional to the solution conductivity. You get industrial grade performance out of this non-contact method in many different types of solutions, but you also need industrial amounts of power the drive the sender coil so it’s hard to implement on the kind of power constraints you see on stand-alone data loggers.  Inductive sensors require a 3 inch radius from any other surface (bio-fouling?) and you see this pretty clearly in the ‘donut on a stick’ sensor heads. It occurs to me that you see very similar components in a wireless charging system, but there’s a lot of devils hiding in those details – like shielding, etc.  It might be possible to press one of the production line proximity sensor chips into service for a low power solution, or simply try measuring changes in inductance due to the presence of salt water.

Off-the-shelf Solutions for Arduino:   (using 2-Electrode Resistance Methods)

TransparentSinglePixl
Atlas Scientific Conductivity Kit
A complete solution including calibration solutions, a range of probes and code libraries. All parts also sold separately: interface boards are ~$35 & EC probes come in around $120 each but they are durable enough for continuous long term submersion.  I2C data transfer is supported, so resolution is not limited to the Arduino’s ADC.  Whitebox labs Tentacle Shields ($35-$110) provide up to four galvanically isolated channels for full hydroponic rigs. Stand alone BNC carriers for $10.
$200
Gravity: EC Sensor Kit for Arduino (K=1)
Another complete K=1 kit solution, but the probes are not robust enough for long term submersion so several people replace the stock probe with the 208DH which is available on eBay for $35. Arduino ADC reads voltage.  The KnowFlow project uses the full set of DFrobot boards.
$70
Vernier CON-BTA EC probe
This 5v K=1 probe covers 0-20,000 μS/cm  in the high range, and the analog voltage output is read by the Arduino ADC. You need an inexpensive adapter board for the BTC connector, and they provide a basic library. One key feature is built in hardware temperature compensation with a 10k thermistor in the probe head. My tests show this reduces the usual 2%/° C reading variation down to about 0.5%/° C, so you still need to do your own calibration to get high accuracy. Like Atlas Scientific, Vernier has many other interesting sensors that are Arduino compatible.
$115
EC/pH Transmitters
This company offers a range of physically bulky turn-key solutions, with the $70 entry level unit claiming 0-5000 μS/cm (fresh waters) and continuous monitoring. Arduino ADC reads voltage. ~$200 units support PH with isolation.
$70-250
Sparky’s widgets MiniEC
An indie who makes several other useful sensor breakout boards, including PH. You have to build or locate your own probes, though they use a standard BNC connector like most EC probes.  Arduino ADC reads voltage output.
$24
EC-Salinity Probe Interface by Ufire
Designed around an ATtiny configured as an I2C slave, probably using the cap-discharge method.
$14.50
Hanna HI 73311 (K=1) Replacement probes
In the past we’ve used used these epoxy&graphite probes from Hanna DIST5 (HI 98311) and DIST6 (HI 98312) testers, which connect to a standard male audio jack.  You can also re-purpose one of the Vernier ABS/graphite probes if you get a used one cheap on eBay, and the Vernier probes have a 10k NTC thermistor built in, which you can read with a divider.
$55
Comercial Standard Solutions
For fieldwork, it’s often easier to transport the dry packets, and mix them on location.  Atlas sells calibration sets, but at the twice the cost of standards when you buy them in larger volumes.  You can find recipes for homemade calibration solutions at Reefnet Central and PublicLab. For a classroom situation, it’s much cheaper to mix secondary “lab standards” in larger quantities, and then test the resulting solutions with a commercial probe that’s been calibrated against commercial solutions. 5.566g of dry NaCl in 1 litre of distilled water will create an ~10,000 μS/cm solution, which you can dilute down for lower concentration standards.
$14/500ml

This photo from Bitnitting gives you a sense of the physical space needed for the Atlas breakouts and a ‘mini’ form factor Arduino.

Hydroponics hobbyists have putting these kits to good use over the years with notable examples like the long running forum thread on Billies Hydroponic Controller, and the well documented adventures over at the Bitnitting Blog.  The people at OpenCTD and other academic projects have put the Atlas boards into real world deployments.

But to me these commercial solutions still leave you stuck with those expensive electrodes which sometimes cost more money than you would pay for a used 4-pole device. More annoying is the fact that those cell constants do not line up with my goal of measuring the entire “fresh” to “marine” range with one sensor, thought if I could extend it a bit the K=10 probe comes close.  This is illustrated by the following graph from Andy Connelly’s Blog which is worth digging through as he has posted lots of other interesting material on calibration, reproducibility, signal detection, etc. 

 

Of course the cell constant changes as your probes get older and dirtier, so you have to re-calibrate them with standard solutions just about every time you want to take a new reading. I’m pretty sure I will end up making my own probes, probably out of Nichrome 80 wire as the vaping fad has made it common on eBay. Some have had good EC results with gold plated PCB traces. Feedback on the Arduino.cc forum suggests that Platinum-Rhodium Thermocouple Wire is another good option.  I’ve also been wondering about Ag/AgCl which is highly resistant to seawater and is commonly used for non-polarizing electrodes in medical/bio applications. (EKG electrodes?) It might also be a good idea to cobble together a DIY magnetic stirrer, based a PC fan and an old hard drive magnet

DIY 2-Probe EC Circuits

The easiest circuits to build yourself are the 555 timer oscillators, but there are plenty of quad-opamp solutions out there for people comfortable with a breadboard. The oldest example I’ve seen is this one by M. Ahmon from the Sept 1977 issue of Electronics magazine which uses the resistance of the solution to modify opamp output:

This circuit uses the first stage of the quad opamp in a Wien-bridge oscillator, reducing errors caused by electrolysis with a 1-kHz signal that gets attenuated by the solution’s resistance before it reaches the driving amplifier A2.  Pot P1 controls oscillator amplitude, and P2 adjusts gain of A2.  A3-A4 form a precision rectifier giving output voltage equal to absolute value of input voltage. This one chip solution seems to have been the basis for many of the current EC projects on the web, including these two exceptionally well documented examples:

Octavia’s EC/TDS/PPM Meter On Limited Budget

Daniel Kramnik’s Digital Salinometer Project

Similar circuits can be found on the breakout modules from Sparky’s Widgets and DFrobot . Using the solution’s resistance in the feedback divider controlling an op-amp is a neat idea, but having only one opamp there imposes hard limits the range you can measure with a given K value probe. There is a more advanced multi-opamp approach over at pulsar.be that can step over several decades.

On more recent EC projects I’m seeing single supply RRIO opamps for the oscillator & gain stages, which are easier to integrate with battery operated Arduino’s. (though any dual supply opamp can be used as a single supply in a pinch; since voltage is relative the opamp doesn’t know whether V- is a negative voltage or ground) To keep using an AC signal, this requires a virtual GND at 1/2 VCC, but the integration also gives you the option of getting rid of the oscillator entirely, since you can use PWM output as your source.

This is beautifully illustrated by the circuit from bhickman’s Conductivity & Temperature Meter over at PublicLab:

Ranging is accomplished with the (red) bank of R1 resistors, and (yellow)R2’s 5/6 can be substituted in for the probe (R8) with those known resistances can be used to track drift. The AC–DC converter stage is built with precision peak detectors. I think this is the best voltage divider approach I’ve seen to date.  To simplify things a bit, you might replace that output stage with an RMS-DC converter; though I’ve not seen any breakouts for those, and I hate working with raw SMD parts.

Sources of Error: 

Even with a clever circuit like the one above you still need to address things like temperature compensation before you get an accurate, repeatable, and stable device. Electrical conductivity measurements are typically referenced to 25 °C using standard temperature compensation factors (α). The conductivity of natural waters exhibit strongly nonlinear temperature behavior, though in practice linear correction factors are most frequently used.  NaCl-based solutions typically have a temperature coefficient (α) of 0.02-0.0214 (~ 2% change/degree C). So to convert your “ambient” conductivity measurement into 25°C “specific” conductivity, the simple linear conversion is:

EC25=ECambient /[ 1 + α (tambient – 25) ], α= 0.02

Field effect errors are significant, causing read errors if bare 2-pole electrode get within 2-5 cm of the solution container: which will completely mess up your calibration and cell constant determination. This is one reason that virtually every EC probe is encased inside a plastic shroud of some sort. That causes field effect errors too, but at least its the same error every the time, rather than one that varies depending on how far you are from the edge of the beaker. Four probe methods also require a fixed volume of solution between the driver electrodes, so the shroud provides that.

Grazing through the hydroponics forums shows plenty of people struggling with cross-sensor interference.  Most notably when conductivity probes affects the accuracy of a PH probe in the same tank.  Any time two devices are immersed in the same environment differences between them can generate ground loop voltages and induce currents which degrade the readings and exacerbate corrosion.  Sometimes you can address these issues with optical or I2C isolators. One helpful contributor at Arduino.cc suggests:

pH electrodes are very high impedance devices and the cabling and connectors are all important – even flexing a decent cable will distort the readings…. Ground loops are the enemy of pH and any other specific ion electrode. I used them a lot in difficult situations and the most trouble-free solution is always to put a buffer op amp (FET type) as close to the electrode as possible – some commercial electrodes come already equipped. Find a decent op amp like the old MAX406, high impedance techniques like PTFE insulators or simply keep the input pin off the board. Modern FET’s take single-sided supplies and run at better than 2-microamps – a 3.6-V lithium cell will give you in excess of 5-year’s trouble and ground loop -free operation. Once you have buffered the signal, you can use any cable you like. As a bonus, you can convert a pH electrode into an ammonia electrode by separating the water from the electrode with PTFE tape as used by plumbers.”

Well, I think that covers most of the stuff I had in my notes, and hopefully gathering it all here saves someone else from burning away that time. I have been experimenting with conductivity quite a bit lately, and I think I might have  come up with an analog approach that will allow people to play with conductivity on shoestring budgets. I just have a little more calibration testing to do before I let that one out of the bag  🙂

Calibrating Thermistors on a 3.3v Arduino

Selecting a thermistor (& series resistor) value

Most of the material you find on thermistors makes the assumption that you are trying to maximize sensitivity and interchangeability. But oversampling gives you access to enough resolution that sensitivity is less critical, and interchangeability only makes sense if you are putting them in a product with good voltage regulation. In that case, precision thermistors like the ones from US sensor are a good option, but according to Campbell Scientific, that choice has other knock-on implications:

“The resistors must be either bought or selected to 0.02% tolerance and must also have a low temperature coefficient, i.e. 10 ppm or preferably 5 ppm/°C.”

Like many better quality components, these resistors are often only available in SMD format, with minimum order quantities in the thousands. If you use a typical 1% resistor with a T.C. of 50 ppm or more, you could introduce errors of ±0.1°C over a 50°C range, which defeats the point of buying good thermistors in the first place.

Still, if I was only building a few sensors, I’d spring for the good ones. But now that I have oversampling working on the Arduino, I’d like to add a thermistor to every logger in the field, and the mix of different boards already in service means I’ll have to calibrate each sensor/board combination. That time investment is the same whether I choose a 10¢ thermistor or $10 one.

Power consumption is also important, making 100kΩ sensors attractive although I couldn’t even find a vendor selling interchangeable thermistors above 50k.  A low temperature limit of 0°C (the units are underwater…) and putting 1.1v on aref to boost sensitivity,  requires a 688k series resistor, which is far from the 1-3x nominal usually recommended:

Here I’ve overlaid an image from Jason Sachs excellent thermistor article at Embedded Related, which shows I will only see about ⅓ of the sensitivity I would get if I was using a 100k series resistor. I highly recommend reading Jason’s post, despite the fact that I’m ignoring almost all of his good advice here…  🙂

Using the internal band-gap voltage as aref improves the ADC’s hardware resolution from 3.22mV/bit to 1.07mV/bit.  This trick gives you a extra bit of precision when you use it at the default 10bit resolution, and I figured I could do it again to compensate for the lost sensitivity due to that big series resistor.

In return, I get a combined resistance of at least 700k, which pulls only 4.7μA on a 3.3v system.  Such low current means I could ignore voltage drops inside the processor and power the divider with one of Arduino’s digital pins.  In practical terms, burning less than a milliamp-second per day means adding a thermistor won’t hurt the power budget if I leave it connected to the rails all the time; which you can only do when self-heating isn’t a factor.  This is quite handy for the bunch of old loggers already in service out there, that I want to retrofit with decent temperature sensors. 

Even 100 ohms of internal chip resistance would produce only 0.5mV drop,  so depending on your accuracy spec,  you could use 16-channel muxes to read up to 48 thermistors without worrying about cable length.  There aren’t many of us trying to connect that many temperature sensors to one Arduino, but using a 100k  thermistor also makes me wonder if you could mux a bank of different series resistor values, pegging the divider output at it’s maximum sensitivity over a very large temperature range.

What is a reasonable accuracy target?

Combining 5¢ thermistors & 1¢ metfilms, means my pre-calibration accuracy will be worse than ±1°C.  Cheap thermistor vendors only provide nominal & βeta numbers, instead of resistance tables, or a proper set of Steinhart-Hart coefficients. So I might be limited to ±0.4°C based on that factor alone.  And it took me a while to discover this, but βeta values are only valid for a specific temperature range, which most vendors don’t bother to provide either.  Even with quality thermistors, testing over a different temperature range would give you different βeta values.

In that context, I’d be happy to approach ±0.1°C without using an expensive reference thermometer.  Unfortunately, temperature sensors in the hobby market rarely make it to ±0.25°C.  One notable exception is the Silicon Labs Si7051, which delivers 14-bit resolution of 0.01°C at ±0.1°C.   So I bought five, put them through a series of tests,  and was pleasantly surprised to see the group hold within ±0.05°C of each other: 

Temps in °CCompared to what I usually see when I batch test temperature sensors, this is pretty impressive for an I2C chip that only cost $9 on Tindie.

Ideally you want your reference to be an order of magnitude better than your calibration target, but given the other issues baked into my parts, that’d be bringing a gun to a knife-fight. 

So my calculations, with oversampling, and the internal 1.1v as aref become:

1) MaxADCReading                  (w scaling factor to compensate for the two voltages)

= ( [2^(OverSampledADCbitDepth)] * (rail voltage/internal aref) ) -1

2) Thermistor Resistance        (w series resistor on high side & thermistor to GND)

= Series Resistor Value / [(MaxADCReading / OverSampledADCreading)-1]

3) Temp(°C)                                  (ie: the βeta equation laid out in Excel)

=1/([ln(ThermResistance/Tnominal R)/βeta]+ [1.0 / (NomTemp + 273.15)]) -273.15

Seeing the error in my ways

I knew that the dithering noise would have some effect on the readings, and all the other source ADC of error still apply.  Switching to 1.1v reduces the absolute size of most ADC errors, since they are proportional to the full scale voltage. But the internal reference is spec’d at ±0.1v; changing the initial (rail voltage/aref voltage) scale factor by almost 10%.  Since all I needed was the ratio, rather than the actual voltages, I thought I could address this chip-to-chip variability with the code from Retrolefty & Coding Badly at the Arduino.cc forum.  This lets Arduinos read the internal reference voltage using the rail voltage as aref.

I started testing units in the refrigerator to provide a decent range for the calibration:

Si7051 in blue vs 100K thermistor in red. The sensors were held in physical contact. ADC was read with 1024 oversamples providing approximately 15bit resolution. Temps in °C.

and strange artifacts started appearing in the log.  The voltage readings from both the main battery and the RTC backup battery were rising when the units went into the refrigerator, and this didn’t seem to make sense given the effect of temperature on battery chemistry:

Si7051 temp. in °C on the left, with the RTC backup battery (V) in green on the right axis. The CR2023 is monitored through a 2x10MΩ divider, using the 3.3v rail as aref. The large number of ADC readings needed for oversampling has the side benefit that it lets you read very high impedance dividers, but by the time you reach 10Meg ohms, you pick up 5-10 points of noise in the readings. Which is why that coincell voltage line is so thick.

I think what was actually happening was that the output from the regulator on the main board, which provided the  ADC’s reference voltage for the battery readings, was falling  with the temperature.

When I dug into what caused that problem, I discovered that temperature affects bandgap voltages in the opposite direction by as much as 2 mV/°C.  So heating from 0°C to 40°C (and some loggers will see more than that…) reduces the 328P’s internal reference voltage by as much as a tenth of a volt. In fact, bandgap changes like this can be used to measure temperature without other hardware.  This leaves me with a problem so fundamental that even if I calculate S&H constants from a properly constructed resistance table, I’d still be left with substantial accuracy errors over my expected range.  Argh!

Becoming Well Adjusted

These wandering voltages meant I was going to have to use the internal voltmeter trick every time I wanted to read the thermistor.  It was mildly annoying to think about the extra power that would burn, and majorly annoying to realize that I’d be putting ugly 10bit stair-steps all over my nice smooth 15bit data. This made me look at that final temperature calculation again:

Temp(°C) =
1/([ln(ThermResistance/Tnominal R)/βeta]+ [1.0 / (NomTemp + 273.15)]) -273.15

which I interpret as:

 =fixed math(  [(ADC outputs / Therm. nominialR ) / Therm. βeta]  + (a #) ) – (a #)

Perhaps tweaking the thermistor’s nominal value (which I only know to ±5% anyway) and changing the (fictional) βeta values would compensate for a multitude of sins; including those voltage reference errors?  Then I could just pretend that (rail/aref) scaling factor had a fixed value, and be done with it:         (click image to expand)

So in my early tests, all I had to do was adjust those two constants until the thermistor readings fell right on top of the reference line.  Easy-peasy!

Well …almost. Repeat runs at 15bit (1024 samples) and 14bit (256 samples) didn’t quite yield the same numbers.  Applying the best fit Nominal and βeta values obtained from a 15bit run to 14bit data moved the thermistor line down by 0.05°C across the entire range (and vice versa). So the pin toggling method I used to generate the dither noise introduces a consistent offset in the raw ADC readings.  While that doesn’t completely knock me out of my target accuracy, I should generate new calibration for each oversampled bit depth I intend to use. It’s still good to know that the dithering offset error is consistent.

Throwing a Big Hairy Fit

I was pleased with myself for the simplicity of the Nominal/βeta approach for about two days; then I pushed the calibration range over 40° with a hot water bath:

Blue=Si7051 , Orange = 100k NTC thermistor.  1024 oversamples = ~15bit. Temps in °C.

This gave me targets at around 40, 20 and 5°C.  But no combination of Nominal & βeta would bring all three into my accuracy range at the same time.  Fitting to the 20 & 40 degree data pushed the error at 5°C beyond 0.2° :             (click image to enlarge)

…and fitting to 20 & 5, pushed the 40C readings out of whack.  After more tests I concluded that tweaking βeta equation factors won’t get you much more than 20° of tightly calibrated range. 

My beautiful plan was going pear-shaped, and as I started grasping for straws I remembered a comment at the end of that Embedded Related article

“… in most cases the relationship between voltage divider ratio and temperature is not that nonlinear. Depending on the temperature range you care about, you may be able to get away with a 3rd-order polynomial or even a quadratic..”

Perhaps it was time to throw βeta under the bus, and just black-box the whole system?   

To find out, I needed to prune away the negative temperature regions where the voltage divider had flat-lined, and remove the rapid transitions since the thermistor responds to changes more quickly than the si7051:                 (click image to inflate)

Then it was time for the dreaded Excel trend line:

Ok, ok. I can hear people inhaling through their teeth from here. But with 15 sigfigs, Excel seems like the height of luxury compared to the constraints in μC land.  I wonder what an advanced modeler like Eureqa would have produced with that dataset? 

The trick for getting workable constants is to right-click the default equation that Excel gives you, re-format it to display scientific notation, and then increase the number of displayed digits to at least six.  

Some people use the LINEST function to derive these polynomial constants but I’d advise against it because seeing the raw plot gives you a chance to spot problems before you fit the curve. When I generated the first Temp vs ADC graph, the horizontal spread of the data points showed me where the thermistor and the reference thermometer were out of sync, so I removed that data.  If I had generated the constants with =LINEST(Known Y values, X values^{1,2,3,4})  I could have missed that important step.

For the following graphs, I adjusted the trend line to display to nine insignificant digits:     

Blue =Si7051 reference, Orange is that 20&40 best fit from tweaking Nominal & Beta values, and the yellow line is the 4th order polynomial from Excel.   Temps in °C. (Click to embiggen)

It took a 4th order polynomial to bring the whole set within ±0.1° of the reference line and 5th order did not improve that by much.  Now I really have no idea where the bodies are buried!  And unlike the βeta equation, which just squeaks in under the calculation limits of an Arduino, it’s beyond my programming ability to implement these poly calcs on a 328 with high bit depth numbers. I certainly won’t be writing those lunkers on the bottom of each logger with a sharpie, like I could with a pair of nominal/βeta constants.

This empirical fit approach would to work for any type of sensor I read with ADC oversampling, and it’s so easy to do that I’ll use it as a fall back method whenever I’m calibrating new prototypes. In this case though, a little voice in my head keeps warning me that wrapping polynomial duct tape around my problems, instead of simply using the rail voltage for both aref & the divider, crosses some kind of line in the sand. Tipping points can only be predicted when your math is based on fundamental principles, and black-boxes like this tend to fail dramatically when they hit one.  But darn it, I wanted those extra 1.1v aref bits! Perhaps for something as simple as a thermistor, I’ll be able to convince the scientist in the family to look the other way.

Addendum 2017-04-28

Seeing that trend-line produce such a good fit to the temperature data, made me think some more about how I was trying to stuff those system side errors into the βeta equation, which just doesn’t have enough terms to cope.  By comparison, the Steinheart-Heart equation is a polonomial already, so perhaps if I could derive some synthetic S&H constants (since my cheap thermistors didn’t come with any…), it would peg that ADC output to the reference line just as well as Excel did?

I rolled the voltage offsets into the thermistor resistance calculation by setting the (rail voltage/internal aref) scale factor to a fixed value of 3, when in reality it varies from slightly below to slightly above that depending on the board I’m using:

1) MaxADCReading                  (w scaling factor to compensate for the two voltages)

=(2^(OverSampledADCbitDepth) * (3)) –1

2) Thermistor Resistance        (w series resistor on high side & thermistor to GND)

= Series Resistor Value / ((MaxADCReading / OverSampledADCreading)-1)

and I went back to that trimmed 40-20-5 calibration data to re-calculate the resistance values. Then to derive the constants, I put three Si7051 temp. & thermistor resistance pairs into the online calculator at SRS:

(Note: There are premade spreadsheets that you can download which will generate S&H constants, or you can build your own in Excel. There’s also coefficient calculators out there in C, Java, etc. if that’s your thing.)

With those Steinhart-Hart model coefficients in hand, the final calculation becomes:

3) Temp °C =1/( A + (B * LN(ThermR)) + (C * (LN(ThermR))^3)) – 273.15

and when I graphed the S&H (in purple) output against the si7051 (blue) and the 4th order poly (yellow), I was looking at these beauties:

and that fits better than the generic poly;  nearly falling within the noise on those reference readings. With the constants being created from so little data, it’s worth trying a few temp/resistance combinations for the best fit. And this calibration is only valid for that one specific board/sensor/oversampling combination;  but since I’ll be soldering the thermistors permanently into place, that’s ok.  I’m sure if I hunt around, I’ll find a code example that manages to do the S&H calculations safely with long integers on the Arduino. 

So even with cheap parts, oversampling offsets & bandgap reference silliness, I still made it below ±0.2°C over the anticipated temperature range.  Now, where did I put that marker…

Addendum 2017-04-27

Just a quick note to mention that you need to tape the thermistor to the si7051 sensor so they are held in physical contact with one another. The thermistors are tiny & react to temperature changes much faster than the si7051’s which have a much larger thermal mass because of the breakout board they are mounted on. So the temp/resistance pairs don’t match up as well as they could if the sensors are in physical contact with one another.

Addendum 2017-06-05

With 1.1v aref in the mix,  my 15bit oversampled resolution on those 100k thermistors varies between 0.002 and 0.004°  from 20-40°C. But I was throwing the bandgap aref in just to see if I could still make it work. From a calibration point of view, it’s better to better to use the rail voltage on aref, and remove that 3x ratio from the MaxADCReading calculation.  This will lower the resolution to somewhere between 0.006 to 0.012C with a 688k series resistor unless you bump up the oversampling to compensate. In addition to tripling my noise/toggle-pin current, how much extra power do I have to pay to get that resolution back if I’m using the 3.3v rail as aref?

In my oversampling experiments, I found that the Arduino ADC works well at 250 kHz, delivering just under 19230 ADC readings /second. For the purpose of estimation, assume the Pro-mini style boards I’m using draw about 5mA during the sampling time, and I take a reading every 15 minutes (= 96 readings per day) :

15bit= 1024 reads/19230 r/sec =0.053s*5mA =0.26 mAs*96/day=~ 25 mAs/day
16bit= 4096 reads/19230 r/sec = 0.213s*5mA =1.00 mAs*96/day= ~102 mAs/day
17bit= 16384 reads/19230 r/sec = 0.852s*5mA =4.26 mAs*96/day= ~408 mAs/day

so it would cost me another 385 mAs/day to reach a resolution slightly better than I was achieving with the 1.1v bandgap on aref. Given that a typical AA battery runs about 2000 mAh = 2000 mAh*3600 sec/hour =~7,000,000 mAs, it would be quite a while before that breaks the power budget.  Removing the ratio dependency also means that your S&H constants are for the resistor/thermistor pair only, making that calibration independent of what system you connect them to.

Using an Rnominial=100k series resistor would give about the same effective resolution boost as going to 17 bit, but that option costs you more power if you are leaving the thermistor powered all the time:

3.3v / 780k combined resistance  = 4.23μA x 86400 sec/day  = 366 mAs/day
3.3v / 200k combined resistance  = 16.5μA x 86400 sec/day  =  1425 mAs/day

You can power the thermistor from a digital pin, but since I’m already using digital-pin toggling to generate noise for the oversampling, I still need to test if I can combine pin power for the sensor with my oversampling technique. It’s possible that the thermistor bridge needs to be powered by the more stable rails, while I’m shaking aref inside the processor, because if the voltage on the divider started moving in sync with the ADC noise, the dithering noise will effectively disappear, and my oversampling would stop working.

Even before doing this test, I have a sneaking suspicion that 100k series vs. oversampling vs. other techniques  will end up converging on the same effective resolution in the end. And I’ll even hazard a guess that the point of diminishing returns is somewhere around 0.001°C, since that’s what you see offered by quite a few high-end temperature loggers.

Addendum 2017-09-24

Just posting an update about pin-powering the thermsitor dividers while using the 3.3v rail as aref: everything works, but as I suspected you need to stabilize the thermistor with a small 0.1uF capacitor or the dither noise vanishes.  This also requires you to take the RC time constant into account, waiting at least 5x T  for that parallel cap to charge before you start reading the divider. You can sleep the processor during this time, since I/O pin states are preserved.

Degree Celsius vs. Time with lines offset for easier visual comparison:  The blue line is over-sampled output from a pro-mini clone reading a 100k Thermistor /100k series voltage divider. Aref was set to the 3.3v rail, with a 100nF capacitor in parallel with the thermistor on the low side.  This RC combination has a time constant of ~10 milliseconds.  A 0.12 mA pin-current provided sufficient noise for dithering 1024 readings: to deliver an effective resolution of ~0.0028° at 24°C.  For comparison, the red line is the output from an I2C si7051 sensor on the same logger, with a resolution of 0.01°C.

So using a 100k series resistor with 3.3v aref really does deliver the same effective resolution as the 680k series/1.1v aref combination, and it does not suffer the problem of bumping into the aref voltage at a certain temp.  I’m using 100k termistors so the pin resistance (~40 ohms) will introduce less than 0.05% error over the range; though this pin-drop error would be higher for therms with lower Rnominal values.

Since I’m using cheap eBay 100k’s and a host of other no-name components, I have to calibrate each logger/thermistor/O.S. bit-depth combination.  This isn’t much of a burden for the overall workflow, since I always give new loggers a shake-down run, in fact, I usually do a fast sampling burn for at least a week before considering a unit ready for deployment:

That Degree vs Time image above was an excerpt from a calibration run like this. I’ve found that Freezer (morning)->Fridge (afternoon)->Room (overnight) is easier to manage than the reverse order, and gives enough time at each temperature to deal with thermal lag differences between the thermistors and the reference sensors.

As before, when I do the thermistor resistance calculation I make the assumption that everything in the system is behaving perfectly (which is obviously not true). So errors from things like pin drops, temp. coefficients, ADC gain, etc., are getting rolled into the S&H constants.  Essentially, I’m eliminating a host of irritating corrections in exchange for the interchangeability between sensors that I might have if I took all those factors into account. This makes it easier to standardize the code between loggers, and is a reasonable trade-off for loggers that I won’t be seeing again for several years, but if I have to swap some components at that time, I’ll need to do  another calibration.

Addendum 2017-11-05

Looks like Sensirion’s new STS35 has ± 0.1°C accuracy like the si7051 I’m currently using as a calibration reference.  Hopefully that shows up on Tindie soon.

Improving Arduino ADC resolution with Oversampling & Noise

Thermistors are really twitchy, so you need to put them inside a big lump of thermal inertia before you start.

The slightest breeze makes glass bead thermistors jitterbug like crazy, so put them inside something with a decent amount of thermal inertia before you do any oversampling experiments.

While I was figuring out how to read thermistors, I came across claims that you can improve the resolution of any Analog-to-Digital converter with a technique called oversampling & decimation. I had already doubled the number of ADC bits covering my target temperature range by powering a thermistor divider from the rails and using the internal 1.1v as the analog reference.  My gut feeling was that aref based ADC bits were somehow better than any I could synthesize, but I was still curious to see if I could add oversampled bits to the ones obtained by the bandgap aref trick.

At first bounce, the method appeared to be incredibly simple, to get n extra bits of resolution, you need to read the ADC four to the power of n times.  Generally you have to add three extra bits (43= 128 samples) to see approximately an order of magnitude improvement in your real world resolution. With thermistor dividers, you typically get about 0.1°C from the default ADC readings, and 128 samples bumps that to 0.012°C.  Taking (46= 4096) samples would bump that up to ~0.0015°C which, as the saying goes, is good enough for government work… 

I usually over-sample one power more than needed for my target resolution, so I’d use for four extra bits to be sure of that order of magnitude improvement, which requires the sum of 44= 256 readings:

uint32_t extraBits=0;    // use an unsigned integer or the bit shifting goes wonky
for (int k = 0; k< 256; k++) {
extraBits = extraBits +analogRead(AnalogInputPin);
}

which is then decimated by bit shifting right by n positions:

Oversampled ADC reading = (extraBits >> 4);

This combination lets you infer the sub-LSB information provided there is enough random noise in the signal for the lowest ADC bits to toggle up and down while you gather those readings. But without the noise, all of the original ADC readings are the same, and the oversampling technique does not work. It’s a good idea to keep in mind what you are aiming for with your oversampling. Generally you have to take enough samples for three extra bits (43= 128) to see approximately an order of magnitude improvement in your the real world resolution. With thermistor dividers, you typically get about 0.1°C from the default ADC readings, and (43= 128) samples gets you to 0.012°C.  Taking (46= 4096) samples would bump that up to 0.0015°C which, as the saying goes, is good enough for government work… 

To show you what that kind of failure looks like, here is oversampling & decimation being done over 4096 readings with no noise or dither signal applied to a 10k NTC thermistor divider read with 1.1v aref:

This is an example of oversampling with no dither signal being applied. So this is the nul result

These are readings from a 10k NTC thermistor divider, and I’ve offset these records from each other by 0.1° for easier comparison. The one-shot ADC readings of the thermistor bridge in purple are converted to °C, as are 4096 sample readings at the default 125kHz(ps64) in grey,  250kHz(ps32) in orange and 500kHz (ps16) in green. With such a large number of samples, the averaging produces some smoothing whenever the raw ADC readings near a transition point, but if you see “rounded stair steps” like this then oversampling is not working properly  the curves shown above are all FAILURES.

Some microprocessors have enough jitter in their readings to use oversampling technique with the natural toggling of the least significant bit.  A few brave souls have even tried to improve the AVR’s crude internal temperature sensor with the technique.  But most of the time, there is not enough naturally occurring noise, and you need to add a synthetic dithering signal to force those LSB’s to toggle.  This is mentioned from time to time in the forums, with a number of references to AVR121: Enhancing ADC resolution by oversampling, but I found frustratingly few implementations using an Arduino that were described in enough detail for me to replicate them.  Most of the technical docs were focused on audio applications, and I was quickly buried under thick mathematical treatments warning me not to interpret the Effective Number of Bits (ENOB) as Effective Resolution (what?), and describing a host of other quid pro quos like signal synchrony.

This is qwerty's original dither circuit from the freetronics forum post at: http://forum.freetronics.com/viewtopic.php?t=5589#p11126

This is Qwerty’s original dither circuit from the freetronics forum. If you are using an UNO, this works well. Of course the ratio between the 5v rails, and the internal bandgap reference,  means you also have extra ADC resolution available without oversampling if you use the 1.1v aref trick, but oversampling gives you more bits for your effort.

About the only useful thing I got out of most of those refs was the apparent consensus that any synthetic dithering signal needs to be at least 2x the voltage per bit on your ADC (although you can use a larger dither signal without causing problems) and triangular dither signals work better than natural noise.  But few of those references said anything about extending ADC resolution, as they were primarily focused on improving the ADC’s signal to noise ratio.

And then there was the fact that several of the older hands seem to dismiss the whole idea as not worth the bother because you had to add so much additional circuitry that using an external ADC was a simpler, cheaper approach.  In fact the subject triggered the closest thing to a flame war I’ve ever seen at the usually staid Arduino playground.  So I was about ready to give up on the idea when I came across a post by user QWERTY at the Freetronics forum explaining how he used a simple RC filter to turn an Arduino’s 480 Hz PWM output into a 9mv p-p triangular dither, which he patched directly into the center of a thermistor bridge.

Yes it is possible to add a jumper on the Aref line of a pro mini.

You can patch into the aref line on a Promini by soldering a jumper to the end of the little stabilizing capacitor.

Holy cow! A solution that only needed a few cheap parts and couple of pins. What the heck were those other guys gassing on about?   My first thought was to try to take the output from Qwerty’s RC filter, and put it onto the Aref as they did in AVR 121.  A compelling idea since putting the dither directly on aref means you don’t have to interfere with the sensor(s), and the same dither circuit would work for all of the analog inputs.  In addition, I was using large resistance voltage dividers to monitor Vbat without wasting power and the high impedance forced me to add a capacitor to feed the ADC’s sample and hold input.  I knew that low esr cap would kill any dither signal that was applied directly to the main battery divider.

fig35avr121

This L-P filter from AVR121 works too, but modifying the circuit to give you other aref voltages is a bit of a pain.

I tried many different combinations, but I never saw the voltage on aref that I expected.  It took ages to discover that ~32k of internal resistance gets connected when you place an external voltage on the aref line, and that forms a ‘hidden’ voltage divider with your circuit. Grrr…

I did eventually get a few of those circuits working, but that internal resistor  seemed  to be slightly different on each board I tried, and I didn’t know if it was going to be stable with temperature, time, etc.  Another important issue was that I was switching from the internal 1.1v aref to read the thermistor, back to using the default 3.3v for other readings during the logger operation. So to put the dither directly into aref meant I would also need some way to modify the baseline aref voltage on the fly.  

Tune the resistor ratio, and roll PWM2 duty cycle and I'm pretty sure this circuit form Open Labs would give you variable Aref voltages.

Tweak the resistors & this circuit could give you variable arefs AND dithering.

I suppose that a truly elegant solution would do that with a PWM/RC filter circuit generating a variable DC voltage, and using a second PWM input to add the much smaller dither signal.  You could tune the dithers pk-pk amplitude to match the adjusted LSB, by the way you varied PWM2’s duty cycle (or by using the tone function)  during the readings.  But I knew working that out would probably give me a host of other problems to resolve (esp. with timing) and I was after a simple solution, with the smallest number of parts.  So I eventually abandoned the “dither on aref” approach.

This brought me back to Qwerty’s method of putting the triangular dither signal on the center of the thermistor bridge. My first task was to change that RC filter to lower the 9mv swing on that 5v circuit to match the much lower 1.1mv/LSB you get when using the internal bandgap as aref.

The power supply ripple calculator at OKAWA Electric was a perfect tool for this job:

oklowpassfilter

3.6mV was just an arbitrary 'close enough' point for me to start at as I had those components in the parts bit already.  But if you see random flat spots in your oversampled readings at the default ADC speed, then try increasing the ΔV pk-pk of your dither signal a little bit.

3.6mV was just an arbitrary ‘close enough’ point for me to start at as I had those components in the parts bit already.  But if you see random flat spots in your oversampled readings at the default ADC speed, then try increasing the ΔV pk-pk of your dither signal a little bit.

…which revealed that a 4.7MΩ/0.1uF RC combination would take the 3.3v 480Hz PWM on D6 and bring it down to  ~3.6mv peak to peak.  I immediately  hopped over to the Falstad circuit simulator to see the see how this worked.  To simulate an Arduino’s positive PWM, I used a 3.3v square wave source with an offset of 3.3v.  The little 10nf coupling cap prevents the pins DC voltage from affecting the thermistor reading, and the 2k2 bridge resistor prevents the dither signal from being grounded out when the 10K NTC thermistor resistance gets very low.  One of the coolest features of this simulator is that if you build a circuit with it, you can export a web link (like the ones above) that rebuilds the circuit instantly, so you can compare different versions simply by keeping those links in your log.

rcrisetime_png

The RC settling time is shown on the Okawa calculator’s step response graph, or you can watch the voltage rise on the scopes in Falstad by restarting the simulation with the buttons on upper right.

I love using Falstad for “What happens if I do this?” experiments. Of course these usually fail, but in doing so they show me that there are things about a circuit that I still don’t understand.  One thing that gave me a lot of grief when I started working with these  dithering circuits was that I did not appreciate how much time they need to stabilize.  This gets worse if you start disconnecting the thermistor  divider to save power between readings.  

So although I was getting smoother curves, and resolution that looked about 10x better than my raw ADC readings:

excerpt from 1024 oversampled temp record on Arduino ADC with triangular dither , 100kthermistor

Here I’ve converted these 1024 sample curves to °C , and artificially offset each curve by 0.05° from the next to it for easier comparison. The one-shot 10bit ADC reading at the default 125kHz (ps 64) is in purple, with other ADC speeds:  250 kHz (ps32) in orange,   500 kHz (ps16) in green, and 1 MHz (ps8) in blue.

At the height of my coupling capacitor lunacy I produced this beast, thinking I could simultaneously read a reference bridge, and correct away any offsets.

At the height of my coupling capacitor infatuation I produced this beast, thinking that if I could simultaneously add dither to a reference bridge I would be able to correct away ADC offset & gain errors, along with the offset caused by the dither signal, at the oversampled bit depth. But all those capacitors added artifacts to the readings when I reconnected GND through that mosfet, producing weird spikes in the data if I took readings less than two minutes apart (?)

…in any set of successive readings, the offset between the oversampled readings and the one shot ADC reading was changing depending on how long the PWM had been running.  No problem I thought, I’ll just throw in another coupling cap to block that slowly rising DC voltage, and connect the ADC input on the thermistor side. Unfortunately replacing the 2k2 bridging resistor with a coupling capacitor forms a high pass filter with the thermistor itself, forcing you to increase the size of the cap to raise the filters cutoff frequency above your 480Hz PWM. But that increases your RC time constant so then the filter starts to act like a differentiator: distorting your nice triangular dither signal (see pg12 of this pdf), and in some cases even reverting it back to the original square wave you started with… Argh!

So the result of all that trial & error is the basic PWM->triangular dither method works well, but you have to wait for the RC filter’s output to stabilize or it messes with your accuracy. And you still end up with a small offset in the ADC readings of 1/2 your dither signals peak to peak, because the original PWM square wave can only be positive.

Crank it up

But no one wants to see a data logger burning away precious milliamp-seconds just twiddling its PWMs!  With guidance from Nick Gammon’s fantastic ADC page, I had already been messing around with pre-scalars to increase the temporal resolution of my UNO DAQ.  I was further encouraged by this line from AVR120    “For optimum performance, the ADC clock should not exceed 200 kHz. However, frequencies up to 1 MHz do not reduce the ADC resolution significantly.  …and there were some tantalizing hints that cranking up the speed might also increase the internal noise enough to make oversampling work better. 

To figure out how fast your ADC is running:

System clock / prescalar = ADC clock,  ADC clock /13 = # of ADC reads/second

The core clock speed on 3.3v promini style boards is 8 MHz, providing:

8 MHz / 64 = 125 kHz /13 ticks    = 9600 /sec      (256 reads =27.6ms, 1024 =106ms, 4096 =426ms)  (default) 
8 MHz / 32 = 250 kHz /13             = 19230 /sec     (256 reads = 13ms,  1024=53ms, 4096=200ms)
8 MHz / 16 = 500 kHz /13             = 38000 /sec     (256 reads = 6.7ms, 1024=27ms, 4096=108ms)
8 MHz /   8 = 1 MHz /13                 = 76900 /sec     (256 reads = 3.3ms, 1024=13ms, 4096=53ms)

Your sensors output must be stable while you gather these samples and this limits what kind of phenomenon you can measure. At the default ADC clock speed, trying to add six extra bits of resolution (46 = 4096 readings) means you can only capture about 2 samples per second. That’s pretty darned slow for data acquisition! In fact, it’s so pokey that some people implement ring-buffer schemes to provide access to an oversampled reading at any time, without having to grab a whole new set of samples. A neat trick if you are continuously monitoring a sensor that changes slowly, and you have enough memory to play with.  Given the powers of 4 relationship between the different bit depths, it’s easy to see how you might hop-scotch through shorter 64 sample readings, and then combine those into a sort of rolling average version of a 256 sample reading if you don’t have quite enough ram for the full ring buffer approach.

enobs

My tests agree with the results posted at Open Labs. You can only push the ADC clock so far before you lose hardware bits, and this defeats the resolution gained from oversampling by making your accuracy worse. You can see this effect in the 1MHz line in the previous 1024 sample graph. Most AVR’s are lucky to get above 9 ENOB’s at their default settings.

200 kHz is the ‘official’ ADC speed limit for 10 bit accuracy, but I didn’t see any  significant difference between oversampled readings taken at the default 125kHz clock (ps 64), and those taken at 250kHz (ps 32).  At 500kHz (ps 16) the readings were good most of the time, but during rapid temperature transitions the readings started to ‘wiggle’ as though the dither signal was occasionally dropping out.   At 1MHz (ps 8) the curves wander around quite a bit, and I was seeing errors of ±0.05°C or more with some prolonged flat spots starting to appear. What’s interesting about this is that the triangular dither RC filter puts a capacitor across the thermistor, which should reduce the input impedance seen by the ADC and allow for faster readings.  But this did not reduce the 500kHz wiggle / 1MHz wandering in any of my test runs.  The ATmega328P datasheet quotes 2 LSB’s (typical) of absolute accuracy with an ADC clock at 200 kHz, but 4.5 LSB’s (typical) at an ADC clock of 1 MHz, and there is no point in pushing clock speeds if the accuracy gets worse by that much in the process.

So you can always double the ADC clock speed for oversampling, but going up to 500kHz depends on whether you can live with the accuracy errors that prescalar creates.  Those 500kHz wiggles become less evident as you progress from 256, to 1024, to 4096 readings, but that’s probably just an artifact of the smoothing.  The other thing to keep in mind is that one full cycle of the 480Hz PWM takes  ~2 milliseconds, but 256 readings at a 500kHz ADC clock takes only 6.73 milliseconds – so there is a high probability that dither signal synchrony issues creep in at the higher ADC speeds to produce offsets that affect the entire curve. Ideally you’d want the time you spend gathering the over-samples to be an exact multiple of the dither cycle time…

Let’s make some noise!

Hotter prescalars cut the oversampling time down dramatically, but I could not see how to avoid that RC settling time, which seemed to require about 50-60ms of PWM operation before the offsets became tolerable.  So I went back to the proverbial drawing board and asked myself, what if forget about the triangle dither signal, and try oversampling with some sort of random noise?

The first hurdle there was:  How was I going to generate this noise if the processor was already busy taking ADC readings?  The beauty of PWM based dither is that it just chugs away in the background, leaving the processor free.  As usual, Nick Gammon provided an elegant solution to this problem with code on his page about interrupts which showed how to read the ADC asynchronously:

// Note: Before calling this function, I change to the internal 1.1v aref and set the ADC prescalars
// but you can leave them at the defaults: see: https://www.gammon.com.au/adc for more details
volatile int adcReading;
volatile boolean adcDone;
boolean adcStarted;
unsigned int  adc_read;

unsigned long asyncOversample(int readPin, int extraBits)

    {
int i=0;int j=0;
int var=256;                                  //default is 4bits worth of oversampling
if(extraBits == 5){var=1024;}
if(extraBits == 6){var=4096;} //I’ve only included three options here, but hopefully you see the pattern
unsigned long accumulatedReading = 0;
adc_read=analogRead(readPin);   // a throw away reading to connect the ADC channel
//delete me:  simply as spacer
pinMode(5, OUTPUT); digitalWrite(5, LOW);  // set the pin you are toggling to OUTPUT!
//delete me:  simply a spacer a spacer comment for blog layout
while(i < var){    // asynchronous ADC read from  http://www.gammon.com.au/interrupts
  if (adcDone)
  {adcStarted = false; accumulatedReading += adcReading; adcDone = false;i++;}
  if (!adcStarted)
  {adcStarted = true; ADCSRA |= bit (ADSC) | bit (ADIE);}

  PORTD ^= B00100000;  // XOR toggle D5 w green LED & 30k limit resistor (see  below for details)
}   // end of while (i < var)

pinMode(5, INPUT);digitalWrite(5, LOW);  //turn off the toggle pin
if(extraBits == 4){accumulatedReading=(accumulatedReading >> 4);}  // Decimation step for 4 extra bits
if(extraBits == 5){accumulatedReading=(accumulatedReading >> 5);}  // 5 bits
if(extraBits == 6){accumulatedReading=(accumulatedReading >> 6);}  // 6 bits
return accumulatedReading;
}   //end of asyncOversample function

ISR (ADC_vect)     // ADC complete ISR needed for asyncOversample function  
  {  adcReading = ADCL | (ADCH << 8);adcDone = true; }

 

Next I had to generate the noise itself. People use Zenner diode breakdown to produce random number seeds, but thought I would see if I could generate noise inside the processor, since there seemed to be no end of people complaining about the Arduino’s ADC in the forums. However when I actually tried to do this by connecting pull-ups,  changing I/O settings, an every other kind of processor toggle I could think of, I got nothing.  That ADC was solid as a rock until I started flipping the pins connected to the external indicator LED.   Even then, the early results were wildly inconsistent, with the same code producing good oversampling on one unit, but not another.

Like the hidden resistor problem, it took me a while to notice that the random bunch of LEDs on my breadboard test units had significantly different forward voltage drops from one LED the next, and from one RGB color channel to the next.  Once I realized how much that was affecting the results,  it didn’t take long to determine that that the noise generating sweet spot was somewhere around 0.04mA of pin current:

An example of oversampling with pulsed pin current of 0.038mA to generate ground line noise.

One-shot ADC reading shown in purple, with oversampled readings taken at 125kHz (ps64 default)  in grey, 250kHz (ps32) in orange, 500kHz (ps16) in green. All readings are converted to °C, and I’ve offset these curves for clarity, as they would otherwise be on top of one another. You can clearly see the PS16 wiggle as the temperature falls, and the sharp eyed will notice there are still offsets between the different runs which were all taken in quick succession. These seem to be more apparent in the longer slower oversampling runs than they are in the the shorter faster ones… darn it…

Unlike triangular dither techniques, which will tolerate a fairly large ΔV, this noise based method stopped working (ie: flat spots started appearing) when the toggled pin current went below 0.02mA, and the curves became pretty scratchy above 0.06mA  indicating there was too much noise.  That’s a fairly tight range, and it was sheer luck that the 30k limit resistor I was using on my indicator LED’s brought me close enough to spot the effect.  So my current target is ~0.04mA of pin current for 1.1v dithering. And there was nothing special about the LED being there either, as tests using a simple 82.5KΩ resistor from the  PORTy ^= _BV( PDx/PBx );    toggled pin to ground produced good results.  This is pure conjecture on my part, but if you assume the mosfets on the I/O pins have about 40Ω of internal resistance with 3.3v control, then 0.04mA pin current would produce a voltage drop of ~1.6 mv – which is suspiciously close to the 1.1mv/LSB resolution of the ADC with the internal bandgap set as aref.  That puts this dither noise right into the 1-2x voltage per bit recommendation from the literature.

rtcdividerreadings

Here I’m oversampling with 1024 readings from a 2x10MΩ divider which cuts the voltage of the RTC’s backup coin cell in half. 250kHz (psS32) in orange, and 125kHz(ps64) in grey. These are the raw readings with aref set to the default 3.3v and there is no capacitor on the divider. This is far beyond the 10k input impedance the ADC was designed for, but I think the many repeated readings you do with oversampling helps the 14pF sample&hold caps do their job. At this resolution, the CR2032 seems to be acting like another temperature sensor (?) UPDATE: So this actually was the battery responding to temperature rather than the dithering method, which does not work with the rail voltage on aref unless you add a cap to the voltage divider.

This pin-toggling noise technique is not exactly a one size fits all solution, and the exact current required to induce ADC bit toggling will vary depending on which board you are using, and especially on which capacitors are being used smooth the output from the voltage regulator.  So you will have to noodle around a bit to find the correct resistor value to use for your particular Arduino. I’d start with a resistor value that draws enough current to give you a voltage drop on the digital pin mosfet to approach 2x your ADC’s mV/LSB resolution. With 3.3v as aref (so 3.22mV/bit), I would use a pin resistor of  about 27.5k to generate a pin current of 0.12mA, and a pin vdrop of ~4.8mV.   With 5v control logic, the mosfets controlling the digital pins are more fully turned, so the pin resistance is somewhat lower; around 25-30 ohms. With 5v on aref your resolution is about 4.88mv/bit, and the dither resistor would have to pull around 0.39 mA to shake the rail with a vdrop twice that mv/bit, so the dithering resistor would need to be somewhere around 12.8 kΩ.

On new builds I will measure the forward voltage drop of the indicator LEDs and change the limit resistor to give me the current through those I need to generate dither noise. That way I don’t need to any new digital lines for the oversampling process, though this will entail checking every LED, as there is significant vf variation between batches.  The blue channel on the RGB’s I have lying around have a vf of ~2.473v, so 0.827v will be left for the resistor to cover with a 3.3v rail.  To achieve a target pin current of 0.12mA the limit resistor would have to be 0.827v/0.00012mA = 6.89kΩ.

This method is also critically dependent on the tiny capacitor stabilizing the aref voltage. When I tried it on the units I had left over from the ‘dither on aref’ experiments, the method did not work at all if the aref stabilizing capacitor had been removed.  I also suspect that the voltage on the capacitor ‘adjusts’ to the noise pulses over time, which might be causing the 0.02C difference between the 256 & 1024 readings shown above. So there could be another settling time issue if you take a large number of over-sampled readings in rapid succession. Larger caps stabilizing the rail voltage on breakout boards may also affect the method.

This technique will work with any resistive sensor being read with a simple voltage divider, provided there are no capacitors nearby to smooth out the noise which is vital for oversampling to work.  I’m not going to pretend to understand all the math behind it,  but it’s probably safe to say you can add somewhere between 2-5 extra bits of resolution to your ADC before the technique suffers from other limiting problems.  Although the 256 sample curves are a bit gritty, you can make that many samples with the ADC clock at  250kHz in ~13milliseconds, which doesn’t impact the power budget too much. If something interesting starts happening, you can dynamically enable another bit or two of resolution in the code to zero in on it.

Overall, I’d say the results from oversampling with toggled-pin noise are not quite as smooth as the curves you can get with a well tuned triangular dither, but I’m happy to trade that last bit of synthetic resolution for a method that’s instantly available for all of the ADC inputs with less chance of a synchrony problem.  The icing on the cake is that I don’t have to add any extra circuitry to use oversampling on the fleet of loggers already on deployment, because all I have to do is toggle the indicator LEDs they have on board since their limit resistors were already in the current range I need…YES!

Addendum 2017-04-26:

I’ve moved on to calibration, and in the process I learned that regulator & bandgap voltages change a fair bit with temperature. So it’s probably not a good idea to use the internal bandgap on aref with this oversampling method if you want thermistors calibrated over a wide temperature range. But I did it anyway.

In those tests I used a 688k series resistor with a 100k thermistor, so I was far from divider’s optimum of Rseries=RTnominal. I was taking 1024 oversamples, adding five oversampled bits to ADC, and I was using the internal bandgap voltage on aref, which added another bit.  Since I was on the tail end of the divider sensitivity curve, the effective resolution changed quite a bit over the range. The output shifted from ~0.0018°C/bit at 20°C, to about 0.0038 °C/bit up at 40°C, and I will have to do some homework to figure out how much of that signal is being lost to noise. This is better resolution than some people achieve reading thermistor bridges with the 16bit ADS1115, though gathering all those readings means I can only capture 18 samples per second – even with the ADC clock at 250kHz.

I have a long way to go before I reach the accuracy levels you see at the geotechnical high end, but I think that’s still good for readings with a humble Arduino ADC!

Addendum 2017-09-24:

Several people have contacted me about their attempts to get this ‘pin-toggling noise’ method working with different Arduinos at higher voltages.  If I had to summarize the kernel of understanding that was missed in the unsuccessful cases it is this:

If you jiggle one part of the system with noise – stabilize the other part.

It does not matter if the noise shows up on aref, or on the sensors output, so long as it is not present in the same form on both.  With the bandgap 1.1v as aref, you can rely on that to be the stable side, so you want the voltage divider with your sensor not to have a capacitor on it, since the sensor side needs to shake by ±2 LSB volts with the pin toggling. The internal reference is slightly different on each individual chip (from 1V to 1.2V), so you’ll need to “calibrate” if you go this route. Don’t forget to throw away the first reading after changing the analog channel, and if you have a high resistance voltage divider, add one ms delay after that first analog read.

If you use the rail voltage as aref (the default) with an un-stabilized voltage divider then your pin toggling current shakes the aref ground in perfect synchrony with the ground line on your sensor, and no matter how many samples you read & decimate you will never get beyond the 10 bit resolution of the ADC. So to use the rail as aref you need a small capacitor across the lower half of your divider so the sensor input to the ADC becomes the stable side: 

Degree Celsius vs. Time with lines offset for easier visual comparison:  The blue line is over-sampled output from a pro-mini clone reading a 100KΩ  NTC Therm/100KΩ series voltage divider. Aref was set to the 3.3v rail, with a 100nF capacitor parallel to the thermistor on the low side.    A 0.12 mA pin-current provided sufficient noise for dithering 1024 readings, delivering an effective resolution of ~0.0028° at 24C. For comparison, the red line is the output from an I2C si7051 sensor on the same logger, with a resolution of 0.01C.

The question of which side should be treated as stable also comes into play when you want to over-sample analog output from more complex sensor circuits. If the sensors supporting electronics are already doing a good job of stabilizing output, say with feedback, caps and some sort of buffer at the end of an amplification cascade, then you have no choice but to set aref to the rail voltage and shake that. I’ve had success with this approach and a complex sensor circuit on a 5v Nano, by pulsing a pin connected to ground through a 12KΩ resistor (~ 0.4 mA of pin current). Of course everything else in your system is feeling this noise to some extent, and this may cause issues with sensitive sensor IC’s, or with micro-controllers other than the 328p.  The effect of the pin current is being limited by capacitance distributed throughout the system, which varies from board to board, so this is definitely a “try it an see” method : when it works it really works, producing smooth curves with no hint of the underlying 10-bit ADC peaking through.

If you see any flat spots or rounded stair steps in your temp. data, especially in areas where the changes are occurring slowly over time, then you know the dithering is not working:  

This is an example of the natural noise problem: the (blue line) thermistor readings achieved high bit depths the refrigerator, but developed flat spots in the room where the changes were happening more slowly. This was a test run with the noise circuit disconnected,which I followed with run using the same code +noise applied so I could compare the two. Doing two runs (with & without dithering) is good general approach to use when testing a circuit that uses oversampling.

Any natural temperature variation over your sampling interval will make it look like your generated noise is sufficient for oversampling, when it is not.  The photo above shows how that this test is almost impossible to do in the refrigerator, because the natural on/off cycle of the compressor generates enough change/time to make oversampling work without dithering. 

With stabilizing capacitors you also have the trickier problem of spotting is the influence of the RC time constant when you only power the voltage divider during readings.  Oversampling before the cap is fully charged will provide more than enough change in the readings to hide inadequate dithering.  In fact, if you scale the capacitor/series resistor combination, and the timing, to sample over the 3T-5T interval, you get good oversampling results with no noise in the system at all.  In some ways, using RC rise time is better than pin toggling when you are using the rail as aref, since it does not fight against the other capacitance distributed around the system to produce a delta on the ADC readings.  I’d use this rather than pin toggling with aref=rail  if it weren’t for the fact that capacitors can have the worst variation coefficients of any electronic component you are ever likely to run into.

Garden variety Y5V ceramics vary by up to 82% over their rated temperature range, and even the X7R’s that most engineers use vary by +/-15%. I could calibrate that thermal variation away, but from an environmental monitoring point of view, the drift over time is a much bigger problem; with caps commonly loosing 10-15% of their rating over the first year (~8900 hours) of operation. There are stable NPO rated ceramic caps out there, but they are only available in relatively small pF sizes, and a good 0.1uF NPO cap will set you back about $7 each even if you buy them in quantity, so that part alone costs more than a decent IC based temperature sensor.

Plastic film capacitors have much better thermal coefficients: Polyphenylene sulfide (PPS ±1.5%) or Polypropylene (CBB or PP ±2.5%)A quick browse around the Bay shows those are often available for less than $1 each, and the aging rate (% change/decade hour) for both of those dielectrics is listed as negligible. The trade off is that they are huge in comparison to ceramics, so you are not going to just sneak one in between the pins on your pro-mini. 

For most rail-as-aref situations, Qwerty’s PWM based dither method (mentioned at the beginning of the post) is a more robust way to dither with cheap ceramic caps, since it can tolerate significant variation in a way that does not affect your accuracy that much – but you still have to keep an eye on the dither circuit settling time. 

Addendum 2017-10-15:

Just came across AN2668 from STMicroelectronics which sums the input signal and trianglar dither signal through an opamp before sending it to the ADC:

Still seems like a lot of work to me, although that apnote does have me wondering if the pin toggle dither noise is actually Gaussian…

Field Report 2016-07-09: I2C pressure sensors work on 20m cables!

Peter Carlin, Jeff Clark, Alex, Trish, and Gosia.

Peter, Jeff, Alex, Trish, and Gosia.    Jeff, Gosia, (and Natalie) took time off work to do some of the more intense installation dives, which helped tremendously.

With the term prep taking up everyone’s time, I almost forgot to post about the wonderful field season we had this summer.  We really covered the bases on this one: from surface loggers, to cave sensors, to new deployments out on the reef.  And there were plenty of new toys in the show, including a couple of “All hands on deck” days for the deployment  and retrieval of several POCIS (Polar Organic Chemical Integrative) samplers.

 

Dual MS5803 pressure sensor unit for tide gauge & Permeameter

A dual MS5803 pressure sensor unit with the same cable & waterproof connectors I use on the DS18b20 chains.

Potted with E-30cl

Potted in E-30Cl epoxy.

Most of the new instrument deployments on this trip were DS18b20 temp chains and deep pressure loggers. While those under water units continue to give us great data, I’ve added a new model that can record water level with a  MS5803 pressure sensor at the end of a long cable.  That sensor has two selectable bus addresses, and I was very happy to discover that with one on the housing, (recording atmospheric pressure) and one on the end of an 18m cable, both sensors will read OK with 4K7 pull-ups if you lower the bus speed to 100 kHz.  Slowing things down to 50kHz (with TBWR=64; on my 8Mhz 3.3v loggers) let’s me extend that out to 25m, again with the default 4k7s. I’m sure you could stretch that even further with lower pull-up resistor values.  I honestly didn’t remember anything in the specs that said an unmodified I2C bus could be extended out to the kind of run lengths you usually see with one-wire sensors…

Peter Carlin did all the heavy lifting, including several long nights feeding mosquitos...

Peter did all the heavy lifting for the permeamters, including some late nights checking all the stations.

This opens up tide monitoring from  stations above water, and will let us capture some decent bore-hole records.   And since I mounted the pressure sensors inside threaded fitting, we could attach the them to a reservoir for other interesting experiments. What we actually used them for on this trip were falling-head permeameter tests.  One of Trish’s undergrad students planted a veritable forest of PVC tubes in locations all over the field area.   Though he built a couple of the loggers himself in the instrumentation course, it was interesting to see him working through all the other things it takes to run an experiment in the real world. Some of the limestone mounted tests took many days to run, as compared to the much shorter times you see with soil, or cement. So being able to let the data loggers record those slow level changes was a real help.

Checking on one of our water level recorders

One of our older in-water level recorders, with the pressure sensors directly on the housing. This station has been in place since Kayleen recorded the big floods in 2013.

While he was out mixing cement & feeding mosquitos, our room turned into a rolling conveyer belt of incoming an outgoing loggers. With many of the drip logging stations approaching two years in service, I was expecting some attrition  in the set at Rio Secreto. To my surprise the majority of sensor failures were from the newest units installed last December. I had used more expensive Adafruit  breakouts for those builds (while the older drip loggers were built with $2 eBay boards) I’d love to say this is an anomaly, but after building & deploying more than a hundred of these things,  it seems that IC sensor longevity can be unpredictable, no matter where you buy them.  And we are not exactly treating them nicely…

As usual there was lots of great diving, and we even got back up to the north coast to replace those opportunistic mangrove deployments from the last trip. I still can’t get over how lucky I am to be able to see the diy loggers going out in the wild like this.  But for Trish, all this is just, you know, another day at the office…

Of course by the time we reach that point, my work is pretty much done. She’s the one who has to wrangle with all the data, and writing a good paper is a lot harder than building a few loggers…

Addendum 2016-11-23

Not that I need them at this point, but I just stumbled across some I2C extenders over at Sandbox electronics. They claim up to 300m with their differential extender.  Those NDIR CO2 sensors also look interesting, but with the caves over 95%RH for significant periods of time, there is some question  about whether those sensors would work.

Addendum 2016-12-20

A borehole installation for one of the dual pressure sensor loggers

We finally got one of the dual 5803 units set up in an unused well. This has been on the to-do list since mid year, but as you might imagine, there are not that many wells that get drilled without being used right away, so we are very thankful to the land-owner.  Of course there is so much pumping going on in the general area, I have a niggling concern that what we will really be recording is the draw-down, rather than the level of the aquifer itself.

 

 

Using the Arduino UNO for Data Acquisition

Using an UNO and an ADS1115 ADC module to monitor current use by a pro mini logger

Here, I’m using the basic UNO Logger as a tethered Data AcQuisition device, recording the current used by a second data logger. Since the second logger is ‘floating’ with no connection to the UNO’s ground line, the voltage drop across the 5Ω shunt resistor is recorded using a differential channel on the ADS1115. Differential readings are also useful for sensor applications that use a wheatstone bridge arrangement.

I recently picked up a ADS1115 breakout board, and it was fairly easy to use that with the serial data plotting capability of the Arduino IDE. It’s not often that something works this well on the first try, and I thought I would post about using  the combination as a kind of  ‘poor mans oscilloscope’.   The plotter’s vertical axis auto adjusts as the value of your output increases or decreases, while the X axis is fixed at 500 points, with each tick of the axis equal to an executed serial println command.  Having the ability to spool data to the screen with a simple print statement, turns the exercise into a “What happens if I do this?” kind of process, which is perfect for providing feedback to students learning how to program Arduinos.  I posted the code used to generate these graphs on GitHub, but you will have to noodle around with it to figure out what the threshold settings should be for your particular application.

Although I already have a good method to estimate the overall power consumption of my loggers, I was motivated by this Jeelabs post to see if there was a way I could look at individual events. Even if you have a nice Rigol to play with, it can still be tricky to get all the settings and timings right because the loggers can draw anything from 0.15mA while sleeping, all the way to up to 100mA during SD writing events.  Jeelabs elegant solution to this ranging problem uses two shunt resistors and a diode, but with 15bits of differential range on the 1115, the 8x gain setting can do the job with a single resistor. (…provided I don’t exceed the ±0.5 volt limit that PGA setting imposes…) 

After installing Rowbergs I2Cdev library, and running a couple of jumpers, my drip sensors were generating raw count output like this:

Cave Pearl data loggers

Of course that was scrolling by like crazy, and the re-scale feature meant that the y-axis was jumping round like a bullfrog on a hotplate. You can bring the vertical axis under control by adding a few delimiter separated constants before the final data println:

Serial.print(4000); Serial.print(” “);     // sets a stable upper value line
Serial.print(0); Serial.print(” “);             //this constant sets a stable lower value

But to prevent the x axis from scrolling forever, I had to setup a trigger threshold, that would only capture a new set of readings when an actual event was occurring:


void loop(void)
{
int sensorOneCounts=adc0.getConversion();  //note we already set the mux to differential in setup
if(sensorOneCounts >= LoopThreshold) {      //rapid sampling loop to capture the event
start=millis();
for (int Cycle = 0; Cycle < ADCcycles; Cycle++) {
currentADCreadings[Cycle]=adc0.getConversion();
}
elapsed=millis()-start;// elapsed gives you the timebase for your samples
}
//A separate loop to output of the samples to the text monitor or serial plotter
for (int Cycle = 0; Cycle < ADCcycles; Cycle++) {
Serial.println(currentADCreadings[Cycle]);
}
#ifdef ECHO_TO_SERIAL    //TEXT only output of the time it took for your data acquisition loop
Serial.print(F(“Time for “));Serial.print(ADCcycles);Serial.print(F(” readings: “));Serial.print(elapsed);Serial.println(F(” milliseconds”));
#endif

}

A few runs with text-only output showed the sleeping logger generating around 65 raw counts on the ADC, so I set the LoopThreshold to 75, and the plotter started behaving like an oscilloscope’s triggered sweep: scrolling 500 new readings across the screen each time the drip sensor woke up.  With a known shunt resistance, and the millivolt resolution per bit from the datasheet, I could convert the raw ADC counts to μA in the printing loop. Just for fun I also tried speeding up the I2C bus to see the temporal resolution of that sample loop at 400Khz:

Running ADS1115 with UNO as DAQ - with I2C bus running at 400HZ, the system outraces the ADCSo I was asking for data faster than the 1115’s top speed of 860sps could deliver it.  A few more trials showed that even with the Arduino’s default 100Khz bus, I had to add delayMicroseconds(400);  into the capture loop to keep the UNO from outpacing the ADC module.  I expect that will be different for Arduinos with different clocks like the promini, so one would have to make manual adjustments based on the elapsed times for each system.

These drip logger events are the result of straightforward code that follows the steps:

1) Wake from interrupt & clear the registers, update count variable
2) Turn indicator LED on & sleep for 15ms
3) Wake, turn LED off & sleep for another 32ms (to let impact vibrations die down)
4) Set accelerometer registers & go into deep sleep: wait for next accelerometer interrupt

Since you can drop as many constants on the plotter as you want, it’s easy to determine the current level at a specific point on the plot by intersecting it with a new line:

Cave Pearl data loggers

Though there were no surprises in the pattern per se, it was handy to find out that the “in-event” sleep current was about twice the normal standby state. (which I confirmed at 0.2mA using a DMM)  Now I can go digging through the code to figure out what’s drawing  that extra power.

After resetting I2C bus, I tweaked the delays to exactly match the 860 sps output from the ADC. Then I sat back and waited for one of the real time clock alarms to fire:

Cave Pearl data loggers

This was my first view of that complex logger event, with variable juggling, time stamp creation, eeprom buffering, counter resets, etc. With I2C devices being able to stretch the clock whenever they need to, I had no idea how long these kind of events actually took, and I certainly never managed to capture one so easily on the old clunker scopes I could get my hands on.  The comb pattern you see there is a result of sleeping the processor for 15ms after sending data to I2C device registers; instead of using a 6ms delay while the Arduino waits for the devices to respond.  This lengthens the overall duration of the event, but the time spent running at full power is reduced.

By how much? I reset the logger to use delays after I2C writes rather than sleeps and re-captured the event:

Cave Pearl data loggers

The horizontal axis is at the same scale as before. The start of the event now looks like a solid block and the overall event is about 50ms shorter.  For comparison, I’ve inset the previous graph with the sleep gaps cut out and the spikes moved together, and the two white rectangles at the top represent area equivalent to sleep gaps that were removed.  The grey area shows the power saved by using 15ms sleeps instead of 6ms delays.  To do this properly I would need to take another reading of the 3xAA power supply voltage and calculate the total power used (minus the loss on the shunt resistor), but you can see at least 25% improvement simply by comparing the plots. That kind of feedback is very helpful when you are trying to optimize your code.

Now, I can already hear people scoff that even $3 is too much to pay for a scope that won’t reach 1Khz. Indeed, with a minimum temporal resolution of just over a millisecond, the ADC frequently misses brief current peaks. I know my 8mhz promini based loggers usually draw more than 4mA while running, and you see the longer operations in the RTC/delays event pushing up towards 6 mA.

Cave Pearl data loggers

But even with the upper third lopped off those spikes by timing errors, I can still use this method to make a reasonable estimate of the power being consumed by logger events provided they don’t last more than 580 milliseconds. ( The serial plotter provides a window for only 500 samples and the UNO barely has enough variable memory for a 16bit integer array that size.) If you add the SD card library to your script you are only left with enough memory for about 100 of these samples in your capture loop, although you could try reducing the ADC readings down to 0-256 with a well tuned map function. Then you could use an uint8_t integer array, and store twice as many readings in the limited variable memory of the Arduino.  If you were just doing a classroom demo, that range would probably still give you a decent display on the plotter.

Despite these limitations, the setup is so simple that I think the serial plotter tool will become a go-to for quick looks at sensor data, perhaps even replacing some processing based demos.  The ability to connect four of these ADC modules to a single Arduino makes the platform look quite respectable as a classroom level data acquisition system, especially for environmental data that rarely requires high frequency sampling.  

If you switch to a beefier 1280 based Arduino like the Mega, or the Moteino, those memory limitations disappear and recording much longer events would become possible.  With the delays I had to put in, my gut feeling is that you could juggle a couple of these boards at a time; interleaving the readings to double or triple the effective sampling frequency. (at least until the input impedances start to mess with the shunt resistor…)  And finally, I think adding a couple of well chosen diodes to clamp the input voltage to the ±limits imposed by your PGA setting would be a good idea in the classroom, or you might loose a few of those ADS1115 modules.

It is worth noting that there is a comparator function in the ADS1115 that can be programmed to send a pin change alert when a given threshold is crossed. If I can figure out how to get ADS1115’s built in comparator working with the differential modes, this would even let me put the DAQ to sleep between sampling runs. The ADC module also has a ‘data ready’ alarm that can be enabled to tell you exactly when it has the next reading available.  I did not go that route as the generic approach used here is applicable to sensors without these features, including the ADC built into the Arduino. 

There are plenty of other serial data plotting programs, but using the free IDE version like this is an iterative exercise, forcing you to switch between the serial text monitor & the serial plotter a few times to tweak the delays and constants. If you are using an UNO in a classroom setting, this in itself is a useful activity.  And given how cheap basic DAQ modules are these days, choosing to use an Arduino is always about the showing the process rather than just getting a final result.

Addendum 2016-08-15:     A MUCH faster DAQ with the native UNO ADC

 Here I’m reading the voltage across a 10Ω resistor with the UNO’s built in ADC. To do that I need to join the ground lines because the Arduino can only take single-ended readings. That also forces me to put the shunt on the low side line so I don’t exceed the 1.1 vref. This arrangement causes the drip sensors’s effective ground line to jump around as the current through the resistor changes.


Here I’m reading the voltage across a 10Ω resistor with the UNO’s built in ADC. To do that I need to join the ground lines because the Arduino can only take single-ended readings. That also forces me to put the shunt resistor on the low side line so I don’t exceed the 1.1 vref. This arrangement causes the promini logger’s effective ground line to jump around as the current through the resistor changes.

The folks over at Measuring Stuff  posted a page called The Arduino DAQ Chronicles, which goes into some detail on the process.  It’s a good chunk of background reading even if you are using the ADS1115 ADC described above. The Arduino’s native ADC delivers 10-bit readings, (ie 0-1023) and compared to the relatively pokey ADS1115, a typical UNO can take several thousand analog readings per second; outperforming the ADS1115 by a fair margin. The challenge is balancing the reduced resolution with the other limits imposed by the method itself. Even if you increase the sensitivity of the native 10-bit ADC by changing to the internal 1.1vref, you only reach 1.1v/1024 steps = 1.07 millivolt per bit. Remember that my loggers are sleeping at 0.2 milliamp x 5Ω shunt resistor = 1 millivolt so we are operating right on the lower limit of the ADC resolution. With that 1.1v cap on the ADC input, and a potential 100mA peak current (during SD card data saves), the largest shunt resistor I can use is 1.1v/0.1a= 11Ω. So to get the Arduino to discriminate the low currents we need to use a pretty  large shunt resistor value, and allowing the voltage drop to get that high imposes another challenge in that the drip logger would only be left with 4.5v (the 3xAA battery) 1.1v drop on shunt = 3.4v. That’s right at the minimum input voltage needed by the 3.3v regulator on the Arduino promini board, risking a potential brown-out in the middle of the SD card writing process.

But hey, lets pop in some new batteries, and see what we get:

FirstRaw

Not too bad, but that’s a noisy plot with lots of jitter on the base line. Having 1-2 bits toggle like that is typical for ADC’s so there is probably nothing I can do to get rid of it. And the drip sensors resting state reads at zero when it should be registering at least two counts. It is also common for an ADC to under or over read by a few bits, so I added an offset adjustment into the code to bring that sleeping baseline up to the 0.2mA that I have already confirmed with a multimeter.

Then I realized I still had the 400us delay in there from the ADS1115, so I took that out:

Cave Pearl data loggers

Crikey! Even with the code overhead, that thing is taking over 8000 samples a second! That’s far too many readings for the serial plotter to handle if I want to view long events.  And that first spike goes upt to about 5mA – exactly where it should have been on the ADS1115 readings if that module had not missed them at 860sps.  So how do I maintain the Arduino’s ability to spot those rapid sample peaks, but only send 500 readings to the screen?

Instead of delays, I decided to create an “oversampling loop” that creates a running average using a low pass filter, but also checks for peaks throughout the process. You can dig into the code for this over at GitHub, but the basic idea that is if a peak occurs, then the highest reading in an interval becomes the final reading. If a decreasing trend is found, then the output from a leaky integrator becomes the final reading for that interval. So the readings get smoothed on the way down, but not on the way up.

Here is a try with the oversampling interval set to six ADC readings:

BothEvents6xoversampling

That’s the RTC event (with delays) and the drip triggered event at the same scale.

Now here’s another run with a 15:1 ratio:

OS15-900msPerloop

Squashing more than 7000 raw readings into a 500 line display really starts to distort the curve, but everything is still recognizable, and more importantly, the peaks did not get chopped off like they did with the ADS1115.  Woot!

Now we have a method that lets us adjust the serial plotter’s display to match the event we are trying to see, provided we don’t over-interpret the accuracy of those heavily averaged interval numbers. And rapid sampling can introduce other issues to deal with. For example: Dustyn Roberts has a useful post on creating very precise time stamps with the micros(); function when you are sampling faster than 1 millisecond. 

So within limits, a standard Arduino gives you good temporal resolution but can not detect less than 1 millivolt of change, while the ADS1115 gives you precision down to an impressive 0.0078 millivolts (@16x gain) but is not so great for tracking really brief events.  If you need both resolution & speed, you need opamp circuits designed for the specific situation, as in the Jeelabs example. It’s also worth mentioning that the Arduino ADC also supports a comparator that you could use for the threshold function more elegantly than I’ve done here.  And with such a low impedance input, you can also double or triple the sampling rate by changing the ADC clock prescalers (this is also in the code on GitHub) Both of those things are described in detail over at Nick Gammons excellent page on ADC conversion on the Arduino.

Addendum 2016-08-16

Hopefully they will add export options (like other serial tools) to the plotter in the future, but for now you can manually copy data out of the serial text monitor and paste it into Excel.  Before you start a long run for export to a spreadsheet, comment out the constants that you were printing to stabilize the plotters y – axis, so that only the numbers you are interested in are sent to the serial output.  If you are waiting for an event to happen, it often helps to add a threshold test so that the serial monitor is not just filling up with uninteresting data that you would have to prune away later anyway:thresholdprintline

Then after your DAQ has been capturing events for a while, click inside the serial monitor window,  select all the data & copy it (on a windows machine that would be [CTRL]+[A] followed by [CTRL]+[C] ) then open a new spreadsheet and simply paste the data into an empty column.

This lets you compare the events side by side:

CopySerialdataintoExcelIf you want to graph more than one number, then add a comma between them with Serial.print(“,”);  remembering to use println on the last variable being sent.  Then you need to add the intermediate steps of pasting the data into a text editor like notepad, and saving that text file. Then change the ending from .txt to .csv and Excel will open it directly. If you can’t change the filename, you can still import the text file and specify comma delimiters:

TextImportExcel

to put the numbers into adjacent columns. If you don’t have excel, there are plenty of other data plotting options out there. And programs like coolterm, can save you from having to do those cutting & pasting steps.

Addendum 2016-10-13

Gokul Shrinivas just posted a neat little bit of code at hackster.io that reads the Arduino ADC at a cracking 68k samples per second out of a Arduino Due by directly accessing the ADC ports as soon as the DataReady interrupt is triggered. 

He starts with defines:

#define ADC_MR * (volatile unsigned int *) (0x400C0004) /*adc mode word*/
#define ADC_CR * (volatile unsigned int *) (0x400C0000) /*write a 2 to start convertion*/
#define ADC_ISR * (volatile unsigned int *) (0x400C0030) /*status reg -- bit 24 is data ready*/
#define ADC_ISR_DRDY 0x01000000
#define ADC_START 2
#define ADC_LCDR * (volatile unsigned int *) (0x400C0020) /*last converted low 12 bits*/
#define ADC_DATA 0x00000FFF 

And the sampling loop becomes

ADC_CR = ADC_START ;
  for (i=0; i<320; i++){
    // Wait for end of conversion
    while (!(ADC_ISR & ADC_ISR_DRDY));
    // Read the value
    analog_data[i] = ADC_LCDR & ADC_DATA ;
    // start next
    ADC_CR = ADC_START ;
  }

He used an LCD screen for the scope, but I am wondering if it would be possible to refine my oversampling trick to squeeze that many data points on the serial plotter somehow…

And I wonder if this trick is even available on the 328p? I guess I will have to go looking at other Arduino Oscilloscope projects. The problem is that most of them seem to be using 8-bit signals from “free-running” modes, and I rarely have an application that needs less resolution. Here’s a project doing Interrupt-Driven Analog Conversion  with an ATMega328p, and Nick Gammon mentions it here, but no one does a speed comparison with the ten analog 10-bit readings you get per millisecond using plain old analogRead();

Even if there is no speed gain, perhaps I could do calculations while I was letting the ADC read continuously to speed up the overall sampling loop.  I could see myself getting out of sync pretty easily though, unless I deactivate the general interrupt flag and activate it again (with  cli() and sei()) or I would randomly miss readings.

Nick posted a code example of reading the Analog-to-Digital converter asynchronously that seems to solve the synchrony problem.  He’s also posted an interesting ‘sleep during ADC conversion’ example buried in there, but that’s only useful if you are not using timers & PWM at the same time, and I often do that…

Update: the Hackster.io project listing magically evaporated… and I just found exactly the same code at Bruce Land’s Hackaday project from 6 months ago. So I think proper attribution for the script above should go to Bruce.

Addendum 2017-01-10

I’ve been playing around with the ADC clock prescalars, and with the asynchronous reading code available over at Nick Gammon’s site to try to get better temporal performance out of my little UNO-scope.  With such a low value shunt resistor the ADC seems to read fine even if you crank it up to about 40k samples/second. Unfortunately you pass a point of diminishing returns due to the processing overhead with the interrupt based asynchronous readings,  so async reading actually reduces the number of readings per millisecond at the higher clock speeds. If you want to play with this approach yourself, I’ve posted the update with the prescalar settings to GitHub.

Addendum 2017-02-27

Anyone interested in using an Arduino as at data acquisition device will probably want to see my recent post: Enhancing ADC resolution with Dithering & Oversampling
For an unmodified UNO,  Qwerty’s triangular dither method is probably the easiest one to use, but for 3.3v boards I’d go for the Toggled pin dither method I outline at the bottom of the post.

Better Thermistor Readings with an Arduino: Series Resistors & Aref

      This ADC adventure was new territory for me and I am still learning my way around.  If you’re in the same boat, then try the introductory videos on Digital & Analog I/O by Jeff Feddersen and Tom Igoe.  Also check out the Jeremy Blum’s tutorial where he mentions the constrain and map commands which come in handy during prototyping.  From there move to Nick Gammon’s excellent reference on ADC conversion on the Arduino and then wrap up the set with Bil Herd’s ADC tutorial over at Hackaday.

Up to this point I’ve been using IC thermometers (TMP102, DS18b20, etc) because they are easy get going, and seemed to offer better resolution than I could get out of the Arduino’s humble 10bit ADC.  But several of the projects I’ve been working on (like Masons hygrometers) have run into their 0.0625 resolution limit.  A few of our tide gauges used MS5803 pressure sensors and seeing those gorgeous 24-bit time series beside the record from an MCP9808 showed me just how much more system behavior information becomes available with those extra bits:

12vs24bit-2

So I began looking for other high resolution temperature sensors, and found many people using thermistors with external ADC’s like the ADS1115,  adding a shunt regulator on one of inputs for calibration.  Then you can double the sensitivity by connecting opposing divider pairs in a bridge configuration, putting the output on two differential channels. That’s pretty much the textbook solution, made easier with a bridge calculator and the ubiquitous TL431.  But to me that seemed like throwing money at the problem, and if I’m going to do that why not just calibrate the MS5803’s since they don’t cost much more than a differential ADS1115 or a delta-sigma MCP3424, and they give you a fantastic pressure record, and they consume < 0.15µA in standby mode…

Now, I’m not going to make the mistake of thinking the Arduino’s ADC will reach the accuracy of commercial instrument,  but with temp. logs providing such a good sanity check when my other sensors go wonky, it would be really handy to add this high res. capability to every logger. It would also be nice to do this without breaking the bank:  I want the Pearls to be more like a Beetle than a Ferrari.

Another look…

I ignored thermistors initially because most of the tutorials I found repeated the same 10&10 divider recipe even though that combination results in a pretty crummy resolution of about 0.1 °C.   There were hints that you could do better by changing the value of the series resistor, but that information was obscured in the forums by mountains of stuff about shifting the point of inflection in the thermistor response curve around.  These seemed to focus on bringing the response curve close enough to linear that  slope/intercept formulas could be used, avoiding the Steinhart-Hart equation.

Eventually I found this post over at electronics stack exchange, which suggested that you’ll get the best overall resolution by setting your series resistor to the geometric mean of the thermistor resistance values that bracket your temperatures of interest:GeometricMean

I knew that my target range was 20-40℃, but when I tried to find the data sheet for the cheap 10k thermistors I had in the parts bin, I discovered that Electrodragon provided only three temp/resistance pairs [ -40℃ /190.5kΩ,   12℃ /18.1kΩ,   65℃ /2.507 kΩ ]  and an unusually low beta value of 3435, which did not seem to agree with the part number.

Fortunately for me the people at Stanford Research Systems Inc produced an online calculator that only needs three temperature resistance pairs  to generate a set of Steinhart constants:   

SRScalculator
The calculated coefficient of 3880 convinced me that the website had a typo, and that these were probably just standard 3950 NTC’s.  With that beta value I could find the resistance values that bracketed my range with the NTC Resistance Calculator over at Electro Tech:

NTC calculator top

Using the geometric mean method with those two temps suggests that my optimum series resistor would be 8179 ohms.  Plugging that and the pro-mini’s 3.3v rail voltage into the next calculator provides your divider outputs:

NTC calculatorbottom

So the delta between those two targets is 1.03 volts, or 31% of the pro mini’s default ADC’s range. That’s an improvement over the 10k ballast, however 0.09°C/LSB still isn’t enough write a blog post about it.

JippiesEquation

This is the general case equation which lets you use any arbitrary Vcc & aref combination.

But in that same StackExchange post user jippie explained that if you power the thermistor bridge from the rails, but set aref to the internal 1.1v band gap, (with  analogReference(INTERNAL); )  you can use significantly more of your ADC’s range.  Putting the thermistor on the high side (see Vout2 in the diagrams above) means the divider voltage rises with temperature, and it reaches the 1.1 aref when the ballast value is 1/2 of my lowest target resistance; which in this case is 4348Ω at 45°C.  That’s would mean a serial resistor of 2174 ohms, or the nearest standard 1% value of 2k2 unless I wanted to go hunting for a perfect match with IN30TD’s non-standard resistor calculator.

Checking those endpoints again with the ElectroTech Calculator:

15C (T min) 45 C (T max)
15837 Ohms 4348 Ohms
Vout2:= 0.4 V Vout2:= 1.1 V ( max )

So the delta is now only 0.63 volts, but after the aref reset this represents 57% of the ADC’s total range.  On the back of the envelope that’s 1024*.57 = 583 bits spread over 30 degrees = 19.4 bits/°C ≈ 0.05°C/LSB.  At the beginning of the post, I mentioned that most of the 12 bit IC sensors offer a resolution of 0.0625°C/LSB and now we have comparable resolution with the Arduinos 10-bit ADC, and a couple of penny-parts.

In fact I don’t reach 0.0625°C/LSB till the temp falls below ten degrees:

SweetSpotThe trade-off here is that we are far from the ‘optimum linearity’ point, so the true resolution of the measurements changes significantly as the temperature falls, which probably causes a heap of trouble for some types of analysis.  I am also throwing everything below 0°C under the bus, but since my loggers are going to be deployed under water, anything below freezing will cost me more than just temperature data…

I set up a quick test of this configuration with a MS5803 to provide a reference line for comparison:

OVERreading_selfHeatingjpg

Y axis=°C temp.  Most of the jitter in the thermistor line is an artifact of the S-H calculation.

Yikes! I didn’t realize that thermistors can have significant self heating problems when you use small series resistors.  Electro Tech have a handy plotter that shows how much power you are dissipating (in mWatts) through your thermistor at the different temperatures.

PowerDissapation

A typical dissipation constant for a small glass bead thermistor is ~1.5 mW/°C and some ceramics go up to 7 mW/°C.  With a consistent 1°C positive offset, I was probably driving too much current through my thermistor. But when I tried switching up to a 100k NTC /22k series combination, they all gave me consistent under-reading problems. It seems that Arduino’s ADC has trouble filling it’s sample & hold capacitors if you connect inputs with more than 10K impedance, and I was more than twice that.  (…though in all fairness I should also admit that I was also pushing the prescalars around…)

Self heating is somewhat less of a problem if you can cut power to your thermistors when you are not reading them, and I will need do some experiments there. I’d also like to speed up the ADC: keeping mcu up time to a minimum, and that makes me want lower input resistances.   Interestingly there are some sensor applications that take advantage of thermistor self-heating for air/water flow detection. 

Cutoffw40KSo choosing my series resistor ends up being a balance between different factors: Self heating, impedance, and in this case, keeping the divider output below the 1.1v Aref with a 2/1  ratio.  I eventually settled on using a 40k series resistor as a pullup, with the thermistor (hopefully) keeping the input impedance low enough to prevent under reads.  Flipping  the arrangement meant that now the voltage divider would hit the ADC’s 1.1v maximum when the temperature fell below 10°C. At that point I will have to fall back on the crude temperature record from the RTC.

Using the the internal 1.1v means that the ADC relies on the stability of the 328’s bandgap, which often gets panned in the forums.  But it seemed to have reasonably good thermal stability in the 20-45C range I’m after (Figs 31-34  pg 335) and I’m curious how bad that really is compared to something like the LM4040 if you didn’t also shell out for expensive high stability 0.1% resistors to go with it.

DividerBridge_byJasonSachsRoutine maintenance:

Most thermistors are only guaranteed to be within ±0.2°C absolute accuracy over a limited temperature range. While I don’t expect that much from these thermistors, I do care about the consistency of the readings over time. Jason Sachs over at the Embedded blog describes how a simple three resistor bridge can monitor your ADC’s Offset and Gain.  With 1% tolerance resistors you can auto calibrate to ±0.02% of fullscale and heck, who uses A6 & A7 anyway. Right?

Then it’s a matter of:

Gain = ( ideal VrefH ideal VrefL ) / ( ADC Measured VrefH – ADC Measured VrefL )
Offset = ideal VrefL   ( Gain * ADC Measured VrefL )
Corrected ADC reading = (Gain * raw ADC reading) + Offset

Unfortunately, I don’t have direct access to the internal 1.1 Vref, so I can only use this technique with the external 3.3v, and then find some way to convert the readings?

Riding the rails:

With the thermistor between the rails & the ADC using the internal ref, the difference between those two is important, especially if Vcc changes but the bandgap does not.  Retrolefty & Coding Badly worked out an elegant bandgap based method to monitor the line voltage so that you can compensate for variations. (especially in battery powered systems)    If you don’t want to use their capacitor method to pin down your chips internal vref, the folks at OpenEnergyMonitor  produced a utility called CalVref.ino that calculates the bandgap voltage by comparing it to DVM readings.  As this needs to be done when the logger is powered by a computers wandering USB line voltage, it is probably a bit less accurate than the capacitor method.

Both seemed to work well enough for me, though they did not always produce the same number(?)  Fortunately, I just want to know the relationship between main regulators output and the internal bandgap voltage, so the ‘true value’ is not critical and I can just insert 1100mv in the RL/CB code.  The resulting Vcc gives me a conversion factor (BandgapVcc / 1.1v) which allows me to adjust the 3.3v reference bridge readings to their post 1.1v changeover equivalents.  Then I can use a modified offset value to correct the thermistor readings after my 1.1v changeover:

Actual ADC w 1.1aref = (Gain * raw ADC read) + [(Offset@3.3v) * (BandgapVcc / 1.1v)]

With TL431’s being so cheap, it would be reasonable to ask why not use them as a reference instead?  Their 1mA minimum current is a bit of a problem for data logging applications, and the dynamic stability that they were designed for prevents you from trying certain oversampling techniques. (… more on that later …)

After many run tests, my experience of this reference ladder approach is that it gives you good gain correction, but at first it seemed to be somewhat less reliable providing ADC offsets.   Even after getting the series resistors sorted out, it still took a batch of process of elimination trials before I realized that with cheap thermistors the majority of the offset is due to variation between the sensors.  In my case this effect was several times larger than errors from the ADC offset, and you can only figure out what the individual thermistor’s offset value is by calibrating against a known reference…and even then  it’s probably not linear…    Of course if you buy interchangeable thermistors with closer tolerances, you quickly reach the price of high resolution IC sensors.

And the result:

Once you’ve plowed through all that you can convert your corrected ADC readings into temperatures using the Steinhart-Hart Formula.  It requires the preliminary step of calculating the resistance of your thermistor, and there is a brilliant explanation of that over at ArduinoDIY, which ends with:

Rntc = Rseries * ((ADCmax/ADCreading)–1)       // with Rseries connected to ground
Rntc = Rseries / ((ADCmax/ADCreading)–1)       // w Rseries in pull-up configuration

And then you pop the calculated resistance value into one of the many code examples out there like the one in Adafruit’s thermistor tutorial though I prefer to do all that later in a spreadsheet to save memory & power on my loggers. (Not to mention the calculation errors that I usually make on the Arduino…)

I did a test including a 24-bit MS5803, a 12-bit MCP9808 and the thermistor so I could compare the output:

ThreeSensorRunThermistors are really twitchy due to their low thermal mass, so I did this test inside of a large ceramic pot with a lid to smooth out the changes.  At first bounce I though that jitter on the thermistor line was due to poor resolution, but it turned out to be an artifact of  the calculations I was doing in Excel.

When I compared the raw output of the Arduino ADC and the 9808:

Though the left axis is inverted for the thermistor, the scale on both is the same showing that the effective resolution is better than the 12-bit sensor. (click the image for a larger version)

Though the left axis is inverted for the thermistor, the scale on both is the same showing that the effective resolution is better than the 12-bit sensor. (click the image for a larger version)

So perhaps correcting the initial 3.3v VrefL offset reading with that Vcc ratio was not such a good idea, and I should avoid mixing different resolutions by taking a post 1.1v read of VrefL for offset correction.  Even if that is the case, tracking the positive rail still seems like a good idea for a data logger, so I will add it to the once per day events that get triggered by the 24 hour rollover.

So the job’s done with  four resistors and bog standard 10K NTC right?

Uh uhh… In fact this is just the stuff I had to get a grip on before starting my quest for the ADC holy grail.  I didn’t want something as good as the 12-bit sensors, I wanted something better, and the semi-mythical technique of oversampling promised to deliver all the resolution I could ever want from a humble Arduino… in theory

But this post is already miles too long, so for now I will just leave you with a teaser from a recent run-test showing output from an 24-bit MS5803 vs256 sample average using the Arduino ADC and that same 10k NTC thermistor:

Teaser_14bitditheredjpg

Y axis=temp in °C    Note: I have offset the curves here for easier visual comparison. The resolvable feature size is already well below 0.01°C and I am sure that I can push that a bit farther…

You have to throw in another resistor, and a couple of capacitors, and I still have some niggling details to work out optimizing the technique to use the least amount of power. When I get all that sorted, I will post the gritty details…

Addendum 2016-06-23:

There is a thermistor based Compost Sensor project by kinasmith at Instructables which uses wireless Moteinos and a cellular module to relay the data. Cool stuff.  Also there is a discussion of the lookup table method to address the accuracy of your thermistor readings (which I did not really talk about in this post) over at Mike’s Lab Notes.

Addendum 2016-08-01:

In this blog post, Ejo puts an ADS1115 / thermistor combination through its paces, using a combination of single and differential readings to remove voltage bias.  His resolution  reached 0.00427°C    And here is another group combining the ADS1115 with a bridge.

Addendum 2017-02-27:

Well it took a lot longer than I expected, but I finally got the post on How to do Oversampling with an Arduino out the door. The pin toggling method I’ve come up with is pretty darned easy, and gives you access to at least 4-5 more bits of ADC resolution.

Addendum 2017-04-26:

I’ve moved on to calibrating the thermistors, and in the process I learned that it’s probably not a good idea to combine the 1.1v aref, with the oversampling method. But I did it anyway.

Addendum 2017-09-26:

You could get another bit of hardware resolution with a two element varying bridge and then doing a Pseudo-Differential reading with the Arduino ADC.  I still haven’t wrapped my head around the math for that yet, which would get tricky if you were simultaneously using 1.1v aref – since your bridge could not be symmetrical.