Field Report 2015-08-10: Diagnosing a TMP102 Sensor Failure

Over the next few days we did cave dives to exchange units at sites along the coast. Some of these systems were too deep for our camera, and in the shallower systems the visibility could get a bit grim because if you stay in one place too long, air bubbles rise up and cause “percolation” of little pieces of rock from the ceiling.  So to dismiss any illusions people might have that science diving looks like what they show you on TV, here’s a clip from one of those claustrophobic installations:

Even so, things went well.  Each night I had a new crop of loggers to download and check for my “standard set” of parameters (battery, RTC coincell, sleep current, clock drift, etc.) which I compared to the readings taken before the deployment. The recent retrievals were from the last generation of 3-inch pvc builds, and while they all looked great on the outside, some of them stopped running long before we pulled them out. It can be tricky to figure out the problem when a logger has several I2C sensors on it because the very things that make it so easy to use those breakout boards, also hide problems from you: essentially they don’t just run digitally, they also tend to die that way.  Issues tend to show up much earlier in the analog data, and one useful diagnostic is the voltage log from the main battery pack (read with a simple resistor divider).  Alkaline AA’s usually provide a nice smooth decline over time, so when I see something like this:

011_PS&Temp

Unit w BMA180, TMP102, HMCL5883l. Deployed in fresh water at 5m.

…it’s an immediate red flag.

But unit #011 had a complete set of data logs, with no hint in the sensor records that something strange occurred.  That peak in temperature was pretty large, but it also corresponded to the June 13/14 event that showed up my Rio Secreto & Akumal Bay records, so I accepted it as valid data.  My next thought was that the unit had suffered a μSD failure, but this logger was still successfully going to sleep at a reasonable 0.26 mA. Rapidly entering a low current sleep state is the best indicator I’ve found so far that a card’s control circuitry is still working as it should. Hmmm….

Then next unit with problems was also from a relatively shallow fresh water deployment, and it ran for only two weeks before shutting down:

BMA180, TMP102, HMCL5883l, Deployed in fresh water at 14.5m Depth

Unit w BMA180, TMP102, HMCL5883l.  Deployed in fresh water at 14.5m.

Leak corosion after sensor failure drained the batteries

The power module from #015:  alkalines almost always leak if something pulls them below 0.5v.

This logger was positively pressurized when I broke the seal, with only 0.22 volts at the battery module. When I connected a new battery the unit would did restart properly, producing a strange pattern on the indicator LEDs. Fortunately even with only two weeks of data, what we did have in the log make it pretty clear who the culprit was. It was extremely unlikely the temperature would have spiked like that on the same day that the battery curve took a nose dive.

The next unit presented another mystery, because it had a nice smooth power curve though it had reached it’s low voltage shut down point too early. The temperature data looked reasonable:

Unit w BMA180, TMP102, HMCL5883l. Deployed in fresh water at 22.5m Depth

Unit w BMA180, TMP102, HMCL5883l. Deployed in salt water at 22m.

These loggers have dual 3xAA power packs, and they normally deliver at least eight months of operation. While #016 seemed to restart OK, with a normal (~o.24mA) sleep current, it then crept up to 1.65 mA.  I left the unit running for about 5 minutes and by that time the sleep current was jumping all over the place, varying from 0.5mA to 4 mA at random. This behavior continued after I swapped in a known good SD card, so I was stumped. Once again the data in the sensor logs looked fine.

I had one more unit from this build generation to open up, and it too had developed a problem over the deployment:

017_PS&TEMP

Unit w BMA180, TMP102, HMCL5883l. Deployed in salt water at 24m.

No rust at all...?

I have to trim the boards a bit to fit them in the sensor wells but I am careful not to cut any traces or vias. There was no visible rust on those breakouts at all.

That hockey stick bend on the power curve was another smoking gun, especially in the context of that temperature data. The deep saline water in these caves is remarkably stable, so when I saw a temperature rise alignment like we had with #015, I started pulling wires. Sure enough, the sleep current problem on #016 disappeared as soon as the TMP102 was disconnected and #015 fired up properly as well.

Since epoxy failures took out a fair number of units in 2014, I did a detailed inspection of the potting, but found nothing to indicate that water had reached the sensors. Since the deeper units were most affected, my current working hypothesis is that these sensors were actually suffering from the effects of pressure, rather than moisture. (…but good luck finding that in your ‘Recommended Operating Conditions’)

Anyway, I disconnected the TMP102’s from all the loggers of that generation, (including the ones that were still working…) and set the code to read the internal RTC temperature register instead. Test runs with that were good, and these units went back out for deployment a few days later even though all that housing mass will put a huge lag into the temperature record.  I don’t think I will be using TMP102 sensors again on my projects, but of course there is nothing to say the MCP9808’s from Adafruit won’t develop a similar problem over time.  So I might end up back at the humble DS18Bs which I used on my earliest builds:  they have proved to be remarkably resilient over time.

Addendum 2016-03-17

The MCP9808’s did indeed suffer a similar fate.

 

This entry was posted in Expedition Reports & Updates, Lessons learned.. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s