Category Archives: Test results & Data analysis

Status updates & ongoing performance trials

Another look at logger battery discharge curves…

I’ve written before about how the main battery discharge record is one of my primary tools to determine what happened when a logger fails, but I thought I would try another kick at that can; delving into some of the things that show up on units that are apparently running OK, but still leave me scratching my head.

I did not really know how to solder at the beginning of the project, so I was using Tinyduino based platforms that drew sleep currents between 0.5-0.6 mA. I was running them right from the battery with no voltage regulation, and using the 328’s internal 1.1v band gap trick  to track the power supply:

Battery Voltage (mV) with

Battery Voltage (mV), 2 x 3AA battery pack, Tiny Duino based logger

These units buffered less than half a day of readings to the small 4K eeprom on the RTC breakout,  so I attributed most of that 100 millivolt  jitter to the frequent SD card writing. Some have pointed out that the internal voltage reading trick itself can suffer from as much as 10% variability from one processor to the next, with additional thermal variations. But at least those curves were predictable, with a 6 cell (2x3AA) pack lasting longer than the four months between fieldwork trips.

The need to provide regulated power to an increasing number of sensors encouraged me to switch over to 3.3v pro-mini style Arduinos,  and adding a 32K external eeprom extended my buffering out to several days. This smoothed the power curves dramatically, and on good builds I now see voltage records like this:

Cave Pearl data loggers

Battery (mV), Rocket Ultra logger, 3xAA Battery, 0.19mA sleep

The steps on that curve correspond to the limits of the ADC (with a 2 x 4.7MΩ divider cutting the battery voltage in half), so that’s as good as I’m ever likely to see. The thing that has me curious is that I also see curves that look irregularly bumpy, and have other intermittent features. For instance, some of the loggers built from the same parts, with the same code as the one above, give me a clear pattern of 100mV dips whenever the data gets transferred to the SD card:

Battery (mV), Rocket Ultra, 2x 3xAA supply

Battery (mV), Rocket Ultra logger, 2x 3xAA Battery, ~3 day buffer

As you see the depth and duration of the dips vary somewhat, although the amount of data being written is always exactly the same, which makes me wonder about random latency delays. And that curve also has a mysterious low voltage anomaly showing up around 9/20 that lasted for the better part of a week.  I see stuff like that in about 15% of the loggers that give no other sign of problems, have low sleep currents, and decent operating lifespans. And this is definitely not temperature related, since most of the cave loggers see less than 1°C variation over a deployment.

Other times I see hump shaped features in the curve that seem to indicate that the problem occurred long enough to pull the batteries down significantly, but then went away, and returned again some time later:

Battery (mV), Moteino based logger, 3xAA supply, 0.22 mA sleep

Battery (mV), Moteino based logger, 3xAA supply, ~5day buffer, 0.22 mA sleep

I took this unit out of service, and am running tests on it now to try to identify what happened. Again there was no hint of a sensor failure in the data, and the sleep current still looks the same as it did at the start of the deployment. I’ve seen this hump/step pattern a few times now, and for all I know it’s an issue with the batteries hitting some kid of threshold, rather than the logger. 90% of my deployments use alkaline AA’s, as I usually catch them on some kind of promotional sale at the local hardware store. I rarely use lithium AA’a as they are more expensive and I’m not convinced they give you longer operating life in low current applications.  They also fall so precipitously at the end of life that I am concerned they will brown out in the middle of an SD card operation, destroying my data before the code has a chance to intercept the low voltage problem. (…and they are not allowed in air travel luggage …)

Probably the longest running power curve mystery are the events that only show up once every month or so. These hits are so widely spaced that they don’t seem to be directly related to the buffering & regular SD card writing events, and I have yet to find anything else in the code that explains them:

Battery (mV), Rocket Ultra, ~3day buffer, New file @~15 days

Battery (mV), Rocket Ultra, ~3day buffer, New file @~15 days

The timing of these long gap events can be regular, as shown above, or more variable. My current hypothesis is that some threshold is being passed inside the SD card controller triggering some kind of huge load balancing event that hits the battery with a sustained power draw much larger than any of the normal logger operations. This makes me wonder if the battery packs would benefit from a parallel supercap to buffer those pulsed loads.  I suspect that the SD cards are responsible for most of the power curve anomalies that I’m seeing. All the loggers are now using Sandisk’s MUVE music SD cards that quickly go into low current sleep, so I am still left wondering what the real cause is.

Anyway, I’d love to hear comments from others who have seen curves like this in ‘otherwise normal’ equipment.

Addendum 2016-03-02:

Just thought I would add another interesting one to the batch. When I build a new logger I like to put them on a shakedown test for a week or two, with rapid sampling and small eeprom buffers to make sure the SD card sees lots of activity to test it out.  So this is in no way a “normal” operation power consumption curve like the ones posted above but it does show another interesting phenomenon:

This unit was taking a sample every 5 minutes, and only buffering 20 readings to the eeprom.

This unit was taking a sample every 5 minutes, and with only 20 readings buffered to the eeprom those numerous 100mV spikes are from the frequent SD write events.

This was a continuous run, so the code did not change even though there was a fairly dramatic shift of the pattern in the voltage readings. Usually I see this kind of thing in reverse when a sensor fails (ie: a smoother curve becoming more and more variable as the sensor goes down).  I have no idea what caused this unit to become more regular as the run progressed?

Addendum 2016-04-07:

Found anther one to add to the collection.  This flow meter is still in service, though the overall curve is crunchy enough to make me suspicious that something is going on, the logger has not replicated the central up/down anomaly in any subsequent deployments:

Caption

Battery (mV), Rocket Ultra 8mhz, ~5day buffer, New file @~15 days

Addendum 2016-04-07:

I monitor the main AA supply with a simple  2 x 4.7MΩ divider  on an analog pin, but Dangerous Prototypes spotted a low voltage indicator circuit that can be built from a couple of comparators. This could come in handy for builds that don’t have enough pins left to keep track of Vbat. This would be handy for minimal builds with AT Tiny’s. There is also some interesting background info on battery discharge curves with boost converter in this TI appnote.

Addendum 2016-07-30:

Reviewing the data from the latest round of fieldwork, and have another interesting power curve to add to the collection:

Rocket based logger,

Rocket Ultra 8mhz based logger, 3xAA, 15min sample, 512 records buffered to eeprom

That’s the first time I’ve seen higher voltages appear in a record where the logger was in a stable thermal environment, and it makes me wonder how that might of occurred. Since there was no change in the cave temperature, I suspect that the voltage regulator on the Arduino had a few low voltage events, and this messed with Aref enough to create artificially high readings on A0.  All the other data in the log appeared to be fine, but I will be keeping my eye on this logger…

Addendum 2016-09-12:

shottky1n5817This is a bit of a tangent, but I’ve been using bog standard 40v 1A Shottky 1N5819 diodes to isolate the 3xAA battery banks on some of my loggers, and I just noticed in the datasheet that 20v 1N5817’s have about 1/2 the Vf. And I just found out that with beefier 3A diodes, I can probably bring the voltage drop even lower during SD card write events, which could push the logger below the 0.1 A line at the bottom of this graph. This makes me wonder how much of the voltage dips I’ve been seeing are a real effect of the load on the battery, and how much is simply an artifact of the isolation diodes?

Addendum 2016-10-17:

1n400x15Continuing on the diode tangent, I’ve recently come across a few mentions of powering Arduino projects with 4xAA batteries in series, using a 1N4001 diode to drop the voltage into the 5.5v range you see on some 3.3v Arduino regulators. I have yet to find any tests of exactly how much power you would be wasting over the diode, vs the extra power you could extract by taking 4 cells down to about 0.85v  than you could with 3 cells in series (~1.16v) . The circuit would cause a voltage fluctuation of about 0.2 volts but I don’t know if that is worse than the Shottkys, as their datasheet does not go down to very low current levels. It’s worth noting that the MIC5205 on the ProMini is rated to 12v input but that comes at the cost of another 0.05mA sleep current which would burn away the benefit of that extra cell fairly quickly.

Addendum 2016-10-27:

EEVblog #140 – Battery Capacity Tutorial
EEVblog #772 – How To Calculate Wasted Battery Capacity

That last one makes me  less concerned about the LDO regulators relatively high 3.5v cutoff when using 3xAA’s, but even more concerned about the curves that show a 100mv pulse discharge drop during SD writing. If those events trigger the low voltage shutdown too early that would loose a significant percentage of the overall battery capacity.  Perhaps the best solution would be to switch to the MCP1703 regulator which accepts up to 16 V, then I could use 4xAA batteries and avoid that diode loss problem. It is available with 3.3V output (MCP1703-3302E  & the SOT23-3 boards to mount them)  although some people have had problems with the 1703 in low power applications. Having the extra input voltage space would let me try putting 3.6v lithium cells in series, which you can buy in “D” size with a whopping 19Ah. Of course with  lithium discharge curves being so flat, I might get away without the regulator at all: I just don’t know how sensitive the SD card would be to that drop at the end, and it’s not worth risking data to find out…

Addendum 2017-02-23:

I finally started tracking down those SD card wear-leveling events with my Uno-scope. It’s hard to catch ’em because they don’t get triggered very often, but after setting a unit to constantly loop through a repeated sequence of short data saving events I managed to snag one:

This is just a brief one, but perhaps others last long enough to cause one of those dips in the voltage log?  I read the battery voltage before and after the data saving, so if the SD card does that kind of housekeeping after the file close event is complete, then SDfat has released the code block and I could be doing that second battery read while the SDcard has the current drain at up near 60 mA or more…

 

 

 

 

 

 

 

How the drag enhancer affected flow meter response

Float configuration deploymnet on new housings

A float configuration deployment with flag on one of the deep saline units.

In March we did an experiment by adding small flags to some of our flow sensors to enhance their response in low flow situations. As with so many of the things we have tried on this project, these thin sheets of ABS were the best solution I could come up with that would (1) flat pack into our luggage and (2) be assembled in the field with zip-ties. One of my pet peeves with commercial equipment is that much of it fails the suitcase test, which can be an  important part of trip logistics.

Now, I’m not even going to pretend I have the kind of skills it would take to estimate drag on those fins, which present a progressively smaller surface as tilt angle increases.  In fact, I probably know less about math than I knew about electronics when I started this project. So everything that follows here is just me just muddling through like always.  If you actually do possesses those skills you should probably look away now. I’d hate to be responsible for another academic drinking themselves into oblivion, while muttering about the internet being taken over by monkeys.

Without an expensive accoustic dopper unit to calibrate against, the best we could do was develop an empirical relationship between the new design and old ones.  So we installed one of the “enhanced” flow sensors beside a similar unit with no flag. Comparing the two data sets would show us how much the flag was increasing the sensors response to water flow.  Since we had no idea what that low end amplification would actually turn out to be, we used a tidally controlled coastal outflow that went from zero flow to peak velocities above 10 cm/s twice a day.

Fortunately, a good storm passed over the system right in the middle of the deployment: pushing that range even farther (probably to about 20 cm/s)Here is a small snippet of data covering that event:

Comparing the drag enhanced, vs the standard configuration
My first impression was that the boosted diurnal response looks like the kind DRC plank that smacks you over the head whenever you turn on a radio these days.  The low end is being boosted by a huge amount, in fact, just before the event there are some spikes in the flagged data there that don’t don’t even rise above the noise floor on the “naked” flow sensor.  Just looking at those tells me we had between 3-4x more signal at the low end. But how do I quantify that?

I started with a plot of the two sensors against each other, which showed a sharp point of inflection: Flag vs No Flag_withExcelFitLine
Note: Logger #012 had the drag enhancer attached, while C4 had no flag.  The loggers bodies themselves presented very similar, somewhat spherical, profiles to the water flow.  My newer builds are cylindrical, which opens another whole can of worms.

I was happy to see that the low end boost looking so linear and I wondered if that elbow was some kind of turbulent flow transition. Who knows, perhaps when the loggers approached sixty degrees  the fin even starts to contribute some lift. (?)  But in terms of relating one units response into the other, even I could see that Excel’s trend line was terrible. You can do better with the solver plug-in, but you have to know the equation you want to use first. If you don’t know what the formula is, it can be a tedious process to figure one out from scratch.

So I went looking for something that would give me a better way to model that relationship. That plot looked like a distorted “S” shape, and google image searching lead me to the entries on logistic functions of the form:  f(x) = a / (1 + b c –x)   These sigmoidal curves start out with a low slope, which increases to an inflection point, then levels off as they approach a maximum value. They pop up frequently in natural systems when people try to model population/cell growth, or EC50 dose response. The Gompertz function was a long-tail variant that also looked like a good contender.

First pass with Eureqa

First pass with Eureqa: meh!

While I was digging through all that, I came across references to a statistical modeler called Eureqa that was developed in Cornell’s creative machines lab a few years ago. I’d seen mention it before in the geek press, but this was the first time that I had a situation where it might be useful to me personally. So I downloaded their free trial version and day-am!  This slick bit of code made me feel like a ten year old who’s been left alone in the cockpit of some large piece of earth moving equipment that still has the key in the ignition. Clearly this was a tool for real scientists, and I should probably wait for that adult supervision. But…well…I’ve been failing that kind of marshmallow test for quite a while now.

And I didn’t get much out of it at first, but after going over their tutorials I found it was fairly easy to change the generic y=f(x) starting point to any model you want. This lets you can derive arbitrary constants from a really disorganized lump of data without having to do all that grunt work in Excel.  I did a couple of runs with the Logistic function, and with the Gompertz curve:

Modeling with Eureqa, starting with Logistic and Gompertz functions

Note: the functions specified with empty brackets: f1(), f2() etc. force the solver to put a constant at that location

My raw data did not really have (0,0) point due to sensor mounting offsets, and the loggers never went beyond 75 degrees of deflection. But I found that by adding an arbitrary point at (90,90) I could move the upper asymptote away from that bulk at the top end of the plot. After seeing the improvement from that change, and after deleting a few outliers, I let Eureqa take another shot from the default y=f(x) starting point:

Eureqa_TweakedPass2

It's turtles all the way down!

Turtles all the way down…

Now that’s starting to look better, and I was not selecting the very “best” solution according to their fit metrics. If you leave the solver running for a long time (say, while you go have dinner…)  it  just keeps chugging away, adding coefficients until you have something large and ugly. But I am sure that if I actually knew what I was doing, the correct solutions would jump right out at me. The press often overlooks this critical step with their hyperbole about Eureqa “replacing” scientists: the real world is not a simple pendulum: it’s a warm, squishy, mess that involves a lot of value judgment.

More electrons will give their lives as I burn through my 30 day trial. It’s too bad Nutonian wants $30/month for their cheapest license. If they went with the WordPress model (ie: $30 a year for the little guys) I’m sure every maker in the world would be using this software to sort out one-of-a-kind build issues like this.  Of course, I’m not sure this exercise actually taught me anything about the physical phenomenon involved. But if blindly applying complex, statistically derived equations is good enough for Wall Street, then it’s good enough for me. What could possibly go wrong?  🙂

And…Happy Birthday to my brother Mike!
As a seasoned Linux system bender, he was one of the first people to bring the Arduino / Open Source Hardware phenomenon to my attention.  And he is also someone who knows how absurd it is for me to post on anything mathematical.  

 

 

 

Testing Power Use with Complex Datalogger Duty Cycles

There is an old saying that goes: “Yesterdays solutions are today’s problems” and I think that now describes this project. You see the first year of development was focused on pretty straightforward hardware issues, and solving each one produced significant gains in performance. Now that I am consistently seeing sleep currents in the 0.1-0.2 mA range (with an SD card (~80uA) & live Adxl345 (~50uA) along for the ride), I am hunting for more elegant ways to extend the operating time while maintaining the simple three component core of the original design. With a 3xAA power pack now providing almost a year of operation for some sensor configurations, I also have the task of developing a method for testing the loggers that can discriminate between subtle code changes with relatively short runs. But artificially reducing the sleep interval between samples distorts the result enough that it’s hard to make good projections. I am slowly coming to realize that testing & calibration are the real heavy lifting when you build any new device.

The new A544 cells arrived at > 7 volts which was too high for the regulator on the Ultras. So I took them down to 5.6 volts with a Zenner. The rare earth magnet soldered to the diode wire gets zapped by the heat from the iron, so you need a second little magnet to hold each battery connection securely. You can also stack a "set" of button cells with these magnets, giving you more options for low power tests.

The new A544 cells arrived at > 7 volts which was too high for the regulator on the Ultra. So I took them down to 5.6 volts with a Zenner that stops the discharge before it goes too far. The rare earth magnet soldered to the leads gets zapped by the heat from the iron, so you need a second little magnet to hold each battery connection securely.

Each new trick I try, like finding another place where I can put the cpu to sleep, adds complexity to code that has “once per day events” and “once per week” events, and soon there will be “only if the delta between two readings is greater than x” events. Most of these are so short that a multimeter can’t catch them but even when a friend donated an old Tektronics to help me try to get a handle on the duty cycle, I faced the challenge of displaying currents ranging from less than 0.1mA to 80mA SD writes with variable duration. To make things more interesting, some of the cheap sensor boards I have been noodling around with have components of unknown origin & dubious quality, which introduce yet another set of variables.

Even with my mediocre scope-skills the forums had convinced me that the SD card was the elephant in the room. So I tried to avoid SD use by adding an external 32k eeprom which let me buffer five or more days worth of data before having to fire up the external storage. Problem solved…or so I thought. I was quite surprised by data from the last deployment that showed using this approach to reduce SD writes by a factor of five only delivered a 5-10% improvement overall.  I had overlooked the fact that the AT24C256 eeprom pulls 3mA for five milliseconds per pagewrite. This was nearly as much current as the Rocket Ultra I was using, not to mention a significant extension of the cpu uptime for multi sensor units that were buffering up to four eeprom pages per record. All of that activity adds up.

So I took another look at buffering data in SRAM, which I flirted with at the beginning of the project. But my script was now much larger than those early versions, leaving barely 500 bytes free.  I know the real coders out there probably laugh at my use of Pstring & Ascii but that lets me add a new sensor by changing a couple of print statements, and adaptability has always been of my primary design goals. To maintain that simplicity I went searching for an Arduino with more headroom and the Moteino Mega over at LowPower Labs seemed to fit the bill with it’s 1284P offering an extravagant 16K of sram (compared to just 2K on the 328p). It also used a low drop out MCP1700 series regulator like the Ultras, and there was support for RFM transceivers.  With the Mega’s larger footprint, I decided to try them first on the larger dry cave platforms:

Rocket Ultra (left) VS Moteino Mega (right) loggers with pin powered RTCs. I break out LED, I2C and one-wire with Deans micro connectors, and you can see the 4.7K one-wire pullup above the main power supply divider on the Mega. The 32K eeprom is tucked under the RTC, which is inverted on the Moteino build to make changing the coin cell easier.

Rocket Ultra (left) VS Moteino Mega (right) based data loggers with pin powered RTCs and 2×4.7MΩ voltage dividers monitoring both the power supply voltage and the rtc backup battery.  I break out LED, I2C and one-wire with Deans micro connectors, and you can see a 4.7K one-wire pull-up above the main divider on the Mega. A 32K eeprom is tucked away under the rtc breakout, which I flipped over on the Moteino build to make it easier to change the CR2032.

For a standardized test, I set both loggers buffering 96 records (= one day @ 15min intervals) in drip sensor configuration. I added the I2C eeprom to the Moteino logger to make the builds as similar as possible, but it does not get used. Instead I store the raw sensor data in integer arrays. So there is no Pstring/ascii use on the Mega logger until I write the data to the SD cards.  With matched acclerometers & cards, both loggers sleep at 0.18 mA so the the only difference between them should be the data handling.  One thing I did not catch from the LowPowerLab specifications was that the 16mhz Mega draws ~12 mA (while awake) in this configuration as compared to the Ultra builds which perk along at just over 4mA. I figured that with SRAM storage the mcu up time would be so much shorter that it would not matter.

With super caps to buffer SD write events, you can drive the loggers with a very small battery. Rare earth magnets let you connect to the ends without a holder and you can make a multi-layer magnet/button cell sandwich to build low power options at just about any voltage. Those are 5v 1farad supercaps in series, so I don't bother to balance them as they should be able to handle leakage asymmetry when the battery input is only 5.6 volts

With a parallel bank of super caps to buffer SD events, you can drive the loggers with small batteries that have high series resistance. Rare earth magnets let you connect without a holder and you can build multi-layer magnet/button cell stacks to create low power options at different voltages. Those are 5v 1farad supercaps so I don’t bother to balance them as they should be able to handle any leakage asymmetry when the battery input is only 5.6 volts. The graphs below had no low volt blips at all, so this series/parallel arrangement of 4 of them was probably more capacity than I needed.

I still had not sorted out the oscilloscope issues but I realized that I could flip the problem around: instead of struggling to display the effect of every little tweak to the duty cycle why not provide a fixed amount of power and just see how long the unit runs. It’s a data logger, so I already have a time stamp and a battery voltage reading with every record.  A couple of people suggested capacitors, but even a 1F supercap only gives you about 0.27 mAh per volt, translating into a few hours of operation for my loggers. I needed longer runs than that because the Moteino was going to loose the data in it’s sram buffer when the unit browned out (I can always dig into eeproms later for the last few records on the Ultra).  A bank big enough for multi day runs was going to be expensive, and is probably a hazard for my little bots.

Fortunately there are a host of small form factor batteries out there for things like fire alarms, medical devices, etc. Energiser’s A544 seemed to fit the bill at 6 volts & 150 mAh: promising to power the Pearls in “sleep current” mode for about 40 days. Even better, they were alkaline cells, so their discharge curve would be more like the AA’s used in real world deployments. There was some risk that these little cells would drop to the low voltage cutoff when the SD write current spikes occurred, so I added a few super caps to buffer those loads. I then set the units up on a book shelf where they would not be triggered and waited for my “baseline” load result.

This is the voltage record from the two different logger platforms, when they were powered by a single 150mAh A544:

A544_FirstPowerTest
(I stopped the test after a month, because I couldn’t take these suspense any longer. There were few sensor interrupts during the test, so this was a baseline power use comparison)

I was sure the SRAM buffering Moteino logger would come out far ahead of the Rocket build that was sandbagged by all that I2C eeprom traffic. But if you correct for the slightly higher starting voltage those two curves are so close to each other they might well have come from the same machine. So there is no longevity boost from SRAM buffering if I use an mcu that draws 3x as much current, but at least I now have a good way to test the loggers without waiting too long for results. This result also agrees with some of my earliest drip sensor results which hinted that the sampling/buffering events were consuming 2/3 of the power budget.

For the next round of tests I will put them on the calibration rigs to see how the A544’s handle the interrupts being triggered all the time. Presumably the Moteinos will draw more power there so I will need to normalize the results to match drip counts. To go beyond the conservative one day buffering I will need some way to capture data from the SRAM buffer before the units power down, so perhaps I will end up using the eeprom on those Moteino Mega builds after all. We will use a few Mega based drip sensors set for very long buffering (8-10 days?) on the next real world deployment. I also have a feeling that the DS18B20 temperature strings would benefit more from SRAM buffering than these simple drip sensors, as they poll up to 40 sensors per record. That’s a lot more data to shuffle around.

Addendum 2015-07-05

Hackaday just posted about [Majek] putting “live” data into Arduino’s flash ram (which is normally not accessible after startup) via a Optiboot hack.  This opens up another possible data buffering strategy, though I am not sure if it could handle the duty cycle of a long deployment. Or it might let you do calculations with the 328p that would otherwise run out of space.  So this is an interesting development that involves no extra hardware, which is usually good news for the power budget. I had already been wondering if calibration data could be stored in flash with Progmem, but that solution only works for data that is not changing all the time.

Addendum 2016-01-06

We finally have some data from the first field deployment of Moteino based loggers which store sensor readings in ram (array variables), rather than buffering all the data as ascii characters in an external eeprom like my 328p based loggers do.

Here is the power curve from a Moteino:

057-M_Drip_BatteryCurve

Battery (mV): 3xAA supply, 0.18 mA sleep current, 5 days of data in ram

And here is a directly comparable build using a rocket scream ultra with a slightly higher drip count (ie: number of processor waking events) over the duration of the deployment.

Cave Pearl data loggers

Battery (mV): 3xAA supply, 0.18 mA sleep, 5 days of data buffered to AT24C256 eeprom

So once again, these performance curves are so close that it makes no odds. But on the bright side, this confirms that accelerated testing with 150mAh A544 batteries does give me results that translate into real world. So this is still pretty good news even if the 1284’s did not deliver the magic performance bullet I was hoping for.

Addendum 2016-02-15

If I wanted something a bit beefier than the 150 mAh in the A544’s, I could hack my way into a 9v battery, and use half of the set of 500 mAh AAAA batteries you find inside. That would give me about 1/4 the power of the AA batteries I typically use on deployment.

Addendum 2016-08-15

I finally figured out how to view  individual logger events using an Arduino UNO as a DAQ with the serial plotter tool built into the IDE:

Cave Pearl data loggers  I’m quite tickled about being able to replicate a task that you normally would need an oscilloscope to see.  Of course my chances of actually catching one of those big unpredictable SD card latencies (from something like  age related wear-leveling) is still pretty low, so I will continue to use this A544 method for solid longevity predictions.

 

High (SD card) sleep current problem solved!

Time for some head-to-head comparison testing.

Wait…do I hear something?

The more I thought about it, the more the high power drain of the flow sensors got under my skin. How had I misjudged the performance that badly, after so many successful bench top tests? I was determined to get to the bottom of this mystery.  The simplicity of the “one sensor” drip loggers, which spend all of their time in sleep mode, meant I had a tool I could use for some process of elimination trials. So I made a few loggers that were identical except for the mcu and put them on my newly arrived µCurrent. But my initial results were all over the place. Some units slept at a nice comfortable 0.2 -0.3 mA, while others drew about 2mA, and a few pulled 5mA or more.

Mcu: Configuration: I(sleep)
Rocket Ultra 128mb SD , adxl345, LED, 2×4.7 MΩ divider 0.26 mA
Sparkfun ProMini 256mb SD , adxl345, LED, 2×4.7 MΩ divider 0.35 mA
Cheap Clone 256mb SD , adxl345, LED, 2×1.0 MΩ divider 4.66 mA
Tiny Duino 128mb SD & shield,LED *Bma250, 5883L, DS18B20 5.42 mA

That last one is my spare flow sensor, and it came in around 5 mA , apparently confirming the high power use seen in the latest field deployment data.

But five milliamp! Was that thing even sleeping at all? I started wondering if adoption of the Rocket Scream sleep library meant that some of the boards were not sleeping properly, so I dug up the older non-RS code and ran those…nearly the same results, with about 0.03 mA more current because because I was not shutting down the BOD in the older sleep routines. But that small consistent difference told me that the units actually were going into sleep mode between sensor readings, so I went back to my original thought that voltage regulators were my energy vampires.

Then, as I was juggling things around, I switched an “apparently identical” 128mb Sandisk microSD card from the Rocket Scream unit into the TinyDuino flow sensor. And suddenly it dropped from an abysmal 5 mA sleep current down to less than one milli-amp.  A furious shell game followed and after locating the “best” 128mb microSD card, and using it (and the same battery module) the test units delivered:

Mcu: Configuration: I(sleep)
Rocket Ultra 128mb SD , adxl345, LED, 2×4.7 MΩ divider 0.24 mA
Sparkfun ProMini 128mb SD , adxl345, LED, 2×4.7 MΩ divider 0.27 mA
Cheap Clone 128mb SD , adxl345, LED, 2×1.0 MΩ divider 0.30 mA
TinyDuino 128mb SD & shield , LED *Bma250, 5883L, DS18B20 0.90 mA

So counterfeit SD cards were causing the high sleeping currents!

Looks like I have been sold several batches of bad cards from eBay, which do not go into the low current sleep mode I was expecting from the Sandisk spec sheets! (typically around 0.15 mA (see sect2)) When I tested each of the cards I have on hand, the Rocket loggers gave me sleep currents ranging from 0.24 to >5mA, with a cluster around 2mA, and another around 4-5mA.  This does not seem to be related to whether the card is 64, 128 or 256 mb.  I don’t see “bad screen printing”, or any of the other tell tale marks of the fakes, and if memory serves, the packaging looked pretty legit when they arrived. To be honest, I figured that 128mb cards are worth so little money that nobody would bother re-bagging them. I was wrong.

At least I have identified the issue, so the shield regulators in the Tinyduino stack are off the hook for the bad power performance of my field units. The only problem with this new information is that I am fairly certain I put a bunch of crummy cards into the newly built loggers that we just deployed.  With six AA’s, the flow sensors should survive all right, but with excess current drain like that, the drip sensors will expire after a couple of months. (and likely go into some kind of weird brown-out power cycle loop until the sd cards are toast…)

Addendum 2014-09-22

If I can’t locate a reliable source for small low sleep current SD cards, I will look into Solarduino’s solution of putting an N-MOSFET on the ground line. A few folks have pointed out that might not be quite as easy as it sounds to turn the power to an SD card on and off like that: “because SD.h wrapper for SdFat has a bug that prevents multiple calls to begin()”. What I really want is a system that protects the data by cutting the power to the SD cards whenever the power supply falls low. I would like this control to latch off (in case there is battery rebound), and use an “independent” circuit that relies on passive components. I have some homework to do there because you obviously can’t pull the plug when you are in the process of actually writing to the SD card. I know from my Vcc logs that the new file creation event is biggest sustained load on the system – making it the thing most likely to trigger the cut-off if the mcu is not in control.

 Addendum 2014-09-24

It looks like some people have managed to test Sandisk microSD cards that sleep down around 70 uA.  If that’s true, then 300 uA sleep currents mean I have other power issues still to sort out on these loggers.  According to fat16lib, SanDisk claims some of their cards draw higher idle current in SPI mode if the unused pins are not pulled high: 

“The ‘RSV’ pins are floating inputs. It is the responsibility of the host designer to connect external pullup resistors to those lines. Otherwise non-expected high current consumption may occur due to the floating inputs.”

Evidently this is a perennial issue for beginners (like me), concisely expressed by MarkT over at the Arduino forum:

“This is the classic floating-inputs problem with CMOS logic.  Never leave an input pin unconnected.  If you do it can either oscillate or end up at 50% of the supply voltage (at which point both input transistors are conducting).  Either situation increases power consumption drastically (3 to 5 orders of magnitude for a single CMOS gate) compared to the normal static situation.  Now this might be a small fraction of current consumption of a running microcontroller, its going to dominate the sleep-mode consumption when nothing is being clocked.”

So I need to do a few tests to see how this helps lower the sleeping currents on my old small SD cards.

Addendum 2014-09-27

 After testing the 64, 128 & 256 mb cards I bought from eBay, I have found that if you have the “good” SD cards, the Rocket based loggers generally gravitates towards 0.2 mA or less, even if connections 8 & 9 are floating.

However many of my cards do need pullups (0r pulldown) resistors on the two data connections that are not used for SPI to keep them from floating, or the sleep currents are much higher. In the forums (and the datasheets) people seem to be recommending 50-100K pullups. I tested about 50 cards, and general result is: If this logger system+Sd card sleeps around 0.22mA with the pins floating, I already have a “good” card and pullups won’t change the sleep current by much. However about 1/2 the time a pulldown increases the sleep current of a low power sleeper, sometimes adding as much as 0.6mA (so total sleep current goes to about 1 mA). If there is no rise, then sleep current is unaffected by the pulldown.

If my system draws between 0.5 mA to 2mA with the two pins floating…then a pull down resistor on those two lines will usually brings the whole system down to about 0.25-0.35 mA sleeping current. A pullup does not change the sleeping current of these bad cards quite as much, usually reducing sleep currents by about 1/3-1/2 as much as a pull down.  So preventing the pins from floating is always good to reduce sleep current, but the worse a card is when pins 8&9 are floating, the more likely it is that pulldown will help it more than a pullup on those lines. This is very odd because no where in the data sheets does it specify to use a pulldown resistor. 

I am setting my pass/fail point for the entire logger at 0.33 mA, and if I don’t get to or below that with either pullup or pulldown, then I’m calling it a bad card and I won’t use it.  If the “sleeping system” current for this logger design is above 2 mA with pullup/down, then I have a REALLY bad counterfeit microSD card, and I just throw it in the rubbish bin (which happened to about 6 out of 50).

The WORST cards of all bounce down to a reasonably low sleep current when the system first goes to sleep, and then slowly increase over the course of 2-5 minutes, as I am watching the meter, even though the logger is completely asleep.  Those cards seem to “creep up”, whether I put a pullup, or a pulldown, on the unused lines. What’s interesting is that they don’t “jump up” like I would expect if they were waking …they just slowly increase the draw bit by bit. Of course I am watching this with a plain old multimeter, so I am only seeing “the average”.  Perhaps its a whole bunch of wake/sleep cycles in some kind of self triggering loop? I found 4-5 of these (of 50) and I am assuming that the card controller itself is NFG. A couple of these eventually went over 5 mA before I lost patience and just pulled the plug, but I might go back and let them run later to see how far they go. It might even be handy to have a duff card like this around when I want to bring the power supplies down to test power fault handling.

Generally, if I have a good card, it goes into sleep state as soon as the MCU sleeps, and you can see that on the meter because the numbers are completely stable right away. The crummy cards seem to wander around for a while, like they have to think about whether they actually want to go to sleep or not. Some keep on jitterbugging, even after they have gone into low sleep current territory. So far, I’d say about 50% of the cards from eBay have been ok, with the ones being sold as “used” being much better than the “new” ones.

Addendum 2015-01-10

Looks like I am not the only one who has had to grapple with SD card power issues.  Luke Miller, over at the Open Wave Height Logger project, spotted another more subtle power problem with some SD cards using his oscilloscope.  The new SD cards that he tested were drawing about 200 uA, which sounds much worse than the 60-70 uA Muve Music SD cards I am been using.  But his loggers operate at an incredibly fast sampling rate, and that means time to sleep is just as important as the actual sleep current. Until I get my hands on an old scope and see for myself, I need to consider the possibility that the cards I am using might not be as good as I think they are….

Addendum 2016-03-29

After experimenting some more with mixtures of hard vs soft pullups (& despite the fact that only CS actually requires it) I found that three lines had to be pulled up (w MISO as INPUT not output!)  before misbehaving cards would sleep properly. So add these lines to the beginning of your setup before you run sd.begin

// pulling up the SPI lines at the start of Setup with 328p’s internal resistors
pinMode([ChipSelect], OUTPUT); digitalWrite([ChipSelect], HIGH); //pullup SD CS pin
pinMode([MOSIpin], OUTPUT); digitalWrite([MOSIpin], HIGH);//pullup the MOSI pin
pinMode([MISOpin], INPUT); digitalWrite([MISOpin], HIGH); //pullup the MISO pin
delay(1);

I found that enabling these three internal 20K pullup resistors raises sleep current by between 1-5 μA, so it should not hurt your power budget too much, and could potentially save far more power by helping the SD cards go to sleep faster.

Addendum 2017-05-21

Well, I finally took the plunge and started cutting power to the SD cards: Switching off SD cards for Low Power Data Logging.   I left that step for last because as I was being cautious about anything that might put my precious data at risk, but so far (and I am still testing it…) is seems to be working OK.

Bench Testing vs. Real world power consumption of our Arduino dataloggers

I finally have a moment to look at the data from the recently retrieved flow meters, and the results are not what I was expecting. This was a five month deployment, with the units operating from March 22, 2014 to August 25, 2014. The three pendulum units had identical data logging hardware consisting of a the Tinyduino, Sd shield, HMC5883L compass & BMA250 accelerometer, with a DS18B20 temp sensor and a 3-color 5050 indicator LED.  These units recorded Vcc using the 328’s internal 1.1 vref trick, and even with the usual caveat about 10% variability there, the flow sensors did not deliver anything like the low power consumption suggested by my dry test runs*:

Units3&4_compare_640

(*One of the pendulums had a NCP1402 voltage regulator on the power module, so I will be ignoring that unit for now although the remaining AA cell voltages from that unit were surprisingly similar to the batteries from other the power modules?)

One key observation is that power consumption was similar in both units although Unit4 was creating three times as much sensor, EEprom & Sd card traffic as Unit3. This becomes more obvious if I project these curves down to 2800 mV, which is my cutoff to prevent unsafe Sd card operations:

Unit4_projection

The vertical lines here represent one month of operation, and Unit 4 (above in orange), which gathered almost 45000 samples in that time,  projects out to another two months of operation.

Unit3_projection

Despite a boost from slightly newer batteries and a longer 15 minute sample interval, the Unit 3 projection (in yellow) has almost exactly the same amount of time left on it’s 6x AA power pack. This would seem to imply to me that the quiescent current draw of my data loggers is far more important than the power used to drive the sensors. If I just do a quick ‘back of the envelope’ here, and I assume that the six AA’s deliver about 2000mA each,  we are burning through 12000 mAh in about 5200 hours (est.at 7 months)  =  2.3 mA average current.  Ouch!

So I went back and looked at the results from my longest bench top test. This was done with a configuration matching units 3&4 above, but racing along with a sensor read every ten seconds for most of the test. I plotted that voltage curve again with one month time indicators added:

BenchTest_against time

Even with ~400 000 sensor read/record cycles it operated for almost four months – more than half of my lifespan projections for unit 3&4 and it was running on only 3 AA batteries. I had assumed that much bus traffic was the biggest load on the system by far, but perhaps it is time for me to re-consider things? The Tinyduino sensors each have a regulator, as does their Sd shield…have I been ignoring the forest for the trees?

We also had a stationary sensor unit on that last deployment, with a single MS5803-02 sensor, recording both temperature and pressure (barometric & water level). I hacked into a TinyCircuits light sensor board to provide the regulation and I2C level shifting needed for the 5803, so that system had one regulator there, and the one for it’s Sd card.  With only two voltage regs, and no power being used by the DS18B20 temp sensor (which draws  for almost 800 ms to load its 12 bit registers), the stationary unit projects out like this:

MS5803_projection

At 11 months this just squeaks back in my  design target of one complete year of operation.

These curves leave me with a couple of impressions:

Any regulator, even one with relatively low quiescent current, will draw at least as much juice over time as any of the bits you are actually powering with it in a long term application like a data logger. I might also need to take another look at the losses on the Shottky diodes isolating the battery banks, because even with all those regulators in play we are no where near spec sheet predictions here.

I needed better acclerometer sensitivity,so the new Cave Pearl builds have the sensors moved away from the main stack, and all of the power for the I2C bus is now runs through the single hacked light sensor board, similar to the stationery unit above.  While power was not my primary reason for doing this, I suspect this was the right way to go for power managment in the overall design.

And finally, the drip sensors have only one single regulator in their build, with the Sd card hanging right off of the pins.  And now I am playing with BOD fuses, probably exposing my precious data to even more hazard.  But it just might be that the humble three component logging platform actually surpasses the TinyDuino logger units, in terms of power use over time.  After some months of waiting, my eevblog µCurrent is finally on it’s way, so I will have more power test results to report soon. In future, I will do my bench tests with two identical units,  one of which will do sensor readings, etc., as normal, while the other one simply sleeps the whole time, so that I can isolate where the power is being used.

Addendum 2015-04-12

There have been quite a few field trials since this test was done, and the short version of the results is: TinyDuinio based loggers draw about 0.065 mA sleep current, which on a 6x AA battery pack will get you between 6-9 months of run time. Loggers built with generic pro-mini style boards draw 0.33 mA which will deliver the same 6-9 months of run time on 3x AA batteries. Having good low sleep current SD card is critical to the success of your data logger and you really need to test to make sure your sensors are going into low current sleep modes as well. Cheap eBay sensors from China often fail this test. 

More Duracell battery failures this week

Houston we have a problem...

Houston we have a problem…

I have been doing run tests on the newest flow sensor builds, and this morning I had a little red error-state led waiting for me on a sensor that I had assembled the night before. I assumed I had botched a solder joint, or broken a wire somewhere while loading everything into the housing, but instead it turned out to be another Duracell battery failure.

The dead power module

The dead power module

Normally I would ignore this but this is the third Duracell battery failure I have had in a week. One is ‘meh’, two is just ‘bad luck’, but three…that’s a problem that requires attention.  My first though was a bad batch, but the batteries are not all from the same package, each being a slightly different variant of the marque.  Interestingly I have a log of the voltage curve during the start of this problem as I track the power supply via the internal 1.1v reading trick on the Atmel processor:     (mV on the vertical axis)

VoltageduringBatteryFail

The logger recorded about 500 samples before going into shut-down mode.  My overnight runs race along at one sample recorded every 15 seconds just to give them a workout, so this graph only represents about two hours of operation. But still, that is one weird looking voltage curve (…especially with those ‘gaps’, and the voltage was increasing??). Since I have not seen one like this before, I thought I would post it here. The power module has 2 banks of 3AA batteries, isolated from each other by shottky diodes, so this probably represents one set trying to keep the logger alive while the other bank went down hill (while venting internal pressure?) Then when the bad battery leaked enough, the alkali affected the others, and the whole power supply pooped out.

Another from this week

Another from this week

I don’t know why the logger stopped at this point either, as all of these voltages were within the units operating range, so I have to suspect that some kind of “burst” occurred around record 500, that pulled the voltage down below the 2800mV sleep trigger in the code. Certainly there was battery goo sprayed all over the inside of the housing.  The cells that leaked were still delivering 1.3 volts each when I pulled the unit apart in the the morning, so this certainly was not caused by running the cells dry.  My current hypothesis is that the new Keystone Electronics battery holders are so strong, that they are exerting pressure on the battery housing causing some kind of containment failure. Until I get a handle on this,  I will have to go hunting for a replacement brand before the next field deployment.

P.S. The Tinyduino logging platform seems to have survived the event, and is now ticking away in the basement for a proper shake-down run.

Project Update: June 19, 2014

NewCropofDataLoggers

All data logging functions are now localized to these interchangeable platforms. Sensors connect via I2C

As I ramp up production for the next round of deployments, I am happy to report that the new crop of logging platforms have all passed bench testing (set RTC, r/w EEprom, r/w SD card, I2C bus coms, etc).  As I have mentioned before, lifting the tiny light sensor board lines is a bit hairy, so I was expecting at least one of these to crap out due to bridging, etc on those tiny traces.

To give them a more realistic workout, I will rotate through these units as I integrate the new sensors into the main codebuild over the next week or so. Hopefully after that I will have enough time for some proper burn tests before we go into the field again.

The 171 holders have so much tension, it usually scratches the battery just putting them in!

The 171 holders have so much tension, it can be a little hard on the fingers putting the batteries in!

There have also been a few tweaks to the power supply modules: I am now using either brass, or nylon bolts wherever possible, and the plastic battery holders have been replaced with aluminum ones from Keystone Electronics.  These holders are easily the strongest ones I have ever seen, should be rugged enough to withstand some serious knocking about.  I still have not really tackled the hard & soft iron calibration issue yet, but hopefully the removal of the stainless parts used in the previous builds will assist compass calibration when I get there.  Still running everything through Shottky diodes, but I have switched over to “Deans” style power connectors after searching the quad-copter forums for something more robust.

<— Click here to continue reading—>