Category Archives: Lessons learned.

If it’s true that people learn from their mistakes, then I guess I qualify as ‘a life long learner’.

Field Report 2015-08-10: Diagnosing a TMP102 Sensor Failure

Over the next few days we did cave dives to exchange units at sites along the coast. Some of these systems were too deep for our camera, and in the shallower systems the visibility could get a bit grim because if you stay in one place too long, air bubbles rise up and cause “percolation” of little pieces of rock from the ceiling.  So to dismiss any illusions people might have that science diving looks like what they show you on TV, here’s a clip from one of those claustrophobic installations:

Even so, things went well.  Each night I had a new crop of loggers to download and check for my “standard set” of parameters (battery, RTC coincell, sleep current, clock drift, etc.) which I compared to the readings taken before the deployment. The recent retrievals were from the last generation of 3-inch pvc builds, and while they all looked great on the outside, some of them stopped running long before we pulled them out. It can be tricky to figure out the problem when a logger has several I2C sensors on it because the very things that make it so easy to use those breakout boards, also hide problems from you: essentially they don’t just run digitally, they also tend to die that way.  Issues tend to show up much earlier in the analog data, and one useful diagnostic is the voltage log from the main battery pack (read with a simple resistor divider).  Alkaline AA’s usually provide a nice smooth decline over time, so when I see something like this:

011_PS&Temp

Unit w BMA180, TMP102, HMCL5883l. Deployed in fresh water at 5m.

…it’s an immediate red flag.

But unit #011 had a complete set of data logs, with no hint in the sensor records that something strange occurred.  That peak in temperature was pretty large, but it also corresponded to the June 13/14 event that showed up my Rio Secreto & Akumal Bay records, so I accepted it as valid data.  My next thought was that the unit had suffered a μSD failure, but this logger was still successfully going to sleep at a reasonable 0.26 mA. Rapidly entering a low current sleep state is the best indicator I’ve found so far that a card’s control circuitry is still working as it should. Hmmm….

Then next unit with problems was also from a relatively shallow fresh water deployment, and it ran for only two weeks before shutting down:

BMA180, TMP102, HMCL5883l, Deployed in fresh water at 14.5m Depth

Unit w BMA180, TMP102, HMCL5883l.  Deployed in fresh water at 14.5m.

Leak corosion after sensor failure drained the batteries

The power module from #015:  alkalines almost always leak if something pulls them below 0.5v.

This logger was positively pressurized when I broke the seal, with only 0.22 volts at the battery module. When I connected a new battery the unit would did restart properly, producing a strange pattern on the indicator LEDs. Fortunately even with only two weeks of data, what we did have in the log make it pretty clear who the culprit was. It was extremely unlikely the temperature would have spiked like that on the same day that the battery curve took a nose dive.

The next unit presented another mystery, because it had a nice smooth power curve though it had reached it’s low voltage shut down point too early. The temperature data looked reasonable:

Unit w BMA180, TMP102, HMCL5883l. Deployed in fresh water at 22.5m Depth

Unit w BMA180, TMP102, HMCL5883l. Deployed in salt water at 22m.

These loggers have dual 3xAA power packs, and they normally deliver at least eight months of operation. While #016 seemed to restart OK, with a normal (~o.24mA) sleep current, it then crept up to 1.65 mA.  I left the unit running for about 5 minutes and by that time the sleep current was jumping all over the place, varying from 0.5mA to 4 mA at random. This behavior continued after I swapped in a known good SD card, so I was stumped. Once again the data in the sensor logs looked fine.

I had one more unit from this build generation to open up, and it too had developed a problem over the deployment:

017_PS&TEMP

Unit w BMA180, TMP102, HMCL5883l. Deployed in salt water at 24m.

No rust at all...?

I have to trim the boards a bit to fit them in the sensor wells but I am careful not to cut any traces or vias. There was no visible rust on those breakouts at all.

That hockey stick bend on the power curve was another smoking gun, especially in the context of that temperature data. The deep saline water in these caves is remarkably stable, so when I saw a temperature rise alignment like we had with #015, I started pulling wires. Sure enough, the sleep current problem on #016 disappeared as soon as the TMP102 was disconnected and #015 fired up properly as well.

Since epoxy failures took out a fair number of units in 2014, I did a detailed inspection of the potting, but found nothing to indicate that water had reached the sensors. Since the deeper units were most affected, my current working hypothesis is that these sensors were actually suffering from the effects of pressure, rather than moisture. (…but good luck finding that in your ‘Recommended Operating Conditions’)

Anyway, I disconnected the TMP102’s from all the loggers of that generation, (including the ones that were still working…) and set the code to read the internal RTC temperature register instead. Test runs with that were good, and these units went back out for deployment a few days later even though all that housing mass will put a huge lag into the temperature record.  I don’t think I will be using TMP102 sensors again on my projects, but of course there is nothing to say the MCP9808’s from Adafruit won’t develop a similar problem over time.  So I might end up back at the humble DS18Bs which I used on my earliest builds:  they have proved to be remarkably resilient over time.

Addendum 2016-03-17

The MCP9808’s did indeed suffer a similar fate.

 

Testing Power Use with Complex Datalogger Duty Cycles

There is an old saying that goes: “Yesterdays solutions are today’s problems” and I think that now describes this project. You see the first year of development was focused on pretty straightforward hardware issues, and solving each one produced significant gains in performance. Now that I am consistently seeing sleep currents in the 0.1-0.2 mA range (with an SD card (~80uA) & live Adxl345 (~50uA) along for the ride), I am hunting for more elegant ways to extend the operating time while maintaining the simple three component core of the original design. With a 3xAA power pack now providing almost a year of operation for some sensor configurations, I also have the task of developing a method for testing the loggers that can discriminate between subtle code changes with relatively short runs. But artificially reducing the sleep interval between samples distorts the result enough that it’s hard to make good projections. I am slowly coming to realize that testing & calibration are the real heavy lifting when you build any new device.

The new A544 cells arrived at > 7 volts which was too high for the regulator on the Ultras. So I took them down to 5.6 volts with a Zenner. The rare earth magnet soldered to the diode wire gets zapped by the heat from the iron, so you need a second little magnet to hold each battery connection securely. You can also stack a "set" of button cells with these magnets, giving you more options for low power tests.

The new A544 cells arrived at > 7 volts which was too high for the regulator on the Ultra. So I took them down to 5.6 volts with a Zenner that stops the discharge before it goes too far. The rare earth magnet soldered to the leads gets zapped by the heat from the iron, so you need a second little magnet to hold each battery connection securely.

Each new trick I try, like finding another place where I can put the cpu to sleep, adds complexity to code that has “once per day events” and “once per week” events, and soon there will be “only if the delta between two readings is greater than x” events. Most of these are so short that a multimeter can’t catch them but even when a friend donated an old Tektronics to help me try to get a handle on the duty cycle, I faced the challenge of displaying currents ranging from less than 0.1mA to 80mA SD writes with variable duration. To make things more interesting, some of the cheap sensor boards I have been noodling around with have components of unknown origin & dubious quality, which introduce yet another set of variables.

Even with my mediocre scope-skills the forums had convinced me that the SD card was the elephant in the room. So I tried to avoid SD use by adding an external 32k eeprom which let me buffer five or more days worth of data before having to fire up the external storage. Problem solved…or so I thought. I was quite surprised by data from the last deployment that showed using this approach to reduce SD writes by a factor of five only delivered a 5-10% improvement overall.  I had overlooked the fact that the AT24C256 eeprom pulls 3mA for five milliseconds per pagewrite. This was nearly as much current as the Rocket Ultra I was using, not to mention a significant extension of the cpu uptime for multi sensor units that were buffering up to four eeprom pages per record. All of that activity adds up.

So I took another look at buffering data in SRAM, which I flirted with at the beginning of the project. But my script was now much larger than those early versions, leaving barely 500 bytes free.  I know the real coders out there probably laugh at my use of Pstring & Ascii but that lets me add a new sensor by changing a couple of print statements, and adaptability has always been of my primary design goals. To maintain that simplicity I went searching for an Arduino with more headroom and the Moteino Mega over at LowPower Labs seemed to fit the bill with it’s 1284P offering an extravagant 16K of sram (compared to just 2K on the 328p). It also used a low drop out MCP1700 series regulator like the Ultras, and there was support for RFM transceivers.  With the Mega’s larger footprint, I decided to try them first on the larger dry cave platforms:

Rocket Ultra (left) VS Moteino Mega (right) loggers with pin powered RTCs. I break out LED, I2C and one-wire with Deans micro connectors, and you can see the 4.7K one-wire pullup above the main power supply divider on the Mega. The 32K eeprom is tucked under the RTC, which is inverted on the Moteino build to make changing the coin cell easier.

Rocket Ultra (left) VS Moteino Mega (right) based data loggers with pin powered RTCs and 2×4.7MΩ voltage dividers monitoring both the power supply voltage and the rtc backup battery.  I break out LED, I2C and one-wire with Deans micro connectors, and you can see a 4.7K one-wire pull-up above the main divider on the Mega. A 32K eeprom is tucked away under the rtc breakout, which I flipped over on the Moteino build to make it easier to change the CR2032.

For a standardized test, I set both loggers buffering 96 records (= one day @ 15min intervals) in drip sensor configuration. I added the I2C eeprom to the Moteino logger to make the builds as similar as possible, but it does not get used. Instead I store the raw sensor data in integer arrays. So there is no Pstring/ascii use on the Mega logger until I write the data to the SD cards.  With matched acclerometers & cards, both loggers sleep at 0.18 mA so the the only difference between them should be the data handling.  One thing I did not catch from the LowPowerLab specifications was that the 16mhz Mega draws ~12 mA (while awake) in this configuration as compared to the Ultra builds which perk along at just over 4mA. I figured that with SRAM storage the mcu up time would be so much shorter that it would not matter.

With super caps to buffer SD write events, you can drive the loggers with a very small battery. Rare earth magnets let you connect to the ends without a holder and you can make a multi-layer magnet/button cell sandwich to build low power options at just about any voltage. Those are 5v 1farad supercaps in series, so I don't bother to balance them as they should be able to handle leakage asymmetry when the battery input is only 5.6 volts

With a parallel bank of super caps to buffer SD events, you can drive the loggers with small batteries that have high series resistance. Rare earth magnets let you connect without a holder and you can build multi-layer magnet/button cell stacks to create low power options at different voltages. Those are 5v 1farad supercaps so I don’t bother to balance them as they should be able to handle any leakage asymmetry when the battery input is only 5.6 volts. The graphs below had no low volt blips at all, so this series/parallel arrangement of 4 of them was probably more capacity than I needed.

I still had not sorted out the oscilloscope issues but I realized that I could flip the problem around: instead of struggling to display the effect of every little tweak to the duty cycle why not provide a fixed amount of power and just see how long the unit runs. It’s a data logger, so I already have a time stamp and a battery voltage reading with every record.  A couple of people suggested capacitors, but even a 1F supercap only gives you about 0.27 mAh per volt, translating into a few hours of operation for my loggers. I needed longer runs than that because the Moteino was going to loose the data in it’s sram buffer when the unit browned out (I can always dig into eeproms later for the last few records on the Ultra).  A bank big enough for multi day runs was going to be expensive, and is probably a hazard for my little bots.

Fortunately there are a host of small form factor batteries out there for things like fire alarms, medical devices, etc. Energiser’s A544 seemed to fit the bill at 6 volts & 150 mAh: promising to power the Pearls in “sleep current” mode for about 40 days. Even better, they were alkaline cells, so their discharge curve would be more like the AA’s used in real world deployments. There was some risk that these little cells would drop to the low voltage cutoff when the SD write current spikes occurred, so I added a few super caps to buffer those loads. I then set the units up on a book shelf where they would not be triggered and waited for my “baseline” load result.

This is the voltage record from the two different logger platforms, when they were powered by a single 150mAh A544:

A544_FirstPowerTest
(I stopped the test after a month, because I couldn’t take these suspense any longer. There were few sensor interrupts during the test, so this was a baseline power use comparison)

I was sure the SRAM buffering Moteino logger would come out far ahead of the Rocket build that was sandbagged by all that I2C eeprom traffic. But if you correct for the slightly higher starting voltage those two curves are so close to each other they might well have come from the same machine. So there is no longevity boost from SRAM buffering if I use an mcu that draws 3x as much current, but at least I now have a good way to test the loggers without waiting too long for results. This result also agrees with some of my earliest drip sensor results which hinted that the sampling/buffering events were consuming 2/3 of the power budget.

For the next round of tests I will put them on the calibration rigs to see how the A544’s handle the interrupts being triggered all the time. Presumably the Moteinos will draw more power there so I will need to normalize the results to match drip counts. To go beyond the conservative one day buffering I will need some way to capture data from the SRAM buffer before the units power down, so perhaps I will end up using the eeprom on those Moteino Mega builds after all. We will use a few Mega based drip sensors set for very long buffering (8-10 days?) on the next real world deployment. I also have a feeling that the DS18B20 temperature strings would benefit more from SRAM buffering than these simple drip sensors, as they poll up to 40 sensors per record. That’s a lot more data to shuffle around.

Addendum 2015-07-05

Hackaday just posted about [Majek] putting “live” data into Arduino’s flash ram (which is normally not accessible after startup) via a Optiboot hack.  This opens up another possible data buffering strategy, though I am not sure if it could handle the duty cycle of a long deployment. Or it might let you do calculations with the 328p that would otherwise run out of space.  So this is an interesting development that involves no extra hardware, which is usually good news for the power budget. I had already been wondering if calibration data could be stored in flash with Progmem, but that solution only works for data that is not changing all the time.

Addendum 2016-01-06

We finally have some data from the first field deployment of Moteino based loggers which store sensor readings in ram (array variables), rather than buffering all the data as ascii characters in an external eeprom like my 328p based loggers do.

Here is the power curve from a Moteino:

057-M_Drip_BatteryCurve

Battery (mV): 3xAA supply, 0.18 mA sleep current, 5 days of data in ram

And here is a directly comparable build using a rocket scream ultra with a slightly higher drip count (ie: number of processor waking events) over the duration of the deployment.

Cave Pearl data loggers

Battery (mV): 3xAA supply, 0.18 mA sleep, 5 days of data buffered to AT24C256 eeprom

So once again, these performance curves are so close that it makes no odds. But on the bright side, this confirms that accelerated testing with 150mAh A544 batteries does give me results that translate into real world. So this is still pretty good news even if the 1284’s did not deliver the magic performance bullet I was hoping for.

Addendum 2016-02-15

If I wanted something a bit beefier than the 150 mAh in the A544’s, I could hack my way into a 9v battery, and use half of the set of 500 mAh AAAA batteries you find inside. That would give me about 1/4 the power of the AA batteries I typically use on deployment.

Addendum 2016-08-15

I finally figured out how to view  individual logger events using an Arduino UNO as a DAQ with the serial plotter tool built into the IDE:

Cave Pearl data loggers  I’m quite tickled about being able to replicate a task that you normally would need an oscilloscope to see.  Of course my chances of actually catching one of those big unpredictable SD card latencies (from something like  age related wear-leveling) is still pretty low, so I will continue to use this A544 method for solid longevity predictions.

 

The 2014 Cave Pearl Project ‘Year in Review’

With the last field reports of 2014 out the door, I feel I should post something about the project having completed it’s first year as a “serious endeavor”.  At the risk of sounding like A.J.Jacobs,  it has been “A Year of Living Inventorly”, and the fact that I had never done anything like this before did not stop me from trying to build one of the simplest underwater dataloggers ever made. Looking back, I am very pleased by how much the Cave Pearls developed, both in terms of the electronics:

Platform_tryptic_Jan2014

and the physical build:

Cave Pearl Physical Build Tryptic

It’s been one long string of learning by doing, and with no “formal” training in electronics, I have been using a basic process of elimination approach to tackle each problem as I went along.  Each time I built a one, it became a little easier to see where the next improvements could be made.  I became fairly obsessed about the project and my wife, with her usual good humor, coined the phrase “nerdling” to describe the days spent cutting & soldering in the basement, instead of pursuing my other passions.  But left brain/right brain tropes have always sounded odd to me because inside my head both tasks feel pretty much the same.

The highlights of the year were the field deployment trips:

Date #deployed Logger / Sensor combination
Dec. 2013 2 Alpha Flow Sensors go under water for the first time.
Mar. 2014 3 Beta Flow Sensors   Tinyduino based loggers with Bma250 accelerometers, housing have no latch clamps
1 1st gen. Underwater Pressure Sensor
Aug. 2014 3 3rd gen. Flow sensors Rocket based logger platforms, Rosetta Stone builds to compare sensors, BMA180 accelerometers adopted, epoxy failure in this generation
3 Beta Flow sensors re-deployed  in the open ocean
2 Pressure Sensors    MS5803 based,   2bar & 5bar
6 1st gen. Drip Sensors    deployed at Rio Secreto
Dec. 2014 10 Flow sensors a mix of 3rd & 4th generation units, most go into deeper low flow saline zones, 1 commercial co-deployment and a fresh/saline co-location at one site
3 Beta Flow units 2 rebuilt for open ocean deployment, one unit has been underwater continuously ~ 10 months
11 Drip Sensors 1st & 2nd generation, @ Rio Secreto
2 Pressure sensors 2bar in water & 5bar @ surface
2 1st gen. R.Humidity, Pressure & Temp. loggers

The first time I make a machine, it usually takes a week or two of experimentation, and plenty of epoxy, wire & PVC goes into the bin. But now that the logging platform is more developed,  it is getting easier for me to come up with new sensor/logger combinations. While the underwater housing remains the most time consuming part of each build, the overall improvements in design, and in my own skill level, show clearly in the number of units we have been able to deploy over time:

Cave Pearl DIY housings made from PVC

So I guess that’s a report card of sorts, and I owe thanks to all the Arduino makers out there who shared their code & experience, because they really made it possible for me to pick it up as I went along.  But even with that help, there were still a few bits of information that would have been really nice to know a year ago, that I that just did not find anywhere on the web.  So I thought I would address some of those questions.

How much does it cost to build this stuff?

As a long time freelancer, I keep pretty detailed records. By the end of 2014, the cumulative hardware cost came to about $10,000  including all electronics, tools, and pvc parts. Yep. Really. The waste generated in the process, as you hunt around for the “right parts” is significant, and in the early stages I build at least two units that never leave the bench for every one that makes it into the field.  Now I try to buy “just enough” material to build the next 3 or 4 units, no matter how ridiculous the shipping charges are. (often more than the parts themselves)  If you try to order the cheaper parts from eBay you end up with 2-3 week shipping delays, which is a real impediment to your prototyping workflow.  The problem is that every design change leaves you with a handful of leftover bits from the “old” version, and I now have a substantial collection these orphans in boxes all over my workshop.  Even with diligent effort about 1/2 of the electronic parts I purchased did not get used. (at least not yet…the prototyping is going faster now that I have a critical mass of components on hand) Also, I would advise anyone to buy better tools right at the start, rather than wasting your money working your way up through crummy soldering irons, and low budget saws. A good table top band-saw & drill press are essential.

Should I make a blog for my project?

At the beginning, I really had no idea if it was worth the time and effort to maintain a blog like this. But I know several people who are smarter than I am, working on equally cool projects that no one will ever hear about because they have not put the word out.  So for them I thought I would post a snapshot of the stats from this little opus, which has not been subjected to any “How to promote your blog” efforts because I don’t have any spare time left.  I post links in forums when I have something useful to contribute, and a handful of people have given a nod to my project, but that’s about it:

CavePearlBlogTraffic_2014

In total there were 6861 visitors in 2014, generating 19,456 page views.
At the start of the new year, the stats look like this:

CavePearlBlog_TypicalWeeklyTraffic

As you might expect, at least 1/2 of all traffic comes from the U.S. The UK and Germany lead the western European contingent, which adds another 1/3 of the total, while the remainder is sparsely distributed around the globe.  I used to get really excited when some new country generated a big spike in traffic to the blog. Then my brother explained how botnet attacks work. This went a long way to curing my daily compulsion to check the wordpress stats because the truth is they probably don’t care that much about underwater data loggers in the Republic of Elbonia.  Still, some stats are useful, like the fact that 99.9% of the traffic comes in from Google searches.  Ignoring the home pages, posts with content on “How to do X with an Arduino” are by far the most popular:

CavePearlBlog_TopPages

I was surprised to see the RTC page rise so high in the ranking and I was probably just the first person to gather that information together in one place.  When a page does rise above the one-per-day hit rate that just about everything seems to get (from search engine spiders?),  I maintain that page with regular updates, and that seems to be paying off. I also take the time to cross-link extensively between old and new posts, because several people have complained that they could not navigate around “my website”. There are still many people out there who do not know how blogs work, and almost no one uses the built in search feature.

Where does the project go from here?

On the physical side, I need to make the process of assembling the underwater housings less labor intensive, and I need to improve the overall response to flow rates less that 1 cm/second because the deep saline circulation in those caves is very slow. Wherever possible the design also needs to make in-water procedures for things like buoyancy adjustment faster and easier.

On the electronics side, I will hunt for ways to reduce sleep power consumption, while continuing to use cheap components in my basic “off the shelf” design criterion. I have to find or develop better sensor calibration methods, and there are still many code issues to sort out. With some basic improvements in buffering and data handling during processor up-time, I think I could still see significant improvements in the power budget. Using other data formats is one obvious approach, because it’s all basically integer data. But I will probably stick with ASCI for a while yet, as PString makes the code very easy to modify when I change sensor configurations. This is important because the “sensor caps” are interchangeable modular units, independent of the logging platform and the rest of the housing.

I am finally getting comfortable tackling new sensors at the datasheet level and I now have a general sense of how to get communication working between breakout boards and the Arduino.  I accept the criticism that so far I have been making DIY versions of sensor & logger combinations that are already available on the market; in essence trading build  time for money.  With a year of work under my belt, I would say that this trade is not worth it for equipment that you only need one of,  because getting that first prototype operational usually costs you more than you would have spent on the commercial sensor. It starts to be worth it when you reach about ten units, because by then you have all your parts sourced and you can build them pretty quickly.  From that perspective, this project is just starting to reach a break-even point, with the added benefit that we can make custom builds for a particular research question.  Of course, sorting out a new instrument is so immensely satisfying that I would be doing it no matter how many we actually needed.   My hope is that by the end of 2015 I have the chops to develop genuinely new devices that no one has ever thought of.

But looking at the page stats again reminds me that I probably shouldn’t spend too much time writing posts like this one.  I need to get back to the bench and figure out how to do more stuff with an Arduino…and then write about that.

What to expect buying cheap electronic components from eBay

The first time I deliberately purchased something with DRM.

The first time I ever intentionally purchased something with DRM on it. But for me, 1 gig is plenty of space! With Sandisks wear leveling, and a 1 million hour MTBF, (yeah…right…)  I am hoping that a few years in someones cell phone before I get them won’t matter. About 10% of these used cards fail to pass my low current sleep tests, and since they only cost $2, I just throw those away.

After playing the SD card shell game for the last week, I think I have finally found a way to make sure that I have authentic Sandisk cards for my data loggers. Muve music microSD cards were custom built by Sandisk for Crickets music service, and they put some kind of blocking routine into them that prevents you from seeing 3/4 of the space on the card. As you might imagine, extra custom circuitry which makes a product less functional affects its market value, so they are unlikely to be seen as a profitable target by counterfeiters.  Of course I put these used cards through a full test routine, formatting them with SDformater, and then making them sweat through a good bout of H2testw’s verify. And even though what I am after is stable low sleep current, I keep an eye on the meter during the testing. Some of the fake cards I ran into earlier draw unusually large amounts of power during write operations, even if they sleep OK.

More adventures in eBay land:

Going through all that motivated me to make a post on the approximate nature of buying cheap electronic components on the internet. Of course there are warnings about this all over the place. But when I was just starting out, very few people took the time to provide details so I had to learn this from scratch.  The easiest way to convey this is with a table showing the photo from the listings  vs  what actually shows up in the mail somewhere between 2 & 8 weeks later. I would say the current hit rate for actually getting what I thought I was ordering from vendors outside the U.S. is about 60%. The rest of the time you get something that resembles the item in the listing…more or less:

Photo Shown Typical substitutions Comment
Square boards have 4 mounting holes, important in tight spaces. roundBoardRTC Don’t count on fine details like mounting holes or board dimensions matching the listing. Batteries can be missing -or- Cr2032 -or LIR2032,  no matter what is shown in the photos.
JY-MCULow profile 3 color & quite visible with 10k limiters. Brightness is highly variable in eBay 5050s. 2 Other leds …or one of 5 other types randomly intermixed on same order. Sometimes they have limiters, sometimes they don’t. Silk screen labels almost never match the actual pin colors indicating defective runs.
MolexBoard flipup If you get the Molex connectors they both work well enough, but if you don’t have the overhead in your build for the flip up style, it’s kind of annoying to get them.

I could go on, but hopefully that gives you a general idea. And this is in no way limited to eBay. But these kind of replacements are usually not hard to work around, and since the parts only cost a buck or two, I’ll make parallel orders from other vendors, just to be sure I get something I can work with. Order different quantities from each vendor when you do simultaneous orders, because the parts could arrive weeks later, and there will be nothing written on the package in English to help you figure out which vendor sent which shipment.  Always do a “test” order of a small quantity, before ordering multiple units from an eBay vendor, and don’t be surprised if the product changes like this from one batch to the next anyway.

More concerning are part substitutions directly on the boards.  The more components on a breakout, the higher the chance that at least one of them will not be what they claim it is. Complex combinations like you see on IMU’s are a red flag:

Photo Shown Comment
IMU GY-81-3205 10DOF IMU modules are usually described as having the a 1g BMA180 accelerometer (which I use with the HMC5883L in the flow meters) but from places like Miniinthebox, Gearbest, Amazon, etc. they actually ship with crappy 2g BMA020 on board. Of 5 vendors I tried, only NYplatform actually sent this board with a BMA180.

If I’m rooting around on eBay for a sensor board to try out, I get one with as few components as possible. For example, I use the ADXL345 in my drip sensors. These things are common as dirt, and almost as cheap, but the boards are often laden with dubious quality voltage regulators, power LEDs, etc.  In general, the less cruft there is, the more likely it is that it will work, and I always look for boards that have a 3.3v power input.

Avoid: Ahh, that’s better…
adxl345-1 adxl345-2 adxl345-3

Of course this is still no guarantee, especially when a conditioning procedure is needed for the sensor to meet spec.  But when the parts are only a couple of bucks, I don’t have worry that much about frying them when I am just noodling around.  (I killed several of these things before I realized it that JB 5-minute plastic epoxy dissolves IC housings before it cures…) And it’s just the magic of eBay economics that you usually pay more, sometimes much more, for the breakouts with fewer parts on them.

Inspect all solder joints. The lower image shows a pretty typical alignment skew on the cheap SD adapters.

Inspect all solder joints. The lower image shows a pretty typical alignment skew on the cheap SD card adapters. Of course I did not discover it until I had already soldered it into place inside one of my loggers . . .

If you paying peanuts for a sensor that normally sells for $20-$30, you should expect a 20-25% failure rate, with about half of those failures being subtle, like for example, one axis on an accelerometer not reading at the same bit depth as the other two, or only reading in the positive, but not negative direction, etc. Always test your cheap sensors against a  a “known good” board and one thing I have found to be particularly diagnostic is how much current an IC draws.  If the milliamps match the data sheet, there is a good chance that the sensor is Ok.  This is especially true of low current modes, as I have run into several cheap sensors that give decent readings, but never go into low current sleep modes. This makes them useless for data logging applications.

It’s usually a much safer bet to get simple passive components on the web, and the difference between low end parts, and brand name stuff is often an order of magnitude or more.  When you are just starting out, finding cheap component kits can get you rolling for without breaking the bank (although the thin wires & non standard labeling will eventually get on your nerves) Sadly, I did not find out about those until I had already paid extortionate amounts to radio$hack, etc.

I was most of the way through the bundle by this point, but by then I was angry enough to take a photo of it.

I was most of the way through the bundle by this point, but by then I was angry enough to take a photo of it.

And speaking of places where staff react like I a speaking another language if I use the words like “capacitor” in conversation,  I have to add that it really does make a difference if  you order stuff from a vendor who actually uses the products they are selling.  For example, I recently ordered a large amount of wire from HobbyKing. Not only did they take quite a while to ship it to me from their US warehouse (for items listed as “in stock” when ordered) but they looped the many different types of wires into a single large intertwined bundle. It took ages to untangle that mess, leaving me with the impression that they are just a warehouse operation, charging a markup simply to repackage stuff that they too are ordering from China.

Everything gets a bath in 90% isopropl alcohol to remove flux residue.

Everything gets a bath in 90% isopropl alcohol to remove flux residue.

I guess that if I was to sum it all up, I’d say I still get lots of stuff from dodgy online vendors if I am risking $5 or less. The more expensive an item is, the more likely I am to go upmarket to a vendor like Sparkfun or Pololu.  For example, when I need a tool like a soldering iron, or a multimeter, I check what they reccomended at EEVblog, then I see what they think about it on Adafruit.  Those guys really know their stuff, and I never have to worry about dropping a couple of hundred bucks on their recommendation. (something I would never do with an eBay vendor from Hong Kong)

Anyway, I hope that helps someone out there who is just getting started (like I was not so long ago). Even with the best research beforehand, I still end up wasting about 30% of the money I spend on components, because there are alot of elements to any item that one just has see & use to understand. And there is always seems to be some “physical” factor that the experts thought was too obvious to mention, that dramatically affects your particular project… like… for instance…never putting your coffee down on the work bench beside your newly built data logger 😉

Drip Sensor Update: The Gamma Build

I suggested in the last post that a new build usually comes together on the third iteration, so I though I would post a few photos of the current drip sensor, to show how much they changed in that short series:

Alpha, Beta & Gamma builds

Alpha, Beta & Gamma builds of the Cave Pearl Drip Sensor

That Alpha would only detect drops in the very center of the housing, and often registered double hits because of the splash back from the complex surface topography, which also caused a buildup of water on the surface, further interfering with the signal. So I removed the pressure and temperature sensor wells from the Beta, and used a heat gun to bow the surface and shed water.  I hand sanded the pvc to make the strike surface thinner and more responsive, which increased the “reliable” sweet spot to the diameter of the circle you see on drawn there. This worked well, but as you might imagine removing that hour long fabrication step rose to the top of the priority list (if necessity is the mother of invention, laziness is surely the father…) I also did not want to penetrate the housing for those standoffs if I could avoid it because this device has to maintain integrity with constant water impact at exactly that spot.

JB Plasicweld bonds the accelerometer to the cap

Plastic-weld bonds the sensor, preserving the integrity of the housing

My solution to both problems was to solvent weld a four inch knock out cap to the top of the hard pvc shell.  The green color you see there is ABS to PVC transition cement, that I learned about back at the beginning of the project when I was mangling Leggo bricks for internal scaffolding. The ABS is translucent, so the LED no longer needs a portal of clear epoxy because the light passes right through.  The knockout is thin and stiff, which eliminates all that sanding and improves the response of the accelerometers so much that now the device even responds to loud noises, and almost the entire surface sensitive to the smallest drip.  As a result I now have to tweak the settings to reduce the sensitivity so we don’t get too many false positives.  But on the third build of a new sensor, that’s the kind of problem you want to have.  I though the flat surface might resurrect the water pooling problem we had on the Alpha, but in the end it just ended up being a lesson in how I still miss things that are really darned obvious, because simply setting the drip sensor up with a slight tilt will shed the water without affecting the operation at all.  (I am just glad I realized this before some three year old came along and pointed it out to me…)

These guys are only $1.5 each, and are much easier to work with than the plastic micro SD adapters

Much easier to solder those jumpers now.

Wirefold

The trickiest part of the build is routing the wires to reduce strain. Thin silicone 26awg wire helps quite a bit there.

There have been a few improvements to the logging platform as well.  I found these really inexpensive  raspberry pi SD card adapters that are mounted on their own pcb, which neatly solves the melting problem that the cheap plastic  adapters give you if you linger too long with your soldering iron.

Drip Sensor Gamma BuildSo this is the new baby, and I am now running burn tests on loggers using a few different clone boards, including a Rocket Scream Mini Ultra, as they have some very interesting power saving features that might significantly increase the operating time, while keeping the overall simplicity of the three component design.  I expect a few more issues will arise in testing, so I will hold off posting the code to Github, till I am sure that they are behaving properly.  I am still a bit stunned that these drip sensors came together so quickly, but perhaps the last six months of work on the flow meters had something to do with that. 🙂

<— Click here to continue reading the story—>  .

Addendum

Adafruit just posted a video from the National Science Foundation showing how water droplets move on various hydrophobic surfaces.  Way cool…

Addendum 2014-12-11:

We deployed our first batch of these drip sensors in August, and when we went back to get them in December, we were delighted to find that the first real world run was a resounding success.

Building your sensors & underwater housings from scratch

I reviewed my build journal recently, and found enough themes in there for another bit of bloggy catharsis.  No one should mistake this as the advice of an expert in anything, as I have a long way to go before I start collecting karma points at the playground. But at least I can claim that these ideas are well tested, because as the saying goes, I never make the same mistake twice – I make it 5 or 6 times… just to be sure.

Prototyping:

Only RTC & interrupt lines get soldered to the mcu.

Only RTC, SD, & interrupt lines soldered to the mcu.

Join the forums: Everyone says read the datasheets first….well from this beginners perspective that’s B.S.  Even when I started to understand all the terms I was reading, it was still a major leap of understanding to realize what the information in the data sheet implied… (and I still go through it with each new sensor)  If you want to get rolling on something start with the discussion forums, because it’s likely that someone there has already walked down your path, or one very close to it. Thirty minutes Goggling Sparkfun, Stack Exchange, or searching through places like the Arduino Playground, will get you farther than several hours reading the datasheet. Of course you will eventually end up doing that too...just don’t start there.  Often the forums lead you to some well commented GitHub code examples.  It is so much easier to understand the data sheet, when you have a piece of relevant code in front of you at the same time. Datasheets are written solely for corporate electronics engineers, because unless you represent the 10 to 100 thousand unit MOQ,  you simply don’t show up on company radar.

( Sometimes it even seems that a company deliberately tries to hide the functions built into their own ICs from the maker community.  For example: the mysterious Digital Motion Processor (DMP) functions of the invensense 9150. I have yet to find a single example of an Arduino script accessing these features though the 9150 is commonly used in quad copters. I’d have thought that having Euler angle & 9DOF quaternion calculations done for free, would have drawn some serious attention from those those guys.)

You need one completely “trusted” set of kit:  Which you will pay dearly for, but you need it to test out each new component you are thinking of adding to your project. (I still haul out my original Uno for this from time to time, because the darned thing is virtually indestructible…)

More than a few of the cheap eBay boards are D.O.A.

About 1 in 10 of the cheap eBay boards are D.O.A. This  ADXL345 board had a pretty typical alignment skew: X & Y axes were ok, Z did not read.

Connect new parts one at a time:  Do not add to your grief by connecting two new untested pieces of equipment to each other at the same time. (like, for example, a new FTDI board, and an Arduino clone of off eBay….grrrrr….) I am slowly learning what it is that you are paying for when a given sensor module sells for $20 from a trusted source like Sparkfun and $2 from China.  It’s probably not the components (although I hear there are fake IC’s out there) but something else that’s just as important.  If you watch the Tiny Circuits promo vid, around the 5 minute mark you see a person manually re-positioning the components that the pick and place machine laid down, and then around 7:15 you see someone testing each board before shipping. At this point I have seen enough of the eBay stuff that I am pretty sure that these two steps are never done on the clone boards. So If you go down that route, you are implicitly agreeing to do those jobs yourself (in addition to cleaning off all the excess flux left on the boards…)  This is not a problem when I am just hacking something together, and then testing it on the bench (or in the bathtub…) but not a risk I will take lightly for the fieldwork units.

And while we are on the topic of those cheap modules, they almost always arrive with connections broken out for both SPI and I2C, but you will likely use them as one or the other, meaning that you often need non adjacent pins soldered onto the breakout. I used to fiddle with crappy metal “helping hands” things (now replaced by a far more effective Panavise Jr)  to get those single pins soldered into place, but now I simply use a bread board to hold the pins in the correct locations before soldering:

Many thanks to my bother Mike for showing me this soldering technique!

Many thanks to my bother for showing me this soldering technique!

Modularize your designs:   Breadboards were never very practical for me because most of my projects get bashed around, and are supposed to run while the whole thing is in motion. But even after the prototype stage my sensors are not usually soldered directly to the data loggers. Yes, it’s a pain constantly making custom all those custom interconnect cables, especially since I color code everything – I2C bus, interrupt lines, …everything. And once you get an early prototype working, immediately build another one, and do all of your tests, etc., on the second unit, with the first working unit just sitting on the shelf.  Then if something stops working, you can use process of elimination with the known good modules to isolate the cause of the problem quickly. Often this technique helps me identify that the problem is actually in the software, and not with the hardware at all.  But I am still wading my way through the wonderful world of crimp connectors for ones that will prove robust enough for fieldwork. (note: I’ve adopted Deans Micro Plugs for my current builds…)

Coding:

Hit the verify button often, even when you make “simple” changes:  Don’t wait till you have been working for half an hour because humans can only remember 7±3 things (for this human, even less than that…)  You will be backtracking allot and error messages in the AVRc programming environment are usually about as helpful as a baby crying. You know something is wrong, but often you cant figure out what the problem is from the feedback it give you.  Something as simple as: “Hey man,  you left out a semicolon at the end of line 17” can be reported with anything from a one line “Expected X before Y” error to twenty (or more…) lines of cryptic compiler faults. Sometimes Google is your only friend when this happens.

The utility of a piece of code is directly proportional to how dangerous it is to use: For example: Global #define statements are tricky…especially when you are using libraries, because you never know when you have tried to “re-define” something hidden in one of those libraries. Interrupt handling is another example of something tremendously useful, and really easy to screw up when you have more than one sensor generating interrupts in the same piece of code, at the same time.

Software faults almost always show a repeating pattern of behavior: At least that’s been my experience so far. Bad wiring, or some other issue with the physical build, (like a v. regulator thermaling out…) hangs the system in a way that is much more unpredictable.  Of course, finding those patterns is a heck of a lot easier when your project is a data logger in the first place…

Physically putting “stuff” together:

Third time’s a charm:  If your alpha works at all; that’s a great success. The holes will be in the wrong place, you will use too much epoxy, and it will be a clunky octopus of dodgy wiring. But even if it only runs for a single day, it has done the job of showing if the idea will work. (and they make such stylish book-ends…right?) The second build lets you sort out the right physical locations for all the components, and eliminates that hour of laborious hand-sanding that you really did not need to do.  This is also the model that lets you work out most of the software bugs, because it is usually much easier to open & close without breaking fragile jumpers all the time.  But for me at least,  it’s the third build that usually comes together in a way that feels right.

This type of bench vise is worth finding. I picked this used one up for $10, and it has been fantastic for holding parts together while epoxies cure & solvents weld PVC parts.

This type of bench vise is handy. I picked this one up for $10, at a garage sale and it has been fantastic for holding parts together while epoxies cure & solvents weld.

Adhesives & epoxies can be one of the most expensive components in your physical build:  (I wasn’t expecting this one) I go through a fair bit of Loctite E30-CL  (or E00CL for better PVC bonding) for my hull pen-etrations, and I love the fine control of the applicator gun.  But the mixing nozzles alone are more than a buck each, and they are elegantly design to waste a large volume of that precious epoxy with every use, which then solidifies inside the nozzles turning them into expensive single use devices.  That irritated me enough to start experimenting, and it turns out that Goof-off  (mainly xylene, and 2-ethanol) cleans out those applicator nozzles relatively easily. (Acetone would also work) I usually buy my adhesives from Zoro Tools. (note: their product search function is terrible, I use Google Shopping to hunt for stuff on Zoro’s site)  There are plenty of cheap Loctite sellers on eBay, but many are hawking expired lots.  In a pinch, I fall back to good old JB weld, and I use PlasticWeld putty absolutely everywhere because it nicely adheres sensor boards to both PVC & ABS in a way that is strong, but still removable (sort of..) if you have to repair something. Be careful not to bridge contacts with your adhesives on really sensitive sensors.  Most of these epoxies do not conduct electricity, so I use JB weld to shore up weak jumpers on I2C lines all the time. I have had problems with some sensors not working after gluing, probably because the cured epoxy added capacitive effects to the circuits. I have also started experimenting with 3M’s VHB double sided tape (for low surface energy plastic & acrylic substrates) to see if that is easier to work with. (Note: after testing the VHB did not work as well on the pvc as the standard 5lb mounting tape)

(…and if paying an “arm and a leg” for your adhesives wasn’t bad enough, those cheerful, brightly colored labels hide some fairly dire warnings about skin contact literally causing cancer, or vapors so dangerous that use of the product anywhere other than a open field is guaranteed to give you brain damage & significantly harm your … uhhh … “reproductive capacity”. Don’t believe me? Get a 200x microscope and read that fine print for yourself… just make sure you do it some place far away from stoves, heaters, electric motors and all other sources of ignition…like soldering irons 🙂 )

The shop course that you only took in high school to boost your marks will finally start to come in handy:  Turns out that working with PVC is almost the same as working with wood, so I was surprised by how much time I spent scoping woodworking forums while figuring out how to cut weird angles and then re-assemble things.

BigIron

An angle grinder pulls the rust off of old school tools in short order.

You can build anything with three pieces of  “iron”: A good soldering iron, a drill press, and a 13″ bench top scroll saw (which cuts PVC and trims circuit boards beautifully).  I have not regretted for one second putting down $100 for my little Haiko from Adafruit. (Wish I had bought the thing at the very start of this adventure.) But my other two irons are vintage Sears Craftsman models which cost next to nothing because when I bought them, they looked like flea-market clunkers from the days before plastic was invented. Cleaning them up was worth the effort because they weigh a ton, and are rock solid stable. In fact, these days I actually go out of my way to get old tools for my workshop. So if you see me one morning, at yet another garage sale, don’t be surprised if my hands are full of weird drill bits,  strange clamps, and other rusty lumps of metal that are even older than I am.

And last but not least:

I know everyone has their own taste, but I find that Pandora’s “Chillout” channel (in the Dance/Electronic genre section) just has a really good vibe for working on the bench. Spacey electronica wallpaper-music smoothly delivers the hyper focus I need to sustain a long soldering session, especially when the clock ticks on into the wee hours… .

Addendum 2014-07-29 

I have another one to add to this list but it stands outside of any category because it seems to apply equally. I call it the ‘Principal of Equivalent Annoyance’, and it’s just the observation that the dumb little things on a project take up at least as much time as the big important things. For example: when you connect allot of parts together, the physical behavior of your wires matters more than you’d expect because the difference between good insulation and stiff hard plastic puts quite a bit of pressure on the wires when you fit everything into the housing. Once and a while I am left with some excess lead wire from a sensor and when I open up the left over multicore cable I sometimes find a meter or so of the perfect wire.  Soft silicone insulation, 12 or more strands, no tinning… the stuff is fantastic to work with and is so flexible that it doesn’t stress the connections. Of course it never has any identification marks on it that might lead me to the source.  So I am slowly working my way through every brand of hook-up wire on the market, searching for this holy grail, and wasting more time on it than I even want to think about.  I suspect this is kind of thing happens on every project.

Addendum 2014-09-13

There are a few contenders so far in my ongoing search for “the perfect wire”: 

Adafruit sells 26AWG in multiple colors. This is my current favorite wire because they sell it in easy to handle packages (~0.95/2m) and they have the only grey & orange colored stuff I have found so far. It probably sounds silly to mention the packaging, but from other vendors like Hobby king, or eBay, you can expect to spend at least an hour untangling the spaghetti when your wire arrives. Their 30AWG is 0.70/2m, and is very handy for running jumpers across the surface of a breakout board.

Sparkfun sells their red & black silicone hook up wire at a good price, I just wish they had more colors.

Cal Test Electronics CT2956 Test Lead Wire:  (24awg $9 for 10 m, green, white) It is soft multi-strand bare copper and there is some variability in the stiffness as they seem to use different stranding depending on the color you order, but all of it is much more flexible than pvc insulated wire.  The insulation is nice and thin, so it crimps well into the standard 0.1″ connectors, and it is a bit stiffer than the Turnigy, so it will hold a bend you put in the wire.

Turnigy Soft Silicone Wire: (24awg $.60-$1 per meter, red, black, yellow blue) which is popular in the RC plane/Quadcoper crowd. Multi-strand “tinned” copper, with lots of colors. The very soft insulation is about twice the thickness of the Cal Test, but it still crimps nicely. (note: Hobby King does not let you mix warehouses on orders…)

There are two drawbacks to to using really soft silicone wire: You can not push the female crimp connector ends into the housings as you normally would with jumper wire. You have to dig in and pull the connector through the enclosure with tweezers.  The other issue is that because the silicone conducts heat so much faster than pvc, you will burn your fingers more often while soldering if you are not careful. In return,  you can stuff as much spaghetti as you want into a very tight housing without putting pressure your the solder joints in the process.  The wire stripper from Pololu, works well on both types of wire.

Addendum 2016-03-11

More eBay vendors for silicone jacket wire are appearing over time, and they are a great place to find a wider selection of colors like brown, purple, & pink. With module & jumper builds like mine, you need as many different colors as you can get your hands on. 

Addendum 2015-01-09 

I received a 10″ craftsman 21400 band saw for Christmas this year. And just like the night & day transition that occurred when I went from a crummy soldering iron to the lovely Hakko, my cuts have now become more accurate and virtually effortless.  No more risking my fingers trying to cut pipe on the table saw! And free-handing with a band saw accomplishes 90% of what I could do with the the jig saw as well.  If you can only afford to own one cutting machine, then I am now convinced it should be a bench-top band saw. Wish I had it from the beginning…

Addendum 2015-01-16

Looks like I am not the only one who is irritated by the waste generated by the Loctite 50ml nozzles. The folks over at RCuniverse confirm that acetone makes a good solvent for cleaning out the mixing tubes, and that its easier to do if you remove the white plastic mixing baffles first. They also mention that putting the nozzles in bags in the freezer will prevent the epoxy from setting, so I might try this trick to see if I can get the tubes to “self clean” by gravity. Looks like All-Spec is the cheapest source for the nozzles, but I keep finding “generic” mixing nozzles on eBay that claim they work with both 3M and Loctite adhesives. Haven’t tried them yet though.

Addendum 2015-02-02

I bought some of the cheep MA5.4-17S 17 element mixing nozzles, and although the fit perfectly onto the 50ml dispenser, they were much shorter than the 15cm, 20 element, Loctite 98623 nozzles I usually use:

TopCheap_BottomLoctite

I thought this was a good thing, as I hate wasting all the epoxy that is left in the dispenser after every use and these smaller ones only retained 1.68ml in the barrel. I did some test pours under nearly identical conditions, and the smaller nozzles left a significant “swirly” pattern when the cured epoxy was examined close up a few days later:

Left side: Loctite Nozzle Right Side: Short Generic Nozzle

Left side: Loctite Nozzle                                                   Right Side: Short MA5.4-17s nozzle

I am interpreting this as a density gradient formed by inadequate mixing in the shorter nozzles. The epoxy has still set hard as a rock so I am not sure if this compromises the integrity too much. There are also listings for MA6.3-20S and MA6.3-21S nozzles, and I will try some of those next time.

(Note: the MA6.3’s worked great, as did the $10 applicator guns on eBay)

Buffering sensor data to an AT24C32 eeprom on I2C

These MS5803 pressure sensors are my very first SMD reflow pieces. Hopefully I did not toast them on the kitchen skillet I was using...

These MS5803 pressure sensors are my first  attempt at diy SMD. Hopefully I did not toast them on the electric skillet I was using to reflow…

While I was waiting on the results of the week-long power consumption test of the SRAM buffering script, I continued thinking about other ways I might extend the operating lifespan of the Pearls. I had originally considered using the processors 512K of internal eeprom for temporary storage, but work over at the solar duino project showed me that using an external 24AA256 is faster and consumes less power than the internal eeprom.  AND the cheap DS3231 RTC boards I had just received from eBay had an AT24C32 eeprom chip: 4k of storage just sitting there waiting to be used…And the data sheets told me that the SD card could be drawing up to 80 ma for who knew how long, while the datasheet for the eeprom listed a paltry 2 ma per block write taking only 5 milliseconds…

Three days of heavy lifting later…I had cobbled together this script, using PSTRING to to dramatically simplify the concatenation of the sensor data into a 28 byte long char buffer (the block write buffer on the AT24C32 eeprom is only 32 characters long and you need room for termination characters, etc).  Pstring uses simple print statements, but more importantly, it never causes buffer overflows if you try to stuff in too much data, like a mishandled sprintf statement could. This also gives me some flexibility while I am still changing my sensors around, as I don’t quite know what the final data is going to look like yet.

So this code buffers each sensor read cycle to the I2C eeprom, using two page writes per cycle.  This simplifies the code a bit, as I have only one set of variables on the go. Then when the countlog = SamplesPerCycle, it does a reverse for loop to pull the data back out of the eeprom, and write it to the SD card. With a rated endurance of 1 million write cycles, I’m not worried about wearing the eeprom out either.

And the result? This script gives me ~700 sensor read cycles per 8mV drop on a 2 AA battery power supply. This is less than half the performance of the SRAM buffering code, which surprised me quite a bit, but I guess that *192 eeprom page writes (with the attendant I2C coms) +1 SD card write per day,  uses 2-3 times as much power as SRAM buffering with 8 SD card write cycles for that same days worth of records. On paper all that eeprom writing represents almost one second per day at 2 mA, which doesn’t seem like much. So either my tiny 128 mb SD cards are very quickly to going back into sleep mode, or the I2C traffic itself is using a significant amount of power…?

So what did I learn:  Well knowing that a fair bit of I2C & eeprom traffic will more than double the power drain is quite handy, as I now jump into connecting temperature, pressure, compass, and perhaps other sensors, to those same I2C lines. It will be interesting to see what the real world performance of these loggers is when the rubber meets the…ummm…cave diver.

Addendum 2015-04-18

I simply let the wires I am already using to tap the cascade port I2C lines poke up enough to give me solder points for the EEprom. Don't forget to remove the pullups on the EEprom board.

I simply let the wires I am already using to tap the cascade port I2C lines poke up enough to give me solder points for the EEprom. Don’t forget to remove the pullups on the EEprom board as you already have pullups on the RTC breakout.

The AT24c32 chip can only hold 4k of data –  if you write beyond 4096 bytes, it rewrites over the old data!  So once you have done 128 page writes, you need to flush to the sd card. In this code, I write two 32-byte pages per record. So I have a upper limit of 64 records before I hit that block limit and start to overwrite the data!  I could bump it up to 32k of external eeprom for only $1.50, so I will have to try a few experiments to see if that helps. That 32k eeprom is a code compatible, drop in replacement. All you have to do is change the I2C address for the eeprom for the new board.

Addendum 2014-07-16

The Code below has been posted to the projects GitHub. Look around as the more recent codebuilds are much more elegant than this crude early version, and include sensor support for an easy to build logger.

//Date, Time and Alarm functions using a DS3231 RTC connected via I2C and Wire lib by https://github.com/MrAlvin/RTClib 

// based largely on Jean-Claude Wippler from JeeLab’s excellent RTC library https://github.com/jcw
// clear alarm interupt from http://forum.arduino.cc/index.php?topic=109062.0
// get temp from http://forum.arduino.cc/index.php/topic,22301.0.html
// BMA250_I2C_Sketch.pde -BMA250 Accelerometer using I2C from http://www.dsscircuits.com/accelerometer-bma250.html
// internal Vcc reading trick //forum.arduino.cc/index.php/topic,15629.0.html
// and http://forum.arduino.cc/index.php?topic=88935.0
// free ram code trick: http://learn.adafruit.com/memories-of-an-arduino/measuring-free-memory
// power saving during sleep from http://www.gammon.com.au/forum/?id=11497
// I2C routine based on http://playground.arduino.cc/Code/I2CEEPROM#.UwrbpPldUyI
// New file name routine from http://forums.adafruit.com/viewtopic.php?f=31&t=17964

#include <Wire.h> // 128 byte Serial buffer
#include <SPI.h> // not used here, but needed to prevent a RTClib compile error
#include <avr/sleep.h>
#include <RTClib.h>
#include <PString.h>
#include <SdFat.h>
SdFat sd; // Create the objects to talk to the SD card
SdFile file;
const byte chipSelect = 10; //sd card chip select

#define SampleInterval 1 // power-down time in minutes before interupt triggers the next sample
#define SamplesPerCycle 5 // # of sample cycles to buffer in eeprom before writing to the sd card: MAX of 64! (do not exceed 128 page writes or data will be lost)
unsigned int countLogs = 0; // how many records written to each file
unsigned int fileInterval = 10; // #of log records before new logfile is made
/* count each time a log is written into each file. Must be less than 65,535
counts per file. If the sampleinterval is 15min, and fileInterval is 2880
seconds, then 96samples/day * 30days/month = 30 day intervals */

#define ECHO_TO_SERIAL // echo data that we are logging to the serial monitor
// if you don’t want to echo the data to serial, comment out the above define
#ifdef ECHO_TO_SERIAL
//#define WAIT_TO_START
/* Wait for serial input in setup(), only if serial is enabled. You don’t want
to define WAIT_TO_START unless ECHO_TO_SERIAL is defined, because it would
wait forever to start if you aren’t using the serial monitor.
If you want echo to serial, but not wait to start,
just comment out the above define */
#endif

char FileName[] = “LOG00000.CSV”; //the first file name

#ifndef cbi //defs for stopping the ADC during sleep mode
#define cbi(sfr, bit) (_SFR_BYTE(sfr) &= ~_BV(bit))
#endif
#ifndef sbi
#define sbi(sfr, bit) (_SFR_BYTE(sfr) |= _BV(bit))
#endif

#define DS3231_I2C_ADDRESS 104 //for the RTC temp reading function
#define EEPROM_ADDR 0x57 // I2C Buss address of AT24C32 32K EEPROM
#define EEPromPageSize 32 //32 bytes for the AT24c32 I am using

#define BMA250 0x18
#define BW 0x08 //7.81Hz bandwith
#define GSEL 0x03 // set range 0x03=2g, 0x05=4, 0x08=8g, 0x0C=16g

//I2C eeprom variables
unsigned int CurrentPageStartAddress = 0; //set to zero at the start of each cycle
char EEPROMBuffer[28]; //this buffer contains a string of ascii
//char EEPROMinBuffer[28]; // this buffer recieves numbers from the eeprom was an unsigned char
//note the data read from the eeprom is binary – not ascii characters!

RTC_DS3231 RTC;
byte Alarmhour = 1;
byte Alarmminute = 1;
byte dummyRegister;
byte INTERRUPT_PIN = 2;
volatile boolean clockInterrupt = false;
byte tMSB, tLSB; //for the RTC temp reading function
float RTCTempfloat;
char CycleTimeStamp[ ]= “0000/00/00,00:00:00”;
byte Cycle=0;

//variables for accellerometer reading
uint8_t dataArray[16];
int8_t BMAtemp;
float BMAtempfloat;
uint8_t wholeBMAtemp,fracBMAtemp;
int x,y,z; //acc readings range to negative values

int temp3231;
uint8_t wRTCtemp,fRTCtemp; //components for holding RTC temp as whole and fraction component integers
int Vcc;//the supply voltage via 1.1 internal band gap
byte ledpin = 13; //led indicator pin not used in this code
// the LED on pin 13 is also shared with the SPI SCLK clock, which is used by the microSD card TinyShield.
// So when you use a SPI device like the SD card, the LED will blink, and the SD library will override the digitalWrite() function call.
// If you need a LED for indication, you’ll need to hook up an external one to a different I/O pin.

void setup () {

pinMode(INTERRUPT_PIN, INPUT);
digitalWrite(INTERRUPT_PIN, HIGH);//pull up the interrupt pin
pinMode(13, OUTPUT); // initialize the LED pin as an output.

Serial.begin(9600);
Wire.begin();
RTC.begin();
clearClockTrigger(); //stops RTC from holding the interrupt low if system reset
// time for next alarm
RTC.turnOffAlarm(1);

#ifdef WAIT_TO_START // only triggered if WAIT_TO_START is defined at beging of code
Serial.println(F(“Type any character to start”));
while (!Serial.available());
#endif

delay(1000); //delay to prevent power stutters from writing header to the sd card
DateTime now = RTC.now();

DateTime compiled = DateTime(__DATE__, __TIME__);
if (now.unixtime() < compiled.unixtime()) { //checks if the RTC is not set yet
Serial.println(F(“RTC is older than compile time! Updating”));
// following line sets the RTC to the date & time this sketch was compiled
RTC.adjust(DateTime(__DATE__, __TIME__));
}

initializeBMA(); //initialize the accelerometer – do I have to do this on every wake cycle?

//get the SD card ready
pinMode(chipSelect, OUTPUT); //make sure that the default chip select pin is set to output, even if you don’t use it

#ifdef ECHO_TO_SERIAL
Serial.print(F(“Initializing SD card…”));
#endif

// Initialize SdFat or print a detailed error message and halt
// Use half speed like the native library. // change to SPI_FULL_SPEED for more performance.
if (!sd.begin(chipSelect, SPI_HALF_SPEED)) {
Serial.println(F(“Cound not Initialize Sd Card”));
error(“0”);}

#ifdef ECHO_TO_SERIAL
Serial.println(F(“card initialized.”));
Serial.print(F(“The sample interval for this series is: “));Serial.print(SampleInterval);Serial.println(F(” minutes”));
Serial.println(F(“Timestamp Y/M/D, HH:MM:SS,Time offset, Vcc = , X = , Y = , Z = , BMATemp (C) , RTC temp (C)”));
#endif

// open the file for write at end like the Native SD library
// O_CREAT – create the file if it does not exist
if (!file.open(FileName, O_RDWR | O_CREAT | O_AT_END)) {
Serial.println(F(“1st open LOG.CSV fail”));
error(“1”);
}

file.print(F(“The sample interval for this series is: “));file.print(SampleInterval);file.println(F(” minutes”));
file.println(F(“YYYY/MM/DD HH:MM:SS, Vcc(mV), X = , Y = , Z = , BMATemp (C) , RTC temp (C)”));
file.close();

digitalWrite(13, LOW);
}

void loop () {

// keep track of how many lines have been written to a file
// after so many lines, start a new file
if(countLogs >= fileInterval){
countLogs = 0; // reset our counter to zero
createLogFile(); // create a new file
}

CurrentPageStartAddress = 0;

for (int Cycle = 0; Cycle < SamplesPerCycle; Cycle++) { //this counts from 0 to (SamplesPerCycle-1)

if (clockInterrupt) {
clearClockTrigger();
}

read3AxisAcceleration(); //loads up the Acc data
DateTime now = RTC.now(); // Read the time and date from the RTC

sprintf(CycleTimeStamp, “%04d/%02d/%02d %02d:%02d:%02d”, now.year(), now.month(), now.day(), now.hour(), now.minute(), now.second());

wholeBMAtemp = (int)BMAtempfloat; fracBMAtemp= (BMAtempfloat – wholeBMAtemp) * 100; // Float split into 2 intergers
//can use sprintf(BMATempHolder, “%2d.%2d”, wholeBMAtemp[Cycle], fracBMAtemp[Cycle]) if we need to recompose that float
RTCTempfloat= get3231Temp(); wRTCtemp = (int)RTCTempfloat; fRTCtemp= (RTCTempfloat – wRTCtemp) * 100; // Float split into 2 intergers
Vcc = (readVcc());
if (Vcc < 2800){Serial.println(F(“Voltage too LOW”));error (“L”);}

//serial output for debugging – comment out ECHO_TO_SERIAL to eliminate
#ifdef ECHO_TO_SERIAL
Serial.print(CycleTimeStamp); Serial.print(F(” Cycle “)); Serial.print(Cycle);Serial.print(F(“,”)); Serial.print(Vcc); Serial.print(F(“,”));
Serial.print(x); Serial.print(F(“,”));Serial.print(y); Serial.print(F(“,”)); ;Serial.print(z); Serial.print(F(“,”));
Serial.print(wholeBMAtemp);Serial.print(F(“.”));Serial.print(fracBMAtemp);Serial.print(F(“,”));
Serial.print(wRTCtemp);Serial.print(F(“.”));Serial.print(fRTCtemp);
Serial.print(F(“, Ram:”));Serial.print(freeRam());
delay(40); //short delay to clear com lines
#endif

//Construct first char string of 28 bytes – end of buffer is filled with blank spaces flexibly with pstring
//but could contruct the buffer with sprintf if I wasn’t changing my sensors so often!

PString str(EEPROMBuffer, sizeof(EEPROMBuffer));
str = CycleTimeStamp;str.print(F(“,”));str.print(Vcc);str.print(F(” “));

Write_i2c_eeprom_page(EEPROM_ADDR, CurrentPageStartAddress, EEPROMBuffer); // whole page is written at once here
CurrentPageStartAddress += EEPromPageSize;

//Construct second char string of 28 bytes to complete the record
str = “,”; str.print(x);str.print(F(“,”));str.print(y);str.print(F(“,”));str.print(z);str.print(F(“,”));
str.print(wholeBMAtemp);str.print(F(“.”));str.print(fracBMAtemp);str.print(F(“,”));
str.print(wRTCtemp);str.print(F(“.”));str.print(fRTCtemp);str.print(F(“,”));str.print(F(” “));

Write_i2c_eeprom_page(EEPROM_ADDR, CurrentPageStartAddress, EEPROMBuffer); // 28 bytes/page is max whole page is written at once here
CurrentPageStartAddress += EEPromPageSize;

// IF full set of sample cycles is complete, run a loop to dump data to the sd card
// BUT only if Vcc is above 2.85 volts so we have enough juice!
if (Cycle==(SamplesPerCycle-1) && Vcc >= 2850){

#ifdef ECHO_TO_SERIAL
Serial.print(F(” –Writing to SDcard –“)); delay (10);// this line for debugging only
#endif

CurrentPageStartAddress=0; //reset the page counter back to the beginning

file.open(FileName, O_RDWR | O_AT_END);
// open the file for write at end like the Native SD library
//if (!file.open(FileName, O_RDWR | O_AT_END)) {
// error(“L open file fail”);
//}

for (int i = 0; i < SamplesPerCycle; i++) { //loop to read from I2C ee and write to SD card

Read_i2c_eeprom_page(EEPROM_ADDR, CurrentPageStartAddress, EEPROMBuffer, sizeof(EEPROMBuffer) ); //there will be a few blank spaces
CurrentPageStartAddress += EEPromPageSize;
file.write(EEPROMBuffer,sizeof(EEPROMBuffer));

Read_i2c_eeprom_page(EEPROM_ADDR, CurrentPageStartAddress, EEPROMBuffer, sizeof(EEPROMBuffer) );
CurrentPageStartAddress += EEPromPageSize;
file.write(EEPROMBuffer,sizeof(EEPROMBuffer));
file.println(F(” “));

countLogs++;
// An application which writes to a file using print(), println() or write() must call sync()
// at the appropriate time to force data and directory information to be written to the SD Card.
// every 8 cycles we have dumped approximately 512 bytes to the card
// note only going to buffer 96 cycles to eeprom (one day at 15 min samples)
if(i==8){syncTheFile;}
if(i==16){syncTheFile;}
if(i==24){syncTheFile;}
if(i==32){syncTheFile;}
if(i==40){syncTheFile;}
if(i==48){syncTheFile;}
if(i==56){syncTheFile;}
if(i==64){syncTheFile;}
if(i==72){syncTheFile;}
if(i==80){syncTheFile;}
if(i==88){syncTheFile;}
}
file.close();
}
// setNextAlarmTime();
Alarmhour = now.hour(); Alarmminute = now.minute()+SampleInterval;
if (Alarmminute > 59) { //error catch – if alarmminute=60 the interrupt never triggers due to rollover!
Alarmminute =0; Alarmhour = Alarmhour+1; if (Alarmhour > 23) {Alarmhour =0;}
}
RTC.setAlarm1Simple(Alarmhour, Alarmminute);
RTC.turnOnAlarm(1);

#ifdef ECHO_TO_SERIAL
Serial.print(F(” Alarm Set:”)); Serial.print(now.hour(), DEC); Serial.print(‘:’); Serial.print(now.minute(), DEC);
Serial.print(F(” Sleep:”)); Serial.print(SampleInterval);Serial.println(F(” min.”));
delay(100); //a delay long enought to boot out the serial coms
#endif

sleepNow(); //the sleep call is inside the main cycle counter loop

} //samples per cycle loop terminator
} //the main void loop terminator

void createLogFile(void) {
// create a new file, up to 100,000 files allowed
// we will create a new file every time this routine is called
// If we are creating another file after fileInterval, then we must
// close the open file first.
if (file.isOpen()) {
file.close();
}
for (uint16_t i = 0; i < 100000; i++) {
FileName[3] = i/10000 + ‘0’;
FileName[4] = i/1000 + ‘0’;
FileName[5] = i/100 + ‘0’;
FileName[6] = i/10 + ‘0’;
FileName[7] = i%10 + ‘0’;
// O_CREAT – create the file if it does not exist
// O_EXCL – fail if the file exists O_WRITE – open for write
if (file.open(FileName, O_CREAT | O_EXCL | O_WRITE)) break;
//if you can open a file with the new name, break out of the loop
}

// clear the writeError flags generated when we broke the new name loop
file.writeError = 0;

if (!file.isOpen()) error (“diskful?”);
Serial.print(F(“Logging to: “));
Serial.println(FileName);

// fetch the time
DateTime now = RTC.now();
// set creation date time
if (!file.timestamp(T_CREATE,now.year(),now.month(),now.day(),now.hour(),
now.minute(),now.second() )) {
error(“cr t”);
}
// set write/modification date time
if (!file.timestamp(T_WRITE,now.year(),now.month(),now.day(),now.hour(),
now.minute(),now.second() )) {
error(“wr t”);
}
// set access date
if (!file.timestamp(T_ACCESS,now.year(),now.month(),now.day(),now.hour(),
now.minute(),now.second() )) {
error(“ac t”);
}
file.sync();
//file.close();
//file.open(FileName, O_RDWR | O_AT_END);

// write the file as a header:
file.print(F(“The sample interval for this series is:”)); ;Serial.print(SampleInterval);Serial.println(F(” minutes”));
file.println(F(“YYYY/MM/DD HH:MM:SS, Vcc(mV), X = , Y = , Z = , BMATemp (C) , RTC temp (C)”));
file.close();

#ifdef ECHO_TO_SERIAL
Serial.println(F(“New log file created on the SD card!”));
#endif // ECHO_TO_SERIAL

// write out the header to the file, only upon creating a new file
if (file.writeError) {
// check if error writing
error(“write header”);
}

// if (!file.sync()) {
// check if error writing
// error(“fsync er”);
// }

}
void syncTheFile(void) {
/* don’t sync too often – requires 2048 bytes of I/O to SD card.
512 bytes of I/O if using Fat16 library */
/* blink LED to show we are syncing data to the card & updating FAT!
but cant use LED on pin 13, because chipselect */
// digitalWrite(LEDpin, HIGH);
if (!file.sync()) { error(“sync error”);}
// digitalWrite(greenLEDpin, LOW);
}

// Address is a page address
// But data can be maximum of 28 bytes, because the Wire library has a buffer of 32 bytes
void Write_i2c_eeprom_page( int deviceaddress, unsigned int eeaddress, char* data) {
unsigned char i=0;
unsigned int address;
address=eeaddress;
Wire.beginTransmission(deviceaddress);
Wire.write((int)((address) >> 8)); // MSB
Wire.write((int)((address) & 0xFF)); // LSB
do{
Wire.write((byte) data[i]);i++;
} while(data[i]);
Wire.endTransmission();
delay(10); // data sheet says 5ms for page write
}

// should not read more than 28 bytes at a time!
void Read_i2c_eeprom_page( int deviceaddress, unsigned int eeaddress,char* data, unsigned int num_chars) {
unsigned char i=0;
Wire.beginTransmission(deviceaddress);
Wire.write((int)(eeaddress >> 8)); // MSB
Wire.write((int)(eeaddress & 0xFF)); // LSB
Wire.endTransmission();
Wire.requestFrom(deviceaddress,(num_chars-1));
while(Wire.available()) data[i++] = Wire.read();
}

void sleepNow() {
// can set the unused digital pins to output low – BUT only worth 1-2 µA during sleep
// if you have an LED or something like that on an output pin, you will draw more current.
// for (byte i = 0; i <= number of digital pins; i++)
// {
// pinMode (i, OUTPUT);
// digitalWrite (i, LOW);
// }

cbi(ADCSRA,ADEN); // Switch ADC OFF: worth 334 µA during sleep
set_sleep_mode(SLEEP_MODE_PWR_DOWN);
sleep_enable();
attachInterrupt(0,clockTrigger, LOW);
// turn off brown-out enable in software: worth 25 µA during sleep
// BODS must be set to one and BODSE must be set to zero within four clock cycles
// BUT http://learn.adafruit.com/low-power-coin-cell-voltage-logger/other-lessons
// MCUCR = bit (BODS) | bit (BODSE); // turn on brown-out enable select
// MCUCR = bit (BODS); // The BODS bit is automatically cleared after three clock cycles
sleep_mode();
//HERE AFTER WAKING UP
sleep_disable();
detachInterrupt(0);
sbi(ADCSRA,ADEN); // Switch ADC converter back ON
//digitalWrite(13, HIGH); this doesnt work because of conflict with sd card chip select
}

void clockTrigger() {
clockInterrupt = true; //do something quick, flip a flag, and handle in loop();
}

void clearClockTrigger()
{
Wire.beginTransmission(0x68); //Tell devices on the bus we are talking to the DS3231
Wire.write(0x0F); //Tell the device which address we want to read or write
Wire.endTransmission(); //Before you can write to and clear the alarm flag you have to read the flag first!
Wire.requestFrom(0x68,1); // Read one byte
dummyRegister=Wire.read(); // In this example we are not interest in actually using the bye
Wire.beginTransmission(0x68); //Tell devices on the bus we are talking to the DS3231
Wire.write(0x0F); //Tell the device which address we want to read or write
Wire.write(0b00000000); //Write the byte. The last 0 bit resets Alarm 1
Wire.endTransmission();
clockInterrupt=false; //Finally clear the flag we use to indicate the trigger occurred
}

// could also use RTC.getTemperature() from the library here as in:
// RTC.convertTemperature(); //convert current temperature into registers
// Serial.print(RTC.getTemperature()); //read registers and display the temperature

float get3231Temp()
{
//temp registers (11h-12h) get updated automatically every 64s
Wire.beginTransmission(DS3231_I2C_ADDRESS);
Wire.write(0x11);
Wire.endTransmission();
Wire.requestFrom(DS3231_I2C_ADDRESS, 2);

if(Wire.available()) {
tMSB = Wire.read(); //2’s complement int portion
tLSB = Wire.read(); //fraction portion

temp3231 = ((((short)tMSB << 8 | (short)tLSB) >> 6) / 4.0);
// Allows for readings below freezing – Thanks to Coding Badly
//temp3231 = (temp3231 * 1.8 + 32.0); // Convert Celcius to Fahrenheit
return temp3231;

}
else {
temp3231 = 255.0; //Use a value of 255 as error flag
}

return temp3231;
}
byte read3AxisAcceleration()
{
Wire.beginTransmission(BMA250);
Wire.write(0x02);
Wire.endTransmission();
Wire.requestFrom(BMA250,7);
for(int j = 0; j < 7;j++)
{
dataArray[j] = Wire.read();
}
if(!bitRead(dataArray[0],0)){return(0);}

BMAtemp = dataArray[6];
x = dataArray[1] << 8;
x |= dataArray[0];
x >>= 6;
y = dataArray[3] << 8;
y |= dataArray[2];
y >>= 6;
z = dataArray[5] << 8;
z |= dataArray[4];
z >>= 6;

BMAtempfloat = (BMAtemp*0.5)+24.0;
}
byte initializeBMA()
{
Wire.beginTransmission(BMA250);
Wire.write(0x0F); //set g
Wire.write(GSEL);
Wire.endTransmission();
Wire.beginTransmission(BMA250);
Wire.write(0x10); //set bandwith
Wire.write(BW);
Wire.endTransmission();
return(0);
}

long readVcc() { //trick to read the Vin using internal 1.1 v as a refrence
long result;
// Read 1.1V reference against AVcc
ADMUX = _BV(REFS0) | _BV(MUX3) | _BV(MUX2) | _BV(MUX1);
delay(3); // Wait for Vref to settle
ADCSRA |= _BV(ADSC); // Convert
while (bit_is_set(ADCSRA,ADSC));
result = ADCL;
result |= ADCH<<8;
result = 1126400L / result; // Back-calculate AVcc in mV
return result;
}

int freeRam () {
extern int __heap_start, *__brkval;
int v;
return (int) &v – (__brkval == 0 ? (int) &__heap_start : (int) __brkval);
}

void error(char *str) {
// always write error messages to the serial monitor but this routine wastes
// everything passed to the string from the original call is in sram!
Serial.print(F(“error in: “));Serial.println(str);
/* this next statement will start an endless loop, basically stopping all
operation upon any error. Change this behavior if you want. */
while (1);
}