Previous in Forum: Setting Earth Fault Relay   Next in Forum: NEC Standards for Fire Alarms, Heat Detectors, and Smoke Detectors
Close
Close
Close
11 comments
Rate Comments: Nested
Active Contributor

Join Date: Feb 2008
Posts: 24

SPI Serial EEPROM Corruption

05/20/2008 2:00 PM

Hello All,

I work for a Medical Equipment manufacturer. Our equipment is sold internationally and is designed to operate on 120/240VAC, 50/60 Hz power. We use a number of Atmel 25256A serial SPI bus EEPROMS, one of which is located on a lamp assembly containing 18 fluorescent lamps, and is used to tally operating time during the interval when the lamps are in operation.

We have recently experienced several failures at a site in Rotherham, England (new site) where the data contained in the EEPROM is corrupted by one or two bits. The power there is 240VAC, 50Hz. These failures have also occurred here in the US (120VAC, 60Hz), albiet much less frequently. The failure seems to occur when to unit is either powered down or up, since the failure is only detected when the unit is powered up after running with no problems during the previous use.

At the time when the unit is shut down, the lamps are not in operation, and the operation time data is collected (EEPROM is written) only during the time when the lamps are on. The lamps are only used after the unit has been turned on for some time during the treatment cycle.

I am of the opinion that we have a power glitch that corrupts the data stored in the device, since the only way to write data into the device involves a fairly complex programming sequence.

I was wondering if anyone has had similar problems with Serial EEPROMS that were found to be due to power fluctuations.

Regards,

Ron

Register to Reply
Interested in this topic? By joining CR4 you can "subscribe" to
this discussion and receive notification when new comments are added.
Guru
Popular Science - Biology - New Member Hobbies - Musician - New Member APIX Pilot Plant Design Project - Member - New Member Hobbies - CNC - New Member Fans of Old Computers - ZX-81 - New Member

Join Date: Jan 2007
Location: Centurion, South Africa
Posts: 3921
Good Answers: 97
#1

Re: SPI Serial EEPROM Corruption

05/20/2008 2:31 PM

I am not in electronics but contactor bounce may have that effect of transmitting false data. What about a delay subroutine to wait until contact is stable?

What baud rate are you using?

What is the binary representation of the wrong bytes? any trains of 0's or 1's?

__________________
Never do today what you can put of until tomorrow - Student motto
Register to Reply
Active Contributor

Join Date: Feb 2008
Posts: 24
#2
In reply to #1

Re: SPI Serial EEPROM Corruption

05/20/2008 2:49 PM

I am in the process of examining the data to determine where the checksum error is. It is not apparent (no unusual trains of one's or zero's). It seems to be a one or two bit difference.

Our system does not have a contactor, and the switch transient has caused no trouble to date. I suspect the power in the location may have something to do with the problem.

Regards,

Ron

Register to Reply
Guru
Engineering Fields - Electrical Engineering - New Member

Join Date: Sep 2006
Location: El Lago, Texas, USA
Posts: 2639
Good Answers: 65
#3

Re: SPI Serial EEPROM Corruption

05/20/2008 6:19 PM

The fact that only one or two bits is corrupted makes me think that you have a noise problem rather than a power glitch. It wold be useful to know where the bad bits are - are they random, or are they always at the beginning or end of your programming stream? What is happening at power up and power down in terms of what commands are being sent to the EEPROM? Is it possible that data is being written during power down, or that data is being read before power is fully up? Do these EEPROMS have a separate Vpp programing voltage pin that might be affected in some strange way by a power glitch - for instance, if the Vpp pin is grounded but you get a -12V glitch on your power it's possible that the Vpp pin might look active to the device.

Do these failures only occur in medical facilities? Is an X-Ray machine or radiation device nearby?

Register to Reply
Guru
Popular Science - Weaponology - New Member Safety - ESD - New Member Hobbies - Fishing - New Member

Join Date: Sep 2006
Location: Near Frankfurt am Main, Germany. 50.390866N, 8.884827E
Posts: 17996
Good Answers: 200
#4

Re: SPI Serial EEPROM Corruption

05/21/2008 3:34 AM

I would guess that you are not able to stop high value spikes getting into your equipment.

May I suggest that you temporarily install a supply monitor at the sites where you have seen this problem. As somewhere nearby, something is generating big spikes.

Probably quite often, but the problem only happens when two or more spikes occur at one and the same time, giving an even bigger spike which is then causing your problem.

Revise your power supply to filter even more spikes of even larger size and duration asap.

The 230volt site will probably allow more spikes through. Plan for spikes of more than 1000 volts at least in the UK......

I have seen lots of problems on US made equipment that was "cheaply" modified to work in Europe, with little understanding of the possible problems - Triacs and SCRs with a working voltage of only 600 volts, quickly degrade in Europe!!!

__________________
"What others say about you reveals more about them, than it does you." Anon.
Register to Reply
Anonymous Poster
#5

Re: SPI Serial EEPROM Corruption

05/21/2008 9:04 AM

OK, Let me add a few bits of information to the discussion. The instrument is designed to work in both environments (US and Europe) and has a medical grade transformer that has connections for several operating voltages. Internally, the AC power delivered to the rest of the instrument is 120VAC. Switching power supplies with a very wide input voltage range supply the secondary DC rails. The instrument is heavily filtered against a wide range of input power disturbances.

The way these particular serial EEPROMs work in the write mode is governed by a very specific series of commands. You must first issue a write enable command, then follow it with a write command with location and data information. Reads are non-destructive to the stored data. The devices are rated for one million write cycles and these failures occur very short of that limit.

We have been able to determine in one unit that was returned for evaluation that 4 byte locations contain data that is incorrect in locations that are normally not written to in normal operation. There are at least two more units on their way here for evaluation with the same failure.

What I am asking for is information from anyone who has experienced similar failures - in any environment - and what was done to correct the problem. This of course does not include problems caused by programing errors.

I have personally observed disturbances on the +5V rail that exceeded the absolute maximum limit, but were of such short duration as to be of no interest other than to prove that they can exist. I spoke to the Applications Engineering Manager at Atmel who described what would take to corrupt data in the device, but so far I have not observed these conditions.

I do suspect that a power line event could cause the problem, but I need additional examples to support my contention, and if possible, well defined conditions that can be duplicated in the lab.

Regards,

Ron

Register to Reply
Guru
Popular Science - Cosmology - New Member Technical Fields - Technical Writing - New Member Engineering Fields - Energy Engineering - New Member Engineering Fields - Electrical Engineering - New Member Engineering Fields - Control Engineering - New Member Engineering Fields - Electromechanical Engineering - Old Member, New Association

Join Date: Apr 2008
Location: Lexington, KY
Posts: 1639
Good Answers: 73
#6

Re: SPI Serial EEPROM Corruption

05/23/2008 4:49 PM

FYI, most fluorescent lamp ballasts are now electronic. These devices now drive your fluorescent lamps at frequencies in the range of 25KHz to 80KHz. Usually this is not a problem as long as you don't have any instrumentation wires too close. But most instrumentation power supply or other filters are not optimized for that frequency band. So it is possible to bleed enough energy into a data or address buss and cause problems. If your ballasts are electronic, check out your proximity between devices. You may need to use some common mode rejection techniques to isolate sensors from your computer.

__________________
A great troubleshooting tip...."When you eliminate the impossible, whatever remains, however improbable, must be the truth." Sir Arthur Conan Doyle
Register to Reply
Active Contributor

Join Date: Feb 2008
Posts: 24
#7
In reply to #6

Re: SPI Serial EEPROM Corruption

05/27/2008 5:56 PM

Joe,

We use three lamp ballasts (electronic - basically switching regulators) in our system each of which drives six lamps in a series-parallel configuration. Ordinarily this is the first place I would look for trouble, but in this case the lamps - and the ballasts - are off (not powered) when the corruption occurs.

I have been doing a bit more information gathering and it's beginning to look like the data is written when the system is shutting down, since it only occurs in fairly specific locations, and there is no warning to the main system CPU (a PIC chip) as to when the power is removed. I'm of the opinion that as the power ramps down, the processor is looping to randon locations in the program as power drops below acceptable operating levels.

Now I have to prove it.

Register to Reply
Guru
Popular Science - Cosmology - New Member Technical Fields - Technical Writing - New Member Engineering Fields - Energy Engineering - New Member Engineering Fields - Electrical Engineering - New Member Engineering Fields - Control Engineering - New Member Engineering Fields - Electromechanical Engineering - Old Member, New Association

Join Date: Apr 2008
Location: Lexington, KY
Posts: 1639
Good Answers: 73
#8
In reply to #7

Re: SPI Serial EEPROM Corruption

05/28/2008 8:39 AM

Sounds familiar. If you check out www.microchip.com you should find all kinds of information about power-down warnings dealing with the issues you seem to be observing. The PIC to which you refer could be one of theirs. They have a big site with lots of well documented information. I have used their products on some projects and they are very helpful. If you have access to the source code on the chip, it may be to your advantage to have them review it for proper shut down methodology. Or take up the concern with the mfg of your equipment using warnings from Microchip as your "probable cause". In any case it is probably a firmware error because many of the Microchip devices have power supply watchdog circuitry that can be used to prevent that situation. That function, however, has to be written into the code.

__________________
A great troubleshooting tip...."When you eliminate the impossible, whatever remains, however improbable, must be the truth." Sir Arthur Conan Doyle
Register to Reply
Commentator

Join Date: Jan 2008
Posts: 83
#9

Re: SPI Serial EEPROM Corruption

07/19/2008 4:13 AM

I would check temperature specifications for the chip. then I would monitor temp long term in the problem sites. check the "clock" chip spec's also.-best of luck.

Register to Reply
Active Contributor

Join Date: Feb 2008
Posts: 24
#10

Re: SPI Serial EEPROM Corruption

07/21/2008 9:10 AM

Hello All,

I thought it might be a good idea to post an update on the problem - and the solution.

The situation wound up deteriorating to the classic hardware vs software conflict, but in this case hardware had an edge. After examining several "corrupt" EEPROMS we found that only certain - predictable - locations were corrupted. After the usual arguments and finger pointing, we set up a test duplicating the conditions that were reported from the field and assumed to be the cause of the problem, and caused several more corruptions under lab conditions.

It happens that the problem only occurred during "power down" events. The brownout setting of the PIC microcontroller had been set too low and as the Vcc drifted down, the uP became senile and code execution was unpredictable.

Score: Hardware 1, Software 0 (for a change)

Actually, we have a pretty good group here and any conflict is more good-natured kidding around than anything else. The goal is to solve the problem, not waste time placing blame.

Ron

Register to Reply
Commentator

Join Date: Jan 2008
Posts: 83
#11

Re: SPI Serial EEPROM Corruption

07/31/2008 3:51 PM

I am glad you solved this and I learned something.

Register to Reply
Register to Reply 11 comments
Copy to Clipboard

Users who posted comments:

Andy Germany (1); Anonymous Poster (1); bhankiii (1); Hendrik (1); NotUrOrdinaryJoe (2); pretzel (2); Ronz (3)

Previous in Forum: Setting Earth Fault Relay   Next in Forum: NEC Standards for Fire Alarms, Heat Detectors, and Smoke Detectors
You might be interested in: Lamps, High Intensity Discharge Lamps

Advertisement