Previous in Forum: Replacing Adobe PageMill 3.0   Next in Forum: Windows XP Home Edition
Close
Close
Close
9 comments
Power-User

Join Date: May 2006
Location: Chennai - India
Posts: 103

Firmware Testing

08/15/2009 3:48 PM

Hi, I work in an organisation dealing with firmware, software and hardware systems for our clients. We are a design house. One of our clients is in the field of auto-mobiles. We design and develop the concept and initial test prototypes for this client related to the auto-mobile industry. The customer of our client has requested him to do software/firmware testing for the future projects and the same request has been put forth to us by our client. We do test and verify our Concepts, algorithms, hardware and the code developed. The problem is that the client wants us to provide them with a firmware test report. The specifications or criteria for the same have not been provided by the client's customer. The client has asked us to find out hoe this can be done. Like software/ middle-ware testing is there also something for the embedded level firmware code? What are the criteria against which the tests are to be carried out? Are there any particular certifications for embedded level coding methodology? How can I go about the firmware testing? Are there any available tools or is it architecture or controller specific. Please respond to this query at the earliest as we need to get back to our client. (And yes I am searching for the same on the internet.)

__________________
I would like to be a human in true sense one day...
Also posted in: also posted in Transportation
Register to Reply
Interested in this topic? By joining CR4 you can "subscribe" to
this discussion and receive notification when new comments are added.

Comments rated to be "almost" Good Answers:

Check out these comments that don't yet have enough votes to be "official" good answers and, if you agree with them, rate them!
Guru
Popular Science - Weaponology - New Member United Kingdom - Member - New Member

Join Date: May 2007
Location: Harlow England
Posts: 16499
Good Answers: 662
#1

Re: Firmware Testing

08/15/2009 4:32 PM

It entirely depends on the application, and how big the code is.
Test it to the spec is the first spec, that's the easy part.
You need to work out data sets and input conditions which represent worst case conditions, exceptions and missuse conditions.
In other words, not only must you test that it does all the things it should, you must test that it does NOT do, all the things which it must NOT do, this may of course be an infinite set of tests, which leads back to the initial question.
The central locking on my Nissan Micra has softare which is barking mad, over complicated, annoying and bug riddled. It is, however not safety critical, as I do at least seem to be able to get out of the darned thing even if I can't always get into it.
I would be VERY careful about certifying it as 'error free' or such like. Let the customer write the spec' else you will just be setting yourself up to fail.
bear in mind there is no such thing as error free software.

How good is the documentation? Have you done documented code walkthroughs for any safety critical areas?

If all else fails delegate upwards!
Del

__________________
health warning: These posts may contain traces of nut.
Register to Reply
Guru
Engineering Fields - Electrical Engineering - New Member

Join Date: Sep 2006
Location: El Lago, Texas, USA
Posts: 2639
Good Answers: 65
#2

Re: Firmware Testing

08/15/2009 7:46 PM

One metric (I hate that word) we use is to verify that your test vectors exercise 100% of the executable code in your firmware. You also need to test your code under all combinations of inputs, if such a thing is possible.

Register to Reply
Guru
Engineering Fields - Systems Engineering - New Member Popular Science - Weaponology - New Member

Join Date: Jun 2006
Location: Borrego Springs
Posts: 2636
Good Answers: 62
#3

Re: Firmware Testing

08/15/2009 9:42 PM

Aviation, medical, and other safety critical functions have standards that are process based, so that even should the individual go astray; the system of checks and balances catches it.

BUT

Software development and hence software test are expensive, and as others have mentioned, prone to endless expansion.

So...develop your internal process first, and target specific levels of effort to the criticality of the code. If it makes the space shuttle go off course it needs more attention than locking you out of your car.

I believe the auto industry has a new set of guidelines (I'm getting the acronym wrong - but its like MISRA?) or you can develop your own.

I like the Carnegie Mellon (CMMI), or the IEEE (12207) but either one involves a process of creating the process the first time. But either method can be tailored to your product.

To get back on track - yes there are methods for firmware depending again on the criticality of the product.

Tools can get expensive and silly depending on how you apply them, so have a goal and criteria in mind before you go shopping. Silly sounds strange, but if you are testing instrumented code on the target environment but the environment is so limited that you have to run a single module at a time with all the calls stubbed out - you are really learning very little but satisfying a requirement. And expensively.

But what you are talking about it a simple extension of what you are already doing:

System Requirement---------------------------------------System Test

software requirement ----------------------------software test

middleware requirement ------------------middleware test

embedded requirement------------embedded test

The HOW of your testing will be very dependent on the exact hardware base. Is your firmware blown onto something inaccessable? or does your board have the complexity to support full Board Support with remote access?

You sound like you have pretty robust processes, how about analysing the product environment, doing a Failure Modes Analysis, and devising System level tests to detect specific failures. Then attempt to stimulate those failures?

That is about as far as I can go without more detail. Most test suites can test either on or off target - some are very expensive, some not so much. You will need to balance capability against cost and TIME. The more penetrating and accurate the testing, the more cost.

__________________
"If you want to get somewhere else, you must run at least twice as fast as that!"
Register to Reply
Power-User

Join Date: May 2006
Location: Chennai - India
Posts: 103
#4

Re: Firmware Testing

08/16/2009 6:15 AM

Thank you all for your prompt replies.

We do test our code for all possible and stray combinations checking what it should and should not be doing. It is simple to generate scenarios and sw checks in Automotive control systems with a few inputs, say 10 to 20 inputs and defined outputs.

We have been documenting our code and hardware designs.

The problem we are faced with is that we dont know if there is any standard report type or test standard that we need to prove our code against.

Or is it sufficient if we, say have 4 inputs and 2 outputs, present the client with the input output combinatorial matrix or truth table of sorts, citing all the exceptions and reserved states.

You know sometimes, rather than being interested in the outcome or even the procedure, people seem to be engulfed and motivated by the 'bling' and 'jargon'. Since it may be the management folks and not the technical team that may deal with it. The problem compounds as I think both the technical and the management teams would evaluate our reports. So I figured that if there exists a guideline that can be followed, it would be easy to put things at rest, given that the system does what it is supposed to and also not do what it is not supposed to.

Thanks again.

__________________
I would like to be a human in true sense one day...
Register to Reply
Guru
Popular Science - Weaponology - New Member United Kingdom - Member - New Member

Join Date: May 2007
Location: Harlow England
Posts: 16499
Good Answers: 662
#6
In reply to #4

Re: Firmware Testing

08/16/2009 8:34 AM

we dont know if there is any standard report type .

Don't worry about that.
Present the information, with a brief explanation/rationale at the start and a sumary at the end.
The chances are that the recipient won't understand it any way.
Don't waste time and effort 'tarting it up' which generally makes it less understandanble anyway.
Let the information speak for itself.
On no account let any market/sales/management anywhere near the report.
Del

__________________
health warning: These posts may contain traces of nut.
Register to Reply Score 1 for Good Answer
Guru
Engineering Fields - Systems Engineering - New Member Popular Science - Weaponology - New Member

Join Date: Jun 2006
Location: Borrego Springs
Posts: 2636
Good Answers: 62
#5

Re: Firmware Testing

08/16/2009 7:22 AM

Call it a Software Verification report, describe the method, and present the results in any summary format you like. Since you mention management, include a Summary or Conclusion.

I know of no standardized report format.

__________________
"If you want to get somewhere else, you must run at least twice as fast as that!"
Register to Reply
Power-User

Join Date: May 2006
Location: Chennai - India
Posts: 103
#7

Re: Firmware Testing

08/16/2009 3:12 PM

Thank you all very much.

I shall go ahead and present the firmware test and analysis report approach to my client. He would understand (he too is unsure of what the customer wants to see/know??)

This should do for now.

If by chance a few weeks down the line if they turn up and ask: "Well the report looks good. So then again, what standard did you say it satisfies(adhere's to)??"

I might knock at the gates again.

The meeting is tomorrow. I shall let you know the response then.

Thank you once again.

__________________
I would like to be a human in true sense one day...
Register to Reply
Guru
Engineering Fields - Systems Engineering - New Member Popular Science - Weaponology - New Member

Join Date: Jun 2006
Location: Borrego Springs
Posts: 2636
Good Answers: 62
#8
In reply to #7

Re: Firmware Testing

08/16/2009 6:24 PM

"Well the report looks good. So then again, what standard did you say it satisfies(adhere's to)??"


"Our Internal Process Standards"

__________________
"If you want to get somewhere else, you must run at least twice as fast as that!"
Register to Reply Score 1 for Good Answer
Anonymous Poster
#9

Re: Firmware Testing

08/19/2009 7:01 PM

Try this link

http://www.logigear.com/newsletter/understanding_metrics_in_software_testing.asp

Register to Reply Score 1 for Good Answer
Register to Reply 9 comments
Interested in this topic? By joining CR4 you can "subscribe" to
this discussion and receive notification when new comments are added.

Comments rated to be "almost" Good Answers:

Check out these comments that don't yet have enough votes to be "official" good answers and, if you agree with them, rate them!
Copy to Clipboard

Users who posted comments:

Anonymous Poster (1); bhankiii (1); Del the cat (2); edignan (3); vish_al210 (2)

Previous in Forum: Replacing Adobe PageMill 3.0   Next in Forum: Windows XP Home Edition

Advertisement