Firmware Testing

It entirely depends on the application, and how big the code is.
Test it to the spec is the first spec, that's the easy part.
You need to work out data sets and input conditions which represent worst case conditions, exceptions and missuse conditions.
In other words, not only must you test that it does all the things it should, you must test that it does NOT do, all the things which it must NOT do, this may of course be an infinite set of tests, which leads back to the initial question.
The central locking on my Nissan Micra has softare which is barking mad, over complicated, annoying and bug riddled. It is, however not safety critical, as I do at least seem to be able to get out of the darned thing even if I can't always get into it.
I would be VERY careful about certifying it as 'error free' or such like. Let the customer write the spec' else you will just be setting yourself up to fail.
bear in mind there is no such thing as error free software.

How good is the documentation? Have you done documented code walkthroughs for any safety critical areas?

If all else fails delegate upwards!
Del

One metric (I hate that word) we use is to verify that your test vectors exercise 100% of the executable code in your firmware. You also need to test your code under all combinations of inputs, if such a thing is possible.

Aviation, medical, and other safety critical functions have standards that are process based, so that even should the individual go astray; the system of checks and balances catches it.

BUT

Software development and hence software test are expensive, and as others have mentioned, prone to endless expansion.

So...develop your internal process first, and target specific levels of effort to the criticality of the code. If it makes the space shuttle go off course it needs more attention than locking you out of your car.

I believe the auto industry has a new set of guidelines (I'm getting the acronym wrong - but its like MISRA?) or you can develop your own.

I like the Carnegie Mellon (CMMI), or the IEEE (12207) but either one involves a process of creating the process the first time. But either method can be tailored to your product.

To get back on track - yes there are methods for firmware depending again on the criticality of the product.

Tools can get expensive and silly depending on how you apply them, so have a goal and criteria in mind before you go shopping. Silly sounds strange, but if you are testing instrumented code on the target environment but the environment is so limited that you have to run a single module at a time with all the calls stubbed out - you are really learning very little but satisfying a requirement. And expensively.

But what you are talking about it a simple extension of what you are already doing:

System Requirement---------------------------------------System Test

software requirement ----------------------------software test

middleware requirement ------------------middleware test

embedded requirement------------embedded test

The HOW of your testing will be very dependent on the exact hardware base. Is your firmware blown onto something inaccessable? or does your board have the complexity to support full Board Support with remote access?

You sound like you have pretty robust processes, how about analysing the product environment, doing a Failure Modes Analysis, and devising System level tests to detect specific failures. Then attempt to stimulate those failures?

That is about as far as I can go without more detail. Most test suites can test either on or off target - some are very expensive, some not so much. You will need to balance capability against cost and TIME. The more penetrating and accurate the testing, the more cost.

Thank you all for your prompt replies.

We do test our code for all possible and stray combinations checking what it should and should not be doing. It is simple to generate scenarios and sw checks in Automotive control systems with a few inputs, say 10 to 20 inputs and defined outputs.

We have been documenting our code and hardware designs.

The problem we are faced with is that we dont know if there is any standard report type or test standard that we need to prove our code against.

Or is it sufficient if we, say have 4 inputs and 2 outputs, present the client with the input output combinatorial matrix or truth table of sorts, citing all the exceptions and reserved states.

You know sometimes, rather than being interested in the outcome or even the procedure, people seem to be engulfed and motivated by the 'bling' and 'jargon'. Since it may be the management folks and not the technical team that may deal with it. The problem compounds as I think both the technical and the management teams would evaluate our reports. So I figured that if there exists a guideline that can be followed, it would be easy to put things at rest, given that the system does what it is supposed to and also not do what it is not supposed to.

Thanks again.

we dont know if there is any standard report type .

Don't worry about that.
Present the information, with a brief explanation/rationale at the start and a sumary at the end.
The chances are that the recipient won't understand it any way.
Don't waste time and effort 'tarting it up' which generally makes it less understandanble anyway.
Let the information speak for itself.
On no account let any market/sales/management anywhere near the report.
Del

Call it a Software Verification report, describe the method, and present the results in any summary format you like. Since you mention management, include a Summary or Conclusion.

I know of no standardized report format.

Thank you all very much.

I shall go ahead and present the firmware test and analysis report approach to my client. He would understand (he too is unsure of what the customer wants to see/know??)

This should do for now.

If by chance a few weeks down the line if they turn up and ask: "Well the report looks good. So then again, what standard did you say it satisfies(adhere's to)??"

I might knock at the gates again.

The meeting is tomorrow. I shall let you know the response then.

Thank you once again.

"Well the report looks good. So then again, what standard did you say it satisfies(adhere's to)??"

"Our Internal Process Standards"

Try this link

http://www.logigear.com/newsletter/understanding_metrics_in_software_testing.asp