When Do You Test Software

Doing it Modern:

Java – Junit – And – Ant

Here is an artical from IBM on how to do it with the latest tools. People are building in Test features befor they do the first build.

This is a lead on how they are doing that.

This piece is long overdue. Many companies treat Software QA much like this: write the code, then throw it over the wall to the QA department which, through testing, somehow imbues the code with "quality" after the fact. And, like this piece mentions, cluebag project managers (frequently) subject designers to moving targets. They release some vague specification - if they release one at all - and then expect the developers to construct a concrete, working system out of vague ambiguity. Then, if the system doesn't work or works differently than the manager had in mind, it's the design team's fault. Nor do many PMs ever seem to be around when the page is blank. If development companies spent as much time and effort on SQA as they do on internal finger-pointing, the quality landscape would today be, IMO, vastly different.

An excellent model of design-for-quality was seen in IBM's Federal Systems Division. FSD designed the early versions of the ultra-reliable Space Shuttle software. Bug rates in commercial software often exceeds one bug per thousand lines of code (Windows was/is considerably worse than this. For instance, Windows 95 was released with over 65,000 known bugs. But it was released anyway. Say hello to CrapWare). In contrast, the code churned out by FSD seldom exceeded one bug per 10,000 lines of code. So it is possible with the right attention to requirements, specification, testing, and measurable quality throughout the development cycle.

IBM FSD is no longer extant, but their example endures. Studying FSD's development process - in which SQA was integrated at every turn - is a worthwhile investment of time for developers/companies of all stripes.

Testing software is easy..

First you test all the things it should do.

Then you test the infinite number of things it shouldn't do.

Then you put it out on field trial for 6months.

Then you release it, and the major bug appears!

You obviously don't work for Microsoft, Del. Their process is somewhat different.

• Put it on field trial indefinitely, or until the next release. Or both. Yeah, both.
• Require the user to pay up front for the privilege of testing our software, but conceal this fact from the user.
  
• Release the software with several thousand known bugs. So what? They're gonna buy it anyway. We're Microsoft. We own the market - and Washington - and so why should we care?
• Let the user test it and provide us with feedback which we will summarily ignore. We've got better things to do - like adding more bells and whistles and DRM code that nobody asked for and that break previously-working code.
  
• Help out Hollywood and the RIAA at every turn by having our OS sniff for illegal copies of stuff. Isn't that what an operating system is for? And besides, they've got much deeper pockets than those detestable herd animals we call "customers."
  
• Change the release number and repeat.
  

Yup I couldn't get away the that...our customers know where we are!

There's nothing quite like accountability. "Here at Microsoft we're accountable to no one but our shareholders. The rest of you be damned."

The simple and obvious answer to the direct question here is: "Test it and re-test it all the time. Start testing it's validity, integrity, and viability, as soon as you write the first code-blocks, and keep doing it until you release your beta for external testing phases"

The testing of software is an endless process. It's inherent in the nature of this medium. Software is a logical and thus a contextual medium, and as such, and according to Gödel's Theorem, above a certain threshold of complexity, any logical system may contain inconsistencies and inherent flaw of integrity.

The associated problem concerning software is layed in the term 'machine-states' or 'number of inherent machine-states'.

Any mechanism whatsoever (Not just logical ones) is characterised with it's given number of 'inherent machine-states'. A toggle has two machine-states; A gun may have three to fifteen, etc., etc.

When it comes to logical mechanisms (often called "Information-Systems") the inherent complexity and thus their inherent number of machine states jumps into a leap:

The best known example is the binary counting system. The addition of each two-state bit doubles the number of inherent machine-states of the number. A mere thirty-element number may contain billions of possible machine-states. Even more so, the decimal counting system, where the addition of each (ten-state) element multiplies the mofo by ten...

Every little, simple, logical complex, not even the size of an app, but just the size of a functional-block within a "program", has a monster of a machine-state-number, thus the potential to contain at least one Gödel's inconsistency, the root of all bugs.

The more additional blocks interact, even within a relatively small program, the more that monster grow, exponentially.

A bug is simply a rare, unchecked machine-state, which was not taken into account, when the program was written. A rare machine-state in the sense that it's rare occurrence and it's inherent consequence, was not pre-calculated and provided for.

In other words: 'A precedence, to challenge the machine's processing capability'.

In simple probability terms it means:

A. It is almost always impossible for a programmer to anticipate all the machine-states their program may produce, let alone it's eventual set of logical consequence.

B. It is almost always impossible for a programmer to provide for those potential bugs, because each such provision adds to the complexity of the program, thus increases the potential of additional bugs to emerge.

C. To ease the burden of this harmful potential, many programmers use a method called "Runner-Demon" or "Construction-Demon" which is a sub-routine written especially or adapted for (but not incorporated in) each routine or functional-block in a written program, and it re-iterates millions or billions of times, to test the 'Input-Process-Output" pipe of the routine through all the possible machine-states inherent to the block tested.

Alas, even this may not be enough, because it deals (and may eliminate) only with what's called "Logical Inconsistencies" not with "Contextual inconsistencies", which may result from the interaction between blocks.

D. "Contextual" are by far the biggest monster of the whole medium, because they multiply the number of machine-states and thus Logical inconsistencies by wild powers.

- - - -

The oldest wisdom of all this is therefore:

- "If it's there and it does the job, don't try to fix or improve it"

- "Better re-write a function than try to fix it, because one of those version may just come out perfect by chance"

- "There is no methodology to debugging, only mere luck"

- - - -

A reference to the subject of machine-states, in logical mechanisms.

(Infinite-state machines, are not in the realm of logical mechanisms, as those are analog mechanism such as wheels, levers, and elastic objects, machines which have infinite or indefinite number of inter-states).