Engineering Management Blog

Engineering Management

The Engineering Management Blog is the place for conversation and discussion about engineering and project management, technology forecasting and planning, productivity tools, and safety and security. Here, you'll find everything from application ideas, to news and industry trends, to hot topics and cutting edge innovations.

Previous in Blog: Facebook Freedoms?   Next in Blog: Three Reasons Your New Hire Will Fail
Close
Close
Close
15 comments

Experiment Results Can Get Worse

Posted February 16, 2011 7:51 AM

Evidence suggests that when you try to duplicate a scientific study or experiment, the spread between the hypothesis alternatives gets narrower with each cycle. You begin to wonder if you did something wrong. Perhaps the original experimenters made the error. Have you ever had that experience? How do you combat these results? How closely did you match the initial experimental conditions? What could you have done better? Could you have done better? In fields that rely heavily on such studies, how confidently can we rely on our conclusions?

The preceding article is a "sneak peek" from Engineering Management, a newsletter from GlobalSpec. To stay up-to-date and informed on industry trends, products, and technologies, subscribe to Engineering Management today.

Reply

Interested in this topic? By joining CR4 you can "subscribe" to
this discussion and receive notification when new comments are added.

Comments rated to be "almost" Good Answers:

Check out these comments that don't yet have enough votes to be "official" good answers and, if you agree with them, rate them!
Anonymous Poster
#1

Re: Experiment Results Can Get Worse

02/17/2011 12:12 PM

Most science deals with what are considered "fixed" systems. The human body, or system, while basically fixed, is also adaptable. At least that's what evolutionists would say. This article is about pharmaceuticals and the human body. The results only demonstrate that as much as chemistry is a hard science, incomplete knowledge of the human body, keeps chemists from being able to predict, in any absolute manner, what their drugs will do. Therefore, the title of the thread is a little misleading. Science involving living systems, is different, from science which does not. One is considered repeatable, while the other may not be, as evidenced by this article.

As an aside... this reminds me of my beliefs about "natural" health remedies, as opposed, to pharmaceuticals. I hypothesize, that pharmaceuticals are much like organ transplants. The body tends to reject them, either by "active" defense mechanisms, or just plain upsetting the apple cart of some "normal" processes in the body. The manifestation of this rejection is labeled "side effects" of the particular pharmaceutical. Using food and plant derived substances as "pharmaceuticals" (not isolated or "standardized") seems better tolerated by the body, in general. No doubt, the body is picky about what one puts in to it. But there seems to be some recognition by the body, that plants are "friendly" systems, based on similar processes to ours. The same can not be said of most pharmaceuticals.

Reply
Guru
Engineering Fields - Manufacturing Engineering - Hobbies - Musician - Engineering Fields - Mechanical Engineering - Popular Science - Weaponology -

Join Date: Jun 2007
Location: Eden
Posts: 1432
Good Answers: 39
#14
In reply to #1

Re: Experiment Results Can Get Worse

03/16/2011 4:10 PM

I can't say that I agree with the statement that plants are naturally better tolerated than pharmaceuticals. There are a multitude of plants, and animals for that matter, that are poisonous to the human body. They gain no special safety significance merely because they are "natural".

In fact, I could almost argue the opposite of what you stated. Pharmaceuticals, by their very nature, are constructed from the ground up. You bake a brand new cake... you know exactly everything that you put into that cake. You can research the effects of every individual item. On the other hand, if you find some new plant... you don't know every single "ingredient" in that plant, until you study it, break it down, hope that you've identified every single component that exists in that plant... and then after all that, you still have to figure out the effects of every component in its makeup.

There is nothing intrinsically safe about a product simply because it is "natural". There are simply more unknowns. I've not crawled onto the "all-natural" bandwagon. I don't believe that it can be said that either is more safe or better for you, than the other. When it comes down to it... everything is made up of chemicals, whether it be manufactured and constructed, or whether it be found in something biological. Research on one is done just as carefully as research on the other.

And the results and conclusions based on this research is just as reliable or unreliable, equally; as are their benefits, effects, and level of safety.

In fact, there may be slightly less research done on natural items, due to the prevailing trust in things that are categorized as natural.

But I will say that "natural" has certainly been beneficial to marketing.

And no... I do not represent, nor do I have any affiliation to any pharmaceutical company

Reply Score 1 for Good Answer
Guru
Panama - Member - New Member Hobbies - CNC - New Member Engineering Fields - Marine Engineering - New Member Engineering Fields - Retired Engineers / Mentors - New Member

Join Date: Dec 2006
Location: Panama
Posts: 4273
Good Answers: 213
#2

Re: Experiment Results Can Get Worse

02/17/2011 1:53 PM

Back in the days when I was involved in R&D, one common feature of every published experiment that I desired to replicate was that insufficient information was provided as to the actual experimental conditions under which the published results were accomplished, no matter whether it involved living or inanimate models. This invariably resulted in the need for consultations with the original experimenters, or additional experiments to identify critical, unpublished constraints, or both. Furthermore, just because the results were publishable and reproducible did not necessarily mean they were universally applicable...

Reply
Guru
Hobbies - CNC - New Member Popular Science - Biology - New Member Hobbies - Musician - New Member

Join Date: Dec 2008
Location: Canada
Posts: 3497
Good Answers: 145
#3

Re: Experiment Results Can Get Worse

02/17/2011 7:48 PM

That's quite an awesome article, in part because it gives equal credence to a number of causes for the 'decline' effect. So although I find it tempting to hold forth on my most cherished a priori convictions and express my bias, I can't get away with it.

Schooler has a really good idea, which mirrors some ethical detail in Helsinki 2004 concerning the need to make the design of all research on human subjects public: .."Schooler recommends the establishment of an open-source database, in which researchers are required to outline their planned investigations and document all their results." This is bound to help, it means there is opportunity for anyone to point out a source of error, as we are so prone to overlook.

John Crabbe's test of replicability speaks volumes about the inadequacy of the scientific method to study behavior (and other in vivo effects). Ultimately, we need to develop a methodology that is designed for complex systems, or accept the fact that the statistical conclusions we are generating cannot be applied in the way that is done, so shamelessly these days, and rubber stamped as "scientific fact".

__________________
incus opella
Reply
Guru
Hobbies - Fishing - New Member

Join Date: Jun 2008
Location: Raleigh, NC USA
Posts: 13529
Good Answers: 467
#4

Re: Experiment Results Can Get Worse

02/18/2011 8:52 AM

Now wait just a darn minute!!!

First of all, you're citing an article from The New Yorker, purporting to bring established scientific method into question?...................Please!

Secondly, almost every experiment sited, related to behavior within a particular species, be it mice, birds or humans. All social beings with brains. Like the mice on cocaine experiments..............do they know that all mice in each experiment had the same reaction, or could it be that one mouse had a different reaction, and the rest just followed suit? They don't.

As far as antidepressants and anti psychotic pharmaceuticals go, one thing has been established, and that's that the experts don't know exactly how they work. No two human beings have identical fingerprints, to think that brain chemistry among humans is close enough to allow experimental trials to be predictable and repeatable is ridiculous..............at least for now.

Hundreds of thousands of experiments, (outside of psychology and behavior), have been done, and replicated, time and time again.

This article is bogus. The author has attempted to throw all established scientific method into question. This is a disservice to every scientist, chemist, biologist, et. al., that both carry out, and replicate experiments on a regular basis.

There are far better publications out there than The New Yorker, for you to get your information. You should have a look.

__________________
Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety. Ben Franklin
Reply
Guru

Join Date: May 2010
Location: in optimism
Posts: 4050
Good Answers: 129
#5
In reply to #4

Re: Experiment Results Can Get Worse

03/09/2011 1:27 AM

Settle down Mark - first of all it's about 'psychology' and 'apothecary' - not 'Physics' or 'Chemistry'.

An obvious question is the reliability of the 'experiment design' by 'non-scientists'

E.g. if you have asymmetry in a species - how could you possibly think the females were using symmetry as a mating priority benchmark?

If you seek to 'prove' ESP and your 'star subject' crashes abysmally on re test - might it not be an idea to examine the test condition variables?

E.g. was the reflection in your eyes or spectacles 'visible to the star' the first time? - and not the second?

There is nothing in that 'extensive speculative journalistic periphrasis' that is not simply saying - these people cannot design an experiment or 'trial' without predilection.

__________________
There is no sin except stupidity. (Oscar Wilde, Irish dramatist, novelist, & poet (1854 - 1900))
Reply
Guru
Hobbies - Fishing - New Member

Join Date: Jun 2008
Location: Raleigh, NC USA
Posts: 13529
Good Answers: 467
#6
In reply to #5

Re: Experiment Results Can Get Worse

03/09/2011 6:08 AM

Right, but both the article and the blogger left the impression that all scientific experiments could be brought into question.

Evidence suggests that when you try to duplicate a scientific study or experiment, the spread between the hypothesis alternatives gets narrower with each cycle.

I've done my own forays into what makes makes people tick, including, reading every bit of published correspondence that I could get my hands on between Freud and Jung, and other books, and actually attending a psychiatry conference. I'm no expert, but I've reached a few conclusions.

Psychiatry/psychology is not science.

Drugs that affect mood/behavior are not fully understood, and are way, way over prescribed.

Psychiatrists/psychologists are some of the most fu%^ed up people on the planet.

I prefer to just see my way through my own madness.

__________________
Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety. Ben Franklin
Reply
Guru

Join Date: May 2010
Location: in optimism
Posts: 4050
Good Answers: 129
#7
In reply to #6

Re: Experiment Results Can Get Worse

03/09/2011 6:32 AM

Ok - on refection - why don't we team up?

I'll dismantle every 'failed un-designed false hypothesis' and you deliver the blows in the emotive form they seem to grasp/work in?

__________________
There is no sin except stupidity. (Oscar Wilde, Irish dramatist, novelist, & poet (1854 - 1900))
Reply
Guru
Hobbies - Fishing - New Member

Join Date: Jun 2008
Location: Raleigh, NC USA
Posts: 13529
Good Answers: 467
#8
In reply to #7

Re: Experiment Results Can Get Worse

03/09/2011 7:57 AM

Sounds like fun, but to what end?

If a tree falls in the forest, and no one is there to hear it, does it still make a sound?

If you're talking across the entire forum, we would be very busy...........and no one would hear the tree fall.

If they did, it would just piss them off.

__________________
Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety. Ben Franklin
Reply
Guru
Panama - Member - New Member Hobbies - CNC - New Member Engineering Fields - Marine Engineering - New Member Engineering Fields - Retired Engineers / Mentors - New Member

Join Date: Dec 2006
Location: Panama
Posts: 4273
Good Answers: 213
#9
In reply to #8

Re: Experiment Results Can Get Worse

03/09/2011 12:15 PM

Did I just hear a tree fall????

That upsets me! Nothing worse in this life than someone cutting down a tree just to make lumber, when we have all these fantastic plastics around...

Just in case there are sensitive eyes perusing this missive, it is intended as a humorous response...please don't take offense!

Reply
Guru
Hobbies - Fishing - New Member

Join Date: Jun 2008
Location: Raleigh, NC USA
Posts: 13529
Good Answers: 467
#10
In reply to #9

Re: Experiment Results Can Get Worse

03/09/2011 1:01 PM

Gotcha! That's why I started using these sometimes, just in case.

__________________
Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety. Ben Franklin
Reply
Guru
Hobbies - CNC - New Member Popular Science - Biology - New Member Hobbies - Musician - New Member

Join Date: Dec 2008
Location: Canada
Posts: 3497
Good Answers: 145
#11
In reply to #8

Re: Experiment Results Can Get Worse

03/09/2011 1:07 PM

For the trees that are falling in this thread, you're gonna need a lumberyard!

__________________
incus opella
Reply
Guru
Hobbies - Fishing - New Member

Join Date: Jun 2008
Location: Raleigh, NC USA
Posts: 13529
Good Answers: 467
#12
In reply to #11

Re: Experiment Results Can Get Worse

03/09/2011 5:41 PM

Fortunately, anyone that could hear them fall has already come to see what all the ruckus is about.

__________________
Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety. Ben Franklin
Reply
Guru

Join Date: May 2010
Location: in optimism
Posts: 4050
Good Answers: 129
#13
In reply to #12

Re: Experiment Results Can Get Worse

03/09/2011 10:06 PM

Umm - now if the trees falling have attracted posters then there must have been 1 around 02/17/11, but none after 02/18/11, until some obviously increased logging on 03/09/11, that has TRIPLED the number!

I have it! - the post numbers the correlate with the number of trees felled! Perfect! A stunning new "scientific study paper" is born!

"Internet post rates linked to CO2 rise"

Now - my fame assured - all I need to do it re-post the OP - and if it doesn't replicate I can write a new paper on the inexplicable nature of diminishing results!

I'll call it;

"So what if it's a repeat! - come up with a new response - NOW!"

__________________
There is no sin except stupidity. (Oscar Wilde, Irish dramatist, novelist, & poet (1854 - 1900))
Reply Score 1 for Good Answer
Guru
Hobbies - Fishing - New Member

Join Date: Jun 2008
Location: Raleigh, NC USA
Posts: 13529
Good Answers: 467
#15
In reply to #13

Re: Experiment Results Can Get Worse

03/16/2011 4:41 PM

Hey 34,

Your response fell through the cracks. I stopped getting email discussion updates for a couple of days.

My response is, don't ever confuse me for a quick thinker. You'll be disappointed.

__________________
Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety. Ben Franklin
Reply
Reply to Blog Entry 15 comments
Interested in this topic? By joining CR4 you can "subscribe" to
this discussion and receive notification when new comments are added.

Comments rated to be "almost" Good Answers:

Check out these comments that don't yet have enough votes to be "official" good answers and, if you agree with them, rate them!
Copy to Clipboard

Users who posted comments:

34point5 (3); Anonymous Poster (1); artsmith (2); cwarner7_11 (2); kramarat (6); Out of Box Experience (1)

Previous in Blog: Facebook Freedoms?   Next in Blog: Three Reasons Your New Hire Will Fail

Advertisement