Challenge Questions

It's my understanding that "the" is the most commonly used word in the English language. So, assuming that it occurs 10x on each page of a 1400 page tome, this would give over 14,000 occurrences of the word. A manuscript page is conventionally accepted as 250 words. Obviously this would lend itself to arriving at over 100 pages of nothing but the word "the". Which would make Tolstoy proud, I'm sure.

14,000/250 = 56

The calcs also need to take word length into account. This site quotes English as having an average word length of 5.1 letters, but I've no idea whether a translation of Tolstoy would have the same average.

Well, I used to be able to do math in my head.....

Ya know, this also begs the question--why would they bring War and Peace along while they were fishing? I think John and Tom need other hobbies.

So if you throw the 1400 page book overboard, what happens to the level of the lake?

War and Peace on a fishing trip ? Try NOT getting into a battle over the discussion of the size of the ONE THAT GOT AWAY ! The ? Duh !

You are right with that one. I read it last winter and found it a 1400 page soap opera, sux.

I ran the first 10 chapters of Book One and got the following statistics:

Words: 16729

Characters: 76972

Chars w/Spaces: 93293

"the" occurrences: 815

Percentages come out to:

"the" / Words = 4.9%

"the" / Characters = 3.2%

"the " / Chars w/Spaces = 3.5% (I used "the" with one space instead of two)

Spaces / Chars w/Spaces = 17.5% (that's alot of empty reading )

Per wiktionary, "the" frequency is about 5.6%, based on frequency counts in a wide range of literature (Project Gutenberg). From my understanding, this is based on word count, so the percentage above of 4.9% is inline with this as well as your estimate based on character count, albeit slightly lower. Thus, 5% x 1400 pages x 250 words/page = 17500 "the"s, which at 250 words/page results in 70 pages. However, since "the" is shorter than the average word, all of them should take up fewer pages than the calc based on word count alone.

It seems I am better at higher math than at counting. It continues to amaze me how often I make the most obvious, simple mistakes. I surely could have counted the 3 letters in "the" and added one for a space to get 4 not 5.

Thanks,

Jim

"To adjust the number of pages for "the " shorter word length, multiply by 0.719 (4 char / 5.56 char):

53 x 0.719 = 38 pages"

I just realized the character count already takes into account the shorter word length, so this correction is not necessary. However, the pagination correction is necessary, so the number of "the" pages is:

53 x 1.164 = 61.6 pages

Still well below 100 pages.

GA for a very thorough analysis.

Depending upon the version you down loaded, the font kerning might be different from the version in the printed book. Sine the word "the" contains three letters which do not require reducing the character spacing to improve the way it looks, the number of pages should be increased. Below are a lines with the words "the" and "ill" repeated 20 times.

the the the the the the the the the the the the the the the the the the the the

ill ill ill ill ill ill ill ill ill ill ill ill ill ill ill ill ill ill ill ill

The difference between these two words is about 80/42.

A more detailed analysis is required to determine the occurrence of each letter. Considering the picky fact might increase the number of pages above 38, but the basic answer should still be that the number of pages filled with just "the" will be well under 100.

You are absolutely right - I am sure the kerning will be different, especially since my original analysis was based on a non-proportional font with no kerning. After I did the analysis, I tried to play around with the effects of proportional fonts, margins and justification, but I found that the text version I downloaded had carriage returns at the end of each line instead of spaces. This only allowed kerning on a line-by-line basis and would not adjust paragraphs when additional space was created at the end of each line.

I wrote a VBA routine to remove all the inter-paragraph CR's and was then able to play around with different fonts and justification. In fact, just getting rid of all those CR's shortened the book by about 5 pages by utilizing the extra space at the end of lines. I also looked at the impact of the 'word + spaces' length after replacing all the CR's, but there was minimal change and only affected the length of "the" book by about half a page.

Finally, I used one proportional & one non-proportional font to create two books that were about 1400 pages each by adjusting font size, page size & margins. Using the page setup & font from each one, I created a book with 34544 "the"s (as MPM did in #13 below) and came up with the following:

"the" Book (proportional font) = 42 pages

"the" Book (non-proportional) = 50 pages

In order to get an accurate answer, we would need to look at line length and numbers of close-spaced lines we can get on a page, rather than trying to count the average length of the characters.

So we would measure the physical length of a "the" measured between a fixed point on the "e" and an immediately preceding "e", and the maximum length of a line. I'm not certain how to handle "The", as this should either follow a full-stop or be at the beginning of a paragraph.

So we've a trade-off between the width of the characters in " the" (including the space) being somewhat above average, and the reduction in wasted space (due to the use of short words and no paragraph breaks). I suspect this would result in rather less than 60 pages.

But there are still ambiguities - Tom may decide the bet included the paragraph break whenever "The" occurs at the start of a paragraph. Such frequent paragraph breaks could easily extend the text beyond 100 pages

Various references say that 'the', 'and' and 'I' have a 10% occurrence in English. Considetring this it seems unlikely that 7% of the book will be 'the' if word length also is taken into account.

May be 40%.

bioramani

Try extracting all the T H & E's in the book & you may find that you get the answer you need.

Very common letters.

I too thought and that would have been close (with the H being minimum would have contributed to approx 73 pages= 0.053*1400 and with the correction factors might have run close. (for T it is 146 and E 184)

But he said re-arrange words not re-arrange letters.

Ask and ye shall receive:

T's - 224,530

H's - 166,522

E's - 313,055

So you could make 166,522 "the"s with this letter distribution. Using my estimate of 42 pages for 34,544 "the"s, this would result in about 200 pages. However, since Tom said to rearrange the words, not letters, I don't think this is the correct approach. I think he was trying to make the point of how prevalent the word "the" is in the English language, and how much superficial reading is contained within the book.

Tom was likely wrong an the utility of the book as an anchor - no surprise that he is also wrong on the density of "the". (On the other hand, if I was floating a bet, I'd probably do something similar to soften up the opposition)

I have to commend the forum members. Whenever I think I'm too much of a Math-Geek to still be called a Mechanical Engineer, I find solace in your Uber-Math-Geekness...

And always such wonderful use of external sources...

... but...

The first thing I thought of when I read, "... rearranged the words...", was rearranging the LETTERS too.

Though it's been shown Tom was wrong using just the 'whole-word' technique, I suspect one could squeeze another 30 - 50 pages of 'the' out of mixed letters. This would fill out Tom's 100 pages.

Thanks.

Wel, well ... it just happend ...

I just read a novel with more than 50,000 words, and not A SINGLE time the word "THE"... I guess this novel pulls down the "the-average" quite a bit ...

check it out !

I believe the answer to the challenge question provides some insight into why we need to look carefully at all of the parameters in what we are analyzing. The analysis assumes all the words are of the same length. "The" is only 3 letters long while the average word length is probably something like 4.5 letters long. Assuming each word has one space, the ratio of "the " length to average word length (plus a space) is about 73%. Taking word length into consideration, would drop the official answer from 116 pages to something less than 100.

Thanks,

Jim

Agreed - it's another example of "the answer" missing any reality check.

It starts with an invalid formula (academic measurements of occurrence find "the" at 6.2% on average including speech. For text it falls to 6% and imaginative text to 5.2%). Even without any other corrections, 6.2% would reduce the result of this calculation to about 87 pages. (And, of course it has been established that "War and Peace" has above average use of adjectives and (long) names, even for imaginative(?) texts.

It continues by ignoring the effect of length of word length - accepted that this depends on the specific font, but it is extremely unlikely to be more than the 86% at which an 8.3% occurrence brings it below 100 pages.

That is of course without considering the effect of paragraph breaks, which would further shorten the length of a bowdlerisation consisting entirely of "the"s.
(On the other hand, asserting that all paragraph breaks have to be retained would be the only way that Tom could avoid losing).

That was I - apologies. Also that I forgot to credit the vastly superior contributions of the community - particularly those of jim, dac and mpm

Regarding the answer:

And the Answer is...

Tom is correct. War and Peace consists of 560,000 words on 1400 pages (in John's paperback). In the English language, "the" tends to be the most frequent. The frequency at which it occurs in a corpus can be estimated by the following formula:

Where k is the frequency rank of the word in question, in this case k=1, since "the" is the most common English word. The s is the exponent for the power law distribution that defines word frequency in the corpus. This value is usually close to 1 and for a rough estimate it's ok to say s~1. HN,s is the Nth Harmonic Number. The general equation for calculating the nth Harmonic Number is:

N is the number of unique terms found in the book. A good rough guess for this number is less than 100,000 unique words (the less unique words there are, the higher the percentage of "the", so we are calculating the minimum number of pages with so large a number. The real number is probably more like 10,000 to 20,000.

Plugging these estimates into equation 1 we get:

HN,s = 12.09; s=1; k=1

f~1/12.09 ~8.3% of the words in the novel would be "the".

Total occurrences of the word "the" in War and Peace= 560,000 x .083 = 46,480

Words per page in War and Peace = 560,000/1400 = 400 words per pg

Total Number of Pages with the word "the" = 46,480 / 400 = 116 pages (Tom is correct)

Thus there would be at least 116 pages of the word "the".

Not only is the notation here muddled up (different uses of 'k', and inconsistent use of HN, H_N, H_n, HN,s [?]) but the conclusion is based on a statistical estimate which is contradicted by the actual values reported by Project Gutenberg (assuming that DAC1267 did not err in his Reply #11).

/Disappointed in this 'Correct Answer'.

The solution does not consider word length. If the average word length is 5 then there would be less than 100 pages of "the".

And the Answer is...

The answer you've given is bollocks. Many of the replies have given practical solutions -not fancy imaginary statistical clap-trap nonsense speculations - which you claim as "the answer", even though it has clearly been shown to be false.

Think about it.

CR4 Challenge Admins - this has left a bad taste in my mouth, and I don't think I'm alone in this.

(the less unique words there are, the higher the percentage

Grammar, my dears, grammar!!!!

That should read the fewer unique words

Just because we're engineers doesn't mean we can't talk and write properly!

I think the usage was perfect Americanish. "Fewer" has gone the way of "whom" here in the states. We use the word "fewer" when we want to sound proper and drop it the rest of the time. Stop trying to force your English on us.

http://www.editrix.us/2008/07/less-and-fewer.html

At least we pronounce our H's.

http://www.youtube.com/watch?v=EAYUuspQ6BY&feature=related

"Fewer" vs "less" is a minor offender.

Since this is an engineering forum, I'd like to let you know that my pet (engineering) peeve of American usage is the expression, "high rate of speed". Cars are routinely reported to be "travelling at a high rate of speed", (sometimes for a long period of time). Even worse is when they travel at a higher rate of speed than 80mph.

Why the s-1 and k=1?

Given that "the answer" is demonstrably wrong by a larger factor than could possibly be explained by the statistics it assumes, there can be no good reason for any of it.

Why probe further? If you are looking to recognise why "the answer" is wrong, you need look no further than assigning unweighted statistics to situations where systematic effects dominate.

Sorry folks - we posted an incorrect answer to the question yesterday. I just got back from a two week vacation and wasn't as focused as I should have been. The correct answer is now available.

Hi Chris

a) I couldn't see the link to the paper.
b) I don't see the evidence for this formula even in the average frequency of words - unless you accept very wide tolerances on relative frequency of occurrence. (N.B. the variance on the ratio must surely increase as the words get rarer)
c) According to this formula, any word that occurs with similar frequency in most documents will be more common in the average of individual documents than in the aggregate of all documents, which is clearly nonsense. If the word does not appear with similar frequency in most documents, you would need to know its rank to make the calculation (even if the formula does work in this case). That means that the method (as applied) fails in principle for all words, not just the rarest and the most common. That in turn means that the basic method is never useful for this purpose, as establishing the rank of a word means you would need to count not only of the occurrences of the word you wish to evaluate, but also of many others.

Regards

Fyz