Authentic: long v short pt 5

Today was a wee bit frustrating.  I spent a solid chunk of time trying to get the ntlk data module installed, and with it the file english.pickle that would have allowed me to do part-of-speech (POS) tagging.  This would have made it almost trivially easy to eliminate the proper nouns and get a genuine look at the real "words" that are of interest to the learner.

Ah well, it looks like it wasn't meant to be.

So I started working towards custom code to eliminate the proper nouns manually, something which would be handy in the future anyway.  The first step was to identify some candidates for further inspection, and seeing as I'm working with English, that's pretty easy: if it's not all in lower case, there's something funny about it.  I wrote the code to identify all the tokens (words) that contained capitals.  Yes, at this point I could have checked whether it was the start of the sentence or not, but that wouldn't have really helped, because proper nouns occur at the start of sentences too, so i'd still need to check.

When I generated my set of candidates, though, it was a little long.  For The 39 Steps, I was looking at 919 tokens to check manually, and that's a fairly short book.  As I'm doing this for fun, it seemed like checking that many would be a little bit boring, particularly in longer books.  (I later checked the candidate set for the 3 books in total, and it turned out to be over 3000 words, which is more than my time's worth.)

My first quick test then was to have a look at the difference in figures.  Eliminating every single item with any capitals in it drops the type:token ratio in The 39 Steps from 14.48% to 13.14% -- that's almost a a 10% drop (it's 1.35 percentage points, but it's 9.27 percent).  Before properly addressing the proper nouns, I wanted to see how big a difference this crude adjustment makes to the figures.  It seemed just a little too high to realistically be led by proper nouns alone.  But can that be?  I mean, how many words are likely to occur only at the start of sentences?

So on I went, hoping that the data I could generate at this stage would start to shed some light on this figure.

The first graph I produced showed me the running type:token ratios and introduction rates for both the full token set, and the token set with non-lowercase words eliminated:

The two pairs of lines follow each other pretty closely, getting closer together as they progress.  But in order to start getting a clear idea of what was going last time, I had to go to another level of abstraction and measure some useful differences.  So here is the difference between the running ratios for all words and lowercase only, and the corresponding difference in introduction rates:

Now you'd be forgiven for thinking that the difference is diminishing here -- I was fooled into thinking the same thing, but then I realised I was dealing with numbers here rather than proper stats, and I redid the analysis but with a difference in percentage:

The overall running type:token ratio does indeed decrease, but it halves (20% down to 10%) then stabilises.  The introduction rate, on the other hand, is all over the place -- there's no identifiable trend at all.  Even subsampling my data didn't give any clear and understandable trends (and since I'm using a desktop office package for my analysis it's a bit of faff to do the resampling automatically -- it's just further proof that I need to get myself familiar with the statistical analysis tools for Python (eg numpy), but my head's full with the NLTK stuff for now, so I'll leave the improved statistical stuff for
another time).  Here's the same graphs, but with 2000 word samples instead of 500 word samples:

So not promising, really.  Still no stable, identifiable trends.

Books as a series
But I had all the infrastructure in place now, so I figured I might as well rerun the analysis on the 3 books as a single body and see what came out.  Let's just go straight to the relative difference between the lines for all words and eliminating all words not entirely in lower case:

Oooh... now where did I leave those figures on where the individual books started...?  44625 and 152034, and there's a notable period of high difference (20-30%) from about 45000 words, and that massive spike you seem on the graph -- which is actually a 63.64% difference -- occurs from 152000-152500.

Bingo: we've got decent support for Thrissel's suggestion that a lot of proper nouns are introduced early on in... at least some novels.

Not the sort of information I was originally looking for, but actually quite interesting.  It's kind of turning the project in a slightly different direction than I had planned.  I'll just have to go with the flow.

What I did wrong today
One of the minor irritations of the day was when I started writing up my results, and after having done the coding, data generation and analysis, I realised a fairly simple refinement I could have made.  It was a real *palmface* moment: I could have simply taken my first list of candidate proper nouns and eliminated any candidates that also appeared completely in lower case.  Having done that, I would have been left with a much shorter list of candidates, and it may well have been worth my time manually checking the results.

>sigh<

But of course, that's as much the point of the exercise as anything: to work through the process and the problems and to start thinking about what can be done better.

It also occurs to me now that I also managed to eliminate every single occurrence of the word I from the books!  Quite a fundamental error, even if it only made a minute difference to the final ratios.

Perhaps I'm being a little too "hacky" in all this.  I'll have to pick up my game a bit soon....

Authentics: long v short - pt 4

Well, I had a nice weekend and visited some friends in Edinburgh for a wedding.  The weather's too good to spend too much time inside, so I'll just write up a few more tests then go and enjoy the sunshine.

Multiple books in a series
Today's figures come from two of the books I've already mentioned -- John Buchan's The 39 Steps and Greenmantle, and the next book in the series: Mr Standfast.

Again, the graphs at different sample sizes show different parts of the dataset more clearly than others:

The graph as 1000 words is too unstable to clearly identify the end of the the first book, and the start of the second book is only identifiable because the introduction rate is greater than the running ratio for the first time in any of my tests.

The 2500 and 5000 word samples give us a clear spike for the end of the first book, but the end of the second book is obscured slightly by noise, and becomes clearer again in the 10000 and 25000 word sample sizes, although the end of the first book is completely lost by the time we reach the 25000 word sample graph.

Having done all that, I went back and verified the peaks matched the word counts -- The 39 Steps is 44625 words long, and Greenmantle is 107409 words long, so ends with the 152034th word.  The peaks on the graphs all occur shorlty after 45000 and 150000.

It was the first graph, from the 1000 word sample set, that piqued my curiosity.  Having spotted the two lines intersecting at the start of the third book, I decided to check the difference between the running type:token ratio and the introduction rate, and I graphed that.  At 1000 word samples, there was still too much noise:

However, given that I already knew what I was looking for, I could tell that it showed useful trends, and even just moving up to 2500 word samples made the trends pretty clear:

Going forward, I need to compare the difference between books in a series, books by the same author (but not in a series) and unrelated books, and I believe that the difference between the running type:token ratio and the introduction rate may be the best metric to use in comparing the three classes.

Problems with today's results
I can't rule out that the spikes in new language at the start of each book aren't heavily influenced by the volume of proper nouns, as Thrissel suggested, so I'm probably going to have to make an attempt at finding a quick way of identifying them.  The best way of doing this would be to write a script that identifies all words with initial caps in the original stream, then asks me if these are proper nouns or not.

By treating the three books as one continuous text in the analysis, it looks like I've inadvertently smoothed out the spike somewhat at the start of each book.  In future I should make sure individual samples are taken from one book at a time so that the distinction is preserved.

Authentics: long vs short part 3

Excuse the slight change of title -- I figured the original long title was probably getting truncated in people's feeds, so I wanted to abbreviate it.  If you've been following my blog recently, you should have already seen my previous two posts on my little project; I am trying to investigate whether my normal advice that long fiction (novels or TV serials) is better than short fiction (short stories and feature films) for the learner.

Sample sizes
Anyway, as I said last time, I wanted to start comparing a fixed length of text, rather than variable-length chapters as my benchmark.  I was looking for a sampling length that would give a clear picture of the overall progression without having too much interference from little local fluctuations.  My first set of results suggests that this is a fool's errand.  The following set of images shows the graphs for the novel Greenmantle by John Buchan, with samples taken ever 1000, 2500, 5000 and 1000 words.

While using larger samples gives a much smoother line, it also unfortunately obliterates some of the most important detail in the graph, in that we start to lose the steep drop at the start -- that's information that's really crucial to my investigation, so I'll have to make put up with various humps and wiggles in the line for now.  However, that's not to say that the other graphs aren't interesting in and of themselves -- the little hump at around 50000-60000 words in the 5000 word sample version suggests that something important may be happening at this point in the story, causing a batch of new vocabulary to be introduced, or perhaps the introduction of a new character with a different style of speech.  Anyway, as interesting as that may be, it would be a diversion from the matter at hand.

Alternatively, I could move away from using linear sampling/projections and start charting using logarithmic or exponential data, and while now would be a good time to start refreshing my memory on that sort of statistical analysis, it also risks diverting me from the task at hand, and I'm following the Coursera.org machine learning course currently, so I should be able to get the computer to do the work itself in a few weeks anyway.  Besides, I've still not got myself a high-frequency word list, and the pattern might be completely different once I've eliminated common words of English from the equation.

So for now I'll stick to working with multiple sample sizes.  I'll admit to being a bit simplistic in my approach to this so far, as I ran my little Python program once for every sample size, rather than just running it once with the smallest sample size then resampling the data.

The program I'm using at the moment is pretty straightforward:

def collect_stats (token_list, step_size):
   i=0
   return_array=[];
   while (i      i += step_size
      running_types_total = number_of_types (token_list[:i])
      if (i         return_array.append([i,running_types_total])
      else :
         return_array.append([len(token_list),running_types_total])
   return return_array

This takes an NLTK token list (it would work with any simple list of strings too, though) and the size of samples to be taken, then builds up a list of lists [[a1,b1],[a2,b2],...] where each a is the number of the last word included in the sample, and each b is the number of unique tokens from the beginning of the text to the ath word.

The number_of_types function just returns len(set(w.lower() for w in token_list)).

This means that at every stage I have a running total of tokens, and it's only when I want to produce a graph that I calculate the number of new tokens in the given slice (= b(n) - b(n-1)), and there's therefore no reason why I can't skip several slices to decrease my sampling rate (eg b(n) - b(n-3)).

Next up
I've taken a running sample of three books from the same series -- The 39 Steps, Greenmantle and Mr Standfast, and run them through as one text, so I'll look at the output of that next, but I don't think it'll be much use until I've got something to compare with -- either/both of: a selection of novels by one author that aren't a series; and a selection of novels by different authors.

Authentic materials for learners: long form or short form? (part I)

Anyone who gets sufficiently far on in learning a language is going to want to start reading, watching or listening to materials intended for natives.  This is what a lot of language teachers refer to as "authentic materials".  (Now, an alarming amount of so-called authentic materials in the classroom are actually very heavily doctored, but that's not the sort of authentic materials I'm going to talk about today.)

I have often claimed that it is better for the learner to use longer materials than short materials (eg when discussing films vs TV serieses in a previous post).  This wasn't an idea I came up with myself, but advice I'd been given when I was at high school, although I can't remember who first suggested it to me.

Anyway, I was told at the time that the first one or two hundred pages of a novel contain most of the language (in terms of grammar, vocabulary and turn-of-phrase) that will occur in the entire book.  It therefore follows that the first two hundred pages of any work are the most difficult, and therefore the longer the book is, the easier the ending will be, because you won't be confused by the language.  This also means that the book is acting as active revision, and that by the end of the book, you will have learnt most of the major vocabulary in it.

A 50 page short story would intuitively sound easier to read than a novel, but this isn't really the case, because you're dealing with something that is going to be littered with new words on every single page.

And what about a piece of flash fiction?  Realistically, we're not going to expect much repetition at all.  Compare with the short extracts of authentic works printed in many classroom language textbooks -- none of the "content words" that are specific to the story are likely to be repeated at all, so they will be looked up by the reader, then promptly forgotten about.

Although I was told this about reading books (as opposed to short stories), I believe this holds for any form of literature, fiction or non-fiction, regardless of medium.

A half-hour documentary will be self-reinforcing in a way that a 4 minute news report on the same topic won't be (on TV or radio).  An 8 hour long TV series similarly will reinforce its language than an 80 minute feature film, or particulary a 10 minute short film.

Anyway, I've been repeating this advice for years, and I've always said that my experience backs it up. Well, in the little gap between finishing my Gaelic course and starting my next job, I was wanting to do a little work with corpus analysis software and it occurred to me that this would be a great little exercise to get me back into the swing of things, so I downloaded several resources: TextSTAT, a concordancer package written in Python at the Free University of Berlin; AntConc, a Linux/Mac/Windows concordancer by Lawrence Antony at Waseda University in Japan; and the Natural Language Toolkit for python, which will allow me to write more flexible, custom queries on my data.

Type:token ratio
One of the most basic statistical measures of diversity in a text is the so-called "type:token ratio".  The number of "tokens" in a text is the number of individual words, the number of "types" is the number of different word forms.

For example, the phrase "the cat chased the dog" has 5 tokens, but only 4 types, because the is only counted once when determining the number of types.

Or again, "I told the woman to tell the man to tell you" has 11 tokens (11 words in the sentence), but as the, to and tell occur twice each, there's only 8 types in the sentence.

The type:token ratio is exactly what you'd expect if you're at all familiar with statistics: the number of types divided by the number of tokens.  In the first example, we have 4:5 = 4/5 = 0.8 (or 80%) and in the second we have 8:11 = 0.727272... (roughly 73%).

Notice how the type:token ratio on the longer sentence is lower than that on the shorter sentence -- in this case it's a matter of my choice of words, but as a general rule, type:token ratio decreases with the length of text examined, which only goes to justify the advice of favouring long-form over short-form materials for the learner.

However, that's still to be proven in practice.

So over the next few weeks, I'll be experimenting with a bunch of public domain texts from Project Gutenberg.  I'll be trying to investigate the basic premise of whether long-form fiction is intrinsically easier than short-form, then investigating whether this extends to reading several books by the same author as opposed to books by different authors, and how much of a difference it makes whether these books are part of a series or individual stories.

The size of this study is going to be very small, as the main goal for me is simply to gain a better understanding of the technology and to reason through the process of designing logically sound research in a corpus, so the conclusions won't be scientific proof of anything, but it will hopefully be interesting (to me at least).

If you're aware of any research that covers the areas I'm looking at, please feel free to drop a reference in the comments, and if you have anything to add or suggest, I'm all ears.