What do we learn when we learn by doing?

The University of Georgia MOOC on online education was starting to look very interesting before it suddenly folded. On one hand, it covered a lot of interesting theoretical pedagogy. On the other, the practical pedagogy of the course itself seemed sub-optimal. As I only got one week in before they closed up, so I can't really say all that much about it.

One academic the course has introduced me to is Roger C. Schank. Roger is an AI lecturer who later specialising in learning. Schank's big idea is that of learning by doing. It's a simple and compelling idea – he claims traditional classrooms don't work because they are far too theoretical and divorced from any real “need” to learn.

There is certainly a lot of truth in this. A book of drills (whether it be arithmetic, grammar, or rote historical facts) does little to demonstrate either why the information is important or the contexts to which it is relevant.

The title of this post is lifted straight from a report Schank wrote in 1995 for his university. It's a huge piece of writing – almost thirty thousand words long – and to be perfectly honest with you, I didn't read it to the end. But why would I? It's called a “technical report”, but in truth it's little more than an oversized opinion piece. There's no technical data: he does not appeal to research, he does not appeal to figures, he just makes unsupported statements. What is most telling is that there are only 5 citations in his bibliography, and four of these are to himself.

As he argues without evidence, I feel perfectly entitled to dismantle his argument without citations. Besides, he's supposed to be an academic, and I'm just a blogger!

First up, Schank fails to address in the first 25% of his essay (which is approximately what I read) the biggest concern that has always been raised against the idea of whole-subject-learning, learn-by-doing or whatever label you chose to attach to the idea: lack of breadth. (A quick skim suggests that that it's not addressed further down, so if it is, it's clearly not given the prominence it deserves. Besides, as it is the single most important concern of most critics, you need to address it early or you lose our attention.) A single task will only lead to a single solution, with some limited exploration of alternative strategies. It teaches the students how to cope with a situation where they lack knowledge, rather than minimising those situations by providing them with knowledge.

There's been research into various aspects of this. I've seen reports claiming to prove schoolchildren given a whole-subject education have a much narrower set of knowledge. This should be pretty obvious, I would have thought... so maybe I'm just displaying confirmation bias and reading the stuff that supports my view.

Of course, there was the study that showed that doctors trained by case-study rather than theory performed better in diagnosing patients, but as I recall it, this was tempered by the fact that their diagnoses took a lot of time and money, because they tended to follow a systematic approach of testing for increasingly less common problems or symptoms. A doctor trained on a theory-based course was more likely to formulate a reasonable first hypothesis and start the testing process somewhere in the middle. The conclusions we can take from this are mixed. You can claim that the traditionally-trained doctor is better at diagnosing on the grounds that he can do it with less testing; or you can claim that only the end result matters, and the case-study-trained doctor is better. You can argue that minimising mistakes is the ultimate goal, or you can argue that the time taken in avoiding mistakes is too great in that it delays treatment for other patients.

So, anyway... Schank does nothing to convince me that it is possible to cover the breadth of a traditional course in learn-by-doing, but there is a video on Schank's site of a course he designed at Carnegie-Mellon about ten years ago, and it alludes to what I think is the only real argument about it. One of the senior course staff and one or two of the students talk about the idea of forgetting everything that's been taught in a traditional course. If challenged, would that be the basis of Schank's response? The logic certainly appeals: if you're not really learning anything in a traditional academic course, the breadth of what is covered is irrelevant.

But is anything ever truly forgotten? I've recently gone back into coding after a long hiatus – even when I worked in IT, I never had any serious programming to do. But when I come up against a problem, I find myself thinking back to mostly-forgotten bits of theory from my CS degree days, and looking them up on the net. Tree-traversal algorithms, curried functions, delayed evaluation... But if I had never encountered these ideas before, how would I even know to look for them?

This is not a mere theoretical problem. I'm not usually one to complain about “ivory tower academics”, but goddamn it, Schank's brought this on himself. And I quote:

"One of the places where real life learning takes place is in the workplace, "on the job." The reason for this seems simple enough. Humans are natural learners. They learn from everything they do. When they watch television, they learn about the day's events. When they take a trip, they learn about how to get where they are going and what it is like to be there. This constant learning also takes place as one works. If you want an employee to learn his job, then, it stands to reason that the best way is to simply let him do his job. Motivation is not a problem in such situations since employees know that if they don't learn to do their job well, they won't keep it for long.

Most employees are interested in learning to their jobs better. One reason for this is, of course, potential monetary rewards. But the real reason is much deeper than that. If you do something often enough, you get better at it -- simple and obvious. When people really care about what they are doing, they may even learn how to do their jobs better than anyone had hoped. They themselves wonder how to improve their own performance. They innovate. Since mistakes are often quite jarring to someone who cares about what they are doing, people naturally work hard to avoid them. No one likes to fail. It is basic to human nature to try to do better and this means attempting to explain one's failures well enough so that they can be remedied. This self-correcting behavior can only take place when one has been made aware of one's mistakes and when one cares enough to improve. If an employee understands and believes that an error has been made, he will work hard to correct it, and will want to be trained to do better, if proper rewards are in place for a job well done. "

Many graduates, particularly CS grads, will immediately be able to identify with me when I say this isn't true. When you leave university, you typically know how to do things correctly, but once you get into the real world, “correctly” is too time-consuming, too expensive. (Except in safety-critical roles, such as avionics or military systems.) In the short term, this is OK – pragmatically, that's the way it's got to be.

In the longer term, though, things start to go haywire. We get so habituated to our way of doing things that we soon learn to identify it as the “right way” of doing things. Someone comes along with a new way, a better idea (probably a recent grad) and we dismiss the idea as wishful thinking.

In computers more than any other field, this problem is easily apparent. I remember suggesting a very simple change to a database system and being told “you can't do that with computers”. A) You can do pretty much anything with computers. B) I was talking about something that is built into the database software we were using!!! Yes, a standard feature of the software, and the team's top “expert” didn't know about it; and because he was the expert and didn't know about it, it was as though it didn't exist.

More generally, the problem becomes visible when a programmer switches languages. A C programmer who learns Python normally ends up writing his Python code using the features that most resemble those of C. The unique features of Python are designed to overcome specific problems and difficulties with a lower-level language such as C, but to the expert C coder, these aren't really “problems”, because he long ago rationalised them away and learned to cope with them. He doesn't realise there is a “problem”, so he doesn't have any reason to go looking for a solution. Even within languages, some people always do things the “hard way” because they've simply never thought to look for an “easy way”.

So Schank is being hopelessly naïve. The key feature of expertise is automaticity – experts have internalised enough that they don't have to think about what they're doing. They close their minds to learning, because there's more value in doing a job suboptimally but quickly than in doing it optimally but slowly. People need to develop breadth before they become experts – before they become “set in their ways”.

Now to answer the question that started this article, what do we learn when we learn-by-doing?

We learn to be adequate, not brilliant. We learn to get by, not to excel. We learn to stop thinking. We learn, paradoxically, to stop learning.

The misunderstandings in MOOCs

I had another article lined up and ready to go, and then I started reading about MOOCs on-line, and in particular a lot of the reaction to the total collapse of Coursera's MOOC Fundamentals of on-line education (#foemooc), one of the biggest and most revealing ironies in history.

One article quoted a law professor as follows:

“Part of what Coursera’s gotten right is that it makes more sense to build your user base first and then figure out later how to monetize it, than to worry too much at the beginning about how to monetize it,” said Edward Rock, a law professor serving as the University of Pennsylvania’s senior adviser on open course initiatives.

Is he mad, or just plain ignorant?  It's this sort of thinking that leads people to compare the MOOC boom to the internet bubble at the turn of the century.  After all, that bubble was based on companies building massive user bases with no idea of how to monetarise them.

We are currently experiencing a similar bubble in Web 2.0.  So-called investors* have ploughed money into companies like Facebook and Instagram, only to see stock prices suffer when the market suddenly realises that they have no business plan.

[* The modern stockmarket makes a mockery of the term "investor".  "Investing" is supposed to mean putting money in so that a company can grow, but most high-profile IPOs, including Facebook and the like, are simply a change of ownership.  The money is for the previous owner, not the business.]

Why do we want to replicate that model at a cost to our valued educational institutions?

I've commented on that side of things before, noting Udacity's move toward corporate sponsorship: MOOCs as adverts for a given vendor.

This MOOC-as-sponsorship model might even work to some extent for the universities too -- the University of Edinburgh's Coursera MOOC E-Learning and Digital Cultures (#edcmooc) is based on a module taken from one of their masters programmes, and it's possible that the increased exposure it gives the course among potential students will get back the cost of putting the MOOC version together and on-line.  The risk, though, is that if too much material is available for free, the adverts will basically kill the market for the product.

One article in particular caught my attention.  Over at the blog online learning insights, blogger and instructional designer Debbie Morrison wrote a post about three takeaways from the collapse of the Coursera Fundamentals of Online Education course.

Her first "takeaway" is, in my ever so humble opinion, completely and utterly wrong.  Totally.  Completely.  Utterly.  Let's take a look:

1) The instructional model is shifting to be student-centric, away from an institution or instructor-focused model.In a massive, open and online course with thousands of students, the instructor must relinquish control of the student learning process. The instructor-focused model is counter intuitive to the idea of a MOOC; in the MOOC model the student directs and drives his or her learning. The pedagogy used for traditional courses is not applicable to a course on a massive scale. With the Web as the classroom platform, students learn by making connections with various ‘nodes’ of content [not all provided by the instructor] on the Web, they aggregate content, and create knowledge that is assessed not by the instructor, but by peers or self. This pedagogy builds upon the constructivist theory, and more recently a theory developed by Downes and Siemens, the connectivist learning model.

Wrong... how so?

Debbie Morrison clearly knows a thing or two about building online courses; unlike me, she's actually written them.  But when she claims that the a tight teacher-led course is "counter intuitive to the idea of a MOOC", she's projecting her views into a reality that is distinctly different.  The origin of the MOOC is in the field of Computer Science and Artificial Intelligence.  The original MOOCs were very much instructor-led, with very specific, well defined tasks for the student to follow.

Because the tasks were very tightly controlled, marking could be automated, and hundreds of thousands of students could be individually assessed by machine.

Morrison is mistakenly applying the standards for her own niche, the small-to-medium online course, to a very different animal.

Morrison's courses are comparable to a seminar-based course in more traditional education.  Every generation has accepted that seminar-based courses are better than lecture-based ones, but that they're more work for the teacher because the uncontrolled variables lead to a massive space of potential outcomes and directions for the class.

Online education for small classes is a good realisation of the seminar-based course, because the online medium removes some of the time pressures on the instructor in terms of organisation, logistics and delivery; time which can be devoted instead to tailored group and individual feedback.

But when you've got one instructor and one teaching assistant for a class of 41,000 students (foe) or 1 instructor for a class of 150,000 (claimed by Ed Tech Magazine for a Udacity course), then that's out the window.  A teacher cannot be a "guide" or a "facilitator" to a group that big.  A teacher cannot even have a concept of the individual students as human beings (we're well beyond Dunbar's Number here).

A massive course cannot be student-centred because there are simply too many students.

And let's go back to the seminar-style class for a minute. Seminar-based classes are far more common in taught postgrad courses than in undergrad courses, and while you might encounter them in undergrad courses, you'll only see them in degree-year modules, or in very specialised, low intake degree schemes.  As I said before, seminar based courses are more work, but more specifically, they take a lot more time to grade.  An open-ended course leads to an open-ended assessment.  If you have a class of 400 students, marking essays is an endless task, and it is very difficult to mark them all fairly and equally, but if you give them a structured exam, marking is quick, easy and completely objective.

But more than that, the standard system provides a gradual shift in abstraction.  Perhaps this is a happy accident rather than by design, but as we progress through our degree scheme, we should be building up a toolkit of useful skills and concepts that can later be applied elsewhere.  As that toolkit expands, we are slowly given greater freedom to apply and think.

My opinion is, and always had been, that the MOOC must be seen as a vehicle for "basic skills" courses.  Many within the educational establishment reject basic skills teaching in favour of "whole topic" teaching, but I personally believe that they've done so on the wrong grounds.

If you were to tell me that basic skills teaching is categorically bad, I would disagree with you.  Basic skills are prerequisite to advanced skills.

But if instead you were to tell me that teaching basic skills properly and exhaustively is good in theory, but unfortunately not practical given class sizes and constraints on classroom time, I think I would be able to accept your point.

And this is where online can really flourish: we can take the basic skills load out of the classroom.  Have the students work on the basic skills individually, using constrained tasks that a computer can assess.  These types of tasks can be completed much quicker without all the usual classroom faff of handing out sheets, explaining the task, checking up on students, going over the answers, addressing individual errors with the whole group etc.  The volume of work that is assessed or otherwise receives feedback can be increased, as the teacher's time is no longer a limiting factor.

Then the students can go into the class with the basic skills they need to address larger, more abstract, more challenging tasks.  Lessons can be more rewarding for both the student and the teacher.

Debbie Morrison proposes turning a course with a 5-figure class roll into a student-centred, open-ended seminar course, but that is precisely what the FOE Mooc attempted to do, and it was a total train wreck.  Debbie's "takeaway" from the incident isn't asking us to learn from their mistakes, but to repeat them.