Original URL: http://www.theregister.co.uk/2008/10/01/stob_open_university_trick_question/

Stob latest: It was a cunning trick, says Open University

Pull the other one

By Andrew Orlowski

Posted in Developer, 1st October 2008 11:14 GMT

Why does the Open University set its students gibberish, Verity Stob asked here recently? We decided to investigate. As our enquiries continued at the Open University, it became harder to find anyone who took the issue seriously.

Two weeks into one of the modules in the OU's Comp Sci postgraduate course - M885 Analysis and design of enterprise systems: an object oriented approach - students were set a tutor-marked assessment. The basis of the questions was a paper published by the IEEEs Software journal. However, the paper contained passages that made about as much sense as a computer-generated spam blog. Some of the words used were relevant to the topic, but sentences and whole paragraphs made no sense - and Open University students were directed to the meaningless passages.

"You would expect higher education to provide you with source information and references that were not only relevant, but also not worded in such a way that gives the impression someone just put a thesaurus through a shredder and taped bits in sequence to pad out their work," reader Steven Raith commented after reading the story.

It seemed incredible that a publicly funded university devising a postgraduate-level course could, with a straight face, use such nonsense.

After Verity Stob raised concerns about the paper, the course tutor admitted he had not read the paper "in detail yet", but advised the student that "overall, the marking schemes for the easy questions are very flexible".

The tutor isn't responsible for setting the course material, the questions or the marking assessments. It's a part-time job. The real responsibility in this case, course M885, fell to Dr Lucia Rapanotti. Dr Rapanotti gave Verity the distinct impression of not being very familiar with the material either, but passed on the assurance that it had been endorsed by "a prestigious peer-reviewed journal", and advised the student to answer it anyway.

But not only was the Madanmohan and De paper gibberish, it was irrelevant to object-oriented analysis: Designing and implementing object models. It wasn't the first time students had been set a puzzling example.

In another question on the same course, students were posed a question based on a paper called Ten Commandments of Formal Methods …Ten Years Later, by Jonathan P. Bowen and Michael G. Hinchey (IEEE Computer, 2006). Formal Methods, which went out of fashion in the mid-1990s, has nothing at all do with object-oriented analysis. One may as well ask students to describe why unicycles are no longer popular.

We mailed Dr Rapanotti and the head of the department, Darrel Ince, requesting interviews.

"I don't even see it as a story," the Open University’s PR guy Louis Delafloret told us. One man's gibberish might be another man's work of genius, he suggested, "like Jane Austen".

"Sorry to disappoint you but I will not be giving you an interview," Ince replied, loftily. "You will be getting a detailed rebuttal." Eventually word was passed down from the elusive Dr Rapanotti, in the form of a 125-word statement.

She explained that, "The article was used with full confidence of authenticity expected of this global leader specialising in peer-reviewed technical articles." Yet she completely failed to address the issue of why the article was nonsense, or why it had been chosen. As an exercise in answering the question, this merited as a “fail”.

We were beginning to think unworthy thoughts: Did they know what object oriented analysis was? And were they actually capable of distinguishing between a coherent paper, and one of composed of gibberish? (You may recall that at one point, authors Madanmohan and De discussed the use of "Artificial Intelligence" in selecting open source code.) The Open University seemed in no hurry to dispel this notion.

Eventually word came on down from high – Darrel Ince.

It began, ominously, "There are two issues intertwined in the query from the Register journalist; the use of an article which plagiarises another article and the practice of asking students to read and comment on research papers."

Actually, no. We wanted to know why gibberish had found it’s way onto the course. Perhaps Ince thought we'd forgotten?

The second issue is that of asking students to read and comment on academic papers. The OU, along with virtually every other university, exposes students to research papers and research articles — usually in the third year of an undergraduate degree course or at any time in a postgraduate degree course. Such articles are challenging to read; for example, they have to tie up every loose knot in describing a piece of research and, hence, can be somewhat convoluted.

But “tying up loose knots” is an exercise in making complicated things comprehensible. The paper Ince's department had put before students was the opposite: It made simple concepts incomprehensible.

Ince then rowed away down the path he had chosen:

There are a number of reasons for exposing students to research papers:

  • In a department where the subject area is advancing rapidly, for example computer science, it would be irresponsible not to give the students an opportunity to read important material before it is incorporated into undergraduate and postgraduate textbooks.
  • An aim of every course, be it undergraduate or postgraduate, is to equip students with the intellectual skills that they will need in the outside world. Not only will some of our students have to read research and development articles but also challenging technical documents. It would be irresponsible not to prepare our students for this.
  • Both undergraduate and postgraduate courses require students to prepare a dissertation. This is a lengthy work which can be as much as 30,000 words in length and which requires the student to consult research and development papers and articles. Developing such a piece of work can be extremely difficult; one of the difficulties being accessing and understanding articles such as the one published in IEEE Software. In order to minimise the culture shock associated with developing a dissertation, the Department of Computing, along with every other scientific university department, exposes students to articles in their courses prior to the student starting a dissertation. Not to do so would be irresponsible.

All worthy boilerplate blather, but irrelevant to the question. Quite amazingly, Ince had managed to avoid the issue we'd raised, and simply substituted one of his own choosing. Questions of general competence were now foremost in our minds.

Then, quite dramatically, another explanation was offered.

Perhaps sensing a PR disaster, the Open University's Communications Director Derek Prior got in touch.

The new explanation? It had been a trap for the unwary, he told us. In order to develop student's critical faculties, bad papers may be deliberately set. These papers may include gobbledygook, or meaningless phrases.

"If you telegraphed ahead that it was a rotten paper, it would not be a critical exercise."

Stob's reaction, he said, had been "exemplary", and the student had evidently tried to understand it from every which way.

Mr Prior told us he’d seen the marking guide from 2006 which noted that the splog paper referenced its own research material. No word of the tutors' views on its comprehensibility or plausibility.

Mr Prior was emphatic: The question required them to be critical of the paper.

Here’s the question again:


Question 2 (15 marks)

Read the following article which you can access from the Reader section of the M885 course website: Open Source Reuse in Commercial Firms, T.R. Madanmohan and Rahul De’, IEEE Software, vol. 21, no. 6, Nov.–Dec. 2004,

http://ieeexplore.ieee.org.libezproxy.open.ac.uk/iel5/52/29730/01353225. pdf?tp=&arnumber=1353225&isnumber=29730

Open source software is always software released under an Open Source Initiative (OSI) certified licence. Each of the licences approved by the OSI meets the conditions of the Open Source Definition [http://www.opensource.org/docs/definition.html]. That definition includes 10 criteria. Perhaps the most important of these are the free redistribution of the software, access to the source code, and the permission to allow modifications to the software and derived works that may be distributed under the same licensing conditions:

http://www.oss-watch.ac.uk/resources/opensourcesoftware.xml

Open Source raises interesting issues. This paper looks at the integration of open source components in commercial applications.

1 Suggest some open source software drawbacks when compared to commercial software development. [3]

2 The paper presents five critical issues for the reuse of open source components. Briefly discuss each of them. [12]

Note that the article mentions COTS, which means Commercial Off The Shelf. This term refers to software or hardware products that are ready-made and available for sale, lease, or license to the general public.


Except, as you can see, it didn’t.

Pardon our scepticism, we told Mr Prior – but wasn’t this a touch convenient? If it was a "trick question" all along, then the proof would be in the marking.

"Would we give a mark to someone who eulogised the article? No,” he replied.

Well there’s one way to find out, we suggested. Why not crack open the marking of Question 2, M885, which would confirm that anyone who failed to spot the trick question had failed.

We’re still waiting for that request to be fulfilled.

We have, however, a strong piece of evidence to suggest that the explanation Mr Prior was asked to pass on to The Register was a misleading, post-hoc piece of damage limitation.

Based on his discussions with the course staff, Mr Prior told us that he was confident that the student, having spotted the irrelevant gibberish, passed Question 2 with full marks. By offering an analysis that the paper was garbage, Verity Stob would have been "right on the money", Mr Prior told us.

In fact, Verity spotted the irrelevant gibberish, noted it in her reply, and received no marks at all. Here's an extract from Verity Stob's reply to Question 2:

After long consideration and much agonising, I am declining to answer this question. My belief is that a large part of this paper is written as deliberate gibberish, designed to deceive superficial readers into believing that they somehow "aren’t quite following" what is going on. This belief is objectively supported by the jumbling of one of the stolen passages before insertion into the paper’s text, as detailed in the PDF I have sent to the OU. But actually I suspect that much larger portions of the paper were generated in this manner. I’d draw your attention to the ‘Customization requirements’ section of the ‘five critical issues’ in the paper, which is not something I can show to be plagiarised. I think this is a particularly striking example.

I could fairly easily create answers to these questions by extracting short, key phrases and contriving to include ideas presented on this course. I feel that to do this would make me a party to the deceit, or at least to a certain level of complacency about an immoral practice. I don’t think it is acceptable to write essays in this way.

Or, there again, I could answer the questions without reference to the paper, and attempt to save at least some of my marks. But then why should I lose any of them, for being, as I see it, true to my standards? It seems to me that only an all-or-nothing stance makes sense.

Contrary to what Mr Prior had been told, Verity Stob's answer forfeited all 15 marks. Someone is being economical with the truth.

Loose ends

There are two footnotes to the ongoing story. We contacted Tony Byrne, editor of CMS Watch, whose 2003 article had provided source material for Madanmohan and De’s plagiarized text. What did he make of all this, we wondered. “I don't know quite what to think about it. First reaction to the plagiarism was annoyance, but then mostly amusement,” he mailed us.

As readers have noted, this paper is conspicuously absent from the many published works listed at the website of co-author, Rahul De. Was this forgetfulness, shame or did he know anything about the paper? We mailed him, inviting him to confirm that he was aware of the paper’s existence – but have yet to receive a reply.

Meanwhile, until we can see the marking for the answers handed in – the Open University can maintain what appears to be a highly convenient cover story. It may just be digging itself in deeper. ®