YouTuber charged loads of fans $199 for shoddy machine-learning course that copy-pasted other people's GitHub code
Oh, and there wasn't a refund policy until folk complained
Special report The AI hysteria has led to a rash of budding engineers hoping to land a cushy job somewhere in Silicon Valley.
So it's no wonder that thousands flocked to an online course titled Make Money with Machine Learning fronted by Siraj Raval, a self-proclaimed AI educator, rapper, and entertainer with nearly 700,000 subscribers on YouTube.
Raval began his internet fame by uploading videos describing how to build a neural network in five minutes, how logistic regression or sentiment analysis works, and how to build Bitcoin or healthcare startups. His work has been praised by revered figures in the machine-learning community like DeepMind CEO Demis Hassabis, and shared by Tesla and SpaceX boss Elon Musk too. So far, so good.
Excellent summary of #AlphaGo Zero from @sirajraval. In general, his videos are a great introduction to AI algorithms: technical yet accessible, and also entertaining - a rare combination!! https://t.co/2LAecUv3pI— Demis Hassabis (@demishassabis) August 23, 2018
But his reputation as a rising pop science star has been called into question after hundreds of students enrolled on his 10-week online course began demanding refunds. For $199, machine-learning enthusiasts are fed weekly video lectures, online quizzes and reading assignments to learn how they can make money by applying AI in various industries like finance, agriculture and healthcare.
Students were also sold access to live Q&A sessions with Raval, promised personal feedback on their code, and invited to join an online network of students interacting with one another on popular chat apps Slack and Discord.
After they finished the course, Raval also said each student would receive a referral to major companies like Nvidia, Google, Amazon or to a startup founder or consultant. After purchasing the course, however, hundreds of them were left unsatisfied.
Ray Phan, a senior computer-vision and deep-learning engineer at Hover, a 3D software startup based in San Francisco, told The Register that the lectures were confusing and contained mistakes. The online projects were too simplistic to be useful in the real world, and Raval was absent and unsupportive.
"As an example, he designed an assignment where you had to use linear regression to predict stock prices," said Phan. "Linear regression in the sense that it's a first-order predictor where you predict stocks with a straight-line approximator. This is utter garbage and not representative of what actually happens in industry.
"The homework assignments – at least from what I saw before I left – were very lacking. The first assignment dealt with copy and pasting from a Tensorflow transfer learning tutorial. Overall, I found the quality of instruction, the assignments and his support extremely lacking and not representative of a true educator."
Refund? What's a refund?
Phan paid for the course a week before it was due to start in late August, but by early September he wanted his money back. Raval had not implemented an official refund policy, however, and unsatisfied customers took to emailing him directly about their concerns.
The whole fiasco kicked off when students began trying to work together on the online projects over Slack. Some couldn't find one another in the workspace channel, and concerns started brewing. Raval had promised to cap the course to 500 people to make sure he was providing enough support to everyone. The Slack channel, therefore, should have contained all 500 students, so it was strange when some students couldn't contact one another.
That's when they realized that Raval had in fact set up two separate workspaces on Slack to hide the fact that he had actually enrolled more than 1,200 students for his course.
"We did some investigation and we noticed that we all got different Slack workspace invites," Phan said. "We all tried out the different invites and to our surprise we saw that there were two separate workspaces that were for the course. The workspace I was in had almost 770 students at the time. The other workspace that some friends of mine were in had about 500 students. We absolutely did not know that another workspace was made."
When the students caught on, some formed another Slack workspace to launch a formal complaint. Raval then apologised for enrolling too many students.
"The course applications hit the 500 student limit within 24 hours, which was pretty surprising to me," he said. "I thought it would take much longer. Immediately afterwards, I started getting lots of emails from students as to why they would still love to join, so I made a few exceptions. And a few exceptions turned into a lot. At the time, I didn't feel like it would diminish the student experience and I vastly overestimated my ability to manage a large number of students alone."
The free version of Slack implements a 10,000 message limit. Once a workspace reaches that number of messages, they start getting wiped. To prevent paying for Slack, Raval moved all his students to Discord, a free chat service popular among gamers.
When students even mentioned the word "refund" on the Discord server, however, their messages got deleted automatically. People began suspecting that someone had written a script to remove all messages asking for refunds to prevent more people asking for their money back.
Raval blamed it on one of his teaching assistants, a small group of volunteers working alongside moderators to help answer any course queries. "I think it was one of the TAs but I'm not sure who exactly. It was not me and I did not ask for that," he told The Register.
At $199 a pop, having 1,200 students means a total of $238,800. That pile of cash has dwindled since Raval began refunding students and at least 250 of them have got their money back so far.
"He just ignored us until it [all] blew up last weekend," one student, who wished to remain anonymous, told El Reg. "In the midst of all this on 8 September, he snuck up a refund policy saying refund has to be asked within 14 days of registration and pretended that it was there all along which itself is ludicrous because there was no refund policy initially and that policy was posted after four weeks of registration start date."
Raval said he'd simply forgotten to add a refund policy. "When I designed the course landing page, I also forgot to add a refund policy," he said. "This was a hard lesson to learn so I got some TAs for the course, and have now given out refunds to every single person who has asked me up to this point today."
In some cases where students had emailed asking for a refund, The Register learned on Thursday that, in an attempt to dissuade them, he had offered to enroll them for free in not one but three future courses.
An email Raval sent to a student, seen by The Register, read: "Since we're more than halfway through the course, the refund period has now ended. However, I'd like to give you a special discount code that gives you 100 per cent off my next three courses. You can use it up to three times and it's valid up until January 2022. Please don't share it with anyone! The code is '------'. I've added you to a private email list, so you'll be the first to know when I release my future courses."
When we approached Raval about this, however, he changed his mind: "Yes, I sent out a few emails with the discount code idea for the first time this afternoon, and five minutes ago received the first reply from a student that they'd still like a refund instead of a discount code.
"Because of that, I've decided to go ahead and refund all students who ask for one moving forward – including the ones I just sent a discount code email to. Looks like they wouldn't be satisfied with the discount code idea."
Ripping off code?
Even if he hadn't taken on so many students, the course was still received pretty poorly. Many claimed Raval had the habit of copying and pasting code taken from other people's GitHub repositories, or linking to some of his old content, and taking reading material publicly available on sources like Medium blogs.
Sven Niederberger, a mechanical-engineering student known as embersarc on GitHub, told El Reg that his code, simulating how to land a rocket with reinforcement-learning algorithms, was used in one of Raval's YouTube videos.
"It is a good example of him exploiting other people's work and purposefully misleading his audience," said Niederberger.
"I had been working on an OpenAI gym environment that simulates a landing rocket. This took me a long time to get right. After that I trained it and uploaded it both to GitHub and Reddit. After presumably seeing it there, he forked the environment for his video in which he did not credit me [properly] at all."
Raval said he did credit Niederberger by adding the line "Credits for this code go to embersarc. I've merely created a wrapper to get people started," on his own GitHub page (here).
"I have 434 repositories on GitHub and for every single one, I perform this same attribution practice. Feel free to browse through my repositories, you'll find that I always credit the author in the GitHub README," Raval said.
A README is, of course, a separate text document describing the code in more detail.
"I think the issue that they have is, I should be crediting the author in the video itself; the GitHub README alone is not enough. And I think that's a fair argument. I am doing better in this regard, if you see my latest videos, when I use code, I mention the tool creators in the video itself," he added.
Udacity and Raval's Nanodegree
It's not the first time that Raval has launched an online machine-learning course. In 2017, he teamed up with educational platform Udacity to teach AI as part of its Nanodegree programme.
The course was rejigged halfway through after Udacity informed Raval that the weekly live coding sessions had to feature software that he had written himself. He told us that for the remaining lessons these live coding classes were scrapped.
"Students liked them, but the replay value was relatively low because it wasn't at the quality level that the recorded content was at. And they figured that later batches of students didn't want to watch prerecorded live content," Raval said.
A Udacity spokesperson told The Register: "In 2017, Udacity collaborated with Siraj Raval to build our Deep Learning Nanodegree program, and we have not worked on building new content with Siraj since.
"The course is still available and has seen more than 26,000 enrollments. We continue to update our courses with new videos, lessons, quizzes, and resources as the industry evolves. Udacity's course development process includes instructor due diligence and extensive quality checks throughout the process to better ensure student success."
Raval received an email from Udacity earlier this month asking to renew his contract so that the course materials can continue to be used.
The spokesperson said: "The contract extension is to compensate Siraj for Udacity's continued use of the content we built with him in collaboration with Udacity's content team and subject matter experts in 2017, not to develop new content. This is standard protocol for content collaborations."
Training workshops, keynote speakers, top advice, and more: Join us at our AI conference MCubed next weekREAD MORE
So, is Raval milking the AI hype, or just a sincere internet personality who simply made a few mistakes?
Well, he's certainly hardworking. He is currently pitching a 12-part AI documentary series to Netflix. "It's a project I've worked on for the past two months," he said. "I am currently leveraging my online audience by tweeting out to Netflix to help generate enough interest for them to reach out to me. No idea if it will work, but I hope it does!"
A spokesperson at Netflix confirmed it is currently not working with Raval.
Steven Edwards, a teaching assistant volunteering to help Raval on his Make Money with Machine Learning, wanted to give the YouTuber the benefit of the doubt. "I believe he is taking the mentality of 'go for it and figure out the rules later'," said Edwards. "I don't believe he has malicious intent. I feel bad for the students, and I feel bad for the damage done to his reputation."
Others like Phan, however, are less impressed: "Some people really like him, and other people really hate him. I decided to give him the benefit of the doubt and try for myself to form my own opinion of him. My opinion of him now is that he's a fake and dishonest crook." ®