OECD should be ashamed; PISA scores announced; doing more damage

Last week I spoke at Online Education Berlin. I thought that I was the opening speaker, but the people who arranged this meeting had set up the OECD representative of PISA, Andreas Schleicher, to be first, so that I could respond to him. I was going to talk about AI and pragmatic learning in online courses, but altered my talk to be an anti-test rant in response to his remarks. He never spoke to me at any time. Here is one the slides I used:

Now, not everyone knows what PISA is, but you should know. PISA is the single worst idea being forced down people’s throats in the world of education. It is sponsored by the OECD, which I had previously thought of as an organization that was started to do good. 

I first became aware of the effects of PISA when I was invited to speak in Bogota, Colombia because they had come in 62nd on the PISA rankings. What that means is that they were in an international competition, the World Cup of testing, and Colombia felt they needed help because they came in 62nd. At this meeting it was clear that they needed help all right, not in getting their PISA scores better but in getting some perspective on education.

Here is a current announcement from PISA: 

Coming Soon: PISA Results

On 6th December 2016 at 11.00 am (Central European Time) the results from PISA's 2015 round of testing of 15-year-olds in science, reading and mathematics in 72 countries and economies will be released. 

http://www.oecd.org/pisa/

How sad. 72 countries are waiting to hear how they are doing in the international math competition. Why would anybody care? Here is the analysis from last year:

"Asian countries outperform the rest of the world", according to the OECD, with Shanghai, Singapore, Hong Kong, Taiwan, South Korea, Macau and Japan amongst the top performing countries and economies. Students in Shanghai performed so well in maths that the OECD report compares their scoring to the equivalent of nearly three years of schooling above most OECD countries.

Of the 64 countries with comparable data up to 2012, 32 improved their reading performance while 22 showed no change and 10 deteriorated. If you look at performance at maths, 25 show an average annual improvement, 25 show no change, and 14 show a deterioration in performance.

Qatar, Kazakhstan and Malaysia recorded an average improvement in maths performance of more than eight points per year. The OECD report also praises Brazil, Chile, Germany, Israel, Italy, Mexico, Poland, Portugal, Tunisia and Turkey who it claims have "shown a consistent improvement" over time in maths performance.

I will keep you in suspense about who won this years competition. But let's think about what these scores might mean in each country. I will start with Greece, a country I know well because I was a consultant to a Greek ship owner (to help him invent software for the shipping industry) for many years.

Greece's main industries are tourism, shipping, industrial products, food and tobacco processing, textiles, chemicals, metal products, mining and petroleum. (This is from wikipedia)

I can tell you that there is not a single course in any Greek school that teaches shipping or tourism. They do teach Ancient Greek and Greek history and, of course, algebra and science. Greece came in #42 in PISA in 2012 and I am sure this made them very anxious. What makes me anxious is that Greek schools didn't care about teaching their kids about the jobs they might actually be able to get in Greece. Instead they are engaged in a competition to answer questions like this one:

DRIP RATE

Infusions (or intravenous drips) are used to deliver fluids and drugs to patients.

  Nurses need to calculate the drip rate, D, in drops per minute for infusions. They use the formula D = dv where

 60n

d is the drop factor measured in drops per millilitre (mL)

v is the volume in mL of the infusion

n is the number of hours the infusion is required to run.

1. A nurse wants to double the time an infusion runs for.

Describe precisely how D changes if n is doubled but d and v do not change.

2. Nurses also need to calculate the volume of the infusion, v, from the drip rate, D. An infusion with a drip rate of 50 drops per minute has to be given to a patient for 3 hours. For this infusion the drop factor is 25 drops per millilitre. What is the volume in mL of the infusion?

Now, I was a math major in college and I figured out math on my own when I was about 5. I can’t answer this question. My mind glazes over. I could see teaching nurses how to do these calculations in nursing school, but everyone in the whole world?  Why? I assume the chief engineer on a ship knows how to do math. I am also sure he went to engineering school and rather sure that he actually never does much algebra.  But PISA ensures that every 15 year old student should be able to answer questions like this. 

Wonder how France did? They came in 25th. This is important because science and math matter so much in France. The Major industries in France include automobile manufacturing, aircraft production, chemicals, electronics, machinery manufacturing, metallurgy and tourism.

Surely these require math you say? Well, if they do, you can be sure that the math needed is covered in the schools that teach metallurgy or chemistry. PISA is for 15 year olds. I would be happy to see France offer metallurgy in high school, or tourism, or aircraft production, but they don’t. They teach the basics which includes Moliere, Robespierre, Louis the 16th, and lots of algebra. My daughter attended high school in Paris when she was 13. She got a B in English.  I asked her if anyone in her class spoke English and she said “no.” “Then why did you get a B,”  I asked. She said she never heard of the subjunctive case. So I am pretty sure they also teach the subjunctive case in France which matters so much to a grammarian and so little to an actual speaker who has non-conscious knowledge of their language and can use it but doesn't know what it is called (nor do they care.)

I was wondering if they tested for knowledge of the subjunctive case in PISA and found this:

In the PISA 2012 Student Questionnaire an OCT was operationalised by asking students to indicate their familiarity – on a 5-point scale from “never heard of it” to “know it well; understand the concept” – with actual mathematics concepts (e.g. “polynomial function”) and foils (e.g. “proper number”). Foils were created by combining a term from grammar (i.e. proper, as in proper noun; subjunctive, as in subjunctive mood; declarative as in declarative sentence) with a mathematical term (i.e. number; scaling; fraction, respectively).  

So they expect it to be taught so that they can use it as a foil in math exams. Pretty clever PISA.

How about Finland? Finland is considered very progressive in education. They are said to be eliminating traditional subjects which would be a wonderful thing. And they win PISA quite often which would imply that maybe they aren't eliminating algebra. But surely this is because “math teaches you to think” which everyone believes and I believe as much as I believe that the Great Pumpkin arises on Halloween eve. 

Metals and engineering now constitute the largest sector of Finnish industry, with motor vehicles and machinery driving much of the growth of the late 1990s.

So, Finland teaches machinery in high school? I hope they do but I doubt it.

As I said, I was worrying about Colombia when I was asked to visit there after they “failed” PISA.

“Colombia is the largest export partner of Aruba (39.4%). The petroleum and natural gas coal mining, chemical, and manufacturing industries attract the greatest U.S. investment interest.”

Do they teach mining in schools in Colombia? No. I was advising in Chile too recently. Mining is major there too. Do they teach mining in Chile? No. I fought with the education ministry there who wanted to teach more algebra in order to do better on PISA tests. 

Today the PISA scores were announced. Singapore won. I was recently speaking in Singapore. The cab drivers told me what an undemocratic totalitarian country it was and I hadn't even asked. I met with the Chairman of Singapore Democratic Party who told me he can never win an election because the party in charge owns all the media and he never gets mentioned. Singapore thinks it has a great education system. I spoke to the teachers there maybe 10 years ago and suggested that maybe learning to think for yourself was more important  than rigidly practicing for math tests all the time. My message was received coldly by the teachers.   And then I went outside and many people on the street were cheering for me. The talk had been televised and average people, the doorman, the taxi driver, the hotel manager were all aware that they had hated school and had been taught nothing of value for their lives. But Singapore won PISA . Hip hip hooray.

After my talk in Singapore last year I got a call from people who wanted my help on a project to teach people to get better scores on TOEFL tests. China and India do well on PISA too. How does this relate to TOEFL tests? Many people in these countries really want to go to university in the US or the UK. They need to show they can speak English well and this is shown by?  Multiple choice tests of course. Prospective students need to have great TOEFL scores and great math scores to get admitted. So congratulations PISA, and to all winning countries, on helping your people get our of you country and study in the US the UK. That will help you get rid of your future scientists, who rarely want to leave the US after they get there PhDs here. (I rememberer trying to persuade a brilliant Indian AI student of mine to go back to his country and help it. He looked at me like I was crazy. Now he is the head of AI at Amazon.)

Does the US and UK have it right? Hardly.

In Santa Fe, New Mexico I asked the people  who ran the Indian school there what curriculum I could build for them and they said “Casino Management.” One can well understand why. Could I do it. No. The Governor vetoed it and said “more algebra.”

In Kansas I asked what they needed and was told aerospace engineering. “Really?” I asked. “Yes” they said, we build Lear Jets here and we don’t have a single aerospace engineering course anywhere in Kansas.” Could I do it? Of course not. We have Common Core instead of PISA in the US. But it is all the same. The 19th century curriculum reinforced by testing companies. 

Getting people to think for themselves, becoming capable of employment, able to live happy lives, would seem to be better goals to me, but instead we have PISA. OECD should be ashamed.

Here is part of what the OECD had to say today:

Around 1 in 10 students across OECD countries, and 1 in 4 in Singapore, perform at the highest level in science. Across the OECD, more than one in five students falls short of baseline proficiency: only in Canada, Estonia, Finland, Hong Kong (China), Japan, Macao (China), Singapore and Viet Nam do at least nine out of ten 15-year-old students master the basics that every student should know before leaving school.

This underlines the challenge that all countries, including some of the wealthiest ones, face in meeting Sustainable Development Goal 4 by 2030 to achieve “inclusive and equitable quality education and promote lifelong learning opportunities for all”.

The report reveals the policies in place that successful countries share: high and universal expectations for all students; a strong focus on great teaching; resources targeted at struggling students and schools; and a commitment to coherent, long-term strategies.

Canada, Denmark, Estonia, Hong Kong (China) and Macao (China) achieve both high standards of excellence overall and equity in education outcomes.  A number of countries have improved equity, especially the United States. But in Australia, Czech Republic, Finland, Greece, Hungary, New Zealand and the Slovak Republic, the share of students performing at the highest levels fell at the same time as the share of low performers rose.

“Achieving greater equity in education is not only a social justice imperative, it also fuels economic growth and promotes social cohesion,” added Mr Gurría.

The OECD PISA 2015 Survey underlines that, in the context of massive information flows and rapid change, everyone now needs to be able to “think like a scientist”: to be able to weigh evidence and come to a conclusion; to understand that scientific “truth” may change over time, as new discoveries are made, and as humans develop a greater understanding of natural forces and of technology’s capacities and limitations.

Everyone needs to know how to think like a scientist. Really? Why? And how would these tests assess that? Let's look at a typical PISA science question:

Bird migration

Most migratory birds gather in one area and then migrate in large groups rather than individually. This behaviour is a result of evolution. Which of the following is the best scientific explanation for the evolution of this behaviour in most migratory birds?

1 Flying in large groups allowed each bird to have a better chance of finding a nesting site.

Flying in large groups allowed other bird species to join the migration.

Birds that migrated individually or in small groups were more likely to find adequate food.

Birds that migrated individually or in small groups were less likely to survive and have offspring.

I found this question in an Australian newspaper today. Here is the headline that accompanied it:

Australian school students two years behind world's best performing systems

Or, in others words, we are all losers. We need to be very afraid. Inciting fear seems to be the major outcome of PISA. I wonder why. Mr Schleiker was proud that all countries will soon be the same and his test will make them be the same in their schools. This is exactly the opposite of what needs to be the case. School should fun. School should be relevant to life after school. And every country and every state is different. Their differences should be cherished. And no one needs algebra or science in high school. What we need is people who can think clearly, understand  scientific thinking, be creative, defend one’s ideas, diagnose problems and come up with solutions, That is what PISA should test. Actually PISA should test nothing. OECD needs too get rid of this idea completely. Perhaps they could put their money into helping countries teach kids things that will useful in their own countries.

The Education Platform I wrote for Mr. Trump

Corey Lewandowski who was, at the time, Trump's Campaign Manager asked me to write an education platform for Mr. Trump. Cory was fired soon after this request, so I am sure Mr. Trump never saw it. Today, I noticed that after the absurd choice of Ben Carson, Mr Trump is now thinking about appointing to secretary of education the head of Success Academy. That is a school that makes sure that all its kids do well on tests. Since testing is the problem in our schools, I find this rather odd, but maybe it is just the usual newspaper rumors. 

 In any case I have decided to share my platform. Here it is:

Platform for Education Policy (proposed by Roger Schank)

some basic assumptions 

1.      Common Core is a bad idea. It reinforces the 19th century curriculum and over emphasizes testing. It needs to be eliminated.

2.      One size fits all curricula are a bad idea. Students have different interests and should be allowed to pursue what interest them (after the 3Rs are learned.)

3.      We have the technological capability to build 100’s of new curricula that would be delivered online with mentoring, to help kids get excited about going to school to learn skills they really want to learn.

4.      Federal control over education policy is a bad idea. Each state is different in many ways. Kids might want to study Oceanography in Florida, or Casino Management in New Mexico, or Aerospace Engineering in Kansas. The role of the federal government should be to fund the creation of such curricula and help find people who can build them after being requested to do so by a State.

5.      High schools have failed. They are all about college prep but they shouldn’t be. It is not the role of the high school system to make admissions decisions easier for colleges. Now, colleges dictate the high school curriculum by “suggesting” that students take certain subjects before they apply. The colleges need to be encouraged to ask for examples of thinking ability, communication ability, and being able to get along with others. Then, the colleges themselves can teach whatever they think their new admits need to know. The system has been built by the elite colleges to make the admissions process easier for themselves. Yale had 29,000 applications for entry into the call of 2017. We must encourage the extended Ivies to not make high school and the SAT all about their needs and start thinking about the needs of the average person. These are more likely to be job skills and life skills than algebra and chemistry skills. The Ivies need to stop telling high school students what to study. No one needs algebra except to meet these requirements.

The very idea of academic subjects in high school needs to be replaced. Most high school students do not go on to become academics. Students need to be helped to learn to think and should learn skills by actually doing them. The skills they learn should relate to actual things adults need to do. For example, why are there no child raising courses, or courses on how to speak well, or courses on how to get a job, or courses on how to deal with personal finance? Everyday skills matter.

I would like Mr Trump to shut down the DOE as he promised and to replace it with Education DARPA. This would be a fund of money offered to people who want to create innovative curricula in any area at all, allowing people everywhere to learn what it interests them to learn. You want to be a doctor? Try it out in an online learn by doing simulation in high school. You want to design aircraft? Do it in high school. Open a restaurant. Do it in simulation first. We can build thousands of them, offer them online with teachers who are available to help no matter where you are.

Please Mr. Trump. You are right about returning control of education to the states. This means fighting big corporate interests who want to sell more tests. We don't need kids who test well, we need kids who are excited to learn because they are following their own interests and can what they have accomplished.

Learning about AI by watching Family Feud

Last week I was in Bogota, Colombia, interviewing experts in the food industry in order to gather video stories for use in a program to be used by that industry. At least 2 of the interviewees mentioned that they were using AI. They didn’t know anything about me so they didn’t mention AI for that reason. They just were saying that they were using AI in their food companies. This is so weird I had no idea what it could mean. What could they know about AI? AI has become the kind of thing that people (and especially the media) mention at the drop of a hat these days. But, they deeply fail to know what AI really is or at least should be.

Also last week, Stephen Hawking yet again made a pronouncement about AI, making clear he has no idea what AI is either.

So, I have come to the conclusion that AI  is a word that many people and businesses say a lot while very few people who say it have a clue as to what it means. 

With that thought on my mind, and being someone who has thought about AI, and human intelligence, for about 50 years, I want to make some remarks here about human intelligence based on some rather mundane observations.

I happened to be watching a silly U.S. TV show, Family Feud, that my wife likes. She likes it because the contestants are often quite stupid and she likes to make fun of them. The show pits two families against each other to see who can best guess answers from a survey of 100 people.

On one particular show we were watching the following question was asked: 

Name something someone might hide in their freezer. 

The first contestant’s answer was:

a dead body

That response matched an answer that 5% of the people had given who had answered the survey. The response was written on the game board  as: body/lover’s head.

So, I started to think about AI. “Why?” you might ask. Because my approach to AI has always been to observe humans and see what they do and wonder how we could get a computer to do that. Based on such observations, my team and I would try to build programs to capture people’s intelligence by mimicking the cognitive processes that we recognized must be occurring.

So, the AI question here is: how does “lover’s head” or “dead body” come to mind when asked this question? You can be sure it doesn’t come to Google’s “mind” nor to Alexa’s “mind” nor to Watson’s “mind” because it is not a matter of text-based search nor statistical calculation.

In order for that answer to come to mind one has to ask oneself about weird things one may have seen in a freezer. Human search of human memory depends on many things none of which are text. What probably happened here, is what often happens; one tries to form an image in one’s mind based upon prior images one has seen. Now I have never seen a human body in a freezer. No wait. Of course I have. I have seen it in the movies. I can see that image now as I write this. I don’t recall the name of the movie, but I can visualize a large freezer in a garage someplace.

So, here we have one principle of human search of human minds:

People have the ability to recall images by trying to imagine something and then connecting what they are imagining to something they have actually seen.

Other answers on this show to this question included these:

2nd answer : jewelry (this matched an answer that 4% of the surveyed had given)

3rd answer: moolah, money (this matched the answer money)

The next three probably were again done by imaging. But they might been done by contestants asking themselves the question: what would I want to hide from someone:

Ice cream

booze 

jewelry

I am not sure why one would like to hide ice cream, but the others seem normal enough. People hide valuables. If you Google this answer you will find that there are texts that say that and therefore Google could come up with jewelry.

You can view this episode of Family Feud here starting at the next answer:

https://www.youtube.com/watch?v=rEAJ9iuU9Bc

The first contestant in the video (which starts in the middle of this question) said:

I am gonna go with a little bit of that ganja weed.

This matched drugs/fat blunt which was an answer given by 8% of those surveyed. Someone who smokes marijuana would have to regularly hide it, but the human search mechanism here is a little different than what we have seen so far.

To answer this, a person would have to ask themselves a question like: “what am I afraid that someone might see in my home?”  (where someone might mean the police or one’s parents typically).

Now, this is a question about one’s personal experience. The person who answered it had to have transformed the initial question into: what do I have that I often hide?

The AI issue here is what we might call question transformation. We hear questions, and in order to answer them we ask ourselves about our own personal experiences or fears or desires.

People typically transform general questions into personal questions in order to answer them.

Would “an AI” have to do that? It would indeed. People try to find memories by asking themselves things and then coming up with new thoughts and old memories. They don’t search texts since they don’t have texts in their minds. But, no one in AI is thinking about that sort of thing anymore. They are too busy with search and machine learning based on tangible data.

The next answer (which is also on the video) was this:

well if you’ve got drugs and you've got booze you got to have something to protect yourself so I am gonna say a weapon  

This did not match any of the answers from the survey, but it caused the host of the show (Steve Harvey) to make the following comment:

“see, the thought process, the beauty of this family, is being how they arrive at the answer”

Oddly enough, Steve Harvey is making an AI observation here. He is noting that in order to arrive at an answer, the contestant had to actually think. Further the contestant was thinking about the answers that had already been given and was picturing a scene where the thing that had been mentioned had actually happened.

Thinking involves imagining events that one may have never witnessed in any way. It involves drawing conclusions from the events that one has imagined. 

People can imagine events and draw conclusions about what would happen next in the imagined circumstance.

Funny how no one is working on that in AI. It is hard to imagine, Mr. Hawking, that a machine that could not do that would be very smart, much less be something to be frightened of.

Now I would like to examine another question from a different episode of Family Feud. This one is a little off color which is not uncommon for this show. I am sorry about that, but the example gives us more food for thought about AI. The question to the families was:

Name something done to nuts that Mr. Peanut’s wife would likely do to him for cheating on him.

1st answer: crack him (this was correct and the top answer from the survey)

The fact that this was the top answer is very interesting because coming up with it is very difficult. It involves recognizing that a word that has two very different meanings can be used as an answer. One has to ask oneself what kinds of things one does with peanuts and then imagine how the answer you came up with might be similar to a different meaning of the word.

What can I do with peanuts? is a question that again requires one to imagine a circumstance. Then, one has to take the word that might be used to describe the action and see if that word can be applied to expressing anger in some way. To do that, one has to infer that the wife is angry and that she might want to retaliate in some way. Then after finding the word “crack” one has to recognize that “crack someone over the head” is an expression that exists in English and is a way of expressing anger. No computer today can do this. People do it easily.

Here are more answers that rely on similar processes:

2nd answer: hide them (not on survey)

3rd answer: throw them out (not on survey)

4th answer: eat them (6th answer on survey)

5th answer: switch foods (not on survey)

6th answer: roast them (#2 answer on survey)

7th answer: chop him to bits

8th answer: make peanut butter

9th answer: boil him

In order to answer questions we must frequently imagine circumstances, draw inferences about the feelings and actions of the participants in those circumstances and then think about words that can be used to describe those feelings and actions.

I will look at one more question from that same show:

Name something the ladies might do if a male stripper preforms at the nursing home

1st answer: faint (this was #7 on survey)

2nd answer: scream (this was considered to have matched the #1 answer which was laugh/cheer)

3rd answer: dance (this matched the #6 answer)

4th answer: get a lap dance (This was deemed to have matched handle goods/spank.)

5th answer: pull out the cell phone and take pictures (This was not on the survey.)

These answers involve putting oneself in someone else’s situation. One has to imagine oneself as an old lady in a nursing home. Since these are not old people on this show and since half of them are men this requires a great deal of imagination. You must ask yourself how someone feels when you may have only minimal knowledge of what that kind of person might feel.

What does a sick old woman feel when faced with male sexuality? is a complex question. How do you think Watson would do with it? My point is that human intelligence is complex indeed and no computer can do very much of it. 

I included this question because of the next answer. I had no idea whatsoever what the answer meant:

6th answer: make it rain (this matched $$$/make it rain which was the #2 answer)

So, not only couldn’t a computer come up with this, but neither could I. And, I could not comprehend the answer. But the audience did. (My wife explained it to me.)

This leads me to my key point. Why did the audience (and my wife) know what this meant? Probably the simple answer is that this is a new expression and I don’t pay that much attention to pop culture. But, and this is the issue, intelligent entities are constantly changing and growing. They learn new things constantly by listening to people, watching movies, and television, texting on the phone, and maybe even by reading. (I am not sure where exactly one might read this expression.)

Intelligence involves the ability to constantly change oneself by updating one’s world view according to new inputs.

I learned this lesson when I was running my AI lab at Yale many years ago in the following way. DARPA (our sponsor) was coming to see what we had done. We had built a program that used the UPI wire as input and read stories, summarized them, and answered questions about them. To prepare for this demo, we knew of many stories that our program had read successfully and so we showed them being read by our program to the visitors from DARPA.

The program did well and our visitors were impressed. But suddenly I found myself getting upset.  One of the stories was about an earthquake in Iran. I knew that our program has read that story many times. I realized that it should have responded:  Enough with the earthquake in Iran story. Or is this a new one? There sure have been a lot of earthquakes in Iran lately. 

Of course, it hadn’t done that because it hadn’t learned anything from that story.

People learn from every experience they have. An AI program would have to change after every experience as well, in order to be considered intelligent.

It was at that point in my career that I switched my interests to learning. I got interested in how to get computers to learn and I also became interested with every discovery about how what I was learning about learning bore very little relation to how learning was taking place in the schools.

So, Mr Hawking, and Food Industry people from Bogota, and Watson, listen up: if your “AI” isn’t changing as a result of every interaction it has it isn’t “an AI” at all. People, even not very bright ones, learn something new from every experience.