This is fantastic, Anna!. I'm already a fan of Chalk & Talk, so it's great to see you contributing at CEP as well.
I'd love to read some stories from you and others about the next step: Basically when you look through the evidence and find shoddy research that doesn't support its own conclusions, how do you present this in a way that gets people on board? Especially for a parent who doesn't have your credentials (not that the credentials should matter more than the evidence!).
Thank you for writing this article. This shines a light on just how murky education research is. Most of the time reports are funded by publishing companies, and this poor practice has made education susceptible to dubious practices for years.
Thanks for writing this article. As a parent and grandparent, it’s hard to see schools struggling with basic reading and arithmetic.
Civilization has been teaching reading and math for centuries. When we set up publicly funded schools in the 19th century, it was assumed we already knew how to educate kids. The idea was to extend that knowledge to everyone, not just the wealthy.
But for some reason, the education establishment set about re-inventing the wheel, and pushed new methods of teaching based on “research”. Apparently the purpose was just to steer more and more money to the research, because it certainly hasn’t been to teach kids.
Great post. The fire hose approach is also called the Gish Gallop after Duane Gish.
Another great find which I can’t find the link to - a UK paper that coined the term “evidence sprinkled”. To describe a work that claims to present an evidence based conclusion but instead presents a conclusion sprinkled with anything that looks like evidence if you don’t scratch the surface.
That paper also recommended a good sanity check - how hard is it to find a good counter point that someone has not addressed?
Someone seeking the truth looks at the best evidence against their conclusion not the worst.
Years ago, when my daughter entered first grade, parents were told that "whole language" was the best way to teach children how to read and write. Having just finished my PhD (in another subject), I rolled my eyes because I knew the academic system: research that shows that current methods are working is rarely rewarded in the academy. To get that doctorate, you need to propose something new, even if the old way is working. My daughter was already reading at that point (she is now a librarian), so I ignored it all. I wish I would have had the courage to speak up and ask for the evidence. As we now know, the evidence wasn't there. My granddaughter is about to enter Kindergarten. Time to start paying attention again!
This is really helpful, thank you. M I’ve found it to be common for my school district to just cite totally unrelated papers for claims and then hope no one notices. Exactly as you pointed out, there are a vast number of reports produced every year so no human can actually read them and verify the citations.
My only encounter with bogus math teaching was a meet-the-teacher night at a well regarded New York public middle school. The emphasis was on letting kids discover math using “friendly numbers” and “clues” to learn, say, percentages and long division. My questions were answered with cheerful vacancy.
At the end we were given a questionnaire which asked inter alia what we learned that night. I said I learned that the school wouldn’t teach my daughter math, and that I would have to.
I remember asking my daughter how any number was unfriendly. What did 7 ever do to her, after all? I remember ridiculing homework assignments with math symbols framing the edge, as decoration.
Sometimes a question on the homework could not be answered from the information provided, or was ambiguously worded. Not because, say, newton’s method for deriving the square root hadn’t been introduced, but because the given input was incomplete. When that happened, I crafted a replacement problem, which she completed. The homework was returned with a note from me explaining the problem with the problem. The teacher never objected or asked to meet. I suppose we understood each other.
The next year my daughter was accepted to a better school, one that understood teaching was more than watching kids guess their way along. She did fine. I feel sorry for her 29 classmates though.
First day of 9th grade math we were introduced to the Common Core math sequence. Instead of the traditional path (Algebra, Geometry, Algebra 2, Trigonometry/ pre calculus, calculus, a 5 year program with advanced students taking Algebra 1 in 8th grade, or doubling up on geometry and algebra 2), they had decided to compress all of those math skills into Math 1, 2, 3, 4. In which every year they would learn some, algebra, some, geometry, some calculus, some trigonometry, and by the end of the 4 years, every child would be able to competently do advanced math instead of calculus, isn’t it wonderful??? I am not a mathematician, but I was on te math team in high school. I love math. And, to me, the math wasn’t mathing. So I asked questions. “What is the scope and sequence of the new math progression?” “Can I see the curriculum?” The answer… it hasn’t been written yet…. We don’t know.
Predictably, by the time my son was in 12th grade, they had started to revert to the traditional curriculum. The results were abysmal and a whole cohort of kids didn’t
learn how to do math… mine were somewhat insulated from the damage because I homeschooled them for a few years and I made sure they had the right skills.
This is great and has tons of parallels in sport. As a layperson/Coach, I’m learning how to read/evaluate research to check claims, but without a Science background it’s very difficult to navigate.
Would love to see the points you made about BS detection expanded, especially how to interpret research and common rhetorical tricks slippery arguers use. This would be a terrific resource for combating this stuff. Maybe a podcast episode or future blog posts?
I've been tripped up by this unverifiable claim myself, when I know the PD is solely focused on one school of educational thought - great rebuttal:
"I’ve been told that inquiry-based instruction is rare and that most classrooms are dominated by direct instruction: the practice is hardly occurring, so why dwell on its effectiveness?...Professional development and school newsletters reflect how teachers are being encouraged to teach. What professional development is being offered? How often does it focus on explicit instruction, retrieval practice, acquiring fluency with basic math facts, or direct teaching of critical math skills versus Building Thinking Classrooms, growth mindset, or inquiry?"
I am a retired educator. I was fortunate during my masters degree to take a class in research and curriculum. First of all, most of the curriculum taught in the US is decided by California and Texas because of the way publishing curriculum works. Reading research can be tedious for most people. Questions parents can ask those administrators that love the phrase "research shows " are : what was the sample size of said research? Who conducted the research and when? How was the control group organized? How long did the research study last? What did the long term results show compared to short term results (Mathew effect)? If the administrator is unable to answer any of those questions without referring you to a book, huge red flag. Easy questions that will tell you exactly how much the administrator actually understands about the curriculum they are presenting. Over the years I have selected curriculum for special needs students and the number of people unable to answer those questions is remarkable. Do NOT allow administration to intimidate you as a parent by touting their "expertise " instrad of answering those simple questions. They picked their curriculum based on cost and the salesmanship of the person presenting it.
Hi. Excellent post. I spent a good part of my career working in behaviour change. In that field, it's well established that facts don't reliably change minds or behaviour. Most often they do the opposite. This very debate is a perfect example :-). What's missing from SoL that could solve this, or what am I missing from my understanding of SoL? SoL says learning is long term memory change, but I can learn every fact about a topic and if my behaviour or attitude doesn't change have I really learned anything? If we look at the medical example of handwashing, perhaps doctors don't go back to not handwashing because the goal of medicine is clear and concise? Do no harm. Perhaps it's not the teaching methods that matter as much as coming up with a clear goal we can measure? If the goal of medicine was to get patients to pass health quizzes would they care about handwashing?
You complain about credential deflection at one point in the article then go on to say that only peer reviewed articles count as evidence. Peer review is just gatekeeping via credentialism.
This is fantastic, Anna!. I'm already a fan of Chalk & Talk, so it's great to see you contributing at CEP as well.
I'd love to read some stories from you and others about the next step: Basically when you look through the evidence and find shoddy research that doesn't support its own conclusions, how do you present this in a way that gets people on board? Especially for a parent who doesn't have your credentials (not that the credentials should matter more than the evidence!).
Yes, we need to talk about the next step. I'll plan on a future episode for that.
Thank you for writing this article. This shines a light on just how murky education research is. Most of the time reports are funded by publishing companies, and this poor practice has made education susceptible to dubious practices for years.
Thanks for writing this article. As a parent and grandparent, it’s hard to see schools struggling with basic reading and arithmetic.
Civilization has been teaching reading and math for centuries. When we set up publicly funded schools in the 19th century, it was assumed we already knew how to educate kids. The idea was to extend that knowledge to everyone, not just the wealthy.
But for some reason, the education establishment set about re-inventing the wheel, and pushed new methods of teaching based on “research”. Apparently the purpose was just to steer more and more money to the research, because it certainly hasn’t been to teach kids.
Great post. The fire hose approach is also called the Gish Gallop after Duane Gish.
Another great find which I can’t find the link to - a UK paper that coined the term “evidence sprinkled”. To describe a work that claims to present an evidence based conclusion but instead presents a conclusion sprinkled with anything that looks like evidence if you don’t scratch the surface.
That paper also recommended a good sanity check - how hard is it to find a good counter point that someone has not addressed?
Someone seeking the truth looks at the best evidence against their conclusion not the worst.
Years ago, when my daughter entered first grade, parents were told that "whole language" was the best way to teach children how to read and write. Having just finished my PhD (in another subject), I rolled my eyes because I knew the academic system: research that shows that current methods are working is rarely rewarded in the academy. To get that doctorate, you need to propose something new, even if the old way is working. My daughter was already reading at that point (she is now a librarian), so I ignored it all. I wish I would have had the courage to speak up and ask for the evidence. As we now know, the evidence wasn't there. My granddaughter is about to enter Kindergarten. Time to start paying attention again!
This is really helpful, thank you. M I’ve found it to be common for my school district to just cite totally unrelated papers for claims and then hope no one notices. Exactly as you pointed out, there are a vast number of reports produced every year so no human can actually read them and verify the citations.
My only encounter with bogus math teaching was a meet-the-teacher night at a well regarded New York public middle school. The emphasis was on letting kids discover math using “friendly numbers” and “clues” to learn, say, percentages and long division. My questions were answered with cheerful vacancy.
At the end we were given a questionnaire which asked inter alia what we learned that night. I said I learned that the school wouldn’t teach my daughter math, and that I would have to.
I remember asking my daughter how any number was unfriendly. What did 7 ever do to her, after all? I remember ridiculing homework assignments with math symbols framing the edge, as decoration.
Sometimes a question on the homework could not be answered from the information provided, or was ambiguously worded. Not because, say, newton’s method for deriving the square root hadn’t been introduced, but because the given input was incomplete. When that happened, I crafted a replacement problem, which she completed. The homework was returned with a note from me explaining the problem with the problem. The teacher never objected or asked to meet. I suppose we understood each other.
The next year my daughter was accepted to a better school, one that understood teaching was more than watching kids guess their way along. She did fine. I feel sorry for her 29 classmates though.
First day of 9th grade math we were introduced to the Common Core math sequence. Instead of the traditional path (Algebra, Geometry, Algebra 2, Trigonometry/ pre calculus, calculus, a 5 year program with advanced students taking Algebra 1 in 8th grade, or doubling up on geometry and algebra 2), they had decided to compress all of those math skills into Math 1, 2, 3, 4. In which every year they would learn some, algebra, some, geometry, some calculus, some trigonometry, and by the end of the 4 years, every child would be able to competently do advanced math instead of calculus, isn’t it wonderful??? I am not a mathematician, but I was on te math team in high school. I love math. And, to me, the math wasn’t mathing. So I asked questions. “What is the scope and sequence of the new math progression?” “Can I see the curriculum?” The answer… it hasn’t been written yet…. We don’t know.
Predictably, by the time my son was in 12th grade, they had started to revert to the traditional curriculum. The results were abysmal and a whole cohort of kids didn’t
learn how to do math… mine were somewhat insulated from the damage because I homeschooled them for a few years and I made sure they had the right skills.
This is great and has tons of parallels in sport. As a layperson/Coach, I’m learning how to read/evaluate research to check claims, but without a Science background it’s very difficult to navigate.
Would love to see the points you made about BS detection expanded, especially how to interpret research and common rhetorical tricks slippery arguers use. This would be a terrific resource for combating this stuff. Maybe a podcast episode or future blog posts?
I've been tripped up by this unverifiable claim myself, when I know the PD is solely focused on one school of educational thought - great rebuttal:
"I’ve been told that inquiry-based instruction is rare and that most classrooms are dominated by direct instruction: the practice is hardly occurring, so why dwell on its effectiveness?...Professional development and school newsletters reflect how teachers are being encouraged to teach. What professional development is being offered? How often does it focus on explicit instruction, retrieval practice, acquiring fluency with basic math facts, or direct teaching of critical math skills versus Building Thinking Classrooms, growth mindset, or inquiry?"
I am a retired educator. I was fortunate during my masters degree to take a class in research and curriculum. First of all, most of the curriculum taught in the US is decided by California and Texas because of the way publishing curriculum works. Reading research can be tedious for most people. Questions parents can ask those administrators that love the phrase "research shows " are : what was the sample size of said research? Who conducted the research and when? How was the control group organized? How long did the research study last? What did the long term results show compared to short term results (Mathew effect)? If the administrator is unable to answer any of those questions without referring you to a book, huge red flag. Easy questions that will tell you exactly how much the administrator actually understands about the curriculum they are presenting. Over the years I have selected curriculum for special needs students and the number of people unable to answer those questions is remarkable. Do NOT allow administration to intimidate you as a parent by touting their "expertise " instrad of answering those simple questions. They picked their curriculum based on cost and the salesmanship of the person presenting it.
You should have noticed the big red flag - a Canadian math consultant.
Hi. Excellent post. I spent a good part of my career working in behaviour change. In that field, it's well established that facts don't reliably change minds or behaviour. Most often they do the opposite. This very debate is a perfect example :-). What's missing from SoL that could solve this, or what am I missing from my understanding of SoL? SoL says learning is long term memory change, but I can learn every fact about a topic and if my behaviour or attitude doesn't change have I really learned anything? If we look at the medical example of handwashing, perhaps doctors don't go back to not handwashing because the goal of medicine is clear and concise? Do no harm. Perhaps it's not the teaching methods that matter as much as coming up with a clear goal we can measure? If the goal of medicine was to get patients to pass health quizzes would they care about handwashing?
You complain about credential deflection at one point in the article then go on to say that only peer reviewed articles count as evidence. Peer review is just gatekeeping via credentialism.
This passed peer review https://www.timesofisrael.com/duped-academic-journal-publishes-rewrite-of-mein-kampf-as-feminist-manifesto/
As I also said, even peer-reviewed articles in education need to be checked carefully.