May 30...Scientifically-based Educational Research
How did the Eisenhart and Towne article leave you feeling about the possibility of educational research to be scientific? What obstacles do you see to realizing the vision of a scientifically-based ed. research and are they insurmountable?
This article was a more in-depth version of what we read in the Paul book last semester, so none of it came as much of a surprise.
As I read the article,I found myself reflecting on the cultural significance of the word "scientific." The American psyche situates that word in white coats and test tubes...I know no one who hears the word "scientist" and thinks of educational researchers videotaping a classroom. I'm not sure we will ever get away from domination by positivist "hard science" science while we use the word "science." In K-12 "science" is natural and hard sciences and social studies is social studies. Social sciences aren't even taught, are they--maybe a little sociology, a little psychology, and anthropology (kind of, maybe, but not much.) It's hard to affect perspective transformation when assumptions about the word "science" goes so very deep into our psyches.
I think educational research can be scientific in the test-tuby, white-coated sense, just as psychology is (if you don't believe me about psychology, I've got a story about a qual-quant argument I got into last semester with some social psychology doctoral students...there was almost violence). But just because I think education could go the way of psychology, "could" doesn't mean the same thing as "should," nor does postpositivism necessarily guarantee that we will learn anything worth learning. At the very least, the avenues of research would be narrowed significantly and some of us might lose all of our joy.
That being said, I can actually live with the definition of scientifically-based research standards as described in the ESRA, IF any pro-positivist assumptions that we as American citizens carry around with us because of our upbringing are bracketed (and that's the insurmountable "IF"). I like systematic and objective methodology that fits our research questions. I like clearly-written, transparent, peer-reviewed presentation of data, data reliability, only making claims of causal relationships reliably. There are no insurmountable obstacles in this list--maybe some better training, maybe some further clarifications of definitions, maybe some cultural reform. The key is educating people--showing them that there are different means to achieve this list of standards and that different methodologies are better for answering different types of questions. Education is the problem, in several different ways.
Kierstyn here....Sorry for the long post!Here goes!
While reading this article I could not help myself from comparing it to other readings for this class. The reoccurring theme in education research is 1) Is qualitative better than quantitative research and 2) What is considered to be research? I became confused because I felt hopeless for a moment. I became saddened because education is so political. The article discussed the prevalence of quantitative research in federal laws such as NCLB. I understand that policy is often impacted by quantitative studies but I think many policies fail to focus on qualitative research.
Before reading this article I was aware that certain fields received more funding, however I was not aware of the history between education research and federal funding. To ensure that federal funding was going to the most beneficial areas, scientific evidence was needed. I consider myself a “fence-sitter” when deciding which type of research (qualitative vs quantitative) is most beneficial when creating education laws. For example if I go to a car dealership to buy a new car, I always research the make and model. I ask people about their experiences with a particular car (qualitative). I also check the miles per gallon of the car (quantitative). In my car buying scenario it is important to use qualitative and quantitative data to make a decision. However, in the case presented in the article I understand why it is so important to have concrete evidence and numerical data to prove that a particular program is beneficial. Unlike buying a car, there is much more at stake if federal dollars are spent on an ineffective program. In today’s economy, everyone wants the biggest bang for their buck.
I do think that education research is moving in a positive direction. I think the six scientific principles listed in the article cover both qualitative and quantitative research. As the article noted “For their part, the SRE principles outline a general form of inquiry and stress that researcher must have the flexibility to choose methods based on their research questions to draw conclusions that are valid for questions and methods used”. The SRE principles outline the general form of inquiry and stress the importance of research flexibility. In my opinion, the six principles are not written in a manner that favors qualitative or quantitative research. The WWC has also made attempts to adopt parallel guidelines for qualitative and quantitative research. The final paragraph on pg. 35 stood out to me. The author stated “In response, the WWC reworded statements about in the introduction to the Study DIAD to emphasize that (a) the work of the WWC focuses on the best methods for assessing causal effectiveness but that (b) the WWC does not believe that quantitative methods are the only methods that can be called “scientific”. This statement makes me believe that the term “scientifically based research” is becoming more broadly defined. In the future I think researcher will be given even greater flexibility. I think a lack of communication, respect and understanding about the field of education is an obstacle that policy makers will have to overcome. I think more collaboration needs to happen between researchers and policy makers.
Carol here......I have a tendency to never accept words like impossible or insurmountable……that seems like an excuse not to try. While it is clear there are vast differences on what types of research may be found “acceptable”……it isn’t clear why educational researcher or practitioners would accept a definition without explanation. Indeed the article clearly states that revisions were made after public comment and testimony. If there are people who disagree with policy and definitions regarding research, then the work of changing it needs to continue. While I get that both practitioners and researchers are busy doing their respective tasks, we cannot ignore the world of policy and politics and simply complain after it begins to affect our work. We must realize our role in the development and assume more leadership in the design of policy. That may mean we have to step outside of our comfort zone and become political. As the article clearly states the ability to influence change exists and indeed a collective voice and organized agenda has an enormous impact. "Never underestimate the power of a small group of committed people to change the world. In fact, it is the only thing that ever has." (Margaret Mead)
When I first learned about the Castle proposal last semester I was rather upset that the government would attempt to regulate (and thus limit) funded research; however, as someone who believes that publically funded research should influence public policy, I agree that there should be standards set for what types of research get funded. This does not mean, necessarily, that I agree with the standards set, but I feel strongly that there are certain standards that should be upheld considering that I hold the belief that research should fuel policy. This being said, policy should not be based upon one study, rather a body of literature. The folly in the post positivist paradigm approach to scientific research, in my opinion, is that it attempts to control for error so much that it misses a lot of error and sometimes produces bold, generalizable claims which appear to be to “scientific” and “valid” but are fraught with problems.
As discussed in Nate Silver’s “The Signal and the Noise”, researchers are perpetually poor at making predictions because they don’t account for extraneous variables. Using the statistician & scout debate in baseball as an example (who is better at predicting the success of young players – statisticians or scouts), Silver being a statistician, admitted that the qualitative data that scouts collect is invaluable when predicting whether or not a player will be an asset to a team. Discrediting their type of data collection methods would be largely erroneous because they get at factors that the numbers cannot capture. The problem with mandating research to be “scientifically-based” is that it’s nearly impossible to account for types of inquiry that are necessary to answer complex questions, and limiting the types of inquiry that are allowed to be funded very well could do the same for education as eliminating scouts for baseball would do to the recruiting process – severely hinder the ability to make quality predictions and judgments.
Heather, I'm going to paste in below the first paragraph of the wikipedia definition of science...Http://en.wiki.org/wiki/Science I know, lazy and unprofessional to use wikipedia but it's actually fairly consistent with the reading I did when preparing a paper on med. education so I'm good with it and it's references. What if we were to back out--oh, maybe 150 years...before those pesky robber- barons and the Carnegies with all their value-motivated philanthropy....what if we loosened and decontextualized our definition of science a little? Does that affect your answer? (I'm looking for that conversation you promised me in the car :))
Science (from Latin scientia, meaning "knowledge"[1]) is a systematic enterprise that builds and organizes knowledge in the form of testable explanations and predictions about the universe.[2][3] In an older and closely related meaning, "science" also refers to a body of knowledge itself, of the type that can be rationally explained and reliably applied. Since classical antiquity, science as a type of knowledge has been closely linked to philosophy. In the early modern period the words "science" and "philosophy" were sometimes used interchangeably.[4] By the 17th century, natural philosophy (which is today called "natural science") was considered a separate branch of philosophy.[5] However, "science" continued to be used in a broad sense denoting reliable knowledge about a topic, in the same way it is still used in modern terms such as library science or political science
How do giant private philanthropic organizations that fund educational programs (and subsequent assessment and evaluation) like Wallace Foundation play into this picture? I can tell you that every Wallace Foundation report or similar that I've ever read (and for full disclosure I've only read thirty or so...I'm no expert) included mixed methodology with a tendency towards the qualitative....I'm just wondering how much of a role private money can influence educational research culture...
I actually read this article when it was released in 2003. It was interesting reading it now, seeing how differently I feel about the topic. The first time I read for information, to understand the process that was changing my profession. This time I read with a professional eye, as a potential researcher whose future practice would be shaped by this content.
When I originally read this article, I looked at the 'big picture' message; educational practices should be grounded in research. I not only agreed with the message but was very excited that it was now policy. Since the establishment of education, students have been treated like lab rats with little thought of long-term affects. Trends in education ebbed and flowed and, especially in the early years of American education, were centered around adults and not students. Eventually, teachers, administrators and even parents began to question the trends and ask 'why are we doing this?' (The accountability movement started earlier than you think.)
So, the task at hand became answer the 'Why?' and shaping how future decisions could garauntee positive outcomes for students. This was a monumental task. There were commissions, panels, experts and lots of talk, talk, talk! We had to start somewhere knowing that the end result would never be universally accepted.
Now we have definitions which restrict the profession of educational research and, as a result, education itself. We should be able to return to the table, continue discussions and adjust the outcomes of the report. Just abiding to our own principle that research is fluid and shaped over time is important to consider here. We no longer live in an era that something written down is final. As a potential researcher I wonder about how to get this conversation started? (The government never changes what they espouse to be true, right?)
Sadly, one must only look at NCLB, 2001 as a whole to discover education in America is being held hostage because politicians refuse to do what I suggested, make changes and update the law that already exists. Rather than go back to the table and make adjustments they are doing twice as much work passing new regulations to simply go around the law.This leaves the discussion on "scientifically-based research' on the books, standing all alone, not being addressed. What can educational researchers do to get it back on the table?
Disclaimer: This post is in no way a political diatribe. The issues facing NCLB reauthorization are bi-partisan. The bar was clearly set at 100% performance achievement and neither side is willing to suggest that this idea was not attainable. Unfortunately, the legislation which binds the profession of educational research is being held hostage as a result.
I am writing at night and this is not when I am able to complete my best work since I am usually up at 5 am…...
In reading this article, I found the standards set forth by the ESRA (2002) to have enough variability to be acceptable for publically funded research. (A great deal of the research completed within my area of interest has been privately funded for the last 30 years.) The process of educational research should not be based solely on one study, but rather the collective body of knowledge and these standards appear to take into consideration various ways knowledge can be obtained. So, I got curious as to what the WWC says currently. I looked up the WWC and the latest version - 3.0 which was accepting public comment through April and the final standards will be adopted in June. I found some interesting information that the WWC contains that I never knew existed (I might be the only one here). Anyway, the WWC on page 8 that a study will be inelgible for WWC review if, “The study does not have an eligible design. The WWC prioritizes findings from studies of effectiveness that use a randomized controlled trial or quasi-experimental design. Studies that use a regression discontinuity design or single-case design may be reviewed against pilot design standards and described in reports. However, the findings from single-case designs and regression discontinuity designs are not included in the evidence findings reported in intervention reports. Studies using other study designs are not eligible for review.” The WWC also has Practice Guides that are intended to give teachers practical recommendations for the classroom based on the research compiled by the WWC. Given this information, it seems that despite what the standards say, when put into practice and with what information makes it to teachers, the “white lab coat” connotation of research is still alive and thriving….or am I wrong? On page 6, they list out what words they use for their literature reviews and the list of words they use also support quantitative research designs though not explicitly stated. Finally, insurmountable– no. Challenging– yes.
I agree, Alison, from an outsider's perspective, the "white lab coat" connotation is alive and thriving as I hypothesize it should be, given our cultural connotation of the word "science." Nothing less than labcoat would make sense in our society. Good to know about your field's private funding...I've been wondering about the impact of private money and would love to hear how that influences things...
In the field of Oral Deaf education, private funding sources have come primarily from one wealthy family who had a deaf child and wanted her to learn to talk. From their initial investments a huge Foundation was formed as well as private Option schools for children to learn to listen and talk. So, in my observations, the funding resulted from a family being personally affected by hearing loss. I wonder if this is true of other private sources of funding. Looking through the WWC, the studies sponsored by this family and the Oral deaf movement were not to be found. However, some very well known computer programs (which were very well funded) were listed as What Works....the difficulty is that the outcomes for HI kids has not changed in relation to reading ability in 40 years. The well funded studies rely on the mechanics of reading and not the basis of language necessary for comprehension....So, to respond to your question, I think the private funding fills in the gaps where the public funding would not dare to go. Just a thought.....
Yes...my experience with private educational funding is based on research and funding regarding system-based afterschool programming. Wallace, Moss, National League of Cities, Forum for Youth Investment (to name a few) are private national funders who have decided how they want education to look and then fund programs...and pay for assessment and eval of those programs in an effort to set up a list of "best practices." To my knowledge they don't have the same personal connection as your field's story--most of these others came from the same ilk as the Carnegie Foundation...guilt money from the people who got rich during the Industrial Revolution (? I'm not sure that's 100% reasonable, but it's what I'm going to write today). They tend to do things like "fight poverty" and "feed children"--all good things...I'm a huge Wallace fan. CF Moss Foundation is highly connected to the government--it helps distribute funds for 21st century community learning centers from NCLB, but the others don't really have much of a government connection. The assessments don't seem to be affected by this pro-quantitative movement much--but then there are definite differences between "assessment" and "research." I agree with you...based on what I've seen, private orgs tend to go rogue, filling in huge gaps in government coverage.
Laura - thanks for a great discussion. I am gleaning more from these discussions - and even more when I go back and read them again! These discussions about "What is Educational Research" really make me question the concept of Altruism in research. Thoughts?
Weighing in on the topic, I'm not sure that there is no one right answer about what is educational research. Too many organizations with too many different definitions makes me crazy. I care more about what works and less about funding (although funding is important), and one problem I see with evidence-based practices is that people who implement them don't always do so correctly. I'm seeing more of a throwback to teachers being autonomous and not working together.
So, can we align definitions and methods and funding?
Like Kurt said about the purposes of education, how can we come to a consensus about what scientifically-based educational research entails? Is it possible?
Like our four-quadrant chart from last class made apparent, educational research is a VERY complex enterprise in making definitive scientific claims. More than many other disciplines, education intersects with a multitude of factors that impose upon research considerations and analysis'.
Pema Chodron, a notable Buddhist philosopher, speaks of the need for humans to cling to certainty. Is this a good thing? We crave definitive answers and outcomes to tie up loose ends. Often, life does not allow for such happy endings.
In the end, the less one is attached to certainty to one's life, the more open one is to the journey life allows...
Is such a viewpoint insurmountable? Of course not!
I wanted to add to my ramble from last night. In my view, the BIGGEST obstacle to realizing a vision of scientifically-based educational research is: what is considered 'science,' and as a follow-up, who is deciding that, and, further, how & why are they proposing such a view of science? Educational research finds itself in a context that includes politics, business/corporate interests, philosophical differences, socio-economic considerations, etc.
'Science' may be crafted to promote many ends that may not necessarily be in the best interests for our democracy, students, teachers, educational leaders, communities, and families.
The current view of science in educational research, highlighted in this article, is definitely conducive to political and corporate interests. No doubt a more multi-faceted and open view should be advocated for...
I couldn't agree more Shawn. The issue is trying to standardize the big 'pile of mush' that is called educational research. Structure can be a damaging thing when you are striving to explore the possibilities.
Our culturally accepted definition of what “scientific” research entails is very “hard” science oriented, and I think the implementation of NCLB in the 10 years since this article was written highlight this. Although there may be a general recognition for the need of different research methodologies in the educational community, it doesn’t seem that public policy has embraced this…just look at the data-driven, standards-oriented schooling on which our country is now focused.
In class we’ve discussed the somewhat unique characteristics of education when it comes to its “soft” and “applied” nature. In my opinion, it is this duality which requires, even demands, that educational research be approached from several perspectives, with the particular research question(s) driving the research methodology. If we, as educational researchers, want this to change, I agree with Carol that we have to be more actively involved in public policy. ~Meg
The Eisenhart and Towne article really helped my understanding of how the intermingling of politics and education has changed, effected, educational research and more in particular the funding for educational research. I do believe that educational research can be scientific and like Eisnehart and Towne believe that finding "constructive ways to encourage and extend the debate and diversity that the current situation highlights" (p.32)is the best approach. I also believe "that the ESRA definitions are more inclusive of various research designs and more sensitive to the realities of research in practice than those in the original Castle Bill" (p.35). I believe this was caused by researchers and the public commenting and constructively adding with appropriate voice (The SRE) concerns, comments and questions to the original bill.
Here lies my concerns, questions, and possible obstacles however. First, The What Works Clearinghouse. This seems to me to be an obstacle because it was awarded a grant in 2001 before the researchers and public voice was integrated into the ESRA and even after receiving comments "many are likely to be disappointed that the changes anticipated by WWC are not more extensive" (p.36). I believe (and have witnessed in my school division) that good instructional and intervention practices may not be supported because WWC has not cleared them, so in the end who had more control over the use of the research and hence, the support of further research the ESRA, the Schools, the researchers, or WWC?
My second great concern or obstacle is who from education practitioners or education research has the time and energy to kkep their finger on the pulse of things like the Castle Bill in Washington, D.C. to ensure the voice of some of the stake holders is heard and valued within our political system. I know we all vote for representative, senators, presidents, and organizations like National Association of Gifted children and ASCD have lobbyists working hard for us, but who is going to be on staff and help focus some of the Washington thinking on true "scientific-based research" in educational practices. There are days that I want to apply to be a congressional aid to the "right" person to start circulating different voices through the offices of congressmen and women. I think the description in the article of how Robert Sweet used research institutions, websites, and researchers (p.32) and the idea that the Castle Bill originally initiated made me even more aware of the question who advises our decision makers about a field they may have no more experience in than the average graduate on the street? Shouldn't we be trying to ensure that is an experienced educator or educational researcher? And how do we encourage this career path or sabbaticals?
Last semester, I worked on a project about including evidence-based research in teacher prep programs, actually special ed teacher prep programs. However, I could argue that the same needs to happen in general ed as well. So, as I was reading the article, everything came back to me.
Here's what the problem is with the WWC, CEC, APA, and the Promising Practices Network (PNP). Each has its own set of standards for what constitutes good research (Freeman & Sugai, 2013). Therefore, bringing these all together in a way that makes sense might be challenging and difficult. And in the Eisenhart and Towne article, the definitions of scientifically based research is similar, but not the same, across several different federal policies (Castle Bill, NRC, NCLB, ESRA).
As far as answering the questions posed, I agree that they posed the issue fairly and objectively based on my previous research. Each group has its own perspective, and bringing them together into a consistent whole may be challenging. I think we need a common definition of what scientific research is in education. We know what it looks like in the physical sciences (Chemistry, Physics, etc.), but is that model translatable to education where people are involved? Simply implementing an intervention without consideration for the human being is unethical (at least, in my view) because our feelings are what makes us different from other forms of life (at least, in part). Not to mention our heritage, culture, languages, etc. So, scientifically based research needs to take into account the human aspect.
I'm not sure if I digress too much in this post, but having thought a lot about scientifically based research (or evidence-based practices) in education over the last 4 months, I believe once we align the standards for what constitutes scientific research and teach teachers how to be informed consumers of research, we'll be headed in a good direction. I keep going back to business and the value of processes and procedures that enable people to complete their assigned tasks efficiently. Education seems to me to be trial and error -- what works for one teacher may not for another due to personality differences, student differences, etc. Yet, we rely on the experience of others to learn how to be teachers. What if educational practices were designed following business practices? But that would require teachers to give up their autonomy and actually learn to work together as partners. So, what works in one setting for one teacher with one student may need to be modified to work with another teacher in another setting and different students.
I understand that time is a critical issue for teachers. But if teachers are taught to be responsible, informed consumers of research, they will be able to implement action research with very little difficulty. For example, I've worked on a project where we teach preservice teachers about self-determination theory and models for implementation. At the beginning of the module, noone has any time to work on self-determination skills with their students. However, a week later, at the end of the module, those same teachers now have the skills to support and facilitate self-determination with their students, and what's more important, actually believe they can within the course of a regular school day and curriculum. So, why does everyone say teachers don't have time for research? If we teach them how, shouldn't they be able to do so in the course of a regular day?
I think the changes in policy are representative of much more than what a group is trying to advocate for, as Eisenheart and Towne were suggesting. I believe they illustrate the Zeitgeist of current educational research. This is just a hunch, I haven’t had time to dig in and gather evidence but I think that when H.R. 4875 of 2000 was passed, the scientific community was debating the standards of qualitative research, and this was reflected in the legislation that stated only what most researchers agreed on, which was sound methodology for quantitative studies, but no further guidelines. The main issue of the time was that quantitative research already had acknowledged standards, but qualitative research, although accepted as a line of research, lacked consensus when it came to picking sound quality standards.
Later on, when consensus was reached on what constitutes acceptable qualitative research, the legislation changed to reflect that in the subsequent authorization of NCLB of 2002. This might very well have been the effect of a group advocating for certain standards, but I think it was more linked to the overall research community perspective on outlining sound research.
Now IDEA is on the roll to be reauthorized (it has been for a few years, actually), so we might see a new development in the definitions of acceptable research when IDEA lays down its evidence-based mandates.
I think the obstacles in a common vision on scientifically based research lie only in helping the members of the scientific community come to an agreement on what they see as sound research, and it seems that they are slowly, but surely, lying the foundation of standards for future research.
Eisenhart and Towne traces how lawmakers attempted to define education research. The need to justify spending public money, and get the most bang for their buck, which is what they are aiming for. However, as the education research community points out that using a positivist philosophy may not be the answer to the problems. Our community is growing more and more aware of the diversity that exists within our community, and we cannot have a one size fits all answer to education (or for that matter anything).
As I read the article, it was very interesting the way different committees defined qualitative and quantitate methods. Most of them were clear on quantitative methods but when it came to defining qualitative methods, it was unclear, and maybe even got murky. As they rightly pointed out that, the research question should point to the research design and the method. I feel that is the maybe the only way to get to the ‘truth’ (t small) as we are working with a soft applied science (Labaree).
I am still worried about education research being more quantitative driven and as investigators we need federal funding (it is the necessary evil). We find ourselves more constrained but the requirements that we have to meet. However, being an optimist that I am, I feel there is still scope for change. Let’s face it, qualitative methods are not always clearly defined, those who do have not have a clear knowledge about the consequences or the process sometimes question their applicability, and lawmakers may be apprehensive about providing funding for those projects. However, as educational researchers we know that qualitative research help answer essential and important questions that cannot be answered using qualitative methods. Moreover, once we can gradually flesh out the idea better, frame qualitative design as a more concrete process, and communicate it better to our lawmakers. They too will realize the benefit. We as researchers need to be careful not to be driven by methodologies, but by the questions that we are asking. It can easily become our obstacle, as we let our methods define how we answer research question. I can very easily say I am a qualitative thinker and will only do qualitative research. But that is not what I have to do as an educational researcher I have to work on answering questions adhering to a scientific process. In the end qualitative, quantitative, quasi experimental or single subject design, will be scientifically-based ed. research as long as they are the best means to answer the research question.
P.s I used Beckers advice and just wrote this chunk. It felt nice.
Jenny here...The issue I have with the Federal government specifying what is acceptable research is that not every research project should be conducted as a scientific experiment that yields quantitative results. The method used to conduct research should be the one that best aligns with the research question. In some cases, qualitative research would be best to use; however, this is not viewed as "favorably" by the federal government as quantitative research. Although the article does state that there were provisions made to include qualitative research, I think the overall consensus of the federal government is that quantitative research is best.
I do think these obstacles can be surmountable if scholars continue to voice their objections. I do have to admit, however, that this issue over which method is the best has impacted my current goals in grad. school. I personally find qualitative research more interesting and a better fit for me as a researcher. However, since quantitative research is more highly regarded by the government and since my goal is to ultimately work for the government, I have decided to focus more on quantitative research since it is most likely that my future job will be in this arena.
This article was a more in-depth version of what we read in the Paul book last semester, so none of it came as much of a surprise.
ReplyDeleteAs I read the article,I found myself reflecting on the cultural significance of the word "scientific." The American psyche situates that word in white coats and test tubes...I know no one who hears the word "scientist" and thinks of educational researchers videotaping a classroom. I'm not sure we will ever get away from domination by positivist "hard science" science while we use the word "science." In K-12 "science" is natural and hard sciences and social studies is social studies. Social sciences aren't even taught, are they--maybe a little sociology, a little psychology, and anthropology (kind of, maybe, but not much.) It's hard to affect perspective transformation when assumptions about the word "science" goes so very deep into our psyches.
I think educational research can be scientific in the test-tuby, white-coated sense, just as psychology is (if you don't believe me about psychology, I've got a story about a qual-quant argument I got into last semester with some social psychology doctoral students...there was almost violence). But just because I think education could go the way of psychology, "could" doesn't mean the same thing as "should," nor does postpositivism necessarily guarantee that we will learn anything worth learning. At the very least, the avenues of research would be narrowed significantly and some of us might lose all of our joy.
That being said, I can actually live with the definition of scientifically-based research standards as described in the ESRA, IF any pro-positivist assumptions that we as American citizens carry around with us because of our upbringing are bracketed (and that's the insurmountable "IF"). I like systematic and objective methodology that fits our research questions. I like clearly-written, transparent, peer-reviewed presentation of data, data reliability, only making claims of causal relationships reliably. There are no insurmountable obstacles in this list--maybe some better training, maybe some further clarifications of definitions, maybe some cultural reform. The key is educating people--showing them that there are different means to achieve this list of standards and that different methodologies are better for answering different types of questions.
Education is the problem, in several different ways.
Kierstyn here....Sorry for the long post!Here goes!
ReplyDeleteWhile reading this article I could not help myself from comparing it to other readings for this class. The reoccurring theme in education research is 1) Is qualitative better than quantitative research and 2) What is considered to be research? I became confused because I felt hopeless for a moment. I became saddened because education is so political. The article discussed the prevalence of quantitative research in federal laws such as NCLB. I understand that policy is often impacted by quantitative studies but I think many policies fail to focus on qualitative research.
Before reading this article I was aware that certain fields received more funding, however I was not aware of the history between education research and federal funding. To ensure that federal funding was going to the most beneficial areas, scientific evidence was needed. I consider myself a “fence-sitter” when deciding which type of research (qualitative vs quantitative) is most beneficial when creating education laws. For example if I go to a car dealership to buy a new car, I always research the make and model. I ask people about their experiences with a particular car (qualitative). I also check the miles per gallon of the car (quantitative). In my car buying scenario it is important to use qualitative and quantitative data to make a decision. However, in the case presented in the article I understand why it is so important to have concrete evidence and numerical data to prove that a particular program is beneficial. Unlike buying a car, there is much more at stake if federal dollars are spent on an ineffective program. In today’s economy, everyone wants the biggest bang for their buck.
I do think that education research is moving in a positive direction. I think the six scientific principles listed in the article cover both qualitative and quantitative research. As the article noted “For their part, the SRE principles outline a general form of inquiry and stress that researcher must have the flexibility to choose methods based on their research questions to draw conclusions that are valid for questions and methods used”. The SRE principles outline the general form of inquiry and stress the importance of research flexibility. In my opinion, the six principles are not written in a manner that favors qualitative or quantitative research. The WWC has also made attempts to adopt parallel guidelines for qualitative and quantitative research. The final paragraph on pg. 35 stood out to me. The author stated “In response, the WWC reworded statements about in the introduction to the Study DIAD to emphasize that (a) the work of the WWC focuses on the best methods for assessing causal effectiveness but that (b) the WWC does not believe that quantitative methods are the only methods that can be called “scientific”. This statement makes me believe that the term “scientifically based research” is becoming more broadly defined. In the future I think researcher will be given even greater flexibility. I think a lack of communication, respect and understanding about the field of education is an obstacle that policy makers will have to overcome. I think more collaboration needs to happen between researchers and policy makers.
Carol here......I have a tendency to never accept words like impossible or insurmountable……that seems like an excuse not to try. While it is clear there are vast differences on what types of research may be found “acceptable”……it isn’t clear why educational researcher or practitioners would accept a definition without explanation. Indeed the article clearly states that revisions were made after public comment and testimony. If there are people who disagree with policy and definitions regarding research, then the work of changing it needs to continue. While I get that both practitioners and researchers are busy doing their respective tasks, we cannot ignore the world of policy and politics and simply complain after it begins to affect our work. We must realize our role in the development and assume more leadership in the design of policy. That may mean we have to step outside of our comfort zone and become political. As the article clearly states the ability to influence change exists and indeed a collective voice and organized agenda has an enormous impact.
ReplyDelete"Never underestimate the power of a small group of committed people to change the world. In fact, it is the only thing that ever has." (Margaret Mead)
When I first learned about the Castle proposal last semester I was rather upset that the government would attempt to regulate (and thus limit) funded research; however, as someone who believes that publically funded research should influence public policy, I agree that there should be standards set for what types of research get funded. This does not mean, necessarily, that I agree with the standards set, but I feel strongly that there are certain standards that should be upheld considering that I hold the belief that research should fuel policy. This being said, policy should not be based upon one study, rather a body of literature. The folly in the post positivist paradigm approach to scientific research, in my opinion, is that it attempts to control for error so much that it misses a lot of error and sometimes produces bold, generalizable claims which appear to be to “scientific” and “valid” but are fraught with problems.
ReplyDeleteAs discussed in Nate Silver’s “The Signal and the Noise”, researchers are perpetually poor at making predictions because they don’t account for extraneous variables. Using the statistician & scout debate in baseball as an example (who is better at predicting the success of young players – statisticians or scouts), Silver being a statistician, admitted that the qualitative data that scouts collect is invaluable when predicting whether or not a player will be an asset to a team. Discrediting their type of data collection methods would be largely erroneous because they get at factors that the numbers cannot capture. The problem with mandating research to be “scientifically-based” is that it’s nearly impossible to account for types of inquiry that are necessary to answer complex questions, and limiting the types of inquiry that are allowed to be funded very well could do the same for education as eliminating scouts for baseball would do to the recruiting process – severely hinder the ability to make quality predictions and judgments.
Heather, I'm going to paste in below the first paragraph of the wikipedia definition of science...Http://en.wiki.org/wiki/Science
DeleteI know, lazy and unprofessional to use wikipedia but it's actually fairly consistent with the reading I did when preparing a paper on med. education so I'm good with it and it's references. What if we were to back out--oh, maybe 150 years...before those pesky robber- barons and the Carnegies with all their value-motivated philanthropy....what if we loosened and decontextualized our definition of science a little? Does that affect your answer? (I'm looking for that conversation you promised me in the car :))
Science (from Latin scientia, meaning "knowledge"[1]) is a systematic enterprise that builds and organizes knowledge in the form of testable explanations and predictions about the universe.[2][3] In an older and closely related meaning, "science" also refers to a body of knowledge itself, of the type that can be rationally explained and reliably applied. Since classical antiquity, science as a type of knowledge has been closely linked to philosophy. In the early modern period the words "science" and "philosophy" were sometimes used interchangeably.[4] By the 17th century, natural philosophy (which is today called "natural science") was considered a separate branch of philosophy.[5] However, "science" continued to be used in a broad sense denoting reliable knowledge about a topic, in the same way it is still used in modern terms such as library science or political science
I applaud the attempt to be good stewards of public money (I recognize my naiveté in believing that politicians desire to be good stewards of our tax dollars and are not otherwise motivated). I believe that there needs to be well defined criteria for federally funded research and federally acceptable research. That being said, I do not necessarily believe that federally funded research should be the only research within a field nor do I believe that the current federal definition of research is perfect. I find it logical that the federal government would struggle to define research given that researchers themselves cannot often agree upon an acceptable definition of research. It is encouraging that revisions of the definition of research have been made/accepted with public opinion. I believe that the field of education needs to define appropriate educational research practices(scientific or not)and then continue to lobby for their inclusion within educational policy. Bruce
ReplyDeleteHow do giant private philanthropic organizations that fund educational programs (and subsequent assessment and evaluation) like Wallace Foundation play into this picture? I can tell you that every Wallace Foundation report or similar that I've ever read (and for full disclosure I've only read thirty or so...I'm no expert) included mixed methodology with a tendency towards the qualitative....I'm just wondering how much of a role private money can influence educational research culture...
DeleteOk, just like Kierstyn I apologize upfront...
ReplyDeleteI actually read this article when it was released in 2003. It was interesting reading it now, seeing how differently I feel about the topic. The first time I read for information, to understand the process that was changing my profession. This time I read with a professional eye, as a potential researcher whose future practice would be shaped by this content.
When I originally read this article, I looked at the 'big picture' message; educational practices should be grounded in research. I not only agreed with the message but was very excited that it was now policy. Since the establishment of education, students have been treated like lab rats with little thought of long-term affects. Trends in education ebbed and flowed and, especially in the early years of American education, were centered around adults and not students. Eventually, teachers, administrators and even parents began to question the trends and ask 'why are we doing this?' (The accountability movement started earlier than you think.)
So, the task at hand became answer the 'Why?' and shaping how future decisions could garauntee positive outcomes for students. This was a monumental task. There were commissions, panels, experts and lots of talk, talk, talk! We had to start somewhere knowing that the end result would never be universally accepted.
Now we have definitions which restrict the profession of educational research and, as a result, education itself. We should be able to return to the table, continue discussions and adjust the outcomes of the report. Just abiding to our own principle that research is fluid and shaped over time is important to consider here. We no longer live in an era that something written down is final. As a potential researcher I wonder about how to get this conversation started? (The government never changes what they espouse to be true, right?)
Sadly, one must only look at NCLB, 2001 as a whole to discover education in America is being held hostage because politicians refuse to do what I suggested, make changes and update the law that already exists. Rather than go back to the table and make adjustments they are doing twice as much work passing new regulations to simply go around the law.This leaves the discussion on "scientifically-based research' on the books, standing all alone, not being addressed. What can educational researchers do to get it back on the table?
Disclaimer: This post is in no way a political diatribe. The issues facing NCLB reauthorization are bi-partisan. The bar was clearly set at 100% performance achievement and neither side is willing to suggest that this idea was not attainable. Unfortunately, the legislation which binds the profession of educational research is being held hostage as a result.
This comment has been removed by the author.
ReplyDeleteI am writing at night and this is not when I am able to complete my best work since I am usually up at 5 am…...
ReplyDeleteIn reading this article, I found the standards set forth by the ESRA (2002) to have enough variability to be acceptable for publically funded research. (A great deal of the research completed within my area of interest has been privately funded for the last 30 years.) The process of educational research should not be based solely on one study, but rather the collective body of knowledge and these standards appear to take into consideration various ways knowledge can be obtained. So, I got curious as to what the WWC says currently. I looked up the WWC and the latest version - 3.0 which was accepting public comment through April and the final standards will be adopted in June. I found some interesting information that the WWC contains that I never knew existed (I might be the only one here). Anyway, the WWC on page 8 that a study will be inelgible for WWC review if, “The study does not have an eligible design. The WWC prioritizes findings from studies of effectiveness that use a randomized controlled trial or quasi-experimental design. Studies that use a regression discontinuity design or single-case design may be reviewed against pilot design standards and described in reports. However, the findings from single-case designs and regression discontinuity designs are not included in the evidence findings reported in intervention reports. Studies using other study designs are not eligible for review.” The WWC also has Practice Guides that are intended to give teachers practical recommendations for the classroom based on the research compiled by the WWC. Given this information, it seems that despite what the standards say, when put into practice and with what information makes it to teachers, the “white lab coat” connotation of research is still alive and thriving….or am I wrong? On page 6, they list out what words they use for their literature reviews and the list of words they use also support quantitative research designs though not explicitly stated. Finally, insurmountable– no. Challenging– yes.
I agree, Alison, from an outsider's perspective, the "white lab coat" connotation is alive and thriving as I hypothesize it should be, given our cultural connotation of the word "science." Nothing less than labcoat would make sense in our society. Good to know about your field's private funding...I've been wondering about the impact of private money and would love to hear how that influences things...
DeleteIn the field of Oral Deaf education, private funding sources have come primarily from one wealthy family who had a deaf child and wanted her to learn to talk. From their initial investments a huge Foundation was formed as well as private Option schools for children to learn to listen and talk. So, in my observations, the funding resulted from a family being personally affected by hearing loss. I wonder if this is true of other private sources of funding. Looking through the WWC, the studies sponsored by this family and the Oral deaf movement were not to be found. However, some very well known computer programs (which were very well funded) were listed as What Works....the difficulty is that the outcomes for HI kids has not changed in relation to reading ability in 40 years. The well funded studies rely on the mechanics of reading and not the basis of language necessary for comprehension....So, to respond to your question, I think the private funding fills in the gaps where the public funding would not dare to go. Just a thought.....
DeleteYes...my experience with private educational funding is based on research and funding regarding system-based afterschool programming. Wallace, Moss, National League of Cities, Forum for Youth Investment (to name a few) are private national funders who have decided how they want education to look and then fund programs...and pay for assessment and eval of those programs in an effort to set up a list of "best practices." To my knowledge they don't have the same personal connection as your field's story--most of these others came from the same ilk as the Carnegie Foundation...guilt money from the people who got rich during the Industrial Revolution (? I'm not sure that's 100% reasonable, but it's what I'm going to write today). They tend to do things like "fight poverty" and "feed children"--all good things...I'm a huge Wallace fan. CF Moss Foundation is highly connected to the government--it helps distribute funds for 21st century community learning centers from NCLB, but the others don't really have much of a government connection. The assessments don't seem to be affected by this pro-quantitative movement much--but then there are definite differences between "assessment" and "research." I agree with you...based on what I've seen, private orgs tend to go rogue, filling in huge gaps in government coverage.
DeleteLaura - thanks for a great discussion. I am gleaning more from these discussions - and even more when I go back and read them again! These discussions about "What is Educational Research" really make me question the concept of Altruism in research. Thoughts?
DeleteDiane here . . .
DeleteWeighing in on the topic, I'm not sure that there is no one right answer about what is educational research. Too many organizations with too many different definitions makes me crazy. I care more about what works and less about funding (although funding is important), and one problem I see with evidence-based practices is that people who implement them don't always do so correctly. I'm seeing more of a throwback to teachers being autonomous and not working together.
So, can we align definitions and methods and funding?
To play devil's advocate Diane...How do we know what works when we can't agree on a process that establishes what works?
DeleteLike Kurt said about the purposes of education, how can we come to a consensus about what scientifically-based educational research entails? Is it possible?
ReplyDeleteLike our four-quadrant chart from last class made apparent, educational research is a VERY complex enterprise in making definitive scientific claims. More than many other disciplines, education intersects with a multitude of factors that impose upon research considerations and analysis'.
Pema Chodron, a notable Buddhist philosopher, speaks of the need for humans to cling to certainty. Is this a good thing? We crave definitive answers and outcomes to tie up loose ends. Often, life does not allow for such happy endings.
In the end, the less one is attached to certainty to one's life, the more open one is to the journey life allows...
Is such a viewpoint insurmountable? Of course not!
I wanted to add to my ramble from last night. In my view, the BIGGEST obstacle to realizing a vision of scientifically-based educational research is: what is considered 'science,' and as a follow-up, who is deciding that, and, further, how & why are they proposing such a view of science? Educational research finds itself in a context that includes politics, business/corporate interests, philosophical differences, socio-economic considerations, etc.
ReplyDelete'Science' may be crafted to promote many ends that may not necessarily be in the best interests for our democracy, students, teachers, educational leaders, communities, and families.
The current view of science in educational research, highlighted in this article, is definitely conducive to political and corporate interests. No doubt a more multi-faceted and open view should be advocated for...
I couldn't agree more Shawn. The issue is trying to standardize the big 'pile of mush' that is called educational research. Structure can be a damaging thing when you are striving to explore the possibilities.
DeleteOur culturally accepted definition of what “scientific” research entails is very “hard” science oriented, and I think the implementation of NCLB in the 10 years since this article was written highlight this. Although there may be a general recognition for the need of different research methodologies in the educational community, it doesn’t seem that public policy has embraced this…just look at the data-driven, standards-oriented schooling on which our country is now focused.
ReplyDeleteIn class we’ve discussed the somewhat unique characteristics of education when it comes to its “soft” and “applied” nature. In my opinion, it is this duality which requires, even demands, that educational research be approached from several perspectives, with the particular research question(s) driving the research methodology. If we, as educational researchers, want this to change, I agree with Carol that we have to be more actively involved in public policy. ~Meg
The Eisenhart and Towne article really helped my understanding of how the intermingling of politics and education has changed, effected, educational research and more in particular the funding for educational research. I do believe that educational research can be scientific and like Eisnehart and Towne believe that finding "constructive ways to encourage and extend the debate and diversity that the current situation highlights" (p.32)is the best approach. I also believe "that the ESRA definitions are more inclusive of various research designs and more sensitive to the realities of research in practice than those in the original Castle Bill" (p.35). I believe this was caused by researchers and the public commenting and constructively adding with appropriate voice (The SRE) concerns, comments and questions to the original bill.
ReplyDeleteHere lies my concerns, questions, and possible obstacles however. First, The What Works Clearinghouse. This seems to me to be an obstacle because it was awarded a grant in 2001 before the researchers and public voice was integrated into the ESRA and even after receiving comments "many are likely to be disappointed that the changes anticipated by WWC are not more extensive" (p.36). I believe (and have witnessed in my school division) that good instructional and intervention practices may not be supported because WWC has not cleared them, so in the end who had more control over the use of the research and hence, the support of further research the ESRA, the Schools, the researchers, or WWC?
My second great concern or obstacle is who from education practitioners or education research has the time and energy to kkep their finger on the pulse of things like the Castle Bill in Washington, D.C. to ensure the voice of some of the stake holders is heard and valued within our political system. I know we all vote for representative, senators, presidents, and organizations like National Association of Gifted children and ASCD have lobbyists working hard for us, but who is going to be on staff and help focus some of the Washington thinking on true "scientific-based research" in educational practices. There are days that I want to apply to be a congressional aid to the "right" person to start circulating different voices through the offices of congressmen and women. I think the description in the article of how Robert Sweet used research institutions, websites, and researchers (p.32) and the idea that the Castle Bill originally initiated made me even more aware of the question who advises our decision makers about a field they may have no more experience in than the average graduate on the street? Shouldn't we be trying to ensure that is an experienced educator or educational researcher? And how do we encourage this career path or sabbaticals?
precisely what I was thinking...thanks!
DeleteDiane here . . .
ReplyDeleteLast semester, I worked on a project about including evidence-based research in teacher prep programs, actually special ed teacher prep programs. However, I could argue that the same needs to happen in general ed as well. So, as I was reading the article, everything came back to me.
Here's what the problem is with the WWC, CEC, APA, and the Promising Practices Network (PNP). Each has its own set of standards for what constitutes good research (Freeman & Sugai, 2013). Therefore, bringing these all together in a way that makes sense might be challenging and difficult. And in the Eisenhart and Towne article, the definitions of scientifically based research is similar, but not the same, across several different federal policies (Castle Bill, NRC, NCLB, ESRA).
As far as answering the questions posed, I agree that they posed the issue fairly and objectively based on my previous research. Each group has its own perspective, and bringing them together into a consistent whole may be challenging. I think we need a common definition of what scientific research is in education. We know what it looks like in the physical sciences (Chemistry, Physics, etc.), but is that model translatable to education where people are involved? Simply implementing an intervention without consideration for the human being is unethical (at least, in my view) because our feelings are what makes us different from other forms of life (at least, in part). Not to mention our heritage, culture, languages, etc. So, scientifically based research needs to take into account the human aspect.
I'm not sure if I digress too much in this post, but having thought a lot about scientifically based research (or evidence-based practices) in education over the last 4 months, I believe once we align the standards for what constitutes scientific research and teach teachers how to be informed consumers of research, we'll be headed in a good direction. I keep going back to business and the value of processes and procedures that enable people to complete their assigned tasks efficiently. Education seems to me to be trial and error -- what works for one teacher may not for another due to personality differences, student differences, etc. Yet, we rely on the experience of others to learn how to be teachers. What if educational practices were designed following business practices? But that would require teachers to give up their autonomy and actually learn to work together as partners. So, what works in one setting for one teacher with one student may need to be modified to work with another teacher in another setting and different students.
I understand that time is a critical issue for teachers. But if teachers are taught to be responsible, informed consumers of research, they will be able to implement action research with very little difficulty. For example, I've worked on a project where we teach preservice teachers about self-determination theory and models for implementation. At the beginning of the module, noone has any time to work on self-determination skills with their students. However, a week later, at the end of the module, those same teachers now have the skills to support and facilitate self-determination with their students, and what's more important, actually believe they can within the course of a regular school day and curriculum. So, why does everyone say teachers don't have time for research? If we teach them how, shouldn't they be able to do so in the course of a regular day?
Irina
ReplyDeleteI think the changes in policy are representative of much more than what a group is trying to advocate for, as Eisenheart and Towne were suggesting. I believe they illustrate the Zeitgeist of current educational research. This is just a hunch, I haven’t had time to dig in and gather evidence but I think that when H.R. 4875 of 2000 was passed, the scientific community was debating the standards of qualitative research, and this was reflected in the legislation that stated only what most researchers agreed on, which was sound methodology for quantitative studies, but no further guidelines. The main issue of the time was that quantitative research already had acknowledged standards, but qualitative research, although accepted as a line of research, lacked consensus when it came to picking sound quality standards.
Later on, when consensus was reached on what constitutes acceptable qualitative research, the legislation changed to reflect that in the subsequent authorization of NCLB of 2002. This might very well have been the effect of a group advocating for certain standards, but I think it was more linked to the overall research community perspective on outlining sound research.
Now IDEA is on the roll to be reauthorized (it has been for a few years, actually), so we might see a new development in the definitions of acceptable research when IDEA lays down its evidence-based mandates.
I think the obstacles in a common vision on scientifically based research lie only in helping the members of the scientific community come to an agreement on what they see as sound research, and it seems that they are slowly, but surely, lying the foundation of standards for future research.
Eisenhart and Towne traces how lawmakers attempted to define education research. The need to justify spending public money, and get the most bang for their buck, which is what they are aiming for. However, as the education research community points out that using a positivist philosophy may not be the answer to the problems. Our community is growing more and more aware of the diversity that exists within our community, and we cannot have a one size fits all answer to education (or for that matter anything).
ReplyDeleteAs I read the article, it was very interesting the way different committees defined qualitative and quantitate methods. Most of them were clear on quantitative methods but when it came to defining qualitative methods, it was unclear, and maybe even got murky. As they rightly pointed out that, the research question should point to the research design and the method. I feel that is the maybe the only way to get to the ‘truth’ (t small) as we are working with a soft applied science (Labaree).
I am still worried about education research being more quantitative driven and as investigators we need federal funding (it is the necessary evil). We find ourselves more constrained but the requirements that we have to meet. However, being an optimist that I am, I feel there is still scope for change. Let’s face it, qualitative methods are not always clearly defined, those who do have not have a clear knowledge about the consequences or the process sometimes question their applicability, and lawmakers may be apprehensive about providing funding for those projects. However, as educational researchers we know that qualitative research help answer essential and important questions that cannot be answered using qualitative methods. Moreover, once we can gradually flesh out the idea better, frame qualitative design as a more concrete process, and communicate it better to our lawmakers. They too will realize the benefit. We as researchers need to be careful not to be driven by methodologies, but by the questions that we are asking. It can easily become our obstacle, as we let our methods define how we answer research question. I can very easily say I am a qualitative thinker and will only do qualitative research. But that is not what I have to do as an educational researcher I have to work on answering questions adhering to a scientific process. In the end qualitative, quantitative, quasi experimental or single subject design, will be scientifically-based ed. research as long as they are the best means to answer the research question.
P.s I used Beckers advice and just wrote this chunk. It felt nice.
Jenny here...The issue I have with the Federal government specifying what is acceptable research is that not every research project should be conducted as a scientific experiment that yields quantitative results. The method used to conduct research should be the one that best aligns with the research question. In some cases, qualitative research would be best to use; however, this is not viewed as "favorably" by the federal government as quantitative research. Although the article does state that there were provisions made to include qualitative research, I think the overall consensus of the federal government is that quantitative research is best.
ReplyDeleteI do think these obstacles can be surmountable if scholars continue to voice their objections. I do have to admit, however, that this issue over which method is the best has impacted my current goals in grad. school. I personally find qualitative research more interesting and a better fit for me as a researcher. However, since quantitative research is more highly regarded by the government and since my goal is to ultimately work for the government, I have decided to focus more on quantitative research since it is most likely that my future job will be in this arena.