EVALUATION II: COMPARISON AND IDENTIFYING NEEDS

[Note:  Before getting started with this posting, I wish to express condolences to the survivors and families of the victims in the Marjorie Stoneman Douglas High School shooting.  Since this blog is dedicated to civics education, I would like to convey my admiration of that school’s social studies teachers; their students are expressing, in an articulate way, a level of civic knowledge beyond their years.  Hopefully, our nation can arrive at a workable solution to this problem of repeated mass shootings.]

The current topic of this blog is evaluation.  The aim has been to review a list of functions an evaluation should meet to be effective in a change effort of a school’s curriculum.  To this point, the last posting commented on the functions:  diagnosis and revision.  These functions are offered by the curriculum scholar, Elliot W. Eisner[1] and also includes comparison, anticipation of needs, and determination of achievement.

          This posting will begin by focusing on comparison.  When one thinks about evaluation, one should realize that all evaluations are of one form of comparison or another.  There are those comparisons against objective standards.  For example, if one evaluates a new car, very objectively, he/she could determine that it should last five years without having to be subject to major repairs.  That can be stated because of the state of mechanical advancements that automobile technology has achieved and by comparing how other people’s cars hold up.

          Of course, most efforts at improving school curriculum is not that kind of thing.  For example, say a school opts to change its math curriculum.  It implements a newer approach to teaching geometry.  The school buys textbooks that incorporate a more practical method of instruction with students solving common problems in which geometry theorems are utilized.  How does one evaluate that effort?  There’s no comparable numbers as there is in the car example.

          “Many, but not all, policy changes in schools are based on certain values that the [practical geometry] policy is intended to realize.”[2]  The value here is that a practical application with everyday problems is better in that students can more readily understand the importance of the material – it’s a value for less abstract instruction.  Evaluation would be to see if the belief, upon which that value is based, is true.  But can that determination be made within a school year; how about two school years?  How about a longer amount of time?  One can begin to see the problem when it comes to educational changes in curriculum.

          Here there is an inability to impose such objective standards as with the automobile.  They are subjective most of the time.  Assuming one has a fairly good idea of what is being sought, whether one attains that aim can be often hard to ascertain.  And, assuming further that one at least has a workable sense of that question, how generalizable is it?  It works today, will it work tomorrow?  Will it work in another location, with another set of people, or under another set of supervisors?  These are but some of the questions one will inevitably have to ask in real life situations.

          And to top it off – and this happens often – yes, it works to some extent, but it leaves one with other problems.  If the change is instituted, are the students better off if the school had just kept what was?  So, there are comparisons in terms of time, people, and states of being.  And much of it is attempted with less than optimal information or over a concern that does not lend itself to definite and discreet information.  Yet, change is needed and all one can do is to try to do their best.

          Again, the previously stated standards, this model identifies, centers on effectiveness.  Effectiveness is defined in terms of student conduct and is measured by the levels students can:

·        demonstrate learning curricular content;

·        demonstrate learning skills in acquiring relevant knowledge associated with curricular content;

·        demonstrate dispositional outlook supportive of being a productive member of the student body;

·        perform their student roles in a civil manner;

·        and follow, in a collaborative fashion, those behaviors that abide by the reasonable policies of the school and school system.

Yes, in a real school, with real problems, these standards need to be more nuanced and more concrete.  For example, what does “learning” mean?  How will learning be compared?  By test scores, demonstrated applicability, transferability, or some other demand?  All that depends on what the conditions of a school are, what is the nature of the curricular content, and what are the needs of students in question.

The next function identified by Eisner is identifying educational needs.  This posting will not expand on this function since this blog in a previous posting conveyed most of what Eisner describes when the blog reviewed the problem identification phase.  See “Initial Concerns of the Landscape,” February 2, 2018.  What should be added here is Eisner’s concern for biased evaluations; a concern already mentioned as the Hawthorn Effect.  He summarizes this obstacle with the following:  “… we find what we are looking for.”[3]

The next posting will describe the last of Eisner’s functions, determining what needs to be achieved.  The writer also wants to add an idea associated with problem identification, the first of the phases of this blog’s proposed model.

---

[1] Elliot W. Eisner, The Educational Imagination:  On the Design and Evaluation of School Programs (New York, NY:  MacMillan Publishing Company, 1985).

[2] Ibid., 196.

[3] Ibid., 198.

EVALUATION I: DIAGNOSIS AND REVISION

This blog is currently summarizing the factors that affect the processes of organizational change.  It is offering a model of change that is made up of phases, as opposed to steps.  To date, the phases problem identification, staffing, “unfreezing,” rule making, information gathering, negotiating, conflict ameliorating, and testing have been described and explained.  This posting looks at evaluating.  This will be the first of a set of postings dedicated to evaluation.

Evaluation:  To understand the role evaluation plays in a viable change strategy, one needs to take a broader view of it than what it is usually given.  The aim here is more than giving a grade or something comparable to what is going on.  It is more than determining whether things are going according to plan.  Yes, these concerns are present, but there is more.

Borrowing from the ideas expressed by Elliot W. Eisner concerning evaluation, the following functions of evaluation are drawn:  diagnosis, revision, comparison, anticipation of needs, and determination of achievement.[1]  His application of these functions are mostly aimed at the evaluation of student performance. 

Here, Eisner’s ideas are modified to evaluate a change effort, both in terms of determining values associated with the change before, during, and after the effort is conducted, and also other related concerns which are delineated below and in the following postings.

A word of context:  as it is stated above, the term phase is used to indicate that the various types of activities comprising change efforts are not cookbook steps.  Each phase is broad, and not meant to designate discreet “steps.”  They are actions that have a less designated character.  That includes what will be described as evaluation.

For example, the first Eisner function of evaluation – diagnosis – can well be considered an aspect of problem identification, the first phase, and information gathering, the fifth phase identified in this model.  The diagnoses under evaluation, though, is aimed at acquiring that information that assist subjects at arriving at its desired – previously identified – results.

Eisner indicates that he borrows the term diagnosis from the medical field but points out that one should not expect that this form of analysis has the tests or technological gadgetry that the medical profession has at its disposal.  In education – and one can add in most organizational settings – the act of diagnosing is more an art than a science or an engineering practice.  In addition, at this phase, the review of the school’s effectiveness has been completed and has led to the change effort in question.  What is now being diagnosed is the progression of a change process.

Specifically, one is trying to determine where and what are the problems that are popping up as the testing protocol is being conducted.  These can be of two sources which have already been cited in this blog:  the social realm and the technological realm.  Of course, problems can have both sources at once.

This type of analysis needs to be constant throughout the process.  Solutions or treatments must be devised.  This in turn might cause one to do more research through reviewing the appropriate literature, interviewing appropriate experts, and/or talking to the subjects.  At times, questioning subjects need not happen in that they are more than willing to let an agent know what the problem(s) is/are.

The agent is looking for “bridges;” that is, those bits of information or changes in the plan that can get the subjects from where they are to where, for the sake of the change, need to be.  As much as possible, the solutions or treatments need to be tailored to the personalities, dispositions, other value orientations of the subjects or to the capacities of the technology being used. 

While posing its own challenges in trying to maintain the integrity of the change, individualization – catering to the biases of those involved – should be sought to the extent possible.  Here the concerns can range from personal problems subjects bring to the work place to more professional commitments the subjects harbor.

Accordingly, the information contained in Topic II of this book – the philosophies, the approaches, the psychological and sociological beliefs of the subjects – come in handy.  That information can serve as a starting point by which to understand where the subjects are mentally as the change effort begins and as it progresses.[2]

If one thinks that this sort of diagnosis should be part of the information gathering phase or even the problem identification phase; this writer does not necessarily disagree.  The writer’s reluctance of using the term, diagnosis, in the earlier phases is its connotation of trying to find problems.  Information gathering and problem identification phases should be more objective in its approach.  Yes, that goes for problem identification.  Those phases should be able to find there are no significant problems to “fix” or even to address.

Here, evaluation presupposes a “measuring” outlook.  That is, how well are things going as the change effort unfolds?  After all, one does not go to a doctor for a diagnosis unless one suspects something is wrong.  A prudent approach assumes problems exist with the change effort – the track record of such efforts would justify such an assumption.

The next function identified by Eisner is revision.  This function is usually referred to in the education literature as formative evaluation.  Its aim is to improve an educational effort and, as oppose to diagnosis, the emphasis is on changing the change process, not garnering the information, per se. 

Its process is to insert review points and questions during development and implementation of an educational effort to determine whether there is progress toward the stated objectives the effort has pursued.  Therefore, it is not a day-to-day revision, but one inserted at key points.  Everything associated with the effort is fair game; that includes reviewing the aims and goals of the effort itself.

One concern of such a review is to detect any Hawthorn effect.  That is, an agent should place heighten interest to see if those who are engaged in the change, due to their probable increased buy-in, might be prone to overlook problems.  People who exert energy in trying to accomplish what they see as important, tend to perceive its implementation through “rosy” glasses.  Hence, they become poor evaluators.  Being on the lookout for such a bias can help in meeting this obstacle to conducting an objective evaluation.

Obviously, formative evaluation is highly dependent on conversations between and among agents and subjects.  Subjects in a curricular change is not just the involved teachers but affected students, other teachers, administrators, parents, in some cases, the custodial staff, etc.  All affected parties – directly and indirectly – need to be heard.  They will each have a reaction and, given their strategic role, can play a determining part in the effort’s success.

One-point Eisner makes is that, contrary to the way earlier curricular change efforts viewed teachers, they are not interchangeable elements – they are not mere “tubes.”  “Teachers cannot and should not be bypassed.  Materials guarantee nothing.  What they do is to expand the range of resources with which teachers and students can work.  They are not a substitute for teachers.”[3]

In this regard, a realistic view of student and parent performance need to be kept in mind.  If they are not behaving, as Eisner phrases it, within “conventional assumptions,” then customary curricular effort will not be successful.  In turn, such a situation indicates what needs to be addressed and changed; i.e., this very dysfunctional behaviors on the part of students and/or parents.  Some might consider this a non-curricular issue.

This is a good definitional concern.  It is possible that certain curricular changes could address extreme student and parental attitudes and behaviors that can be determined to be counterproductive.  But if not, this calls for qualitatively, differing approach.  Frankly, such an extreme case needs its own theoretical treatment beyond the purposes of this book.

To return to the current concern, revision is what it is.  This level or aspect of evaluation is essential and, as with many of the other aspects of change, deserves focused attention.  The agent should spend meaningful levels of time and interest in the questions that are asked and the information that those questions generate to make those revisions in the change strategy. 

It is this writer’s experience that informs him that an agent is not given too many opportunities to implement change, so he/she needs to strive for success when the opportunity is there.  He/she should not experience failure due to the lack of a timely revision.

---

[1] Elliot W. Eisner, The Educational Imagination:  On the Design and Evaluation of School Programs (New York, NY:  MacMillan Publishing Company, 1985).

 [2] This writer estimates that in a given school, about 80% are essentialists/behaviorist, just under 20% are progressives/humanists, 1 or 2 Perennialist/academics, and, if any, a reconstructionist/reconceptualist.

[3] Elliot W. Eisner, The Educational Imagination:  On the Design and Evaluation of School Programs, 195.