Education 3604 Professional Semester 2 Spring, 2003

Evaluation

Instructor:	Robert Runté	Office: B-850	Phone: 329-2454
E-mail:	Runte@uleth.ca		Fax: 329-2252
Secretary:	Barb Krushel	Office: B-868	Phone: 329-2260
Web Pages:	Supplementary: http://www.edu.uleth.ca/~runte/tests/ Instructor pages: http://www.edu.uleth.ca/~runte/
Schedule :	Thursdays 1:00 – 4:00 PM Room A844

Course Goals

By the end of the course, students will be able to:

W list and apply basic principles of systematic evaluation

W develop their own reasoned position and policies on ethical issues related to evaluation

W Demonstrate an awareness that assessment is integral to instruction and learning by developing tables of specifications as part of unit planning.

W Demonstrate test construction skills through practical, real-world application.

W Select, develop, and use paper and pencil evaluation strategies that are valid and reliable.

W Gather, organize, analyze, interpret and communicate test data to students, parents, and relevant administrators in a manner that will promote further learning (through helping students to assess their own progress and set appropriate goals, through improving follow-up instruction, and through program evaluation).

W Compute and interpret an item analysis of a test they have developed and administered in a real-world situation.

W Develop and use in a real-world situation a self-assessment instrument to further their own professional development.

Course Rationale

The evaluation component of Professional Semester I concentrated on assessment within the context of a single lesson. The evaluation module of Professional Semester II will expand on these principles to develop evaluation strategies for entire units. Similarly, as PSI focused on a variety of non-test assessment strategies (e.g., oral questioning, conferencing, checklists, rating scales, portfolios, peer editing, and various performance based assessments) we will now turn our attention to the techniques of test construction, term assignments, and essay grading.

Provincially Mandated KSAs Related to Education 3604

The Minister of Education has established a list of knowledge, skills, and attributes (KSAs) required for Interim Certification as a classroom teacher in Alberta. Graduates may be asked to document that they possess these KSAs, and to this end, the KSAs applicable to the evaluation module of Professional Semester II are listed below:

9. Gather & use information about students’ learning needs & progress.

Teachers monitor students’ actions on an ongoing basis to determine and respond to their learning needs. They use a variety of diagnostic methods that include observing students’ activities, analyzing students’ learning difficulties and strengths, and interpreting the results of assessments and information provided by students, their parents, colleagues and other professionals. Teachers select and develop a variety of classroom assessment strategies and instruments to assess the full range of learning objectives. They differentiate between classroom and large-scale instruments such as provincial achievement tests, administer both and use the results for the ultimate benefit of students. They record, interpret and use the results of their assessments to modify their teaching practices and students’ learning activities. Teachers help students, parents and other educators interpret and understand the results of diagnoses and assessments, and the implications for students. They also help students develop the ability to diagnose their own learning needs and to assess their progress toward learning goals. Teachers use their interpretations of diagnoses and assessments as well as students’ work and results to guide their own professional growth. They assist school councils and members of the community to understand the purposes, meanings, outcomes and implications of assessments.

4. Know there are many approaches to teaching & learning.

Teachers appreciate individual differences and believe all students can learn, albeit at different rates and in different ways. They recognize students’ different learning styles and the different ways they learn, and accommodate these differences in individuals and groups of students, including students with special learning needs. Teachers understand the fluidity of teaching and learning. They constantly monitor the effectiveness and appropriateness of their practices and students’ activities, and change them as needed.

5. Engage in a range of planning activities.

Teachers monitor…their instruction, and monitor and assess students’ learning on an ongoing basis, and modify their plans accordingly. Teachers strive to establish candid, open and ongoing lines of communication with students, parents, colleagues and other professionals, and incorporate information gained into their planning.

6. Create & maintain environments conducive to student learning.

Teachers establish learning environments wherein students feel physically, psychologically, socially and culturally secure. They are respectful of students’ human dignity, and seek to establish a positive professional relationship with students that is characterized by mutual respect, trust and harmony.… Teachers work…to make their classrooms and schools stimulating learning environments. They maintain acceptable levels of student conduct, and use discipline strategies that result in a positive environment conducive to student learning. They work with students to establish classroom routines that enhance and increase students’ involvement in meaningful learning activities.

Topic Outline

I. Introduction:

A review of ethical principles, and an introduction to the Alberta Diploma Examinations, Achievement Tests, Diagnostic Kits, and CAMP materials

II. Assessing Student Achievement with Teacher-Made Tests

A. Planning the Classroom Test

1. Constructing a Table of Specifications (designing unit objectives based on the Alberta Curriculum and Bloom’s Taxonomy)

2. Choosing the appropriate type of test

B. Objective Type Tests

(includes True/False, Matching, Multiple-Choice, Short Answer/Completion, and Interlinations)

1. Advantages and Disadvantages

2. Basic Item Writing (knowledge items)

3. Advanced Techniques (assessing higher thinking skills)

C. Essay Style Tests

1. Advantages and Disadvantages

2. Item Writing

3. Essay Scoring (Analytical, Primary Trait, and Holistic approaches)

4. How to Increase Objectivity & Reliability

D. Administering the Test

1. Basic Principles

2. Test-Anxiety

3. Cheating

4. Test-Wiseness & Guessing

5. Teaching to the Test

E. Summarizing and Interpreting Test Results

1. Frequency Distributions

2. Measures of Central Tendency (Mean, Median, Mode)

3. Measures of Variability (Range, Standard Deviation)

F. Improving the Test Through Item Analysis

1. Item Analysis and Interpretation

2. Item Banking

III. Assessing Achievement with Standardized Tests

A. Types of Standardized Tests

B. Uses (and Abuses) of Standardized Tests

C. The Alberta Achievement Tests and Diploma Examinations

D. The Alberta Reading Diagnostic and Math Diagnostic Kits (time permitting)

IV. Program Evaluation

A. Using Alberta Achievement Test and Diploma Examination

Results for Program Evaluation

B. Using Classroom Evaluation for Diagnostic Teaching

Textbook

The required textbook for this course is the Ed 3604 Course Reader (available through the University of Lethbridge bookstore) and consists of relevant excerpts from Making the Grade: Evaluating Student Progress and materials written specifically for this course module.

Assignments

There are three major assessments in this course module:
Course Examination	1-3 PM, March 6		30 marks
Unit Test Assignment: Unit Test Specifications Unit Test Blueprint (Mini) Unit Test Analysis of Test Results	Option 1 Due Feb 13 Due April 12 Due April 12 Due April 17	Option 2 Due Feb 13 Due Feb27 Due Feb 27 Due March 6	60 marks 10 marks 5 marks 30 marks 15 marks
Self-Evaluation	Due on or before April 17		10 marks

Practicum based assignments may be emailed to the instructor at Runte@uleth.ca or faxed to 329-2252. Include your own name in the title of any emailed documents, e.g., "FredA.doc". Clearly label all faxes "For Dr. Runte"

Major Practicum Assignment - Option 1

The purpose of the major project is to synthesize what you have learned in your evaluation courses by developing a table of specifications related to a specific unit plan, implementing an appropriate evaluation strategy (i.e., an end-of-unit or mid-unit test), and analyzing the results.

The major project should relate to your subject major and/or the unit you are submitting to your methods instructor. Naturally, your evaluation strategy should be appropriate for the grade, subject, and student group of your practicum placement. For example, it would NOT be appropriate for teachers in primary grades placements to develop a multiple choice test for their students, even though a major component of this course will be devoted to multiple-choice techniques. You may also wish to consult with your methods instructor and Teacher Associate to ensure that you have chosen an evaluation strategy consistent with their needs. (See boxed note, on the next page) Students in primary grade placements should give serious consideration to choosing Option 2.

If two or more students find themselves teaching the same unit in their practicums, they may complete this assignment as a collaborative team, provided everyone contributes equally. (Note that taking credit for an assignment to which you have not contributed constitutes plagiarism.)

Table of Specifications (10 marks)

Construct a table of specifications for a unit you intend to teach during your practicum. Provide a rationale for the choices that you have made.

A table of specifications identifies the content to be taught and tested, its taxonomical level, and the relative importance (% weighting) of each objective.

A chart format is generally most convenient and successful.

(a) Appropriate title/identifying information (1 mark)

(always label everything so you can find it again quickly next time you teach that unit)

(b) Subject matter definition of cells (2 marks)

(i.e., appropriate titles to reflect objectives, and at an appropriate level of specificity)

(i.e., Bloom’s taxonomy or subject specific equivalent, grouped appropriately)

(d) Allocation of emphasis (2 marks)

(i.e., weightings reflect the relative importance of the objectives in the unit)

(e) Rationale for table design

Explain the decisions you made in designing the table this way. How did

you decide which objectives to include, and what weightings to give them?

(Refer to the Curriculum Guide.) Explain why you decided to group Bloom’s

categories this way? Are there aspects of this unit which cannot be tested by

pencil & paper test? What changes, if any, will you have to make in the test

blueprint for a paper and pencil test for this unit? (3 marks)

Subtotal = 10 marks

Major Practicum Assignment - Option 1 (cont.)

Test Blueprint (5 marks)

Map (i.e., match) the questions on your test to the table of specifications to create a test blueprint. (You can either map the blueprint directly onto the table of specifications or create a separate page for the blueprint, whichever is more convenient.)

If your table of specifications includes topics that cannot be assessed through paper & pencil tests, it may be necessary to design a separate test blueprint that relates only to those topics that are amenable to testing.

Questions must be cognitive level claimed for full marks. Subtotal = 5 marks

Test Construction (30 marks)

Construct a test based on your table of specifications/test blueprint.

The test may be an end-of-unit test, a mid-unit test, or a quiz.

(Note that if the quiz is short, you may need to include additional items to demonstrate your mastery of the skills required in this course – see boxed note below.)

Be sure to include an answer key with your submission.

(a) Appropriate use and balance of a variety of question types* (5 marks)

(e.g., variety of multiple choice, matching, written response, and/or

PBA appropriate for subject and grade; questions vary in difficulty

and taxonomic level; questions are grouped in logical fashion;

test increases in difficulty; etc.)

(b) Test items free of construction flaws* (15 marks)

(e.g., questions are clear, concise and well constructed; the vocabulary

is appropriate, questions follow correct format, multiple choice have

plausible alternatives, etc.)

(d) Comprehensive marking key for written response items (5 marks)

(e) Clear designation of marks within key (2 marks)

Subtotal = 30 marks

*NOTE: Since the length and complexity of your examination will be determined by the grade level and subject for which it is intended, you may need to submit additional test items to demonstrate the full range of your capabilities. For example, if you choose a unit that introduces a lot of new terminology, you may feel that the test for your students should concentrate on knowledge recall. That may be acceptable, but you would then need to submit additional samples of higher order learning questions, separate from the test, to receive full marks in this category. Similarly, you may feel the best design for a particular test would be pure multiple-choice, but would then be responsible for submitting additional short answer and essay items to demonstrate your mastery of those forms, and so on.

In some placements, you or your Teacher Associate may feel that any written test is inappropriate for the particular unit you are teaching (e.g., oral presentations). In such cases you may develop a test for a unit you are teaching outside your major. If your practicum affords no opportunities for a written test or quiz, this will be apparent by Feb 6, and you must complete Option 2 for this assignment instead. Option 2 due dates will then apply.

Major Practicum Assignment - Option 1 (cont.)

Test Administration

Administer the test to your students.

There are no marks for this portion of the assignment, pre se, but is obviously part of one's practicum responsiblities and is required for the next part of the assignment.

Analysis of Results (15 marks)

Analyze the results of your test or quiz.

(Student surnames should be stripped from materials submitted in this assignment to preserve confidentiality.)

(a) Summarize the results of your test through the use of a (4 marks)

frequency distribution, graph (histogram, bar graph, or

frequency polygon of the frequency distribution), mean, ,

mode, medium, range and standard deviation.

(b) Perform an item analysis on four consecutive multiple choice (8 marks)

items, indicating the strengths and weaknesses of each item.

Where analysis indicates need for revision, provide appropriate

revisions.

What decisions about the class, unit, your teaching, or your

evaluation techniques have you reached based on your students'

examination results? For example, is the class ready to go on to

the next unit? Have you identified students requiring extra help?

In terms of evaluation, what, if anything, would you do differently next time?

Subtotal = 15 marks

Major Assignment - Option 2

Option 2 for the major assignment uses the same scoring criteria as option 1, but instead of producing a unit test for one's practicum, one develops a test for use in Ed 3604. The table of specifications must be based on the course description, KSAs, and topic outline for this course, but one may assign different weightings than the instructor. The test blueprint and test construction assignments are identical to those in Option 1. Instead of administering the test, however, students choosing this option will be provided with mock data that they are to analyze as if it were the results for their test questions.

The purpose of this option is to (a) provide a viable alternative for those assigned practicums in primary grades or other situations where tests are not considered appropriate; (b) allow students the opportunity to complete the evaluation module prior to beginning their practicum; (c) provide input into the development of the Ed 3604 course examination; and (d) promote mastery of course content by having students learn the material well enough to develop a test on it.

Note that dues dates for this option require that all work be completed before the practicum. No course work may be undertaken in the practicum unless it is fully integrated with practicum duties (as in Option 1).

Course Examination

In keeping with the spirit of this course, the course examination will incorporate a variety of question types: true/false, matching, multiple-choice, short answer & completion, and extended written response. A significant proportion of the test will involve your creating test items based on the skills acquired in this course. The test may include some items originated by students in this or other Education 3604 item-writing workshops. It will be a two hour examination written in class March 6 from 1:00 to 3:00 PM.

Minor Practicum Assignment

Self-Evaluation Instrument (10 marks)

Design a questionnaire to collect feedback on your teaching performance from your students near the end of your practicum. (An anonymous questionnaire is most likely to obtain candid responses.)

Scoring will be based on the following:

(a) Appropriate length (maximum of 15 minutes) and (1 mark)

appropriate vocabulary, reading level, etc. for

grade level and subject

(b) Majority of questions posed are quantifiable (1 mark)

but with some open-ended questions included*

*For Division I (i.e., primary grade) placements,

the mark is for avoiding the use of open-ended items

which require a written response.

regarding your teaching performance: e.g.,

planning, appropriateness of visual aids,

communication skills, relationship with

students, classroom management, general

attitude as students see you. (Again, these

must be appropriate to the student's grade level.)

(d) Appropriate procedures used to administer (1 mark)

the questionnaire (e.g., not immediately

before or after an exam, etc.)

(e) Tally and Analysis of results (3 marks)

(i.e., summary of what they think about you)

(f) Recommendations for future based on this feedback (3 marks)

(i.e., what you think about what they think)

Total = 10 marks

You only need to hand in your analysis of one class set for this assignment, but may wish to use the form you develop with all your classes to get useful feedback before continuing on to PSIII.

Grading

Excellent

Good

Satisfactory*

Poor**

Failing

97 - 100 A+

93 - 96 A

90 - 92 A-

87 - 89 B+

83 - 86 B

80 - 82 B-

77 - 79 C+

73 - 76 C

70 - 72 C-

67 - 69 D+

63 - 66 D

<63 F

*Note that although a "C" represents a passing grade in any particular module, students are required to maintain a 2.5 average in their professional semesters.

**Note also that a "D" is an unsatisfactory grade for your professional semester and will likely lead to your being asked to withdraw from the program.

The Fine Print:

Failure to meet a deadline without the prior consent of the instructor (based on medical or extenuating circumstances) may result in a lower grade for that assignment.

With the exception of the course examination and in-class workshops, students are required to type all assignments.

Please note that attendance is compulsory in this course. If you are going to be absent for a class, you must notify the instructor, or leave a message with the course secretary, Barb Krushel, at 329-2260.

Faculty policy requires that assignments must be picked up from B. Krushel’s office (B868) by Oct 15, 2003. Assignments left after that date will be discarded.

The Very Fine Print: Realizing that no one ever reads all the way through the course outline (even though course outlines generally contain useful information and might even provide a useful role model for planning out one’s own courses — though admittedly you won’t need to acquire that particular skill until PSIII) I thought I would point out here that about half of my speaker notes for this course have been uploaded in their raw, unedited form to the website under construction at http://www.edu.uleth.ca/~runte/tests/ Knowing this may or may not be helpful in dealing with PSII, but I mention it because if you happened to really enjoy this course and were interested in doing more along this line, and also happened to be interested in learning about the new communication technologies (e.g., world wide web), then you might consider undertaking an independent study with me at some time in a future semester in which you essentially took responsibility for developing a portion of the evaluation website. Might look good on a résumé. Just a thought.

Oh, and if you send me an email that contains the phrase “I want fries with that” by January 11th, I’ll give you a 1/2% bonus for reading the course outline thoroughly. (Oh and don’t tip off the others, let them find it on their own.)

Additional References

Etobicoke Board of Education. Making the Grade: Evaluating Student Progress (Scarborough: Prentice-Hall Canada, 1987. 272 pp. ISBN 0-13-547191-5).

W. James Popham, Classroom Assessment: What Teachers Need to Know 3^rd Edition(Boston: Allyn & Bacon, 2001; ISBN 0205333044)

Gilbert Sax, Principles of Educational and Psychological Measurement and Evaluation, 4^th Edition. (Belmont, CA: Wadsworth, 1997; ISBN 053457486)

Principles for Fair Student Assessment Practices for Education in Canada. (Available on-line from http://www.education.ualberta.ca/educ/psych/crame/research.html and as an appendix in the Wilson text below.)

Robert J. Wilson, Assessing Students In Classrooms and Schools. (Scarborough: Allyn & Bacon Canada, 1996.