Thursday 14 November 2013

GCTLT - Evaluation Plan


Evaluation

Link to development and implementation plan

I intend to have informal discussions with my colleagues in the department (College of Enterprise and Development), particularly those that teach on the project based learning papers.  I would like to find out from them whether they use 'cooperative learning' as their model, 'active learning', 'collaborative learning' style or 'problem based learning' (Prince, 2004) and compare my style of teaching with their format.
I have already had discussions regarding whether they mark the group work individually or as a group. My plan is to encourage cooperation but once roles and work have been assigned then the final assessment will be marked individually, evidence based on the blog summaries.

Aim:

  •  To conduct a formative evaluation in the form of interviews to look at the design of a mixed media paper in the Diploma of Travel and Tourism for delivery to students synchronously.
  • To prepare a set of questions for a focus group of industry professionals using their observation and feedback on the performance of student interns (summative)

Goals:

  •  To measure whether the team building instruction and exercises used in the paper are likely to support cooperative learning (formative)
  • To collect feedback about the design of the paper (formative)
  • To establish whether the content is relevant to travel and tourism learners (formative)
  • To measure the relevance of this design in preparing learners for the Tourism Industry (Summative)
I plan to  use a mix of qualitative/quantitative approaches using an interview process as formative feedback and industry focus groups as part of my summative evaluation.  The reason for this is the over-all design is using an "interactive learning system(s) for education and training"  where the rationale is that in order to get constructive feedback, this approach " emphasises the utilisation of information by a wide range of people to improve decision making".(Reeves & Hedberg, 2003, p.3)
The groups I will focus on are fellow lecturers, students and Industry professionals.

Reeves and Hedberg (2003) say "we define Interactive learning specifically as a process involving some form of digital mediation between a teacher or designer and a learner".  This approach to me would make more sense than gathering figures to indicate the effectiveness of the design, however I will use this as one form of evaluation.  
"The 'qualitative' aspect represents the emphasis on the human being as the primary evaluation instrument, rejecting the mathematical modeling of phenomena upon which the quantitative paradigm depends to heavily".

Questions for interviews with three lecturers( One on-line expert, one wiki-educator expert and one with limited technology use) (formative).
Big picture question;

A. How effective is the design of a mixed media paper in The Diploma of Travel and Tourism, for encouraging cooperative learning for students?

  1. Do the learning outcomes match the course design?
  2. Effective team work is an essential part of  being involved in the Travel and Tourism Industry.  Do you think the 'cooperative learning' approach is the right method for this paper ?  Other approaches include 'active learning' 'collaborative learning'  and 'problem based learning'. Link to definitions of learning
  3. Do you think a blog is the most effective forum for students to summarise and reflect on their work?  What other media do you feel would be more appropriate?
  4. The idea is to change the grade from a pass mark to a percentage grade and to mark the assessment individually.  What do you see as the advantages and disadvantages of this approach?
  5. The assessment is based on the completion of a magazine as an e-book with articles written by individual students.  In your opinion, is this an effective way of assessing students?
Students at the beginning of the course will be given a series of open-ended questions to answer once they have set up the blog and had an overview of the paper.  These can be answered in email format to me; (formative).  These questions are based on the Canadian Education Association (2013) "When students believe that you have a "correct answer" in mind, they are slow to respond.  On the other hand, a true open-ended question sincerely invites authentic reflection and discussion."  Based on Bloom's Taxonomy, " asking an open-ended question is a way to elicit discussion, brainstorm solutions to a problem, or create opportunities for thinking outside the box". (Canada Education, 2013).
  1. What are your first impressions of the design for this paper?
  2. How was your experience with setting up the blog and interacting with your team?
  3. Do you enjoy participating in the team activities? If not, why not?
  4. How do you think you can effectively contribute to the team's success?
  5. Do you understand the requirements of the paper?
Methods: (How to gather and analyse data)
I have always found the course/lecturer/WES evaluations at Otago Polytechnic a very effective way of analysing data.  The method used is a Likert Scale, its easy to read and sorts the information into simple, straightforward data.  This type of rating scale is measuring a person's attitude.  Another possibility here would be to ask the questions to the students as a series of statements about the design of the course and measure the responses through a likert scale "in terms of the extent to which they agree with them, and so tapping into the cognitive and affective components of attitudes".( McLeod, 2008).  Example of Likert Scale:


Likert scale displaying feedback on quality of facilitator

This way the responses can be measured as a linear scale on a continuum from strongly agree to strongly disagree, these results would be quantitative, timely and feedback to the students would be relevant within a reasonable period.

Industry specialists: (Summative)At the end of the year the students complete a period of 80 hours in the industry, explore the organisation they are working with and write a report.  Part of the requirement is that the students receive feedback from the employer about their progress while on work experience.  I have included in this feedback form some additional questions which ask specific questions regarding the students ability to work effectively in a team environment.
Example of questions for employer feedback :
  1. Did the student work effectively in the team environment? If not, why not?
  2. What particular skills did the student display? e.g. initiative, social skills, awareness of others, conflict management, assertiveness, problem solving?
  3. Did the student communicate effectively
 Decisions: (what to do with the information)
Lecturer feedback:  I will use this in the form of recommendations from my colleagues and will make on-going improvements periodically.
Student feedback:  The students will email (or blog?) me their answers, my feedback could be individually via email, on the blog or as a class discussion, dependent on the response.  The other approach with the likert scale would be more formal and processed through the department that sends out course evaluations.  This way the students will have two forms of formative evaluation throughout the course and one summative at the end, (student/peer/course evaluation).
Industry specialists feedback: This will be collected, along with the student evaluations at the end of the paper and will inform my planning and preparation for the paper for 2014.

Example of Design matrix

Design matrix for Destination Diploma paper



References
Canada Education, (2013).  Engaging students through effective questions. Retrieved from http://www.cea-ace.ca/education-canada/article/engaging-students-through-effective-questions

McLeod, S. (2008) Likert Scale. Retrieved from
http://www.simplypsychology.org/likert-scale.html

Prince, M. (2004). Does Active Learning Work? A Review of the Research. Retrieved from http://www4.ncsu.edu/unity/lockers/users/f/felder/public/Papers/Prince_AL.pdf

 Reeves, T. C. & Hedberg. J. G. (2003). Interactive Learning Systems Evaluation. Englewood Cliffs: Educational Technology Publications.



1 comment:

  1. A thorough evaluation plan with a mix of methods for collecting data and a variety of target audiences who will participate. You should get some very valuable information on which to make your decisions - a great list by the way. Fabulous to see so many references which indicates that you have read widely and taken a critical analytical approach to your planning. great work. :)

    My only suggestion is that you decide on some 'big picture' evaluation questions - these will link to your aims (they are like research questions). For example: For the lecturers it might be: How effective is the design of a mixed media paper in the Diploma of Travel and Tourism for encourage cooperative learning for students? Then the specific questions you have listed fit under this.

    Do the same for each group of participants.

    ReplyDelete