Assessment 2: Prepare a new or revised assessment strategy

I have modified the assessment strategy for the course I evaluated in assessment one, (see earlier blog Link to existing assessment ) which is the packages assessment for the Certificate in Travel and Tourism.

This new assessment has been designed to allow the student to work through and research a set of questions about hotel packages from different advertising sources/media.  They are then required to pick two hotel packages and describe them, giving reasons why the package may have been created. The assessment tools used are in the form of a word processed document and a speech.  Two marking sheets have been designed, a formative peer evaluation and a summative sheet for their word processed work(assessment methods).
The students have three presentations to do over a 6 month period.  These speeches can be a nerve racking experience for the students, so in order to make the process less daunting I focus on a particular aspect of the presentation for the first two assessments before the final one.  Effectively stair-casing the students learning and development.  By doing this I can ensure that the feedback is related to the learning goals and I focus on the task not the individual.

• Presentation 1 - Unusual hotels.  The focus is on the content of the speech (July)
• Presentation 2 - Hotel packages. The focus is on their ability to sell the package (August)
• Presentation 3 - Personal presentation induction speech to new receptionists.  The focus is on the whole thing, uniforms worn. (November)

The assessments have been designed to form a staircase of integrated learning, linking and building on the last one, so that by the end of the year, the students are applying and analysing " breaking information into parts to explore understandings and relationships" Bloom's Revised Taxonomy,(2001).

This obviously happens at different stages for different students and some may stay at certain stages while others are forging ahead, by breaking the assessments down into smaller components this will cater to all learning abilities. 

By outlining the assessment structure at the beginning of the paper, the students are able to see 'where am I going?(what are the goals?) (Hattie&Timperley,2007).  This structure helps the student focus on a particular aspect of the presentations by directing the feedback to these areas and withholding feedback, "to allow students to internalise and process the demands of the task". (Hattie&Timperley,2007).  In this way there is a 'feed-forward' process happening that focus's the student on future learning, the "where to next? (What activities need to be undertaken to make better progress?).  The idea is to "prompt some degree of self reflection and management......feedback will demonstrate appropriate ways of enhancing the performance on the task, and offer strategies that invite more learner responsibility to improve." (Spiller, 2009). 

http://www.waikato.ac.nz/tdu/pdf/booklets/6_AssessmentFeedback.pdf (p.8)

I felt the original assessment did not align with the learning outcomes and the format was wordy and confusing.  I also wanted to produce a feedback form to be used during the students speeches, one that could be used by both the lecturer and the audience.  In this way a fair and reliable form of formative assessment would be made and the other students would be involved in the process, further developing their observation skills and their understanding of the requirements of the assessment.
The students receiving the feedback should have time to process and reflect on their presentation and get a sense of  'how am I going?"(what progress is being made towards the goal?), Hattie and Timperley(2007, as cited in Spiller, 2009).
"Learning can be deepened if the feedback additionally prompts some degree of self reflection and management". (Spiller, 2009).


The new assessment has been re-designed with 'chunks' of information to help the students draw on their own experiences to formulate some ideas about how they will approach the assessment.
The 'I' part of the ICE Model, Fostaty&Wilson 2000. The assessment is designed to start them gathering information (ideas) allowing recall and repetition (similar information from different sources i.e. newspaper, magazines, radio, t.v.) where they can develop 'discrete skills' (question one of the assessment).  The idea is for them to use these skills as stepping stones in their learning to formulate links of comprehension to the next question. 

http://wikieducator.org/Hospitality_Operations/Activities/Packages

By using this narrative form of assessment the student can start to make 'connections' from the clues in the text to information they already know, maybe from modules learnt in the previous semester.  This is the 'c' part of the ICE Model.
The 'Extension' component of this phase would be the process of the student 'doing' the assignment, giving the speech and participating in the feedback process of their own work and others in the group.  This would be the "how am I going?"  (what progress is being made towards this goal?) part of Hattie and Timperley(2007) cognitive process that would lead to the student starting to self -regulate their learning.


The marking criteria has been modified as well.  I have decided to make this very basic as there is the feedback form that will be used to help form the final result.  As this is part of an integrated assessment the marks are passed/not passed, so the peer feedback form will be used as a formative evaluation sheet, designed to focus the students and help them critique/observe another persons work.  It will also assist me in providing evidence and making an informed decision.

http://www.scribd.com/doc/114795032/Marking-Sheet-Revised-Hotel-Packages2
(This has been revised on 29/11)

The important part of the marking criteria will be the written feedback that is provided.

The feedback form has been modified from an internet and colleague's example and will formalize the speech process, where the 'audience' will be actively evaluating the speaker.  This can be a stressful experience but it is a necessary part of the Industry and is also used as a learning experience for the whole class.  This will build on from a previous assessment where the students are being evaluated just on the content.  In this assessment the students are being evaluated on their ability to sell as well as the quality of the content.


http://www.scribd.com/doc/114795029/Hospitality-Operations-feedback-form-packages-%E2%80%93-2012
(This has been revised on 29/11)

I investigated the use of rubriks for the feedback form but decided that the form would be more useful as a formative tool, to inform and link the learning, rather than a process of grading the student and their work.

The additional instructions and success criteria, have been added to give the students a starting point for this assessment.  I have linked the additional instructions to each question so they can refer to this to give them more clarity when attempting the assessment.
http://www.scribd.com/doc/114795028/GCTLT-Assessment-Activities

Once the assessment has been completed the students can use the self/peer assessment sheet to tick off their or others work to ensure they have completed all the questions.
http://www.scribd.com/doc/114797193/Self-Assessment-Checklist-for-Packages

Summary:
Through the process of evaluating this assessment, I have discovered the importance of breaking down the assessment into smaller 'chunks' of learning, aligning the feedback to the learning goals and focusing on the task, not the individual.  In designing the feedback form this focuses the evaluator on particular areas of the performance not on the individuals mannerisms.  I have been able to critique the assessments for the whole paper and look at how these effectively stair-case the students learning as they progress through the course.

References;

Victoria Business School, Victoria University of Wellington. (2010). Bloom's Revised Taxonomy. Retrieved from http://www.victoria.ac.nz/fca/teaching/steps-to-teaching-success/bloom-revised-taxonomy)

ICE model - Ideas, Connections and Extensions - Fostaty Young, S., & Wilson, R. (2000). The ICE Model of qualitative assessment. 

Spiller, D. (2009). Assessment: Feedback to promote student learning. Hamilton, New Zealand: Teaching Development, The University of Waikato. 

www.scc.spokane.edu/_.../5_Feedback_form_for_individual_speech.

Feedback process for group work

This post follows on from examples in my flexible learning course regarding the Conference and Events paper I teach in the Diploma of Travel and Tourism.  I am just in the process of evaluating, marking and providing feedback to the class.  They formed 2 groups and I have been deliberating how to offer feedback as a group and individually.

The course;

• Work on a theoretical 3 day conference and submit a portfolio and presentation
• Design, plan and work on a themed dinner
• Produce a blog which tracks their progress
• Collaborate in a group to produce the end result.
• Produce a reflection of the process based on these questions;

Write a paragraph reflecting on how well your team/you performed (critical group/self reflection) e.g. what you did well, what you could have done better, what you found difficult, what was easy.  How well did you as a team problem solve in an effort to deal with difficult issues.  

These reflections were sent to me via email and will form the basis of my individual feedback to them.

I intend to meet with each group and offer verbal feedback on their portfolios.  Then I will provide individual feedback via email and provide information on any re-sits for those students that have not put enough team effort into the group work.

There is plenty of evidence to show which students did not contribute effectively to the group and there is opportunity for them to submit further work.  So far these new processes have worked and the students feel that the process has been fair and equal.  This has been possible by the fact that there are only 10 students in the class, I will have to re-think this process if the class size increased dramatically.

What feedback models or processes do you prefer?Question 2 for assessment 3

What feedback models or processes do you prefer?

I read and connected immediately with the I.C.E. model (FostatyYoung & Robert J. wilson 2000) for assessing and thought of how it had similarities with Kolb's (1984) cycle of Adult learning. See my earlier post, Link to activity Eight.  His suggestion is that there are 4 stages to learning which link in a circle.
Concrete experience - reflection - abstract conceptualisation - active experimentation - and back again.
If this is a basic concept of how a person learns, then having an assessment model that aligns closely with this cycle e.g; formulating ideas or facts - making connections - and applying it (I.C.E.), follows the learning patterns and behaviours of adults , in theory should have a process of growth and deeper learning.  If the assessment type aligns and matches the learning outcomes then the process of feedback is an important component of the procedure.  Matching the type of feedback given and at the appropriate time would be important factors to consider.

Much of the facilitating I do is practical based and I assess by observing an interaction or activity.  I find the best process of feedback in these situations is an immediate verbal, one on one interaction.  Hattie & Timperley's (2007)  table would show this as feedback at Task level, correcting at a surface level.
One of the techniques I use I learnt from a colleague who has an outdoor education background and finds this useful to feedback in the first instance and then to encourage the learner to use it to self monitor their progress as they develop the skill(s).  By encouraging the learner to use this process themselves this would help with their deeper learning. (Self regulating).

• Start - I would like to see you start doing........
• Stop - I would like to see you stop doing......
• Keep - I would like to see you keep doing.......

( Similar to Start,Stop,Continue)

Another form of feedback I use which has a reflective component to it is in a similar vein to the previous example;

• What ( What did you do?  How was your performance?)  Task
• So what ( What does it mean?)  Process level
• Now what ( How can this be improved? what can you do differently?) Self regulating

Adventure Education, Miles & Priest (1990)

Why is feedback important?

It is an important part of the learning process, gaining effective feedback helps the learner to reflect on the experience and improve next time.  If there is no feedback then there would be no opportunity for self regulation or awareness of where improvements/changes/adaptions where needed and change would not happen.

What are the challenges or issues with assessment feedback?

1. Often difficult to give feedback in a timely manner, particularly if it is verbal, if not given at the right time then the moment is lost
2. It is easy to be too general with feedback, 'great job' doesn't really offer any real information for improvement 
3. It can be hard to get the right mix of positive and negative feedback without being too nice or too harsh
4. 'By being honest and straightforward, and by offering balanced feedback, the people you influence can build skills and confidence at the same time. ' Gail Anderson Giving and receiving feedback

How do you balance how much feedback to give?

This is difficult to do especially if there are big classes and many practical activities. There must be an effective feedback process that is easily adaptable and straightforward.   One that can be used to produce a consistent outcome for the learner.

Evaluation of an existing course assessment

The assessment I have decided to concentrate on forms one of twelve small evaluations within the Hospitality Operations course in the Certificate in Applied Travel and Tourism.  A one year, level three programme with minimum level 2 NCEA and an interview prior to entry.

1st Semester (All papers)
Day in the life of a Tourist
Day in the life of a Conference coordinator
Day in the life of a Travel agent
Day in the life of a Tour guide

2nd Semester (Choose 2)
Hospitality Operations
Travel agency
Tour guide
Each paper must gain a pass mark in order to obtain the Certificate in Travel and Tourism.

The paper is an introduction to the Hospitality Industry with an emphasis on Front Office and Reservation roles.  The students have access to the course outline, information, activities and assessments through Wiki educator http://wikieducator.org/Hospitality_Operations and a computerised virtual Hotel reservation system.  This system enables them to check customers into a hotel, add expenses to their account and check them out, this is assessed by providing role play opportunities, written assessments and a virtual hotel closed book evaluation.

A brief outline of all the assessments within this paper;

• Unusual Hotels (presentation)
• Impacts of mass Tourism (case study)
• Hotel packages (presentation)
• Hotel reservation/cancellation policies (word processed)
• Reception Theory assignment (word processed)
• Telephone reservations (role play)
• Checking guests into the hotel (role play)
• Checking guests out of the hotel (role play)
• Business correspondence (word processed)
• Alcoholic beverages (practical/closed book written assessment)
• Virtual hotel assessment (closed book computer assessment)

Grading; The course is set up in a logical, sequential manner so that there is rich learning on the way and there are opportunities to build on and reinforce their learning opportunities at different stages.  As the emphasis is on building on what they have just learnt rather than the final outcome the grading is pass or not passed.  

( Having said that, with the move towards statistical outcomes and lecturer accountability, I do struggle with the ethics of pushing students through at the last minute just to catch them up, how much learning has been done along the way for them? )

My focus for this topic is the 3rd assessment in the group namely, Hotel packages.  The students are required to do some background research, collect evidence and explain their own hotel package in the form of a presentation.

 http://wikieducator.org/Hospitality_Operations/Activities/Packages

 What is the purpose of this assessment?

Good question as often I think to myself, I know what I want to achieve but does this actually do it?

The aim of this assessment is twofold and hopes to provide an opportunity for the students to do some

research and to start to create an awareness of where and how these packages are advertised.

Secondly, to create their own package and present it in such a way as to sell the dream to their audience.

I see this assessment as a fair way to test their 'understanding by explaining their ideas and concepts'

(Bloom's revised Taxonomy;2001)which, with the appropriate level of feedback will provide
opportunities for the students to 'apply their understanding in another familiar situation'. (Bloom, 2001).

Does this happen?
In general, the quality of the presentations is very high and the students really enjoy producing
their own package, however some do find that doing a presentation and having it marked at the
time quite stressful, I think this is an area for improvement.  I would like to create a clear feedback
form to be used during the presentation so that guidance can be given one on one rather than in the group.
To extend this further I would like to design a simple peer feedback form for the other students to use 
during the presentation, which may help to focus their attention while their colleague is speaking.

How does it enhance students' learning?

I think the concept of this assessment helps to enhance the students learning if adequate research

is done and adequate exploration of hotel packages is pursued. The assessment is relevant

in helping the students to understand why packages are produced, the importance of

creating a sales concept to actively promote the hotel during the slower months/periods.

Formative feedback is given directly after the presentation, I may need to work on my technique

for this after Bronwyn's workshop on Friday!

How effectively does it measure student's learning outcomes?

This is where I feel I need to do some work on revising the assessment. The concept is good but the way

the assessment is laid out and the wording needs work. I do not think it is very easy to understand and

it needs to align with the learning outcomes more closely. The marking sheet needs to offer a better

guideline for the students in its layout and design.

Example of marking sheet;

http://www.scribd.com/doc/104841431/Marking-Sheet-Hotel-Packages

I think that the assessment has a reasonably good basic structure but the instructions are not clear
and the marking criteria is not accurate enough to enable feedback to be given in an easy, clear and
transparent manner.  There is confusion with what is required for the other questions in the assessment
and work handed in has sections missing and has to be handed back for re-sits.  This is time consuming
and unproductive. 

2nd Paper;Assessing and Evaluating for learning; Discussion1 for assessment 3

What does cultural diversity in relation to assessments mean to you?

What factors need to be considered? • How can consideration of diversity benefit the learner? • What are the challenges?

Whilst reading Maori Approaches to Assessment, I found myself identifying with the principles and approaches that the Te Whariki  Early Childhood Curriculum advocates, 'that children" grow up as competent and confident learners and communicators, healthy in mind, body and spirit, secure in their sense of belonging and in the knowledge that they make a valued contribution to society" (Rameka, 2007).
The underlying principle here is that all children will grow up and integrate under the guidance of the bi cultural curriculum into this modern society no matter what their cultural diversity is, e.g. ethnicity, gender, sexual orientation, differently-abled, etc.

Within the pre-school curriculum the concept of the inter-woven mat as an analogy recognises the importance of respecting the language, culture and knowledge of the individual and using portfolios to illustrate their growing skills and knowledge while actively participating in their own learning.

Using this concept as a guide to constructing assessments it is important to give a variety of choices to our learners helping them to integrate into the group using team based activities drawing on the strengths of each individual in different situations.  Analyse the most effective way of assessing for each situation ensuring it meets all of the values e.g. authentic, fair, transparent, etc.

If diversity is considered then there will be choice within the curriculum for the learner to explore their own method of being assessed, if the learning process has led them to that point .

The challenges are relying on the facilitator to try, test and try again if the processes are not working, something we are all doing all the time.  

Summary of my Flexible Learning Plan

Flexible learning presentation2

View another webinar from hgeytenbeek

This is in the form of a powerpoint presentation, comprising 7 slides.

Slide 1;  Definition of flexible learning  - Link to Activity 2

Slide 2 ;     What is the Plan? - Link to Activity 6

Slide 3;      Why have I chosen to do this?  - Link to Activity 4
Creating a safe environment where students have equal access to course materials, activities, interactions and assessments.  Within an environment that fosters respectful behaviour, inclusive of different cultures.  Using Ice breakers activities to help the groups go through the process of forming, storming, norming and performing

Slide 4;      Who are my Students?  - Link to Activity 3

Slide 5;      Honey and Mumford; Typology of Learners - Link to Activity 8
Within Kolb's Learning Cycle.  4 stages of learning which follow on from each other, Honey and Mumford have built learning styles around this sequence, identifying individual preferences for each stage.

Slide 6;      How will I do this?   - Link to Activity 7
Socio-cognitive learning perspective, Vygotsky's(1978) Zone of Proximal Development, embedded in this perspective is the idea that understanding results from personal interactions in social contexts.  Thus, students are viewed as participants in the construction of meaning through their interactions with others (Black, 2005).  To promote a higher level of thinking (critical), students will provide feedback to each other, and thus, co-construct knowledge and understanding(Roehler & Cantion,1997), which is the Active experimentation part of Kolb's Learning Cycle.
This will sit within the framework of Otago Polytechnic's Strategic Plan and the process for inspiring capability in our students.

Slide 7;      References


Thanks, Bronwyn for all your advice and guidance, I have learnt so much in a short space of time, lets just hope I can pass it on to my students now!

Good luck to all those that are yet to present!!  Its not that bad really!!

Activity Twelve - Organisations

How do your ideas for flexible learning, and those of your organisation or consultancy fit (or not) with the TES for NZ?

My Son had his cross country today and I could see the relief on his face when he could see the finish line, I know how he feels!!!!
I think this post title should be 'losing the plot! me or them, or both!

The Government's vision is that all New Zealanders should have ;
'Access to high-quality tertiary education (that) enriches people's lives, increases their employment opportunities and helps to build a productive skills base to drive economic growth.'

Part of that vision is to move New Zealand away from being a low wage economy, with many people doing unskilled jobs.
The strategy is how the government proposes we get there and whether the organisation I work in provides relevant and efficient processes that support that strategy for the future.  The TES and many other sectors will need to 'live within its means and do more with less', which prompts us to be proactive and look at new and innovative ways of teaching many people with fewer resources, less contact time and less man-power, is flexible learning the answer?  It certainly provides opportunity to think outside the box and be resourceful with new technology so that the same quality education is provided in new and surprising ways.

After reading the TES, OP's strategic plan/charter, the Maori strategic plan and the Literacy/numeracy one, I got a sense that they all aligned very neatly, in fact almost intertwined with each other in their visions and strategies.
Accessible learning for all, any time, any place, any where;


'We will continue to develop our courses for flexible delivery to facilitate learning in regional centres, in workplaces and in people's homes. ' OP Charter.


What do you need to include in your plan to fit with the TES for NZ?

• Continue to work with Industry to ensure that the learning meets Industry needs. (PEAC)
• A reliable and relevant moderation process that ensures Industry relevant skills
• Continue to offer STAR (Secondary Tertiary Alignment Resource) programmes to encourage the flow of secondary students into further education
• Continue to embed literacy, language and numeracy skills throughout the plan

I need to ensure that I continue to enhance my teaching role with up-dated quality resources, self moderation to ensure that the training/education is providing skills for employment, and creating a flexible approach where learning new technological skills will provide the flexibility to access  the wider community. 

Short and sweet, now on to weaving it all together!!!!!

References;

http://www.minedu.govt.nz

http://www.otagopolytechnic.ac.nz

The Otago Polytechnic Charter.

http://www.tec.govt.nz/Tertiary-Sector  Draft Literacy and Numeracy Implementation Strategy released