Collecting Indirect Evidence
The following table outlines a variety of methods for collecting indirect evidence on student learning and indications for their use. Following the table are more detailed on notes on a variety of methods for gathering indirect evidence of student learning.
(Click on below image to enlarge, click image again to minimize)
[singlepic id=37 w=450 h=523 mode=watermark float=]Northwest Centre for Public Health Practice. Data Collection Toolkit.
Student self-reports
Because attribute-based assessment invites the involvement of all stake-holders, we’ve included information on including students in the assessment process. Here you’ll find a Graduate Attribute Self-Assessment Questionnaire that was adapted from one used at the University of Dalhousie and an article on the use of e-portfolios from the students’ perspective.
Surveys (current students or recent grads)
Surveys can be a valuable means of collecting indirect data on students’ learning and can be created for students’ themselves or for other stakeholders such as co-op supervisors, alumni or industry employers.
Some guidelines to consider when creating surveys:
1) Keep your purpose in mind, it’ll help to ground you when you start to get lost in your question-writing
2) If in doubt, throw it out, if you don’t have a sound reason for asking a question or don’t know what you’ll do with the data it generates, don’t ask it.
3) Keep your questions simple, break complex questions down into component parts
4) Include only one topic per question
5) If a question can be misinterpreted, it will be, be clear, and concise and pilot your questions before going live
6) Avoid leading questions or questions that suggest a preferred response
7) Make sure respondents have enough information to answer the questions you ask. If you want to know about employers’ perceptions of how well your students demonstrate Graduate Attributes, you might need to provide them with a list of what the attributes are and include descriptors of each
Focus Groups Interviews
Focus group interviews are typically conducted with 7-12 individuals from the target stakeholder group (in this case, students from your program). The purpose is to identify trends and patterns of perception about the program and its effect on learning. The group moderator/facilitator sets the tone for the interview and ensures that each member of the group has ample opportunity to contribute to the discussion. Focus group interviews are useful for purposes of triangulation to supplement other findings.
Advantages
- Provides more in-depth information on issues identified in a survey
- Can inform the interpretation of survey results
- Interaction among focus group participants often leads to new
- Insights as contributors piggy-back on other’s ideas
- Allows the probing of unanticipated issues
Disadvantages
- Moderators require training to be effective
- Face-to-face meetings may be difficult to arrange in a timely manner
- Differences in participant responses may be difficult to reconcile
- Data analyses may become quite complex
[expand title=”Learning Activity“]Using the following guidelines, create three focus group questions
[/expand]
- Narrow your focus to a single theme (i.e. graduate attributes)
- Decide what it is exactly that you want to find out
- Ensure questions are concise and invite discussion (i.e. are opened ended and judgment free)
- Anticipate possible responses and prepare enabling, follow-up questions
[expand title=”Summary Activity: Based on the following chart and the time and resources available to you, which methods of data collection best suit your needs and context? “]
Using the chart at the top of the page as a guideline and considering the time and resources available to you, which methods of data collect suit your needs and contexts. Keep in mind all stakeholders you maybe want to consider including: Co-op placement or community service learning supervisors/clients and employers.
[/expand]