{"id":31,"date":"2011-06-06T15:43:37","date_gmt":"2011-06-06T15:43:37","guid":{"rendered":"http:\/\/egad.engineering.queensu.ca\/?page_id=31"},"modified":"2014-12-12T14:53:41","modified_gmt":"2014-12-12T19:53:41","slug":"identifying-and-collecting-data-on-student-learning","status":"publish","type":"page","link":"https:\/\/egad.engineering.queensu.ca\/?page_id=31","title":{"rendered":"3. Collecting Data on Student Learning"},"content":{"rendered":"<h3>Module 3 &#8211; Collecting data on student learning<img loading=\"lazy\" decoding=\"async\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/><\/h3>\n<p>Because students\u00e2\u20ac\u2122 success in acquiring the Graduate Attributes is the single strongest indicator of a program\u00e2\u20ac\u2122s success, the ability to collect and report on the results of effective, reliable and valid learning assessment is critical.<\/p>\n<p>Your purpose in collecting data is to obtain pertinent information in demonstrating the relative success of your programs and to use for program decision-making. The most important issue regarding data is\u00c2\u00a0fit for purpose\u00c2\u00a0\u00e2\u20ac\u201c selecting evidence and information on program effectiveness.<\/p>\n<p>This module provides information on direct and indirect evidence of students\u00e2\u20ac\u2122 learning,\u00c2\u00a0 methods of data collection, aligning methods of assessment with intended learning, and setting\u00c2\u00a0targets\u00c2\u00a0and\u00c2\u00a0thresholds\u00c2\u00a0for student\u00e2\u20ac\u2122s learning achievement.<\/p>\n<p>In this module you will:<\/p>\n<ul style=\"padding-left: 30px;\">\n<li>Identify the types of information needed to demonstrate the degree to which students have acquired the Graduate Attributes.<\/li>\n<li>Identify\u00c2\u00a0the stakeholders from whom that data may be collected.<\/li>\n<li>Match assessment tools to intended learning outcomes<\/li>\n<\/ul>\n<h4>Types of Evidence of Student Learning<\/h4>\n<p><img loading=\"lazy\" decoding=\"async\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/><br \/>\nData or evidence of student learning is categorized in two ways: direct evidence and indirect evidence.<\/p>\n<p>Direct evidence of learning is observable and measurable.<\/p>\n<p>E.g. Student exams, reports, projects, essays, portfolios, etc.<\/p>\n<p>Indirect evidence, on the other hand, relates to opinion or self-reports of student learning, or educational experiences.<\/p>\n<p>E.g. grades, student surveys, faculty surveys, focus group data, graduation rates, reputation, etc.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/><\/p>\n<p><!--nextpage--><\/p>\n<h4>Collecting Direct Evidence<\/h4>\n<p><img loading=\"lazy\" decoding=\"async\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/><br \/>\nFor accreditation purposes, schools will be asked to provide a range of examples of completed and graded student work as direct evidence of learning. Direct evidence must be related to the indicators defined by the program. It is essential that the assessment tool used aligns with learning objectives. Possible assessment tools include:<br \/>\n[singlepic id=36 w=557 h=264 mode=watermark float=]\n<p>Can you identify which assessment tools will yield direct evidence of student learning?<\/p>\n<blockquote>[expand title=&#8221;<strong>Learning Activity: Assessments tools<\/strong>&#8220;]\nUsing two of the indicators you wrote in Module 1, select at least two different ways you might choose to conduct assessment:<\/p>\n<ul style=\"padding-left: 30px;\">\n<li>Determine three ways that they could be assessed (a list of assessment tools are on summary sheet), at least one done using a direct assessment tool<\/li>\n<li>If any are difficult to measure, consider whether the criteria should be modified<\/li>\n<\/ul>\n[\/expand]<\/blockquote>\n<p>Because the accreditation process requires submission of a range student work at varying levels. This range includes: below, at and above threshold levels. As a result, a threshold for success must be determined.<\/p>\n<h4>Targets and thresholds<\/h4>\n<p><img loading=\"lazy\" decoding=\"async\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/><br \/>\nAn example of how targets and thresholds could be incorporated into an assignment is shown below. It shows a rubric used to assess specific indicators where the threshold and target descriptions are highlighted.<\/p>\n<p>Example:<\/p>\n[singlepic id=40 w=485 h=264 mode=watermark float=]\n<blockquote>[expand title=&#8221;<strong>Learning Activity<\/strong>&#8220;]\n<p>Using the indicators you wrote while completing Module 1, and with the methods of assessment you selected to gauge students\u00e2\u20ac\u2122 success in mind, describe the characteristics of a minimal successful performance (threshold).<\/p>\n[\/expand]<\/blockquote>\n<p><img loading=\"lazy\" decoding=\"async\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/><\/p>\n<p>Below is an alternative example that does not use rubrics, rather it has well developed targeted questions in assessing specific indicators.<\/p>\n[expand title=&#8221;<strong>Examples: Knowledge assessment\u00c2\u00a0(Click to expand) <\/strong>&#8221; rel=&#8221;expendable-highlander&#8221;]\n<p>Calculus instructor asked questions on exam that specifically targeted 3 assessment criteria for \u00e2\u20ac\u0153Knowledge\u00e2\u20ac\u009d:<\/p>\n<ol>\n<li>\u00e2\u20ac\u0153Create mathematical descriptions or expressions to model a real-world problem\u00e2\u20ac\u009d<\/li>\n<li>\u00e2\u20ac\u0153Select and describe appropriate tools to solve mathematical problems that arise from modeling a real-world problem\u00e2\u20ac\u009d<\/li>\n<li>\u00e2\u20ac\u0153Use solution to mathematical problems to inform the real-world problem that gave rise to it\u00e2\u20ac\u009d<\/li>\n<\/ol>\n[expandsub1 title=&#8221;Indicator 1&#8243; expanded=&#8221;true&#8221;]\n<p>The student can create and\/or select mathematical descriptions or expressions for simple real-world problems involving rates of change and processes of accumulation (overlaps problem analysis)<\/p>\n<p>Context: calculating\u00c2\u00a0intersection of two\u00c2\u00a0trajectories<\/p>\n[singlepic id=41 w=378 h=284 mode=watermark float=]\n[\/expandsub1]\n[expandsub1 title=&#8221;Indicator 2&#8243; expanded=&#8221;true&#8221;]\n<p>Students can select and describe appropriate tools to solve the mathematical problems that arise from this analysis<\/p>\n<p>Context: differentiation\u00c2\u00a0similar to high school\u00c2\u00a0curriculum<\/p>\n[singlepic id=42 w=378 h=284 mode=watermark float=]\n<p>Context: implicit\u00c2\u00a0differentiation, trig\u00c2\u00a0inverse<\/p>\n[singlepic id=43 w=378 h=284 mode=watermark float=]\n[\/expandsub1]\n<p>Physics course instructors administering the Force Concept Inventory (FCI) before and after course in mechanics to assess conceptual understanding<\/p>\n<p>Allows for benchmarking, which is difficult to do for most other indicators.<\/p>\n[\/expand]\n<p><img loading=\"lazy\" decoding=\"async\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/><\/p>\n<h5><span style=\"text-decoration: underline;\"><strong>View the Live Workshop on Assessment Tools:<\/strong><\/span><\/h5>\n<div class=\"x-video player\" data-x-element-mejs><div class=\"x-video-inner\"><video class=\"x-mejs x-wp-video-shortcode advanced-controls\" id=\"video-31-1\" width=\"640\" height=\"360\" poster=\"http:\/\/.\/wp-content\/themes\/platformpro\/images\/assessment_tools.jpg\" controls=\"controls\"><source type=\"video\/mp4\" src=\"http:\/\/.\/wp-content\/Videos\/EGAD_4.mp4?_=1\" \/><source type=\"\" src=\"http:\/\/video-js.zencoder.com\/oceans-clip.webmm?_=1\" \/><\/video><\/div><\/div>\n<p><img loading=\"lazy\" decoding=\"async\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/><\/p>\n<p><!--nextpage--><\/p>\n<h4>Collecting Indirect Evidence<\/h4>\n<p><img loading=\"lazy\" decoding=\"async\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/><br \/>\nThe following table outlines a variety of methods for collecting indirect evidence on student learning and indications for their use. Following the table are more detailed on notes on a variety of methods for gathering indirect evidence of student learning.<\/p>\n<p>(Click on below image to enlarge, click image again to minimize)<\/p>\n[singlepic id=37 w=450 h=523 mode=watermark float=]\n<p><em>Northwest Centre for Public Health Practice. <\/em> <a href=\"http:\/\/www.nwcphp.org\/docs\/data_collection\/data_collection_toolkit.pdf\" target=\"_blank\"> Data Collection Toolkit<\/a>.<\/p>\n<p><strong>Student self-reports<\/strong><br \/>\n<img loading=\"lazy\" decoding=\"async\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/><br \/>\nBecause attribute-based assessment invites the involvement of all stake-holders, we&#8217;ve included information on including students in the assessment process. Here you&#8217;ll find a <a href=\"\/wp-content\/documents\/GASelf-Assessmentedited.pdf\" target=\"_blank\">Graduate Attribute Self-Assessment Questionnaire<\/a> that was adapted from one used at the University of Dalhousie and an article on the use of <a href=\"http:\/\/www.cjlt.ca\/index.php\/cjlt\/article\/view\/97\" target=\"_blank\">e-portfolios<\/a> from the students&#8217; perspective.<\/p>\n<p><strong>Surveys<\/strong> (current students or recent grads)<br \/>\n<img loading=\"lazy\" decoding=\"async\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/><br \/>\nSurveys can be a valuable means of collecting indirect data on students\u00e2\u20ac\u2122 learning and can be created for students\u00e2\u20ac\u2122 themselves or for other stakeholders such as co-op supervisors, alumni or industry employers.<\/p>\n<p>Some guidelines to consider when creating surveys:<\/p>\n<p style=\"padding-left: 30px;\">1) <strong>Keep your purpose in mind, <\/strong>it\u00e2\u20ac\u2122ll help to ground you when you start to get lost in your question-writing<strong> <\/strong><\/p>\n<p style=\"padding-left: 30px;\">2) <strong>If in doubt, throw it out, <\/strong>if you don\u00e2\u20ac\u2122t have a sound reason for asking a question or don\u00e2\u20ac\u2122t know what you\u00e2\u20ac\u2122ll do with the data it generates, don\u00e2\u20ac\u2122t ask it.<strong> <\/strong><\/p>\n<p style=\"padding-left: 30px;\">3) <strong>Keep your questions simple, <\/strong>break complex questions down into component parts<strong> <\/strong><\/p>\n<p style=\"padding-left: 30px;\">4) <strong>Include only one topic per question<\/strong><\/p>\n<p style=\"padding-left: 30px;\">5) <strong>If a question can be misinterpreted, it will be, <\/strong>be clear, and concise and pilot your questions before going live<strong> <\/strong><\/p>\n<p style=\"padding-left: 30px;\">6) <strong>Avoid leading questions or questions that suggest a preferred response<\/strong><\/p>\n<p style=\"padding-left: 30px;\">7) <strong>Make sure respondents have enough information to answer the questions you ask. <\/strong>If you want to know about employers\u00e2\u20ac\u2122 perceptions of how well your students demonstrate Graduate Attributes, you might need to provide them with a list of what the attributes are and include descriptors of each<strong> <\/strong><\/p>\n<p><strong>Focus Groups Interviews<\/strong><br \/>\n<img loading=\"lazy\" decoding=\"async\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/><br \/>\nFocus group interviews are typically conducted with 7-12 individuals from the target stakeholder group (in this case, students from your program). The purpose is to identify trends and patterns of perception about the program and its effect on learning. The group moderator\/facilitator sets the tone for the interview and ensures that each member of the group has ample opportunity to contribute to the discussion. Focus group interviews are useful for purposes of triangulation to supplement other findings.<\/p>\n<h5 style=\"padding-left: 30px;\">Advantages<\/h5>\n<p><img loading=\"lazy\" decoding=\"async\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/><\/p>\n<ul style=\"padding-left: 30px;\">\n<li>Provides more in-depth information on issues identified in a survey<\/li>\n<li>Can inform the interpretation of survey results<\/li>\n<li>Interaction among focus group participants often leads to new<\/li>\n<li>Insights as contributors piggy-back on other\u00e2\u20ac\u2122s ideas<\/li>\n<li>Allows the probing of unanticipated issues<\/li>\n<\/ul>\n<h5 style=\"padding-left: 30px;\">Disadvantages<\/h5>\n<p><img loading=\"lazy\" decoding=\"async\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/><\/p>\n<ul style=\"padding-left: 30px;\">\n<li>Moderators require training to be effective<\/li>\n<li>Face-to-face meetings may be difficult to arrange in a timely manner<\/li>\n<li>Differences in participant responses may be difficult to reconcile<\/li>\n<li>Data analyses may become quite complex<\/li>\n<\/ul>\n<blockquote>[expand title=&#8221;<strong>Learning Activity<\/strong>&#8220;]\n<p>Using the following guidelines, create three focus group questions<\/p>\n<ul style=\"padding-left: 30px;\">\n<li>Narrow your focus to a single theme (i.e. graduate attributes)<\/li>\n<li>Decide what it is exactly that you want to find out<\/li>\n<li>Ensure questions are concise and invite discussion (i.e. are opened ended and judgment free)<\/li>\n<li>Anticipate possible responses and prepare enabling, follow-up questions<\/li>\n<\/ul>\n[\/expand]<\/blockquote>\n<p><img loading=\"lazy\" decoding=\"async\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/>[expand title=&#8221;<strong>Summary Activity: <\/strong>Based on the following chart and the time and resources available to you, which methods of data collection best suit your needs and context? &#8220;]\n<p>Using the chart at the top of the page as a guideline and considering the time and resources available to you, which methods of data collect suit your needs and contexts. Keep in mind all stakeholders you maybe want to consider including: Co-op placement or community service learning supervisors\/clients and employers.<br \/>\n[\/expand]\n<!--nextpage--><br \/>\n<strong>Triangulating Data on Student Learning<\/strong><br \/>\n<img loading=\"lazy\" decoding=\"async\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/><br \/>\nUsing multiple sources of data when documenting students\u00e2\u20ac\u2122 learning is an effective way of validating initial analyses of a single piece of data.<\/p>\n<p><strong>1. <\/strong><strong>Include opportunities for informal assessment, students\u00e2\u20ac\u2122 self-reports of learning, and even unsolicited data from placement supervisors or employers.<\/strong><\/p>\n<p>Students\u00e2\u20ac\u2122 performance in the classroom can become a data point if teachers adopt a habit of keeping session logs to record ways that students respond to instruction in different learning situations. Data gained through student focus groups or interviews may also provide an alternative perspective on how and what students are learning.\u00c2\u00a0 Placement supervisors and employers are likely to be able to comment on integrated, authentic, practical learning.<\/p>\n<p><strong>2. <\/strong><strong>Use more than one type of assessment when analyzing data<\/strong><\/p>\n<p>When you know what it is that you\u00e2\u20ac\u2122re looking for in students\u00e2\u20ac\u2122 learning, the more likely you are to want to gather data from different sources to bolster your claims about what they\u00e2\u20ac\u2122ve learned. Looking at multiple sources of data on students\u00e2\u20ac\u2122 learning is analogous to taking multiple snapshots of a single event \u00e2\u20ac\u201c it provides a more complete picture of reality. Using data from multiple assessment sources enables teachers to more accurately determine students\u00e2\u20ac\u2122 strengths and identify areas for improvement<\/p>\n<p><strong>3. <\/strong><strong>Value all assessments not just major events<\/strong><\/p>\n<p>Any assessment activity that was worth having your students do is worth using as a data point.<\/p>\n<p><strong>4. <\/strong><strong>Use the data gained from assessment to answer questions about authentic learning<\/strong><\/p>\n<p>Typically, authentic learning is described as learning that mimics the real-life knowledge and skills that are required in industry. Just as typically, questions about complex learning are unlikely to be fully answered using just one piece of data.<\/p>\n<p><strong>5. Look at data across time intervals<\/strong><\/p>\n<p>When looking at data on students\u00e2\u20ac\u2122 learning over time, look for patterns and changes in performance and achievement in relation to the demands of the curriculum.<\/p>\n<blockquote>[expand title=&#8221;<strong>Learning Activity<\/strong>&#8220;]List the types of data you have already collected or have access to. What types of data are you missing (that is where are the gaps in your understanding of the strengths and limitations of your programs) and how might you access it?[\/expand]<\/blockquote>\n<p><img loading=\"lazy\" decoding=\"async\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/><br \/>\n[expand title=&#8221;<strong>Other useful\u00c2\u00a0resources: <\/strong>&#8220;]\n<a href=\"http:\/\/condor.depaul.edu\/acafflpc\/aahe.htm\" target=\"_blank\">Assessing professional skills (DePaul University) [link<\/a>]\n<p><a href=\"http:\/\/www.informaworld.com\/smpp\/ftinterface~content=a921570553~fulltext=713240930~frm=content\" target=\"_blank\">Development of a scale to\u00c2\u00a0measure life-long learning (Kirby, Knapper, Lamon &amp; Egnatoff) [pdf]<\/a><\/p>\n<p><a href=\"\/wp-content\/documents\/Rubrics.pdf\" target=\"_blank\">Assessing integrated learning creating rubrics [pdf]<\/a><\/p>\n<p><a href=\"http:\/\/www.cjlt.ca\/index.php\/cjlt\/article\/view\/97\" target=\"_blank\">Student-maintained portfolios e-portfolios [link]<\/a><\/p>\n<p><span style=\"line-height: 16px;\">Designing the right assignment<\/span><\/p>\n<p><a href=\"http:\/\/www.brookes.ac.uk\/services\/ocsld\/resources\/methods.html\" target=\"_blank\">http:\/\/www.brookes.ac.uk\/services\/ocsld\/resources\/methods.html<\/a><\/p>\n<p><a href=\"http:\/\/aei.uoregon.edu\/handouts\/2004_bloom_polygon.pdf\" target=\"_blank\">http:\/\/aei.uoregon.edu\/handouts\/2004_bloom_polygon.pdf<\/a> (note: Bloom\u00e2\u20ac\u2122s taxonomy depicts learning as occurring in three distinct domains. The polygon deals only with cognitive learning. Integrated learning will require instructors to include elements of skill (psychomotor) and attitudinal (affective) development as part of assessment design. \u00c2\u00a0(information on psychomotor and affective domains can be found at <a href=\"http:\/\/www.nwlink.com\/~donclark\/hrd\/bloom.html\" target=\"_blank\">http:\/\/www.nwlink.com\/~donclark\/hrd\/bloom.html<\/a><br \/>\n[\/expand]\n<img loading=\"lazy\" decoding=\"async\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Module 3 &#8211; Collecting data on student learning Because students\u00e2\u20ac\u2122 success in acquiring the Graduate Attributes is the single strongest indicator of a program\u00e2\u20ac\u2122s success, the ability to collect and report on the results of effective, reliable and valid learning assessment is critical. Your purpose in collecting data is to obtain pertinent information in demonstrating the relative success of your &#8230;<\/p>\n","protected":false},"author":19,"featured_media":0,"parent":857,"menu_order":3,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-31","page","type-page","status-publish","hentry","no-post-thumbnail"],"_links":{"self":[{"href":"https:\/\/egad.engineering.queensu.ca\/index.php?rest_route=\/wp\/v2\/pages\/31","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/egad.engineering.queensu.ca\/index.php?rest_route=\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/egad.engineering.queensu.ca\/index.php?rest_route=\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/egad.engineering.queensu.ca\/index.php?rest_route=\/wp\/v2\/users\/19"}],"replies":[{"embeddable":true,"href":"https:\/\/egad.engineering.queensu.ca\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=31"}],"version-history":[{"count":120,"href":"https:\/\/egad.engineering.queensu.ca\/index.php?rest_route=\/wp\/v2\/pages\/31\/revisions"}],"predecessor-version":[{"id":233,"href":"https:\/\/egad.engineering.queensu.ca\/index.php?rest_route=\/wp\/v2\/pages\/31\/revisions\/233"}],"up":[{"embeddable":true,"href":"https:\/\/egad.engineering.queensu.ca\/index.php?rest_route=\/wp\/v2\/pages\/857"}],"wp:attachment":[{"href":"https:\/\/egad.engineering.queensu.ca\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=31"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}