{"id":33,"date":"2011-06-06T15:44:07","date_gmt":"2011-06-06T15:44:07","guid":{"rendered":"http:\/\/egad.engineering.queensu.ca\/?page_id=33"},"modified":"2013-03-25T14:10:25","modified_gmt":"2013-03-25T19:10:25","slug":"analyzing-and-interpreting-the-data","status":"publish","type":"page","link":"https:\/\/egad.engineering.queensu.ca\/?page_id=33","title":{"rendered":"4. Analyzing and Interpreting Data"},"content":{"rendered":"<h3>Module 4 &#8211; Analyzing and Interpreting the Data<strong><img loading=\"lazy\" decoding=\"async\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/><\/strong><\/h3>\n<p>The previous module,\u00c2\u00a0Collecting Data on Students\u00e2\u20ac\u2122 Learning, dealt with the process of generating findings. Findings are merely the facts and empirical evidence of your investigation. To be of use in the process of curriculum evaluation, those\u00c2\u00a0findings will have to be interpreted so that\u00c2\u00a0judgments\u00c2\u00a0and then\u00c2\u00a0recommendations for improvement might be made.<\/p>\n<p>This module differs from the ones that come before in that the  content is restricted to the practice of question-generation. We did so  because the quality of the questions asked during the process of  program evaluation influence the types of data collected and the  interpretations and recommendations made. Use your program\u00e2\u20ac\u2122s purpose  statement as the basis from which to ask questions about the  effectiveness of programming and students\u00e2\u20ac\u2122 learning. When the answers to  your questions begin to emerge, the next step in the process is  identifying recommendations and actions for program improvement.is a platform on which to build additional inquiry.<\/p>\n<p>These primary eight questions are designed to guide you in analyzing and interpreting your data. Each of the eight primary questions is supplemented with additional considerations to guide interpretations, judgements and program recommendations. The questions in this module will start your process in:<\/p>\n<ul style=\"padding-left: 30px;\">\n<li>Identify themes, patterns, associations, gaps, and redundancies in your programs\u00e2\u20ac\u2122 curricula<\/li>\n<li>Interpret trends and discrepancies in data from different sources<\/li>\n<\/ul>\n<p>In an outcomes-based curriculum such as those now required for engineering schools in Canada it is important to be able to show a direct relationship among the Graduate Attributes, indicators, instructional methods and activities, assessment activities and student performance. That being the case, evaluating the curriculum means asking the question: To what extent does the design of the curriculum accomplish what it was intended to accomplish? Once you\u00e2\u20ac\u2122re able to determine the ways in which the curriculum was successful, it\u00e2\u20ac\u2122s equally important to follow up with the question of \u00e2\u20ac\u0153what went wrong and why?\u00e2\u20ac\u009d.<\/p>\n<h4><span style=\"text-decoration: underline;\"><strong>View the Live Workshop on Analyzing Graduate Attribute Data:<\/strong><\/span><\/h4>\n<div class=\"x-video player\" data-x-element-mejs><div class=\"x-video-inner\"><video class=\"x-mejs x-wp-video-shortcode advanced-controls\" id=\"video-33-1\" width=\"640\" height=\"360\" poster=\"http:\/\/.\/wp-content\/themes\/platformpro\/images\/analyzing_data.jpg\" controls=\"controls\"><source type=\"video\/mp4\" src=\"http:\/\/.\/wp-content\/Videos\/EGAD_5.mp4?_=1\" \/><source type=\"\" src=\"http:\/\/video-js.zencoder.com\/oceans-clip.webmm?_=1\" \/><\/video><\/div><\/div>\n<p>&nbsp;<\/p>\n<p><!--nextpage--><br \/>\nPresented are a series of 8 questions that might serve as a starting point in focusing the analysis, interpretations, judgements and recommendations that may arise from your curriculum mapping process. For each of then8 primary questions we have included starter questions to focus attention on things to look for in your findings, other questions to focus interpretations, judgements and recommendation. In each case, the questions we\u00e2\u20ac\u2122ve provided are just a start; you are encouraged to refine and supplement the questions we provide with others that represent your own programs\u00e2\u20ac\u2122 context and purpose.<\/p>\n<h4>1) On which Graduate Attributes is programming focused?<\/h4>\n<p><img loading=\"lazy\" decoding=\"async\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/><br \/>\n<strong>Findings:<\/strong><\/p>\n<ul style=\"padding-left: 30px;\">\n<li>Which Attributes are most often cited as being supported programming?<\/li>\n<li>Are faculty and students perceptions congruent?<\/li>\n<\/ul>\n<p><strong>Interpretation: <\/strong><\/p>\n<ul style=\"padding-left: 30px;\">\n<li>What explanations might be offered as to explain any discrepancy between faculty and students\u00e2\u20ac\u2122 perceptions?<\/li>\n<\/ul>\n<p><strong>Judgment: <\/strong><\/p>\n<ul style=\"padding-left: 30px;\">\n<li>Of what relative importance is it for faculty members\u00e2\u20ac\u2122 and students\u00e2\u20ac\u2122 perceptions to be congruent?<\/li>\n<\/ul>\n<p><strong>Recommendation: <\/strong><\/p>\n<ul style=\"padding-left: 30px;\">\n<li>What <em>specific<\/em> changes will you make to address the incongruence?<\/li>\n<\/ul>\n<p><img loading=\"lazy\" decoding=\"async\" style=\"border-style: initial; border-color: initial;\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/><\/p>\n<p><em>What other questions for findings, interpretations, judgements and recommendations will be pertinent to your own context?<\/em><\/p>\n<h4>2) Where are the focus attributes being best supported?<\/h4>\n<p><img loading=\"lazy\" decoding=\"async\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/><\/p>\n<ul style=\"padding-left: 30px;\">\n<li>In which courses?<\/li>\n<li>In which years?<\/li>\n<li>By what instructional methods?<\/li>\n<li>Assessed by what means?<\/li>\n<\/ul>\n<p>Are instructional and assessment strategies aligned with intended learning and the purpose of the program<img loading=\"lazy\" decoding=\"async\" style=\"border-style: initial; border-color: initial;\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/><br \/>\n<em>What other questions for findings, interpretations, judgements and recommendations will be pertinent to your own context?<\/em><\/p>\n<h4>3) Which Graduate Attributes are reported as getting the least attention?<span style=\"font-size: 14px; line-height: 22px;\"><img loading=\"lazy\" decoding=\"async\" style=\"border-style: initial; border-color: initial;\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/><\/span><\/h4>\n<ul style=\"padding-left: 30px;\">\n<li>What are the quantitative differences between high and low profile attributes?<\/li>\n<\/ul>\n<ul style=\"padding-left: 30px;\">\n<li>Is there agreement among all stakeholders regarding the strengths and limitations of attribute support throughout the program?<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p>The main deficiencies identified in university courses were not related to teaching and learning but to assessment practices (Quality Assurance Agency for Higher Education (2003)<img loading=\"lazy\" decoding=\"async\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/><\/p>\n<p><em>What other questions for findings, interpretations, judgements and recommendations will be pertinent to your own context?<\/em><\/p>\n<p><!--nextpage--><\/p>\n<h4>4) What program-specific attributes are supported by your curriculum? (i.e. attributes not identified by CEAB) <img loading=\"lazy\" decoding=\"async\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/><\/h4>\n<p><strong>Findings:<\/strong><\/p>\n<ul style=\"padding-left: 30px;\">\n<li>What are those attributes<\/li>\n<li>Where and how are they supported?<\/li>\n<\/ul>\n<p><strong>Interpretation:<\/strong><\/p>\n<ul style=\"padding-left: 30px;\">\n<li>In what way are those program-specific attributes related to the purpose of your program?<\/li>\n<li>In what way are those attributes related to professional engineering?<\/li>\n<\/ul>\n<p><strong>Judgments:<\/strong><\/p>\n<ul style=\"padding-left: 30px;\">\n<li>How important are the program-specific attributes to the character of the program and to your students\u00e2\u20ac\u2122 development?<\/li>\n<li>Should the support of these program-specific attributes be enhanced?<\/li>\n<\/ul>\n<p><strong>Recommendations:<\/strong><\/p>\n<ul style=\"padding-left: 30px;\">\n<li>What changes to the program might be made to better support the program-specific attributes?<strong> <\/strong><\/li>\n<\/ul>\n<p><img loading=\"lazy\" decoding=\"async\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/><br \/>\n<em>What other questions for findings, interpretations, judgements and recommendations will be pertinent to your own context?<\/em><\/p>\n<h4>5) What gaps exist in the program?<\/h4>\n<h3><strong><img loading=\"lazy\" decoding=\"async\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/><\/strong><\/h3>\n<p><strong>Findings:<\/strong><\/p>\n<ul style=\"padding-left: 30px;\">\n<li>Where do they exist?<span style=\"font-size: 14px; line-height: 22px;\"> <\/span><\/li>\n<\/ul>\n<p><strong>Interpretations:<\/strong><\/p>\n<ul style=\"padding-left: 30px;\">\n<li>What is causing the gap?<\/li>\n<\/ul>\n<p><strong>Judgments:<\/strong><\/p>\n<ul style=\"padding-left: 30px;\">\n<li>What is the impact of the identified gap(s)?<\/li>\n<\/ul>\n<p><strong>Recommendations:<\/strong><\/p>\n<ul style=\"padding-left: 30px;\">\n<li>How might the gap(s) be addressed?<\/li>\n<\/ul>\n<h3><strong><img loading=\"lazy\" decoding=\"async\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/><\/strong><\/h3>\n<p><em>What other questions for findings, interpretations, judgements and recommendations will be pertinent to your own context?<\/em><\/p>\n<h4>6) How are your students doing?<\/h4>\n<h3><strong><img loading=\"lazy\" decoding=\"async\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/><\/strong><\/h3>\n<p>After contrasting student\u00e2\u20ac\u2122s self-reports of learning with data collected on their actual performance, here are some questions you may want to consider:<\/p>\n<p><strong>Findings:<\/strong><\/p>\n<ul style=\"padding-left: 30px;\">\n<li>What percentage of students is reaching target levels of\u00c2\u00a0performance?<\/li>\n<li>What percentage of students is reaching threshold\u00c2\u00a0levels?<\/li>\n<li>How congruent are the results of indirect (students\u00e2\u20ac\u2122 self reports, co-op supervisors\u00e2\u20ac\u2122 reports etc) and direct (results on tests, assignments and projects) methods of\u00c2\u00a0assessment?<\/li>\n<li>What are the trends?<\/li>\n<\/ul>\n<p><strong>Interpretations:<\/strong><\/p>\n<ul style=\"padding-left: 30px;\">\n<li>What explanations can be made for the discrepancies between direct and indirect methods of assessment?<\/li>\n<\/ul>\n<p><strong>Judgments:<\/strong><\/p>\n<ul style=\"padding-left: 30px;\">\n<li> What is the significance of the discrepancies between direct and indirect reports of learning?<\/li>\n<li>What effect is the discrepancy having?<\/li>\n<\/ul>\n<p><strong>Recommendations:<\/strong><\/p>\n<ul style=\"padding-left: 30px;\">\n<li>What specific strategies will be suggested to bridge the gaps that have been identified?<\/li>\n<\/ul>\n<p><img loading=\"lazy\" decoding=\"async\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/><\/p>\n<p><em>What other questions for findings, interpretations,\u00c2\u00a0judgements\u00c2\u00a0and recommendations will be pertinent to your own context?<\/em><!--nextpage--><\/p>\n<h4>7) Where in the program is students\u00e2\u20ac\u2122 development being best supported?<\/h4>\n<p><img loading=\"lazy\" decoding=\"async\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/><br \/>\nHere, you might choose to contrast students\u00e2\u20ac\u2122 reports with faculty members\u00e2\u20ac\u2122 perceptions and cross contrast those findings with course outcomes, instructional and assessment strategies. What courses, activities and learning opportunities do students identify as providing ideal opportunities to develop the graduate attributes?<\/p>\n<p><strong>Interpretations:<\/strong><\/p>\n<ul style=\"padding-left: 30px;\">\n<li>What are the characteristics of the learning and assessment opportunities that students identify as critical to their learning?<\/li>\n<li>How well aligned are instructional and assessment activities with the outcomes, indicators and stated purpose of the courses?<\/li>\n<li>Are the methods of assessment authentic?<\/li>\n<\/ul>\n<p><strong>Recommendations:<\/strong><\/p>\n<ul style=\"padding-left: 30px;\">\n<li>What specific recommendations can be made to increase alignment of stated program purpose, course outcomes, instructional methods and assessment?<\/li>\n<li>What educational development support might faculty members require to increase the alignment between instructional and assessment strategies and expected outcomes?<\/li>\n<\/ul>\n<p><img loading=\"lazy\" decoding=\"async\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/><\/p>\n<p><em>What other questions for findings, interpretations, judgements and recommendations will be pertinent to your own context?<\/em><\/p>\n<h4>8 ) In what areas are students doing less well?<span style=\"font-size: 14px; line-height: 22px;\"><img loading=\"lazy\" decoding=\"async\" style=\"border: 0px initial initial;\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/><\/span><\/h4>\n<p>Contrast students\u00e2\u20ac\u2122 perceptions with faculty members\u00e2\u20ac\u2122 perceptions and achievement records Interpretation: What are the characteristics of the environments in which students are doing less well? Recommendations:  What supports might you put in place to better scaffold students\u00e2\u20ac\u2122 learning?    Alert &#8211; Are the instructional and assessment methods reportedly in use congruent with the attributes and indicators that have been identified as targets? When instructional and assessment methods in use are inconsistent with intentions for learning, students\u00e2\u20ac\u2122 learning achievement can be significantly compromised.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/><\/p>\n<p><em>What other questions for findings, interpretations, judgements and recommendations will be pertinent to your own context?<\/em><\/p>\n<p><!--nextpage--><\/p>\n<h4>Tips To Remember When Interpreting Your Findings<\/h4>\n<p><img loading=\"lazy\" decoding=\"async\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/><\/p>\n<ul style=\"padding-left: 30px;\">\n<li>Interpret evaluation results with the goals of your program in mind.<\/li>\n<li>Keep your audience in mind when preparing the report. What do they need and want to know?<\/li>\n<li>Consider the limitations of the evaluation:<\/li>\n<\/ul>\n<p>Possible biases Validity of results Reliability of results<\/p>\n<ul style=\"padding-left: 30px;\">\n<li>Are there alternative explanations for your results?<\/li>\n<li>Have the different data collection methods used to measure your progress\u00c2\u00a0shown similar results?<\/li>\n<\/ul>\n<ul style=\"padding-left: 30px;\">\n<li>Are your results similar to what you expected? If not, why do you think they\u00c2\u00a0may be different?<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<h3>Module 4 Summary:<\/h3>\n<p><img loading=\"lazy\" decoding=\"async\" src=\".\/wp-content\/themes\/platformpro\/images\/line_separator.jpg\" alt=\"\" width=\"605\" height=\"2\" \/><\/p>\n<p>To reiterate, we restricted the content of this module to the practice of question-generation. We did so because the quality of the questions asked during the process of program evaluation influence the types of data collected and the interpretations and recommendations made. Use your program\u00e2\u20ac\u2122s purpose statement as the basis from which to ask questions about the effectiveness of programming and students\u00e2\u20ac\u2122 learning. When the answers to your questions begin to emerge, the next step in the process is identifying recommendations and actions for program improvement.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Module 4 &#8211; Analyzing and Interpreting the Data The previous module,\u00c2\u00a0Collecting Data on Students\u00e2\u20ac\u2122 Learning, dealt with the process of generating findings. Findings are merely the facts and empirical evidence of your investigation. To be of use in the process of curriculum evaluation, those\u00c2\u00a0findings will have to be interpreted so that\u00c2\u00a0judgments\u00c2\u00a0and then\u00c2\u00a0recommendations for improvement might be made. This module differs &#8230;<\/p>\n","protected":false},"author":19,"featured_media":0,"parent":857,"menu_order":4,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-33","page","type-page","status-publish","hentry","no-post-thumbnail"],"_links":{"self":[{"href":"https:\/\/egad.engineering.queensu.ca\/index.php?rest_route=\/wp\/v2\/pages\/33","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/egad.engineering.queensu.ca\/index.php?rest_route=\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/egad.engineering.queensu.ca\/index.php?rest_route=\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/egad.engineering.queensu.ca\/index.php?rest_route=\/wp\/v2\/users\/19"}],"replies":[{"embeddable":true,"href":"https:\/\/egad.engineering.queensu.ca\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=33"}],"version-history":[{"count":94,"href":"https:\/\/egad.engineering.queensu.ca\/index.php?rest_route=\/wp\/v2\/pages\/33\/revisions"}],"predecessor-version":[{"id":113,"href":"https:\/\/egad.engineering.queensu.ca\/index.php?rest_route=\/wp\/v2\/pages\/33\/revisions\/113"}],"up":[{"embeddable":true,"href":"https:\/\/egad.engineering.queensu.ca\/index.php?rest_route=\/wp\/v2\/pages\/857"}],"wp:attachment":[{"href":"https:\/\/egad.engineering.queensu.ca\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=33"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}