Graduate Attributes - McGill University

Loading...

Graduate Attributes Assessment and Reporting For 2016 Accreditation October 2015

Accreditation 2016 • 10 of 11 Engineering programs up for accreditation in 2016 • We “must demonstrate that the graduates of a program possess the [12] attributes” and “there must be processes in place that demonstrate that program outcomes are being assessed in the context of graduate attributes, and that the results are applied to the further development of the program.”

2

Graduate Attributes Graduate attributes are the qualities, skills and understandings a university community agrees its students should develop during their time with the institution and consequently shape the contribution they are able to make to their profession and society. They are qualities that also prepare graduates as agents of social good in an unknown future. (Bowden et al., “Generic Capabilities of Australian Technology Network University Graduates”, http:/www.clt.uts.edu.au/ATN.grad.cap.project.index.html, 2000)

3

Graduate Attributes - CEAB “Generic characteristics, specified by the Accreditation Board, expected to be exhibited by graduates of accredited Canadian engineering programs at the time of graduation.” 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12.

KB: PA: IN: DE: ET: IT: CS: PR: IE: EE: EP: LL:

Knowledge base for engineering Problem analysis Investigation Design Use of Engineering tools Individual and teamwork Communication skills Professionalism Impact of engineering on society and the environment Ethics and equity Economics and project management Life-long learning

The graduate attributes broadly define the required outcomes, but they are difficult to measure directly. 4

Indicators • To measure graduate attributes, all universities must define their own measurable descriptors called “indicators”: o “Descriptors of what students must do to be considered competent in the attribute; the measurable and pre-determined standards used to evaluate learning (i.e. measureable characteristics of attributes or components of attributes).”

Note: We are required to “measure” indicators, not GAs 5

Complete Process Defining Graduate Attribute Indicators

Curriculum Mapping

Collecting Data of Student Learning

Analysis and Interpretation

Datainformed Curriculum Improvement

• Core Graduate Attribute indicators are defined o

Some graduate attribute indicators will ultimately be customized for each department

• Pilot projects have been undertaken to evaluate how best to report results of Graduate Attribute assessment • Graduate Attribute curriculum maps have been prepared for each program • Graduate Attribute indicators must be assessed and reported for 2015-2016 academic year (as a minimum) 6

Objectives • Two principal objectives in developing accreditation procedures for the Engineering programs: 1) Maintain the accreditation of our programs in the new CEAB accreditation context 2) Do so with as little additional work imposed on instructors

7

Procedure •

Quantitatively assess all GAs at the level of “D” (Developed) or “A” (Applied) in a courses, in all years, for all students, by correlating performance in “assessment tools” with that in GAs / indicators



However, to get satisfactory data for all GAs in a given program, departments must also o use other approaches to evaluate “soft skills” GAs (e.g., IT, CS, PR, IE, EE, EP, LL) o employ other approaches to evaluate GAs other than just correlating GAs with graded work



A special emphasis on obtaining data for the “soft skills” GAs will be undertaken in FACC 100 and FACC 400 (which are common to all programs).

8

Procedure (cont.) • The faculty is providing general support to assist departments in meeting the necessary requirements to obtain, analyze and present their GA data. • However, every course, program and department is different, so the process cannot satisfy all situations o Adaptation and customization will be required by each department o CEAB guidelines and expectations are very vague  Although the faculty can provide guidance, professors and departments will sometimes need to use their judgment, as many questions do not have clear answers!

9

Overview Department Program General approach - infer GAs from assessment tools in courses (i.e. correlating grades with individual GAs)

myCourses (most courses, Faculty does the analysis)

Alternate approach - ensure good coverage of GAs is obtained - used to evaluate GAs by other means

Excel (analysis is the responsibility of the prof. / dept.)

10

Reporting Assessments myCourses Gradebook Setup Excel Format

General Approach Use myCourses for the majority of courses to correlate individual student performance in certain GAs with the various assessment tools used by the instructor • The gradebook setup can be done centrally, if requested by the instructor • Professors (only) to associate graduate attribute indicators with each graded item • Professors (or TAs) are responsible for entering data (grades) in myCourses at a level that is sufficiently refined to distinguish between different graduate attribute indicators • Analysis of the data will be done centrally

12

myCourses Setup Options Two options to setup myCourses gradebook 1) The gradebook setup can be done centrally, if requested by the instructor o Information is submitted through a general questionnaire

2) The gradebook setup can be done by the instructor or TAs o One-on-one consultation with professors will be available o Hands-on TA training will be offered in mid-October

13

14

Pre-Setup Tasks • Define the role of your TA: o http://kb.mcgill.ca/#tab:homeTab:crumb:8:artId:4131:src:article

15

Pre-Setup Tasks (cont.) • Define the gradebook rules: Student View Display Options

How to treat ungraded items

16

Information for General Setup • The assessment tools for your course • The grade distribution

17

Information for General Setup (cont.) • The grade distribution within each category: Example:

Assignments 9%

Final exam 45%



Even distribution



Distribution by grade point

Even distribution of grades Assignment#1 Graded out of 20 Assignment#2 Graded out of 80 Assignment#3 Graded out of 80 Assignment#4 Graded out of 50 Assignment#5 Graded out of 100

Distribution based on grade points Question#1 Graded out of 20 Question#2 Graded out of 20 Question#3 Graded out of 20 Question#4 Graded out of 60 Question#5 Graded out of 80

20% 20% 20% 20% 20% 100%

1.80% 1.80% 1.80% 1.80% 1.80% 9%

10% 10% 10% 30% 40% 100%

4.50% 4.50% 4.50% 13.50% 18% 45%

18

Information for General Setup (cont.) Determine the Graduate Attribute indicators for each component within a graded assessment tool

19

General Setup Step 1 – Create the assessment tool categories

Step 2 – Create the sub-categories (items) & linking the Graduate Attribute indicators

Graduate attribute indicators

20

Step 1 – Create Assessment Tools Categories

21

Example

22

Step 2 – Create the Sub-categories (items) & Linking the Graduate Attribute Indicators

23

Example Case1: The grade distribution

Case2: The grade for the

for the Final Exam is based on the grade points

Assignments category is evenly distributed between the items

Final Exam

Assignments

Question 1:

Assignment 1:

• • •

Graded out of 50 Graduate attribute indicator: KB.7 KB.7 - Recalls and defines information, first principles and concepts in specialized engineering science

Question 2: • • •

Graded out of 100 Graduate attribute indicator: PA.2 PA.2 - Develops models from first principles to analyze complex engineering problems

• • •

Graded out of 10 Graduate attribute indicator: KB.3 KB.3 - Recalls and defines information, first principles and concepts in the natural sciences

Assignment 2: • • •

Graded out of 100 Graduate attribute indicator: PA.1 PA.1 - Identifies and formulates complex engineering problems

Question 3: • • •

Graded out of 80 Graduate attribute indicator: PA.4 PA.4 - Critically evaluates the validity and accuracy of existing solution methods

24

Final Exam Question 1: • •

Graded out of 50 Graduate attribute indicator: KB.7

Question 2: • •

Graded out of 100 Graduate attribute indicator: PA.2

Question 3: • •

Graded out of 80 Graduate attribute indicator: PA.4

Assignments Assignment 1: • •

Graded out of 10 Graduate attribute indicator: KB.3

Assignment 2: • •

Graded out of 100 25 Graduate attribute indicator: PA.1

Gradebook

26

Remarks • Every course will be different, this is just an example • Inclusion of question weights is critical to the analysis • Prof/TA enters grades on myCourses • Analysis of the data will be done centrally

27

Exceptional Cases - Using Excel Excel can be used exceptionally • Courses that require more flexibility than myCourses can offer, e.g. FACC 100, Capstone design courses o Proper formatting is required o Professors are responsible for calculation of grades for each graduate attribute indicator o Professors are responsible for the analysis of data

28

Excel Setup • Example for the proper structure o First spreadsheet shows grades for all “assessment tools” and the associated indicators • Similar structure as in myCourses is expected, e.g.

o Each attribute is summarized in its own individual worksheet • Each indicator is recorded and presented individually • An overall calculation for the graduate attribute is recorded

29

Format – Excel Excel spreadsheet showing grades for all “assessment tools” and the associated indicators

Different indicators are evaluated on 1 assessment

Different indicators are associated with different assessments

Different questions on the exam are associated with different indicators

Each attribute is summarized in its own individual worksheet 30

Excel - Data Calculation Each indicator is recorded and presented individually

An overall calculation for the graduate attribute is recorded.

A typical Graduate Attribute 31

Excel – Data Analysis

32

Additional approaches • Departments will need to add to the general approach to: o Acquire sufficient quality data for the “soft skills” graduate attributes o Use alternate GA evaluation approaches to satisfy (the unclear) CEAB requirements regarding types and variety of measurement tools o Maria Orjuela-Laverde (Academic Associate, Pedagogical Coordinator) is available for pedagogical assistance to develop alternate GA evaluation methods

33

Additional approaches (cont.)

http://www.engineerscanada.ca/sites/default/files/intro_team_chair_presentation_english-sep19.pptx

34

Additional approaches (cont.)

http://www.engineerscanada.ca/sites/default/files/intro_team_chair_presentation_english-sep19.pptx

35

Recommendations • Recommendations to departments: o Examine your curriculum maps to look for “holes” in the data o Start by focussing on “Intro to XXX Engineering” course (or its closest approximation) and capstone design courses to complement your general approach for GA data acquisition o List GA indicator codes (e.g. “KB.1,” “PA.2,” etc.) for each question/exam sheet for all graded items (assignments, labs, exams) o List learning outcomes on all course outlines (a requirement for the revised course information sheets!) o Remember to collect appropriate sample materials for your courses.

36

Overview Department Program General approach - infer GAs from assessment tools in courses (i.e. correlating grades with individual GAs)

myCourses (most courses, Faculty does the analysis)

Alternate approach - ensure good coverage of GAs is obtained - used to evaluate GAs by other means

Excel (analysis is the responsibility of the prof. / dept.)

37

Learning Outcomes Course Information Sheet

http://www.engineerscanada.ca/sites/default/files/intro_team_chair_presentation_english-sep19.pptx

39

Course/Lab Learning Outcomes • “What the student is expected to have learned - These should be analogous to indicators and may have been used for GA assessment purposes. A small number of outcomes (general measurable skills acquired) are expected - not a list of content.” • “Laboratory learning outcomes - These should be analogous to indicators and may have been used for GA assessment purposes. What are the important specific skills and techniques the student is expected to have acquired from the laboratory experience - not a list of content.” 40

41

Faculty assistance • Setting up gradebook templates in myCourses • Analyzing and processing the data acquired via myCourses • Providing support (Nasim and Maria) to both professors and TAs • Preparing a survey for students in capstone design courses • Obtaining GA data o in common FACC courses o via exit survey of graduating students

• Establish a faculty-wide “program improvement” procedure based on GA data

42

Misc. comments • Some courses not on myCourses • For a “standard” course, the process has been designed so that the additional work required by a professor is small (as low as 2 hours per course) • Subsequent years will require less work (e.g. myCourses gradebook template can be imported from previous years)

43

Resources • Website: (http://www.mcgill.ca/engineering/faculty-staff/accreditation) o List of indicators

• Video clips o How to setup your gradebook

• Instructions o How to setup your gradebook

• TA Workshops o Two sessions were offered in September o Hands-on sessions will be offered in mid-October 44

Thank you!

Keywords for Learning Outcomes • • • •

In–Depth Knowledge Complex Engineering Problems First Principles Research

The Affected Graduate attributes: KB: Knowledge Base PA: Problem Analysis IN: Problem Investigation DE: Engineering Design CS: Communication Skills

Questions to consider for these GAs: • Does the course or components of the course: o address complex engineering problems/open-ended problems? o require the use of in-depth knowledge of mathematics, natural sciences, or engineering fundamentals? o require the use of in-depth knowledge of discipline-specific engineering? o require the development of solutions from first-principles? o require research for the development of solutions? o require research as an investigative tool?

46

Loading...

Graduate Attributes - McGill University

Graduate Attributes Assessment and Reporting For 2016 Accreditation October 2015 Accreditation 2016 • 10 of 11 Engineering programs up for accredita...

3MB Sizes 17 Downloads 12 Views

Recommend Documents

Mapping Graduate Attributes - University College Dublin
Jan 13, 2010 - O'Neill, G, 2009. A Programe wide approach to assessment; A reflection on some curriculum mapping tools.

Nursing | Future Graduate Students - McGill University
Nurse applicants with a Master's degree in Nursing and with the required clinical experience are prepared for nurse prac

Capturing technical competencies and graduate attributes developed
Capturing technical competencies and graduate attributes developed during work placements using online reflection and ep

Can d - McGill University
SUFISM IN INDONESIA: NAWAWÏ AL-BANTENI'S SALALIM AL-FUDALA' ... a great influence on the people of his country, especia

Contractual Obligations - McGill University
Stephen A Smith, Contract Theory (Oxford: Oxford. University Press, 2004) at 56-60; 78-85. - Pierre-Gabriel Jobin with t

Read - McGill University
º A discussion of the spectrum of obligations (moral, natural and civil) will be discussed in more detail infra, at not

1 - McGill University
activities of dakwah: (propagation), and a great number of educational institutions and social ...... apparent expansion

Alfonso X - McGill University
Ridwān, the Libro conplido en los iudizios de las estrellas (Kitāb al‐Bāriʿ fī aḥkām al‐nujūm) by ʿAlī ib

Text - McGill University
See also Harun Nasution, "The Islamic State in. Indonesia : The Rise of the ldeology, the .... Indonesian under the till

McGill University Pension Plan
Apr 9, 2015 - the year ended December 31, 2014 and the independent auditor's report thereon;. (d) receiving the stewards