PSC & NCATE

The American Association of Higher Education’s 1995 Bulletin defined assessment as:

In keeping with this definition, the College of Education has developed an assessment system in which (1) the unit and its programs are aligned to national standards; (2) the unit and its programs systematically collect, analyze, and interpret evidence not only on candidates and programs, but on faculty and the overall operation of the unit to ensure the delivery of quality education; and (3) it is embedded effectively within larger institutional systems at the university, Board of Regents, and state levels. While the unit began to develop its assessment system in 2000, this effort has been ongoing and continues to be reviewed, revised, and centralized. In an organizational structure that previously included nineteen departments each with multiple programs all engaged in different ways to assess candidate performance, the unit’s challenge has been to streamline and centralize both the unit structure and its assessment system over time. The unit and its faculty and administrators in collaboration with arts and sciences faculty and P-12 professionals has worked effectively to build a unit assessment system.

Comprehensive assessment has been a strong commitment across the university for many years. In the early 1990s, the university began the development of a comprehensive assessment system for all undergraduate and graduate majors. One aspect of that system was the requirement that each program develop an assessment plan that included a set of student outcomes and key assessment measures that would be reported to the university as part of its regular program review cycle. These plans were submitted and reviewed by the university in 2000. At the college level, prior to this and prior to the development of the NCATE 2000 standards, candidate and program assessment was left largely to individual programs with little assessment at the college level.

Then in 2000, the college began a major effort to develop its assessment system in preparation for our 2001 continuing accreditation visit as a pilot institution using the NCATE 2000 standards. At that time, every program began an intensive process intended to yield a plan to align current assessment practices with PSC and NCATE standards as well as national professional specialty association standards. That process was designed in such a way as to graphically indicate to the departments and to the college as a whole exactly what candidate outcomes each program had already identified and which standards did not yet have outcomes associated with them. Further, the process was designed to identify outcomes being assessed and whether the assessments were traditional in form (GPA, course grades) or performance-based assessments. Through that process, departments and programs began to elaborate further outcomes, to refine and rethink existing outcomes, and to assure themselves that all of the PSC and NCATE standards were addressed through specific, measurable outcomes (2001 assessment documentation). The NCATE leadership team, a group of faculty and administrators, then worked to design and implement the assessment system.

Unit Leadership/Management Structure for Improvement of Programs and Unit Operations

The unit has an assessment system developed with its professional community that reflects conceptual framework and professional and state standards.

Following our 2001 PSC and NCATE continuing accreditation visit at which the unit’s assessment system was found to meet the standards with no areas for improvement, the assessment system was further refined and developed. Over the next four years, more centralized assessment data was designed and implemented with strong emphases on performance-based measures at both the program and unit levels. This system has become part of our regular unit and department operations. The NCATE Leadership Team, currently comprised of the Associate Dean for Academic Programs; Director of Student Services; Director of Technology and other technology staff; and faculty program coordinators from four of the nine departments, assisted with the continued refinement of the system and sharing with the professional community. In 2006, the Professional Education Advisory Committee established an assessment committee that took up the work of the NCATE leadership team. This committee provided leadership over the past three semesters to evaluate and refine the assessment system. As described in the conceptual framework portion of this report, the work of this team resulted in the Georgia Framework for Educators. The committee led the effort toward a common evaluation scale across the unit for use by all programs as well as a common student teaching/clinical evaluation instrument for all educator preparation programs. While these efforts are still ongoing, much progress has been made toward this end. It is expected that a common evaluation instrument for teaching candidates and another for other school personnel will be implemented during AY2008. Due to its focused attention on assessment, the unit is viewed across campus and at the Board of Regents P-16 Office as a leader in assessment and accreditation.

The unit’s system includes a comprehensive and integrated set of evaluation measures that are used to monitor candidate performance and manage and improve operations and programs.

Our unit assessment system is reflective of the conceptual framework as well as state and national standards. This system provides regular and comprehensive information on applicant qualifications, candidate proficiencies, competencies of graduates, unit operations, and program quality and uses the data and its analysis for the purposes of program improvement and improvement of unit operations. At the core of the process is both reflective practice and attention toward improvement of our practices. The following diagram illustrates our continuous assessment and improvement cycle. The assessment system begins with an alignment to unit, state, and national standards.

All educator preparation programs are aligned to their national specialty professional association standards or the Professional Standards Commission (PSC) standards where national program standards have not been created (such as gifted and creative education). These programs must also be aligned to the University System of Georgia Board of Regents Principles for counselors, leaders, and teachers. At the unit level the programs are aligned to the unit’s conceptual framework as evidenced in their assessment systems.

The next component of the assessment system is the regular collection, analysis, and interpretation of data at both the unit and program levels. Assessments at all levels are designed to reflect the standards described above and are examined for fairness, validity, and reliability. Assessment reports are required of all programs and the unit on regular cycles, and these reports are evaluated at the unit, university, and/or state/national levels. The final component of the unit assessment system is the use of these assessments for improvements of programs and unit operations.

The unit’s assessment system is comprised of four major components to measure the following:

Figure 2.1 Unit Assessment System Flowchart for Candidate Quality, Program Quality, Faculty Quality, and Unit Budget & Resources for Quality Programs

In this section we detail the key assessments included in each of these four major components illustrated with tables summarizing each of the assessments within those components. These tables describe the level, focus, cycle, and responsibility for monitoring and evaluating the assessments within the system as well as their alignment with the conceptual framework and NCATE standards. This system uses multiple assessments that are regularly and systematically summarized and analyzed to improve program quality and, consequently, to improve candidate quality. Since 2000, when the college decided to pilot the NCATE 2000 standards, efforts have been aimed toward coordinating all assessment systems required for the university, state, and national associations into one major. Technology is used throughout the system as a tool to collect, aggregate, and evaluate data across the programs and unit.

Assessment System for Candidate Knowledge, Skills, and Dispositions: Initial and Advanced

In 2007 through the work of USG Board of Regents EPAAC’s Education Deans’ Group, the unit provided leadership in gaining consensus on the decisions points that would be used by all USG educator preparation programs. The deans’ group unanimously adopted these decision points after involvement of their faculties and P-12 partners. Hence, while slightly revised from the 2006 decision points, the unit and its programs are in alignment with the Board of Regents’ policies and use the following decision points to evaluate prospective candidates as well as to evaluate candidates’ progress through their programs of study:

During the course of each educator preparation program, candidates are expected to successfully pass program-based assessments during courses and field experiences for teachers or clinical practice for other school personnel at these decision points. Each program has designated six to eight program-based assessments as part of requirements for the program review process to the specialized professional associations (SPA) or for program review at the Georgia Professional Standards Commission (based on national SPA standards). These key assessments include, in addition to the state’s GACE licensure examination, assessments of content knowledge, clinical practice, planning, effects on P-12 student learning, and dispositions. These assessments can be reviewed within the PSC’s web-based Professional Annual Assessment Report (PAAR). While a recently developed system, these are the same assessments used in previous SPA and PSC reviews. Specific program assessment instruments, aggregated data, and examples of data-based program improvements are available with the program review folders and on the accreditation site.

Unit-Level Review of Initial Candidates

The unit’s educator preparation programs are at both the undergraduate (B.S.Ed) and graduate (M.Ed) levels. The following describes the typical teacher education assessment at the initial level. For other school personnel at the initial level such as school counseling, the decision points remain the same while the assessments vary somewhat by program.

Candidates applying for admission to initial teacher preparation programs must apply by the date specified and provide the following evidence of knowledge of basic skills and content:

The Student Services Office, a unit of the Dean’s Office, reviews all candidates applying to initial programs at the bachelors level to ensure that all requirements are met prior to admission. A form to document successful completion of these entry requirements is used with each candidate and stored in their advising files. This office then admits those qualified students to programs that are not identified as high demand majors (career and technical education, mathematics education, science education). For programs identified as high demand majors, program coordinators and faculty have an additional set of requirements that must be met and for many of these programs the application process is highly competitive with only the most qualified gaining admission (early childhood, English, foreign language education, health and physical education, pre-K to 2nd , middle grades, social studies, special education). Again, the Student Services Offices ensures that candidates for all programs, open and high demand, meet these requirements for entry into the programs.

Currently, a 2007 initiative of the PEAC Assessment Committee is focused on a candidate self assessment for initial teacher candidates that will be administered during the Area F pre-education courses. This assessment measure is aligned with the self-assessment components of the Exit Self-Assessment and Satisfaction Survey (see description below) so that the unit can track candidates’ reflection of their own growth from entry to exit in our programs. While not an assessment designed to be used in the decision to admit the candidates, this measure will be shared with program faculty to assist with professional growth plans to build candidates’ knowledge, skills, and dispositions over the course of the program.

Approval for student teaching provides a key assessment point for initial candidates within teacher education programs. For approval to student teach, candidates must make application to the Student Services Office and be approved by their department. Each department checks to ensure students have met the necessary program requirements and signs formal recommendations to the Student Teaching Office for placement in the student teaching experience. The departments review all teacher education candidates prior to entry to student teaching placement to ensure completion of appropriate program requirements including successful completion of required courses and satisfactory grades in field experiences, maintenance of a minimum 2.5 cumulative GPA. The Student Services office facilitates the criminal background check process, which requires the school district in which the candidate is placed to review and clear the criminal background checks.

Program completion begins with successful completion of all program requirements including satisfactory assessment of student teaching, maintenance of 2.5 cumulative GPA. At program completion, all candidates have a recommendation for certification evaluation. After initial candidates have applied for graduation, the Registrar’s Office issues a graduation Degree Audit Report (DARS) listing all coursework completed including accepted transfer courses. This form comes to the Student Services Office after having been evaluated for completion of the university’s general education requirements. These evaluations are then sent to the candidate’s faculty advisor who certifies that the candidate has completed the approved teacher preparation program, has the appropriate GPA, and is considered ready for certification and when appropriate graduation. The unit’s Certification Officer reviews each evaluation and accompanying certification application when submitted.

Exit Self-Assessment and Satisfaction Surveys. Candidate self-assessment and program evaluation occurs at the end of each semester following completion of the student teaching experience or for advanced programs at the completion of their programs. As part of a college-wide effort initiated in spring 2000, data were systematically collected during student teaching using a survey of all candidates regarding their reflections on their own knowledge, skills, and dispositions as well as their satisfaction with a number of aspects of their education. The survey was developed and the data analyzed by Educational Benchmarking, Inc. (EBI). The EBI survey was used for three years as the unit developed its own exit self-assessment/satisfaction survey for teacher candidates in 2003-04 that was aligned to the conceptual framework, including the newly developed GSTEP Framework for Accomplished Teaching and PSC/NCATE standards. This survey allows for feedback and evaluation on the extent to which candidates believed they were prepared to enact the conceptual framework goals, on the quality of program coursework (including subject area content in arts & sciences), fieldwork, faculty quality, and qualitative feedback on the teacher education program. Since 2000 these data were shared with program faculty and used for program improvements. All data are aggregated at the program and unit level. Program coordinators typically report these data in their undergraduate assessment reports and in the PSC PAAR system. Administrators within the unit, the PEAC Assessment Committee, department chairs, and their faculty review the submitted reports. Portions of exit survey results are included in the university annual reporting process. Since the PEAC Assessment Committee developed the more inclusive GA Framework for Educators in 2007, a new exit survey based on this framework has been developed and reviewed for quality and will be used at the end of each semester by all educator preparation programs. This self-assessment/satisfaction survey will be piloted in fall 2007 with those exiting candidates in initial certification programs across the unit.

PRAXIS II /GACE II State Licensure Examinations. Prior to certification, initial candidates must pass the GACE state licensure subject assessment examination (formerly used PRAXIS) in their particular content areas. Passing this standardized test is required by the state of Georgia for certification and is reported for HEA Title II purposes. At the unit level, these data are tracked for each completer, and summarized at the program and unit levels. Prior to 2000, the unit created an electronic database for compiling and searching PRAXIS data that is sent from the Educational Testing Service (ETS). Since the state moved to the GACE, this database has been reconfigured to accommodate the GACE test results for both GACE I & II. In addition, the Associate Dean for Academic Programs unit established a clear process for tracking program completer licensure examination scores and aggregating those scores at the program and unit levels. PRAXIS/GACE aggregated scores are shared and reviewed by the Dean’s Cabinet, department heads, program coordinators, and PEAC assessment committee. These data are reported as part of the university annual report, program reports (university, SPA, PSC PAAR), and used for the U.S. News and World Report.

Unit-Level Graduate and Employer Surveys. In 2004-05, the unit developed and implemented unit-wide alumni and employer surveys to assist the unit in examining its programs from the perspective of former candidates and their current employers.

These surveys were both aligned to the GA Framework and NCATE standards. Using facilitate.com software, these web-based surveys were sent to graduates and employers in the educator preparation programs. Due to a low return rate the surveys were revised, reviewed extensively, and were piloted in 2007. Efforts were made to communicate more effectively with school districts through the Dean’s Superintendents Advisory Council as to the importance of these surveys. In addition, the unit provided leadership to the USG Board of Regents through its assessment committee to use these surveys across the system for all USG educator preparation programs. In 2006-07 the surveys developed at UGA were reviewed and refined based on feedback from faculty and P-12 educators across the institutions. The surveys were approved at the August EPAAC Education Deans Retreat and will be administered to all completers (one year out) and their employers by the Board of Regents’ P-16 Office starting spring 2008.

Unit-Level Oversight of Board of Regents Guarantee. Since the unit and its programs must assure that our candidates in all educator preparation programs are highly qualified professionals, it provides oversight in any case where this guarantee is invoked. If a candidate is deemed inadequate, the unit works with the department head and program faculty to facilitate a process for remediation of that particular candidate. To date, only one candidate several years ago was returned to UGA for remediation.

Unit-Level Review of Advanced Candidates

Although individual programs may have additional requirements, a minimum set of admission standards for full graduate standing for any education program at UGA includes a bachelors degree, three letters of recommendation, a 3.0 undergraduate grade point average, and GRE or MAT scores that are considered “respectable” as set at the program level. Each program has additional requirements, and initial certification candidates must also pass or exempt PRAXIS/GACE I. Typically to qualify for admission to advanced programs for continuing teachers, candidates must have a minimum combined score of 850 (M.Ed.) and 900 (Ed.S.). To qualify for admission to the unit’s Ed.D. or Ph.D. programs, candidates must typically have combined GRE scores of 950 and 1000 respectively. Programs may set their qualifying scores slightly higher than this if they choose. For example, potential candidates applying to the social studies education program must have a combined GRE score that totals at least 1100.
Candidates apply for admission through the UGA Graduate School using an on-line application form. After review by the Office of Graduate Studies, all admission materials are forwarded to the graduate coordinator for the specific program. The graduate coordinator and program faculty review the application materials, including interviewing the candidate if appropriate, and makes a recommendation regarding the candidate’s qualifications and fit with the program. The graduate dean reviews and approves or disapproves all application materials and the department admission recommendation. The Graduate School then contacts the candidate by letter providing admission details.

During the course of each advanced program, candidates are expected to successfully pass program-based assessments during courses and field experiences. Programs have designated six to eight program-based assessments as part of requirements for the program review process to the specialized professional associations (SPA). All advanced programs are aligned to national SPA standards. These key assessments include assessments of content knowledge (in addition to the GACE state licensure examination where appropriate), effective planning, impact on P-12 student learning, and additional program-based assessments. Assessments, aggregated data, and program improvements are available with the program review folders, graduate assessment reports, and university program reviews.

A preliminary program of study, developed by the major professor and the doctoral student and approved by a majority of the advisory committee, is submitted to the graduate coordinator by the end of the student's first year of residence. The program of study should consist of 16 or more hours of 8000- and 9000-level courses in addition to research, dissertation writing, and directed study.

A final typed program of study is submitted to the Graduate School prior to notification of the comprehensive examination. This program of study must be submitted by the student on the proper form for approval by the advisory committee, the graduate coordinator, and the dean of the Graduate School. The final program of study must show all graduate courses relevant to the doctoral program and not just courses satisfying the minimum degree requirement. Courses from the master's degree and courses taken at other universities should be listed in the "Relevant Master's or Other Graduate Degree Courses" section of the program of study form. The program of study must carry a minimum of 30 hours of course work, three hours of which must be dissertation writing (9300).

The student is responsible for initiating an application for admission to candidacy so that it is filed with the dean of the Graduate School at least one full semester before the date of graduation. This application is a certification by the student's major department that the student has demonstrated ability to do acceptable graduate work in the chosen field of study and that:

After admission to candidacy, a student must register for a combined total of ten hours of dissertation or other appropriate graduate credit during the completion of the degree program. Students planning to graduate the same semester they enter candidacy must be admitted to candidacy by the published deadline for candidacy during that semester and register for ten hours. The student must also meet all other deadlines for graduation in that semester. A student must register for a minimum of three hours of credit in any semester when using University facilities, and/or faculty or staff time.
Once a student has been admitted to candidacy, the department has an ethical responsibility to ensure that appropriate faculty mentorship is provided to the candidate for completion of the degree.

If the advanced candidate is seeking certification or endorsement (ESOL, Gifted & Creative, Reading, L-5 Leadership), internship assessment, GACE II, and recommendation for certification are pursued in a similar manner as indicated above for initial candidates. All graduate degree candidates are required to have completed a comprehensive evaluation before they are approved for graduation. The type of assessment varies based on program requirements and includes such options as professional portfolio, thesis, and/or comprehensive examinations. The graduate coordinator for each program ensures the comprehensive evaluation procedures are followed.

Exit Surveys. In the past, the departments have been responsible for the administration of the advanced candidate exit surveys. In addition, the data from these surveys are aggregated at the department level. As reported above, an exit survey based on the Framework for Accomplished Educators has been developed for use across the unit beginning in fall 2007.

Graduate and Employer Surveys. These surveys are administered by programs and/or departments and are reported to the unit and the university through the university’s Graduate Assessment Report. They are also used to report annual progress in the departmental annual reports submitted to the College.

Assessment System for Program Quality

The unit assesses its program through a variety of formal processes that include reviews at the national, state, and university levels.

National Level Assessments and Evaluations. At the national level, the unit participates in the evaluation of its programs through national specialty association reviews on regular cycles that vary by discipline. In addition, the unit has maintained continuous accreditation with NCATE since 1954. The unit has 12 programs that are approved for national recognition. Also at the national level, the unit participates in regular assessment and reporting to the U.S. News & World Report survey and the federal Title II Report, both of which enable us to track performance measures across time at the unit level.

Regional and State Level Assessments and Evaluation. At the regional level, the unit as part of the university participates in Southern Association of Colleges and Schools (SACS) reviews on a 10-year cycle. The unit participates in the planning and data generation for this effort. For example, two administrators in the college are on SACS committees chaired by the Vice President for Institutional Effectiveness in preparation for the 2009-2010 SACS review. This level of participation enables the unit to ensure that the unit-level assessments are aligned with regional standards and evaluation system and that all faculty are highly qualified. The unit’s assessment system and electronic accreditation site are recognized as an accreditation model at the university level. The unit participated as a pilot for the new electronic UGA SACS Faculty Qualifications system to ensure that all faculty are appropriately qualified to teach the courses they are assigned.

At the state level, the Professional Standards Commission accredits all educator preparation programs on a seven-year cycle in conjunction with the NCATE review cycle. Our programs have maintained continuous accreditation through the state process and all were awarded continuous approval during the 2006 review. With exception of an L-5 certificate program, all programs met all 7 state standards, including Standard 2, assessment system.

Another assessment at the state level that enables the unit to track the number of program completers in each of our programs and across the college as a measure of unit effectiveness is the annual submission of the completers report. These data enable us to track our candidates (teachers, leaders, school counselors, school psychologists, school library media specialists, and speech-language pathologists) into the field if they are employed by the state of Georgia in public schools. Most recently through the submission of our completers report to the Board of Regents Data Mart system, the unit is able to create reports that track our completers, their contact information, place of employment, position, and grade level. This newly created web-based system will support and enhance our ability to conduct induction activities, as well as follow up surveys with candidates and their employers. This system enabled the unit to initiate unit-level alumni and employer surveys on our candidates, an effort that was done at the program level and not consistently across all programs. The Board of Regents’ P-16 Office is in the process of generating institutional reports on retention rates and other measures that are due in October, 2007 for unit review. The unit has served as a leader in assisting the Board in the refinement of this system, especially in the area of data definitions.

Another state level program assessment is the University System of Georgia Board of Regents (USG BOR) required annual assessment report of educator preparation programs in counseling, leadership, and teaching to evaluate the extent that they meet the BOR principles. This assessment began in 1999 and continues on an annual basis. Each BOR report is evaluated by BOR staff and assessed on a four-point scale. Data on progress toward meeting all BOR Principles are then summarized and reported to the USG institutions as well as to the BOR at meetings annually. This annual assessment of programs has enabled the unit to examine the performance of the unit and its candidates, as well as input and results of its programs on P-12 student learning. The BOR reports enable the unit to use unit-level data to assess its programs in response to BOR principles and document the improvements made to programs as a result of this assessment. The unit and the BOR reviews of these reports demonstrate continual improvement in our educator preparation programs. As part of this process, the unit must assure that it is meeting the BOR guarantee of the quality of its candidates for the first two years of induction into the profession.

University-Level Assessments and Evaluations. The university has a comprehensive system of unit and program assessment and review that is part of the unit’s overall assessment system. The UGA Office of Institutional Effectiveness provides oversight for the university’s assessment systems. This office also provides oversight for the university’s strategic planning process and is responsible for the university’s annual reporting process. The College of Education’s Associate Dean for Academic Programs interacts closely with this office in the assessment and accreditation processes. There are five major components of the university’s assessment of its academic units: (1) Outcomes-based strategic planning and annual reporting; (2) Undergraduate Major Assessment Process; (3) Graduate Major Assessment Process; (4) Seven-Year Program Review of Academic Units; and (5) University Academic Advising Survey. Each of these is elaborated below.

(1) The university requires that all units on campus have a strategic plan in place that articulates the unit’s mission, vision, and overall goals from 2000-2010. In 2004, the university administration required that all units review their strategic plans and create a 5-year plan with specific outcomes and performance measures to assess those outcomes. Each year, the unit collects and analyzes performance measures for each of the outcomes articulated in the strategic plan. For much of this process, the web-based Faculty Activity Report (FAR) database is used. Additional program- and unit-level data are maintained through the use of spreadsheets and aggregated at the unit level. These data are analyzed and reported in annual reports submitted to the university and provide the unit with opportunities to assess the extent to which it is meeting its strategic goals.

(2) The Undergraduate Major Assessment Process, instituted in 1993, is designed to use outcomes assessments to “seek reliable, useful evidence about student learning in an academic program and to use that evidence to improve delivery of the program.” The process works according to the criteria and policies established by the faculty Program Review and Assessment Committee (PRAC) and requires program faculty to (a) define clear learning outcomes for the candidates in their programs, (b) identify and implement measures that assess whether their students attain those outcomes, (c) analyze the data gathered through the assessment measures, and (d) use that resulting information as the basis for program improvement. Faculty members from the College of Education regularly serve on the PRAC and participate in the review of academic units and their assessment reports. Program faculty report on their undergraduate major assessment systems on a three-year cycle that is integrated into the Seven-Year Program Review Cycle described below. As part of this process, each program in the university developed and submitted an assessment plan in 2000 that was reviewed both at the unit level and the university level. Feedback on these plans based on an evaluation rubric was provided at both levels. Since the initial plans were submitted, programs have submitted annual reports on their continuous assessment and review of their programs. The annual reports are reviewed at the unit level in the Dean’s Office and at the university level by the PRAC with feedback returned to the unit and programs.

(3) Similarly at the graduate level, beginning November 2000, the Graduate School instituted a Graduate Major Assessment that required all programs to submit the student outcomes expected in the program, assessment measures, analyzed data, and data-based program improvements. The annual reports are reviewed at the unit and university levels. Like the undergraduate major assessment system, the graduate assessments are on a three-year cycle. The Office of Institutional Effectiveness now provides the administrative support to the Graduate School for the administration of this process. College of Education graduate assessment reports were last due November 1, 2005.

(4) The Seven-Year Program Review of Academic Units is an intensive process that is designed to provide “a systematic means of assuring the continuous improvement of the University’s academic, research, and service programs. It should incorporate and build on related, ongoing processes such as annual reporting, learning outcomes assessment, strategic planning, accreditation review, and Five-Year Program Planning. This program review involves a year of self-study that culminates in a written report to the Office of Institutional Effectiveness. A committee consisting of faculty from other units in the college as well as from other colleges on campus conducts a thorough review of the self-study report as well as a multi-day onsite review consisting of interviews with administrators, faculty, staff, current candidates, and graduates. A final report with explicit recommendations is submitted to the Senior Vice President, the Associate Provost for Institutional Effectiveness, and to all administrators with oversight of the unit. A meeting is held with key stakeholders to discuss the final report. One year following the completion of the review, the unit head submits a follow-up report on the unit’s progress in responding to recommendations in the Final Report. In AY2005, three departments in the College of Education went through this review cycle; an additional three departments were evaluated in AY2006, and the remaining departments were reviewed in AY2007. This process is facilitated and monitored at the unit level by the Associate Dean for Academic Programs who provides program and department data from the unit’s assessment system as needed for the self-studies and consults with department chairs as needed both during the self-study period, and prior to university-level meeting to respond to recommendations. In addition, this office nominates one faculty member to serve on each program review team.

Assessment System for Faculty Quality

The unit is committed to recruiting and retaining high quality faculty who are exemplary and reflective teachers and researchers and also serve their professional communities at the local, state, national, and international levels. The unit’s assessment system includes a strong component focused on faculty quality that involves both university level as well as unit level assessments. The assurance of high quality faculty begins at the recruitment and hiring stage. All potential faculty candidates are screened thoroughly before hiring through a rigorous credential review, letters of recommendations, and a two-day interview process including a research/teaching presentation. Each faculty search process is provided unit oversight by an associate dean to ensure appropriate process and selection of the highest quality candidate from the best possible pool.

Student Evaluations of Faculty. The college routinely collects course and instructor evaluation data from candidates each semester on every course taught. These evaluations are completed during the last two weeks of the term. If using paper versions, Test Scoring and Reporting Services scans the forms, and the results are returned to the individual faculty with a copy to his or her department head. If electronic, the Office of Information Technology aggregates the online course evaluation data and feeds it automatically into the Faculty Activity Report. Data for both versions (paper and electronic) are in the form of numerical summaries (means, standard deviations, and percentages) for each item plus any written comments the candidates provided. Since before 2000, the unit has maintained unit-wide summaries of these data. The summaries are mean faculty ratings across all items for departments and for the college. These data are used within the college to study teaching effectiveness as viewed by the candidates in these courses (see standard 5.2). These faculty evaluation data are used in annual reviews of faculty performance as a measure of quality and to assist with the determination of merit pay within departments and at the unit level. These data are also used as part of the information collected when new faculty members undergo third-year review and become part of a faculty member's dossier for promotion and tenure.

In addition to the more traditional paper format for evaluations of faculty, candidates have access to online course evaluations. This option for course evaluation was developed in 2003 to support online delivery of courses. The course evaluation was designed using evaluation suggestions from the Faculty Senate. A faculty committee finalized the instrument, which includes standard questions, instructor-developed items, and open-ended response options for candidates. Use of the evaluation instrument has grown over the last three years and includes evaluations of online courses and face-to-face or mixed-delivery courses. The evaluations are captured and used for program and instructional improvement opportunities. The COE course evaluation has been implemented or used as a design template by three colleges at the university: Terry College of Business, Franklin College of Arts and Sciences, and the School of Forestry.

In 2007, the Faculty Senate began an initiative to review and revise the current faculty course evaluations and a committee of this body is expected to propose a new format for adoption during the current academic year. The new format has been piloted and is now undergoing psychometric testing to ensure validity, reliability, and fairness.

College and University Faculty Evaluation Process. As per university and Board of Regents policies, faculty undergo regular review through annual reviews, third-year reviews of untenured faculty, a promotion and tenure process, a graduate faculty review process, and a post-tenure review process. The Dean and Associate Dean for Faculty and Administrative Services provide oversight of all faculty evaluations within the unit. Departmental-, college-, and university-level committees review faculty in each of these review processes. The Associate Dean for Faculty and Administrative Services sits on the College Promotion and Tenure Review Board to ensure compliance with university, college, and departmental policies. In addition, her office conducts periodic studies focused on the success rate of candidates requesting promotion and tenure and identifies any issues or concerns related to this process.

College Annual Faculty Evaluations. The unit developed a web-based Faculty Activity Report (FAR) that is used annually to assess faculty accomplishments in teaching, research, and service. This tool has been in place since AY2003 and provides a mechanism to aggregate faculty activities and accomplishments at a departmental and college level. While an annual evaluation of faculty has been in place in the college for many years, this FAR system enables better departmental- and unit-level review. Each spring, faculty are asked to submit their FARs to department heads who then write evaluations of individual faculty members as part of an annual review process used for with merit-raise increases. In addition to the unit requirements for annual faculty evaluations, each department has developed more specific guidelines for evaluating faculty effectiveness in teaching, research, and service with some utilizing a peer-review process and others relying solely on the department head’s review. Department heads share those results with the college administration as merit-pay determinations are made. The web-based FAR enables the college to aggregate data at the department and college levels for reporting specific performance measures in the annual report submitted to the university (described above under strategic plan) as well as other required reports. As a result of faculty feedback on the FAR system, the unit totally revised the system in 2006 and is continuing to revise it in response to user feedback. This system is a web-based queriable system that enables easy access to aggregated faculty data across performance in all areas of teaching, research, and service.

Assessment System for Unit Operations and Resources

In order to monitor and ensure that the unit has adequate resources to support its programs, it has incorporated in its assessment system a component on Unit Operations and Resources. The Associate Dean for Faculty and Administrative Services provides oversight for the unit and departmental budgets. Her office regularly meets with department heads to ensure adequate resources for quality programming. The Associate Dean for Research and Outreach provides oversight for all resources related to external courses and programs (i.e. online, Gwinnett and Griffin campuses).

Figure 2.5. Unit Budget and Resources for Quality Programs Assessment Flowchart

Undergraduate Academic Advising Survey. A university-wide assessment process is the Undergraduate Academic Advising Survey conducted annually by the Office of the Vice President for Instruction. This survey, initiated in 2004 and administered each spring semester, examines student satisfaction of the quality of academic advising for UGA’s undergraduate candidates. Results for individual units on campus are reported to the unit heads for review and to address areas for improvement. The unit incorporated this assessment into its system to monitor student satisfaction on advising within the college. In response to the low student-response rate in the College of Education, the unit took steps to increase response rates by providing computer terminals in the Student Services Office and requesting candidates to complete the survey when they arrived for their advising appointments, as well as sending out multiple reminders and encouragements to complete the survey. The college’s return rate increased from 4% in 2004 to 19% in 2005. With the new computer terminals installed in summer 2005, a higher return rate was anticipated, however it continued to be 19% on the 2006 advising survey. The college did see an increase in FY07 when the return rate was 25%.

Unit and Departmental Budget Spreadsheets. The Assistant to the Dean who provides oversight to the Business Office developed an electronic database system to enable the college to regularly track finances within and across departments and at the college level. This system was developed since the 2001 NCATE visit and has revolutionized the way the college operates fiscally. Quarterly reports of all college budgets are prepared by the Business Office and reviewed by the Dean’s Cabinet. Multiyear analyses of data from quarterly reports are conducted periodically. Most recently such analysis led to the adjustment of O&E allocation to each department to reflect net gains or losses of faculty lines. The system has been a model for other colleges across campus and at other universities.

Credit Hour Production Reports. Annual tracking of credit-hour production and the ratio of credit-hour production to faculty EFT at the department and college levels is conducted to assess this measure of productivity. Using data from the Office of Institutional Research, the unit aggregates credit hour production data by faculty, program, department, and college. These data are tracked over time, shared with department heads and faculty and used to make critical resources decisions at the unit level.

Research Funding Tracking. Annual tracking of research funding divided by faculty EFT to assess the amount of research funding per faculty.

Faculty Database by gender and ethnicity. Because of the unit’s commitment to diversity and its recruitment efforts, faculty are continually tracked by gender and ethnic diversity at both the department and college levels.

Unit Field Experience Database. The unit provides oversight for field- experience placement to ensure that all candidates have experience with P-12 students from diverse cultural backgrounds and with special needs.

Unit Classroom Level Diversity Survey. In spring 2004, the unit began the survey of classroom diversity in field experience and clinical practice. Each semester, candidates complete this one-page survey that asks for both mentor teacher/clinician diversity as well as P-12 student diversity. The data is collected and analyzed at the college level.

Survey for Student and Faculty Technology Access, Use, and Performance. This survey was developed and administered in fall 2005 to assess the extent to which faculty and candidates in the college have access to and use technology in their teaching, research, and service. This web-based survey is used to review the technology integration in the unit.

Survey of Technology Integration in Courses. This survey was conducted in spring and summer 2005 to assess the extent to which technology was integrated throughout the courses.

OIT Assessment of Services. In 2002, The Office of Information Technology (OIT) conducted an assessment of the service and training aspect of its services. This resulted in a streamlined organization with service teams reduced from four to three and the appointment of team leaders in (1) instructional services, (2) applications and special projects, and (3) client support and infrastructure.

The unit consistently reviews its assessments with an eye to fairness, accuracy, consistency, and lack of bias. Key assessment are developed and reviewed by key stakeholders, including P-12 faculty, and piloted before they are considered fully implemented into the unit’s assessment system. The following brief summaries are provided as specific examples of the unit’s steps to insure quality assessments:

Georgia Framework for Accomplished Teaching: This framework went through extensive scrutiny, focus groups, and a validation study to ensure the quality of the instrument. See http://www.coe.uga.edu/gstep/documents/Framework_History.pdf for details.

Georgia Framework for Accomplished Educators: This more inclusive version of the framework went through an 8-month development review cycle to ensure its quality. After the language of the framework was made more inclusive than the Georgia Framework for Educators, it was reviewed in focus groups of P-12 teachers, teacher educators, and P-12 school personnel (school psychologists, counselors, and leaders) for further refinement.

Exit Self-Assessment and Satisfaction Survey Quality: A table of specifications was prepared to ensure the alignment between the survey and the unit’s conceptual framework. The new exit survey was piloted in spring 2004 and at that point was reviewed for fairness, accuracy, consistency, and bias. A quantitative methodologist in the Research, Measurement, Evaluation, and Statistics program conducted an analysis of the pilot survey to ensure its validity and reliability. His suggestions for improvement were incorporated into the final, published version of the survey that has been used every semester since that time. In 2007, the unit conducted psychometric testing of the data collected using this instrument over the past three years. Information from this analysis is being used to inform a new version of the exit survey that will be inclusive of all educator preparation programs.

Graduate and Employer Survey. A table of specifications was prepared to ensure the alignment between the survey and the unit’s conceptual framework, and the surveys were reviewed for accuracy, fairness, consistency, and lack of bias. When these surveys were developed, they were sent out for review to all deans of educator preparation programs in the USG system who distributed them widely to the educational community for review and comment. Feedback was incorporated on at least two rounds of review before the USG deans group formally adopted them. In addition, the surveys were piloted prior to full on-line development.

GACE I & II Licensure Exams: UGA was an active participant in the item analysis and review process for the new state licensure examinations. The unit facilitated the distribution of draft examination to faculty in the appropriate content and pedagogical areas for review and critique. In addition, the unit facilitated a series of pilot examinations with the College of Education to test the new examinations.

At the program level, faculty ensure quality of their assessment instruments through review by multiple faculty members and testing for fairness, reliability, and validity. For example, in several programs, multiple raters are used for scoring portfolios. In science education, when the portfolios are graded, at least two individuals grade each portfolio. If there is a significant discrepancy between the scores a third scorer is enlisted to score the portfolio again. Also if a student does not receive a passing score from at least one scorer on their portfolio, a third scorer is recruited to score it again. In many preparation programs field supervisors are trained to use observation instruments so that there is common understanding of programmatic expectations and consistency across multiple raters.

In all initial and advanced programs, the Associate Dean for Academic Programs provides oversight for the assessment system described in section 2.1. This office works closely with programs to ensure that program-level assessment systems are aligned and integrated into the unit assessment system. Unit-level data is shared with department heads and program faculty according to the cycle in which the data is collected as well as more broadly as appropriate.

Two offices provide a central location for the collection, summarization, and analysis of data within the unit related to educator preparation programs: The Associate Dean for Academic Programs and the Office of Student Services, which reports to her. This associate dean has significant responsibilities for assessment within the unit. With the help of an administrative assistant, she coordinates the collection of assessment data through regular collection of assessment instruments and data. While this office has maintained responsibility for data collection and analysis since the 2001 NCATE visit, the nature of the data collected and the methods used have expanded significantly over the last three years. The office has worked to expand its involvement in tracking field experiences, administers a diversity demographic survey every semester, annual graduate and employer surveys, exit surveys, GACE II summaries, program completers data, faculty evaluation data, and through regular meetings of the program coordinators, provides oversight for assessment system in the unit and for program assessment submitted to the university, PSC, and SPAs. This associate dean facilitates the work of the PEAC Assessment Committee and serves at the unit’s contact point for all university program evaluations.

The unit collects multiple assessments from candidates, graduates, faculty, and other members of the professional community and these data are used to make decisions regarding candidate progression through programs, program quality, and unit operations. As in 2.1 above the unit maintains assessment systems in which multiple sources of data are collected on candidate qualifications, proficiencies, and graduate competence, faculty performance, program quality, and unit and resource effectiveness and efficiency. A short sampling of these data include:

As part of the University Annual Reporting Process, all departments write annual reports that are submitted to the college. These annual reports provide performance data on specific measures within their strategic plans and also provide the program improvements made based on these data. All departmental annual reports are reviewed by the Dean’s Cabinet and used to create the College of Education Annual Budget Report, which in turn reports progress toward its strategic goals and initiatives and is the basis for annual budget development. This Budget Report aggregates data from credit hour reports, grant funding reports, GACE pass rates and other performance measures. Performance data is tied to COE 5 year plan and provides an opportunity to ask central administration for needed additional resources. At the unit level, the dean and associate deans closely track all departmental and college performance measures on an annual basis.

The responsibility for collecting data on candidates’ performances while they are in the program at Decision Points #2 and #3 lies primarily with program faculty. While all programs share many common assessment measures, several of which are administered at the unit level, programs have developed program assessment systems aligned with unit, state, and national standards and use multiple forms of traditional and performance-based measures to assess the performance of their candidates at each of the major decision points. These program-level assessment systems are in place for each program and are the basis for the university’s program assessment and review process. Candidates must meet the set criteria to be admitted into the programs and to progress through the programs at key decisions points. Program faculty continually review the progress of their candidates through the programs and as they are inducted into their professional fields and make changes to their programs as they see the need based on candidate feedback. In addition, faculty review assessment data at key points as they aggregate and analyze data for annual brief undergraduate major program reports, undergraduate and graduate major reports on a three-year cycle, state and national accreditation reviews, and self-studies for university program reviews.

Candidate complaints and appeals are always of serious import to the unit and its faculty. UGA is concerned that each candidate has a positive educational experience. Formal procedures of undergraduate student complaints are described on the Student Services web site. Student complaints are forwarded to the director of student services. Upon receipt of a student complaint, the director first determines whether the complaint falls into a category of complaints handled by the College's Academic Appeals Committee (admission into major, admission into teacher education, grade appeals, petition for readmission). Student complaints falling into these categories are considered by the appeals committee only after the appropriate procedural process has been followed. Candidates are requested to first appeal directly to the faculty member involved. If they are not satisfied at this level, they may then appeal to the department chair who facilitates the departmental process. If they are still dissatisfied the college committee considers the appeal. Candidates have the right to appeal the decisions of the Academic Appeals Committee to the university’s Petitions Subcommittee of the Educational Affairs Committee.

The five members of the college committee are education faculty with three members elected by the Faculty Senate and the remaining members appointed by the Director of Student Services. Decisions of the appeals committee are maintained in the Student Services Office for five years and then shredded. The Director of Student Services handles student complaints that fall outside the purview of the appeals committee and maintains information related to these complaints for a period of five years. Candidates in advanced programs that have complaints that fall outside the process described above can appeal to the department head, Associate Dean for Academic Programs or can appeal directly to the Graduate School.

Students with disabilities can contact the Office of Disability Services. Students alleging discrimination or sexual harassment can contact UGA’s Office of Legal Affairs. Both undergraduate and graduate student complaints and issues that are not resolved through other means within the unit can be considered at the Dean’s level by the Associate Dean for Academic Programs.

A regular, systematic approach to the assessment system was carefully built to assess our candidates, program, faculty, and unit operations. The technologies used to maintain the assessment system are identified for each of the key assessments as illustrated in the tables above. The college supports a highly skilled, professional staff of information technology experts to support the use of technology in our assessment system. Information technology used in the system ranges from web-based formats (FAR, on-line surveys, PAAR, online course evaluations, etc.) to unit-wide Access and Excel spreadsheets used for a variety of data aggregation purposes. Some of the programs utilize LiveText for a data collection and aggregation tool.

2.3 Use of Data for Program Improvement

The unit regularly and systematically uses data, including candidate and graduate performance information to evaluate the efficacy of its courses, programs, and clinical experiences. The unit analyzes program evaluation and performance assessment data to initiate changes where indicated. Candidate and faculty assessment data are regularly shared with candidates and faculty respectively to help them reflect on their performance and improve it.

The unit’s assessment system has program/unit improvement at its core. Candidates are required to reflect on their performance and develop plans for improvement where appropriate. Throughout the programs candidates are required to reflect and improve their practice from entry through induction. For example, in all programs candidates are required to reflect on their professional practice during performance evaluations in clinical experiences in both initial and advanced programs. Candidates reflect on their practice and improvements to that practice within program portfolio process. Faculty mentor candidates as they seek to promote high levels of exemplary and reflective professional practice.

There are established processes through which faculty continuously and systematically use data to reflect on and improve their own practice. Course evaluations completed by candidates serve as an important source of data for the annual evaluation of individual faculty members. Faculty work with their mentors, peers, and departments chairs to reflect on their performance and to develop ways to improve their teaching and scholarship.

At the level of curriculum, before any changes in programs become final, such as adding new courses, deleting existing courses, program admission or exit requirements, or adding new programs, or terminating programs they must go through the University Faculty Governance Structure. These changes begin within the online Course Approval Process Automation (CAPA) system with the program faculty initiating a request for change. This is then approved by the department head, and reviewed by the Faculty Senate’s elected College Curriculum Committee. It then goes to the university curriculum committee, the Graduate School curriculum committee where appropriate and to the University council for final approval. All new programs must have additional Board of Regents approval. At the College level, a summary of curriculum changes is developed jointly by the Chair of the Curriculum Committee and the Associate Dean for Academic Programs, then submitted to the Faculty Senate.

Each of the program reports must include data-based program improvements in their regular undergraduate and graduate major assessment reports submitted to the university and reviewed by the unit. In addition, programs report improvements as part of their SPA reviews. Since the reports are maintained on the unit’s assessment web site, programs are able to regularly review their progress. Improvements reported in each of these major assessments are aggregated at the unit level in a spreadsheet and posted to the electronic accreditation web site. The unit’s leadership reviews program and unit improvements on an annual basis. At the unit level, data-based program improvements are made continually. For example, as data are shared with department heads and faculty within our annual performance assessment process, the unit determines where program improvements are necessary and takes action to implement improvements.

Standard 2 - Narrative

STANDARD 2. ASSESSMENT SYSTEM AND UNIT EVALUATION

2.1 Assessment System

Assessment System for Candidate Knowledge, Skills, and Dispositions: Initial and Advanced

Unit-Level Review of Initial Candidates

Unit-Level Review of Advanced Candidates

Assessment System for Program Quality

Assessment System for Faculty Quality

Assessment System for Unit Operations and Resources

2.3 Use of Data for Program Improvement