|
|
|
The Juried Electronic Portfolio
Douglas B. Walcerz Associate Professor of Mechanical Engineering Department of Physical Sciences York College of Pennsylvania, York, PA 17405
Abstract—The juried student portfolio is an outcomes assessment tool where the faculty designate a set of elements, such as term papers, reflective essays, and capstone projects, that students must assemble into a portfolio and submit for evaluation. Faculty compose rubrics for the elements and evaluate the portfolios to determine whether certain educational outcomes are being achieved. The administration of juried portfolios is complex, and the evaluation of portfolios is time intensive. TrueOutcomes, a commercial software program containing other assessment tools, has released a Juried Electronic Portfolio module that is intended to simplify the administration of the process and maximize the efficiency of evaluation. In addition, it provides automatic data analysis and presents the results as "drillable" graphs to permit rapid and intuitive verification of the results. This paper provides a description of the features of the software by explaining the roles of the people who use the system. Index terms—outcomes assessment, continuous improvement, electronic portfolio, student portfolio, juried portfolio, rubric.
I. Introduction TrueOutcomesÔ is a Web-based, software driven outcomes assessment process that was originally designed to provide students with an efficient and appealing tool to create and maintain an electronic portfolio of their work, and to provide faculty with tools for aggregating student portfolios and analyzing student work with respect to educational outcomes, . One of the characteristics of the original design is that students decide what to put into their portfolios and conduct their own evaluations of their work. Two of the advantages of this system are that students become active participants in assessment and the process becomes more effective through their increased awareness of and attention to assessment, and the work done by students relieves the faculty of much of the traditional burden associated with student portfolios. But, the system has the disadvantage that students control it, and the quality of student-driven assessment is generally not as good as the quality of faculty-driven assessment. Faculty and administrators at some universities would like to establish a student portfolio system where the faculty determine what goes into the portfolio, and the faculty or an evaluation board evaluate the quality of the portfolio with respect to educational outcomes. By design, this is a much more rigorous and controlled assessment process than the student-driven portfolios, and by necessity the portfolios will be limited to a relatively small number of submissions. Outcomes Assessment Solutions developed the Juried Electronic Portfolio (JEP) to meet these needs. This paper provides a detailed discussion of the JEP process, especially the responsibilities of program coordinators, gatekeepers, evaluators, and students and the interactions between them.
II. Background Faculty-prescribed student portfolios have existed for several decades in several varieties. We will limit our discussion to portfolios that are used for an institution or a degree program, but not for a single course. The following four examples are presented to illustrate issues of portfolio design, evaluation, and application. Manhattanville College, a private, liberal arts college of about 1400 students, has used faculty-driven portfolios since 1971. Students can submit material to their portfolios only with an advisor’s signature. Portfolios are evaluated by the faculty at the end of the sophomore, junior and senior years. There are about 250 portfolios for each class year, and they occupy ten 5-drawer file cabinets. A committee of eight faculty spend seven or eight days reading and commenting on a set of 250 portfolios, with each portfolio having two readers/evaluators. Truman State University, formerly Northeast Missouri State University, is a public university with about 6000 students. All students are encouraged to keep portfolios, but only seniors are required to prepare a portfolio according to a prescribed format, and the portfolio committee only evaluates senior portfolios. The portfolio committee consists of 20 faculty, and they spend about one week evaluating approximately 1000 portfolios. The School of Education at Samford University requires 11 items for the portfolio: a resume, evidence of membership in professional organizations, a statement of teaching philosophy, copies of teaching units, copies of lesson plans, photos, diagrams, and descriptions of classroom activities conducted during the teaching internship, original worksheets, activities and educational games, a videotape of a lesson, an internship journal, summaries of teacher conferences, and outcomes assessment for the student’s students. In the Department of Communication Studies at the University of Missouri-Kansas City, students must include in their portfolios at least three items from the following: a resume and cover letter, a statement of professional goals for graduation and five and ten years later, a sample of written work from an internship or employment, a letter of recommendation from an internship of employment, a research paper from an upper-level course, a self-assessment of communication skills, an example of written work other than a term paper, an essay test, an essay on the student’s personal theory of communication, a discussion of a valuable theory of communication, an explanation of the student’s choice of courses outside the major department, a list of classical and contemporary literature the student has recently read and why, and a list of periodicals the student regularly reads and why. If we can generalize from these four examples, the design of a student portfolio may be tightly controlled or flexible. Portfolios may be required of all students or just certain class years; and they can contain academic work as well as material from co-curricular activities, employment, and self-evaluation. The submission of work to a portfolio can be at the discretion of the students, or may require the signature of a gatekeeper such as an academic advisor. Portfolios evaluation is done by faculty or external evaluators or a combination of both, and portfolios may be read by a single evaluator or multiple evaluators.
III. The TrueOutcomes Juried Electronic Portfolio TrueOutcomes is a commercial product so it must meet the needs of a wide range of customers. The Juried Electronic Portfolio (JEP) thus contains several user configurable parameters to allow for different portfolio designs, methods of application, and systems of evaluation. Perhaps the best way to describe the JEP and its flexibility is through the roles of the people who use the system. We define a Program Coordinator as a person in charge of an academic program. In general, there is one program coordinator for each degree program, plus there may be a program coordinator (or committee) for the General Education Program. Program Coordinators carry the primary responsibility for administering JEPs. A Student is a person who must complete a JEP. A Gatekeeper is a person who is responsible for certifying student submissions. Gatekeepers are often faculty or academic advisors. An Evaluator is a person who reads JEPs and evaluates them based on a rubric.
Program Coordinators Program Coordinators have the primary responsibility for administering the JEP. They must give the JEP a name and define its contents, identify the group of students that must complete the JEP and the time frame for completion, compose the rubric for evaluating the JEP, create lists of evaluators, and prescribe the evaluation process. Figure 1 shows an example of the name and contents of a portfolio designed to assess the general education curriculum. Creating such a definition requires faculty consensus and can take a substantial period of time. However, once the task is complete it is the program coordinator’s responsibility to enter it into TrueOutcomes. The program coordinator must identify the population of students who are subject to the portfolio requirement. Students are classified by degree program and class standing, so it is possible to create a JEP for "seniors majoring in electrical engineering" and another for "freshmen, sophomores, juniors and seniors majoring in biology", and still another for "all students in all majors". Every JEP is tied to a time period, such as "Fall, 2001 – Spring 2004", and the JEP will apply only to students who fit the classification in the specified time period. TrueOutcomes thus provides the flexibility to create portfolios for programs, departments, schools and entire institutions, with the ability to target individual levels if desired. Students can be subject to more than one portfolio requirement, e.g., a discipline specific portfolio plus a general education portfolio, thus allowing both campus-wide portfolio assessment and discipline specific assessment.
For every portfolio definition there must be a rubric for evaluating the portfolio. A rubric consists of a series of questions that an evaluator must answer. The questions may be multiple choice, T/F, short answer, or essay questions. There can be a different rubric for each element of the portfolio, as well as a rubric for the portfolio as a whole. Like the portfolio definition, rubrics must meet with faculty approval in order to be effective for assessment. Once rubrics are approved the program coordinator is responsible for entering them into TrueOutcomes. An example rubric for the first element of the example portfolio is shown in Fig. 2. In general, students must submit their work to a gatekeeper before it is allowed into the student portfolio. Program coordinators are responsible for creating teams of gatekeepers and assigning them specific elements. TrueOutcomes provides program coordinators with the flexibility to specify individual gatekeepers by name or by position, e.g., the student’s advisor or the instructor in a specific course. Evaluating portfolios is time intensive, so it is valuable to be able to manage the task in ways that maximize efficiency. TrueOutcomes gives program coordinators the ability to define multiple teams of evaluators and to assign one or more rubrics to each team. So it is possible to have one team of evaluators work on one set of portfolio elements/rubrics while a different team works on different elements/rubrics in the same portfolio, thus dividing the work between the teams. Some elements of a portfolio may be more important in the eyes of the faculty than others; a capstone project may carry more weight as an indicator of student ability than a resume. TrueOutcomes provides program coordinators the ability to specify how many times an element is evaluated. For example, the faculty may want three readers for each capstone project to maximize the reliability of the evaluation, but only one for each resume. By tailoring the number of readers of each element the program coordinator eliminates waste due to multiple readings of less important elements. While some colleges may elect to evaluate 100% of their students’ portfolios, others may choose to evaluate only a sample of the submitted portfolios. TrueOutcomes allows program coordinators to specify either the absolute number or the percentage of portfolios to be evaluated. Once the program coordinator has specified the teams of evaluators, the elements/rubrics for each team, the number of evaluations for each element/rubric, and the number or percentage of portfolios to be evaluated, TrueOutcomes automatically selects portfolios and creates individual files of elements/rubrics for each evaluator so that all of the parameters of the evaluation process are met.
Students When students log onto TrueOutcomes and view their Juried Portfolio(s), they can open the portfolios to see which elements have been submitted, which ones have been approved, and which ones are still to be done. Submitting new material is as simple as filling out a short form and attaching the submission. Students are allowed to view their submitted materials at any time, but they obviously cannot change them.
Gatekeepers Gatekeepers are responsible for ensuring that the work in a student’s portfolio is, in fact, the student’s own work. Under some circumstances, gatekeepers may also evaluate the quality of the work; for example, if the submission is a term paper or other class assignment and the gatekeeper is the instructor, the gatekeeper may grant approval only if the work receives a passing grade. Students who do not receive a passing grade must revise and resubmit until they are approved. Gatekeepers have the ability to view a student’s submission and either approve it and submit it to the student’s portfolio, or return it to the student with comments.
Evaluators Evaluators have the difficult job of reading and evaluating student submissions. Evaluators generally have multiple elements to evaluate, and there is a different rubric for each element. TrueOutcomes allows them to evaluate one element at a time to maximize efficiency and consistency. Evaluation is inherently asynchronous. Evaluators do not have to work in the same location or at the same time even though they are evaluating the same portfolios. While this maximizes flexibility and efficiency, it is worth noting that synchronous evaluation provides substantial benefits due to the interaction and discussion that evaluation stimulates. So, while asynchronous evaluation is possible, it may still be desirable to conduct evaluations as a group. The difference is that evaluation is done in a computer lab instead of a conference room. |
|
ASEE 
