Evaluate Administrator Guide

This document is intended to act as a brief technical guide in how to use the evaluate course evaluation system as faculty or department administrator.

Administrative Privileges

The majority of configuration and data analysis in evaluate is done by individuals with administrative access. All individuals with access can currently be found on the Access Directory.

Permissions are divided into Faculty and Department Level permissions, with any permissions granted at the faculty level being inherited at the department level.

Please keep in mind that evaluate maintains its own definition of what departments belong to which faculty; this can be viewed on the access directory.

The obtainable permissions are as follows:

Granting Permissions to New Users

If you have the Promote permission in any group, you can give other users any permissions you also have within that group.

To add a new users to your group, go to the respective Faculty / Department and click “Add User+”. Then enter the user's 8 character user id. These IDs can be found at the UW Authenticated White Pages.

Some users will have usernames longer than 8 characters. Please be sure to truncate these to 8 characters.

Deleting Users

To fully delete a user, a user with the Demote permission should edit another existing user, and simply remove all of their permissions. Upon saving the user will be removed from the group.

Quick Reference for Standard User Permission Levels

Note on your Own Permission Levels

No matter your access level, you cannot edit your own permissions. This means a user with demote cannot remove themselves, and will need to contact an administrator or have another user with the demote permission remove them from the access group.

Adding Courses to Evaluate [Add Courses]

The most common process most admins will be carrying out in evaluate is adding courses.
To minimize the time this process takes, evaluate pulls courses and students from QUEST – this means you should wait until courses have “settled” for the term before adding courses to evaluate. Some time after the “Student Add” date is definitely recommended.

Evaluate will automatically keep class-lists for surveys up-to-date, however it cannot fix if an instructor gets changed in QUEST. This is the primary reason to wait until slightly later in the term to begin adding courses.

To begin the process, click the Administrative > Add Courses link in the evaluate sidebar on the left. You will be taken to a page which will list every department you are able to add courses for. You can select the term you wish to view by clicking the term dropdown menu at the top right. Simply click on one of the department cards to add that department to your course search. Upon clicking a department, it will turn blue. Multiple departments can be selected at any time. Clicking again on a department will deselect it.

By default only Lecture and Online course components will be searched, as they are the most commonly evaluated. To add other components, simply click a gray component to turn it blue. Clicking again will deselect it. When you’ve selected your departments and components, press Search Courses (beneath the list of departments) to request a list of courses that match the query.

If you required to add courses from a department you aren’t affiliated with, you can search for departments by clicking Search Bar at the top of the page. Typing in the department code into the text field and then clicking “Search Courses” will generate a list of courses that match the query.

You will be required to add additional tags to courses which aren’t within your administrative scope. Please be aware of policy and rules in other faculties and departments before proceeding.

You can also access the previous version of the course adder by clicking Old Course Adder Page at the top of the page if are having difficulty with the new tool.

After the search concludes, you should see a list of courses. You can quickly take note of the following:

As an admin, all that will need to be done in most cases is toggling which courses should, or should not be evaluated, what the start and end dates are and which survey template to use. To toggle a course, click the gray box to the left of a course code to check it off. Then, set a survey Template, Start Time, End Time, and Release Time for the survey and its results.

For quick use, if most of your courses will have the same template, start date, and end dates, select these on the top row of the table and click apply to all for the columns to quickly set values on all courses. Clicking Toggle Create All will check all unconfigured courses. To replace any previously reconfigured evaluations, they will need to manually be checked off.

Some courses may not display instructor names. If an instructor is in QUEST and has a valid WATIAM Id, they can be manually added with their name and username.

Once this has all been set up, simply click Submit. After a moment you will see a receipt displaying the surveys created for each course and the configurations. That’s it!

You can make changes to the configurations for course surveys in the Modifying Added Courses section below.

If you run into errors when adding courses (if no surveys get added), contact the sysadmin with your issues to have them quickly resolved!

Merged Courses

As you likely know, occasionally courses have multiple instructors for a single section. In this case, it may be convenient to have a single course evaluation survey for all the instructors in the joined courses. If a course has the same section but different instructors, you can click Merge Surveys between them to join them. Merged surveys will show up with a black outline.

This special evaluation will ask a separate set of instructor-based questions for each instructor, but only ask course-base questions once. When the student hits submit, a response is sent to each instructor’s survey with only their own data.

If, instead, you wish students to get a full separate evaluations for each instructor, you will need to unmerge them by clicking Break Apart.

Cross-listed courses

Cross listed courses will automatically show in evaluate with all the cross-listed sections put together. These cannot be separated!
In evaluate, anybody with permissions in any of the cross-listed sections will be able to work with these courses. This means if you have MATH permissions and add a MATH / PHYS course, admins with PHYS permissions can view or edit these courses.

If you’d like to Remove cross-faculty permissions, click Toggle Permission Tags at the top right of the course list. Click on any green tags to remove them.

Likewise, if you’d like to Add additional permissions to a course, you may add tags by clicking the gray Add+ button. These permissions must be configured individually per course and section. A list of common tags will be displayed in blue to choose from. To manually add tags, type in the department or faculty tags in the text field, separate them with a comma and click Add+.

Keep in mind that department tags are caps-locked short-names (i.e. MATH, PHYS) and *faculty tags are title-cased full words (i.e. Science, Applied Health Sciences).

Green tags indicate successfully added tags.

If you require adding or removing tags after a survey has already been created, you can re-create it, or ask the sysadmin to do it.

Modifying Lots of Courses at Once

From the table view, the following properties can be altered:

To change these variables, click Toggle Modification Tools to bring up the necessary panel.

In the table select (click) on the rows which you wish to change. Selected rows will turn a beige colour.

With your rows selected, set any desired dates in the inputs, and click Apply to Selected. This will turn any altered columns green.

After inspecting the staged changes, click “Save Changes” to apply them. Changes will be pushed to the affected courses, and logs should appear indicating successful changes.

Modifying a Single Course

To modify a single course, click the eye icon on the left side of any course or section. The page you will be taken to will give you a status report of the course evaluation, as well as a preview of the questions.

Press Open Survey Management Panel from here to modify the course. From here the following can be altered:

We don’t recommend you alter Instructor data here unless absolutely necessary. For the integrity of the evaluate’s data it is better to have QUEST data updated with a new instructor, and re-add the course.
The only time you should edit an instructor here is if some surveys have already been completed, or the instructor cannot be correctly entered in QUEST.

After you make any changes, simply click Save Changes to apply them!

Note about managing students

Students should automatically be added / deleted as QUEST updates. Only add / remove students to a survey if absolutely necessary.

Deleting a Survey

Within the “Survey Management Panel” you also have the option to delete a survey. To do this, simply click the delete button and hit save.
There is a known bug currently, where after you delete a survey you get an error page. This is because evaluate tries to re-load the deleted survey.
As such, when you see “Something Went Wrong” after deleting a survey, that means it worked!

Things that Can’t be Modified

Due to various oversights and limitations, the following things cannot be edited after a course has been added and will require re-adding the course to alter:

Viewing Survey Results

To view survey results you will require the Read permission. To view them, simply go to the View Results link on the left sidebar.

From here you will be given a list of terms and filters to construct a query. Simply select your desired term-range, and any desired scope of departments or faculties and click Search. Any courses you teach, or you have administrative access to, will be shown.

Note that surveys that have not yet passed their “Release Time” will not be shown.

One of the filtering options is “Include Flagged Responses”. This is due to an error that happened in a past term where we needed to remove some duplicate responses. At this time flagged responses aren’t used for anything.

Some things to note about the resulting table:

Survey Result Page

Clicking on a single survey will yield the results page which is primarily intended for instructor-viewing.

A breakdown of each question, as well as some aggregate data is available here. Instructors will also be able to view comments.

As mentioned in the Permissions section, comments will generally only be visible by instructors, as this is university policy.

A detailed CSV can be exported this page which breaks down exactly how many students selected each question. If instructors wish to perform analysis on their own data, or if a department wishes to scrutinize a specific course, this information is very useful.

Emailing Functionality

Student email

In our first couple of years we experienced many cases where the vocal minority of students would complain to instructors that “we don’t need in class time, we already did our surveys” due to the initial email going out.
That, combined with Engineering’s continual insistence that we explicitly don’t send students emails at the start of the period (while maintaining excellent response rates) lead us to remove the start-of-period email for students.

As such, students will only get automated emails from evaluate when:

Since most surveys are ~13-14 days long, the aim is for students to get the email around the middle of the second week of surveys. This does mean that surveys taught outside the normal window get automated emails, as long as the window is > 9 days.

This logic also helps us cover students who are taking cross-faculty courses so they don’t get spammed.
When we changed the email logic, we watched for dips in responses and found no notable impact – as we found in our first couple of years with evaluate, in-class time drives the majority of responses, so these emails act more as a reminder than a driver.

Instructor Email

Instructors will automatically get an email a week before their first course evaluation for the term.

This email simply informs them they have upcoming course evaluations, when they are, and recommends giving in-class time if possible. It is recommended that faculties send out official instructions for instructors if more info than this is required.

Anonymity in evaluate

The evaluate system was designed to ensure that student anonymity is protected as much as possible. As such, student information is discarded entirely when a student submits a survey.

As such, there is no possible way to perfectly correlate any one submission to a specific student, unless only one student submits a response for a given course.

Evaluate’s Data

Comment Data

Evaluate does not use any form of filter mechanism for submissions, as such all submitted comments are kept in the system and available to instructors.
This may change with university policy at a later date.

Current policy states that Only instructors should have access to their own comments, meaning chairs and admins (save for the sysadmin) should not be given the Read Comments permission.

Survey Data

Numeric survey data is subject to the rules of the faculty or department who owns them, based on each faculty’s internal policy.

We store this numeric data in “Responses”, which contain which option was selected for each question in the survey for each submission.

This allows us to change the weighting of options or define new aggregate metrics for existing survey templates, and apply those changes to existing survey data within the system if necessary.

A note about Joined Courses’ course data

With a “Joined Course”, multiple instructors are put on a single survey, and the course component of that survey is only asked once.

When the survey is submitted, each instructor will get a separate results page. In order to acheive this evaluate assigns a copy of the course question answers to each of the instructors’ results pages..

Please keep this in mind when performing analysis of data, as the data for course-related questions will be copied to multiple surveys’ result data.

Quirks and Tips

Quick Tips

Known Quirks

Wizard of Oz Stuff

We’ve managed to do away with the majority of the sysadmin does some magic behind the curtain to make it work things frequently found in home-grown systems, however there’s still a couple of points you may need us with!