CSci 435/535: Software Engineering

Project Grading

Summary

In order to help assess the contributions of each person that works on the project, I have implemented an automated grading system into the Issue Tracker task management system. This document explains how it works, and how each developer's grade is computed.

As the project goes on, work that needs to be done is entered into the class issue-tracking system, either by managers or developers. Each issue has a number of points associated with it. Students assign issues to themselves to indicate that they are working on them. After the issue is complete, the manager checks the work and closes the issue, thereby indicating that the associated points have been earned.

The developer's grade is the percentage of points earned, relative to the average earnable points per person. For example, if there are 50 points and 10 people, the average earnable points is 5. If a student earns 4 points, their grade is 4/5*100=80.

At any time, you can determine your project grade by logging into the issue tracker system and viewing the quick report on the left side below the navigation menu. For additional details, you can tell issue-tracker to generate a "Performance Score" report for the current milestone and the Developer group.

Individual Grade Complications

There are several details which complicate the simple computation of the individual grades above. For example, there may be points earned by people who drop the course.

Earnable Points

If a manager completes the work for an issue, the manager will assign the task to him- or herself before closing it. The points for this task are not "earnable" by any student. A similar situation occurs if a student does some project work before dropping the course.

To address this problem, the earnable points are computed by subtracting points earned by inactive developers and managers from the total project points. The effect will be to slightly reduce each developer's target points for the milestone.

The earnable points are used to compute each developer's share of the work. So the target points per person is the earnable points divided by the number of developers. The individual grade is then the percentage of points earned by the developer, relative to the target points per person.

Computing Points for Issues

For the purposes of calculating the value of a task, each task will have an assigned Points value. The points for a particular task will be calculated based on three properties: Difficulty, Priority, and a bonus Modifier. The formula for points will be Points = (Difficulty * Priority) + Modifier. See below for manager guidelines on point values.

If a developer performs some work and feels that the difficulty or priority estimates were incorrect, the developer may ask the manager for the task to review these values, providing some justification for why they need to be changed.

Teamwork

Multiple people can sign up for the same task, in which case the total number of points will be divided evenly amongst the developers. Developers are encouraged to work in pairs, especially early on. You can often get things done more than twice as fast with a partner.

Overachievers

Some people may do more than their share of the work. For example, each developer may have a target of 10 points worth of work, but one person does 25 points worth of work. This does not mean that 15 points of work cannot be earned by the other developers. The reason is that there is always more work that can be done—just create a new task and assign some points to it.

Manager Responsibilities

One situation which may occur is that the outstanding issues may be all finished, but the software may still be unfinished. (This can happen if no one enters new issues to fix problems or implement missing functionality.) In this situation, the developers will receive the grade computed by issue tracker.

Another situation that may occur is that points are assigned unevenly to different types of tasks, or for the same type of task across different managers. Managers should work hard to ensure that easy work is worth less points than hard work, and should cross-check each other periodically to ensure consistency.

Management of issues is the manager's reponsibility. If tasks are missing from the system, developers cannot be expected to complete them. Managers are assigned grades at each milestone by the professor, based significantly on their management of the developers via the issue tracker.

Milestone Grades

Milestone grades for developers are determined by doing the above grading for the tasks created for the given milestone. The points will be reset after each milestone, so as to not unfairly punish people who do poorly on a previous milestone. The milestone grade can be computed by selecting the appropriate milestone from the report form in the issue tracker.

Guidelines For Assigning Points to Tasks

This guide is meant to help managers consistently assign points, so that grading will be fair.

Overview

Difficulty

The Difficulty property will be a representation of the level of effort required for the task. This will be a numeric value that will approximate the number of man-hours required to complete the task by its assigned deadline. One man-hour represents one hour of work by a single worker. The Difficulty value assigned to a task by the managers is unlikely to change over the life of the project.

Priority

The Priority property will represent the importance of the task to the project lifespan. Priority will be numeric, with the following possible values:

Priority 1: Tasks assigned a priority of 1 will be 'nice-to-haves'. These are tasks that could be implemented in the project, but are not absolutely necessary to the completion of any milestone.
Priority 2: Tasks assigned a priority of 2 will be tasks that are required for project completion, but have no significant dependencies on other tasks. These tasks must be completed, but can be completed any time during the assigned milestone.
Priority 3: Tasks assigned a priority of 3 will have specific deadlines and dependencies during their assigned milestone, but are not misson critical to a majority of tasks for the milestone. Most tasks will probably fall in this category.
Priority 4: Tasks assigned a priority of 4 have significant dependencies to other tasks within the milestone or within the project lifespan. These are tasks that may hold up further action by a large number of developers if not completed quickly.
Priority 5: Tasks assigned a priority of 5 are mission critical to nearly all tasks for the assigned milestone. A task given a priority of 5 must be completed immediately, or project development may come to a standstill.

The Priority assigned to a given task may be raised if the task becomes more urgent to the project milestones. For example, a priority 3 task that has not been picked up and has now become a more immediate need due to dependencies may be escalated to a priority 4 to encourage developers to sign up.

Modifier

The Modifier property exists to give the managers some "wiggle room" in the number of points awarded for a task. This allows the managers to reward exceptional efforts with extra points, or to penalize incomplete efforts with less points. For example, if a student is for whatever reason unable to complete a task they have signed up for, they may ask a manager if partial credit can be applied; the manager can then decide what portion of the possible points for the task should be awarded if any.

FAQ

How many tasks can I claim? What happens if someone assigns all the tasks to themselves? What if someone joins my task but isn't working on it?
Issue tracker will limit you to two tasks at a time. If you want to join a task with other developers already working on it, make sure it's okay with them. If you see someone abusing the task list, please notify the task manager.
What if someone sneaks into a task just before it is completed?
You should get an email notification if anyone modifies a task. You should talk to the person, and contact the task manager if that doesn't work.
What if a task isn't really completed?
Managers will have three options. If the task is incomplete due to gross negligence, the manager can reopen it, in which case the developers will lose the points until they complete it. If the developers do not wish to continue working on a task, the manager can leave the task closed but lower the number of points. If the task is done but could be done better, the manager can add a new task.
What if someone actually completes all the "good" tasks?
Early in the semester, you can be assured that there will be many more tasks added to the system. Late in the semester there may be no "good" tasks (or any at all!). However, there is always some way that the software can be improved. Just create a task for whatever you think needs to be done. For peace of mind, it's better to earn your points early.
Hey! I had a good grade and the manager added more tasks, and now my grade is bad!
Managers add tasks for work that needs to be done. If there's more work for the developers, then you need to do more work yourself to maintain your grade.
What if a manager isn't assigning points fairly? What if a manager isn't closing issues?
You can tell the manager why you think the points should be adjusted. If you are unhappy with their decision, you can appeal to the professor. Decisions the professor makes are final. You'll also have a chance to evaluate all the managers at the end of the project.
Another student joined my task late. It's okay by me, but do I really have to give them half the points? How about we both get 60% of the points?
Yes. Sorry, but the system is complicated enough already.
I need points, but all the remaining tasks are taken or are too hard.
- Don't procrastinate next time. With a 2-3 task maximum, this means that there should be plenty of tasks for everyone to sign up for.
- Work in teams. Work with an overachiever and share the points.
- Add more tasks and do the work. You can create your own points!
Another student created a task, and I did the task. But the manager lowered the points before closing it. That's not fair! I wouldn't have done the work for so few points.
Managers always try to make the points consistently fair. If you do a task before the manager has had a chance to make the points appropriate, the manager can't do anything about that. If you are worried that this might happen, contact a manager to make sure that the points are correct before starting the work.
What if I don't want to put an issue into the system?
You really should create an issue for the work that you do, because it helps communication during development. If you fail to create the issue before starting the work, then someone else might do the work and get the points for it before you finish. And if you need the points, you won't get any credit for the work that you've done unless there's an issue for it.
If you don't need the points, you may be concerned that adding a task for some work that you want to do may increase the workload for the rest of the class. Just create your task, assign it to yourself, and set the modifier so that the overall point value is 0. If you ever need the points later, you can ask the manager to set the modifier back to 0.
Finally, managers should keep in mind that they will be evaluated on the final product. As a result, they should be proactive in making sure that the issue tracker list of issues truly reflects the work that needs to be done. Allowing developers to do work "outside the system" increases the chance that the work won't be properly tracked and completed.
Next milestone is really busy for me. Can I do my work for the next milestone in the current milestone?
Absolutely. Work with the managers to create issues for the next milestone, and sign yourself up to do them. Note that you should work this out in advance of the busy milestone, rather than retroactively arguing that you did additional work in the previous milestone for the current milestone. Also, this cannot be used as a method of "pre-claiming" work for later milestones.
In the past you used to factor in a "group grade" that was based on the overall project completion. Why did you remove this?
Originally I thought the group grade would encourage people to help each other. Instead I found that it (1) allowed slackers to pass the course without doing hardly any work on the project, and (2) it created resentment among hard-working students whose grade was hurt by slackers.

Back to the CSci 435/535 Homepage.

Last changed April 23 2007 09:36:39. David Coppit, coppit@cs.wm.edu

There have been 1232687 hits since Thu Jun 9 14:49:55 2005