CSci 435/535: Software Engineering
Project Grading
Summary
There are two key questions to be asked when evaluating a student's project grade:
- How much of the overall work was completed?
- To what extent did the student do their share of the work?
The grading system in this class determines the answers to these two questions
using a group grade and an individual grade. Once these
grades are known, they are combined to determine a student's project grade.
As the project goes on, work that needs to be done is entered into the class
issue-tracking system, either by managers (graduate students) or developers
(undergraduate students). Each issue has a number of points associated with it.
Students assign issues to themselves to indicate that they are working on them.
After the issue is complete, the manager closes the issue, thereby indicating
that the associated points have been earned.
The group grade is essentially the percentage of the total number of completed
points, relative to the total number of points. For example, if there are 50
points for all issues, and 45 points for closed issues, the group grade is
45/50*100=90.
The individual grade is the percentage of points earned, relative to the
average earnable points per person. For example, if there are 50 points and 10
people, the average earnable points is 5. If a student earns 4 points, their
individual grade is 4/5*100=80.
The overall project grade for a student is 50% group, 50% individual. From the
examples above, the student's project grade would be 90*50/100 +
80*50/100=85.
At any time, you can determine your project grade by using
issue-tracker to generate a "Performance Score" report for the current
milestone and the Developer group.
Individual Grade Complications
There are several details which complicate the simple computation of the
individual grades above. For example, there may be points earned by people who
drop the course, points earned by non-developers, and points "stolen"
by people who do more than their share of the work.
Earnable Points
If a manager completes the work for an issue, the manager will assign the task
to him- or herself before closing it. The points for this task are counted
toward the overall group grade. However, note the effect of this on the
individual grades: because the points for this task can't be earned by a
developer, it makes it harder for people to earn the points they need. In other
words, the manager has "stolen" points from the developers. (To avoid
this, make sure managers don't need to do development work!)
On the other hand, any work done by a student who drops the class is not
counted. The student's account will be marked "inactive", and the
points they have earned will count toward the group grade only.
Therefore, the earnable points are computed by subtracting points earned by
inactive developers from the total project points.
Computing Points for Issues
For the purposes of calculating the value of a task, each task will have an
assigned Points value. The points for a particular task will be calculated
based on three properties: Difficulty, Priority, and a bonus Modifier. The
formula for points will be Points = (Difficulty * Priority) + Modifier. See
below for manager guidelines on point values.
If a developer performs some work and feels that the difficulty or priority
estimates were incorrect, the developer may ask the manager for the task to
review these values, providing some justification for why they need to be
changed.
Teamwork
Multiple people can sign up for the same task, in which case the total number
of points will be divided evenly amongst the developers. Developers are
encouraged to work in pairs, especially early on. You can often get
things done more than twice as fast with a partner.
The Zero-Sum Game and Overachievers
Some people may do more than their share of the work. For example, a student
may already have a 100 individual score, but a 70 group score. So the student
decides to help the group (and themselves) by doing extra work. At the first
order, the grading system is a zero-sum game since there are a "fixed" number
of total points. By definition, when this overachiever does more than the
average, someone else must suffer because they can not earn those points.
[Since any developer can create additional points for themselves by adding a
new task to the system, it isn't a true zero-sum game. There are also other
strategies you can use to avoid having points "stolen" from you---see the FAQ
below.]
In order to encourage people to overachieve if they want to, there is an
additional step in the grading process which is designed to reduce the impact
of overachievers. We simply remove all points from consideration which were
earned by overachievers, which results in an adjusted earnable points
value.
The adjusted earnable points are used instead of the total points to compute
each (non-overachiever) person's share of the work. So the target points per
person is the adjusted earnable points divided by the number of remaining
developers. The individual grade is then the percentage of points earned,
relative to the target points per person.
Within the set of remaining people, there will still be some people who earn
more than the average. That is unavoidable---at least the effect will be
smaller. (Besides, these people aren't really overachievers since they didn't
earn the original target average.)
Overall Grade
The overall grade is 50% group grade, 50% individual grade. If a person does
absolutely nothing for the project, the best grade they can hope for is a 75.
If a person gets a 100 individual grade for the project and no one else does
anything, the best grade the person can hope for is slightly higher than a 75.
Group Grade Complications
One situation which may occur is that the outstanding issues may be 100%
complete, but the software may still be unfinished. (This can happen if no one
enters new issues to fix problems or implement missing functionality.) To take
this situation into consideration, the client will estimate the percentage
that the software is complete (based primarily on the stated requirements).
This percentage is used to scale the group grade to compute an adjusted
group grade.
For example, even if 99% of the issues points have been earned, the software
may only meet 85% of its requirements. In this case, the adjusted group grade
is 99*85/100 = 84.15%.
Note that Issue Tracker will use the unadjusted group grade to compute
an estimate of your final grade.
Milestone Grades
Milestone grades are determined by doing the above grading for the tasks
created for the given milestone. The milestone grade can be computed by
selecting the appropriate milestone from the report form in the issue tracker.
Guidelines For Assigning Points to Tasks
Overview
Difficulty
The Difficulty property will be a representation of the level of effort
required for the task. This will be a numeric value that will approximate the
number of man-hours required to complete the task by its assigned deadline. One
man-hour represents one hour of work by a single worker. The Difficulty value
assigned to a task by the managers is unlikely to change over the life of the
project.
Priority
The Priority property will represent the importance of the task to the project
lifespan. Priority will be numeric, with the following possible values:
- Priority 1: Tasks assigned a priority of 1 will be 'nice-to-haves'. These
are tasks that could be implemented in the project, but are not absolutely
necessary to the completion of any milestone.
- Priority 2: Tasks assigned a priority of 2 will be tasks that are required
for project completion, but have no significant dependencies on other tasks.
These tasks must be completed, but can be completed any time during the
assigned milestone.
- Priority 3: Tasks assigned a priority of 3 will have specific deadlines
and dependencies during their assigned milestone, but are not misson critical
to a majority of tasks for the milestone. Most tasks will probably fall in this
category.
- Priority 4: Tasks assigned a priority of 4 have significant dependencies
to other tasks within the milestone or within the project lifespan. These are
tasks that may hold up further action by a large number of developers if not
completed quickly.
- Priority 5: Tasks assigned a priority of 5 are mission critical to nearly
all tasks for the assigned milestone. A task given a priority of 5 must be
completed immediately, or project development may come to a standstill.
The Priority assigned to a given task may be raised if the task becomes more
urgent to the project milestones (for example, a priority 3 task that has not
been picked up and has now become a more immediate need due to dependencies may
be escalated to a priority 4 to encourage volunteers for development).
Modifier
The Modifier property exists to give the managers some "wiggle room" in the
number of points awarded for a task. This allows the managers to reward
exceptional efforts with extra points, or to penalize incomplete efforts with
less points. For example, if a student is for whatever reason unable to
complete a task they have signed up for, they may ask a manager if partial
credit can be applied; the manager can then decide what portion of the possible
points for the task should be awarded if any.
Example
This section shows a sample performance score report. As a developer, you will
only be able to view the details related to yourself. As a result, your report
will look slightly different.
First we break the user list into developers and non-developers.
Next, for each task, we compute the associated points per person. Note that
only closed tasks count toward your score. ("admin" is bold because this report
was generated by the admin user.)
The overall group grade is simply the percentage of total points completed.
Since issue #7 was completed by someone who is no longer in the class, the
points they earned do not count towards the total points that are earnable by
individuals. Note that the points associated with the admin account (a manager)
do not get deducted.
Now we compute the points earned by overachievers, and the resulting adjusted
points per person. Here, developer_3_overachiever has earned 23.75 points
beyond the average 120.75. So we deduct 23.75 from the total earnable points of
483, getting 459.25. The new points per person is then 114.81 instead of
120.75.
The individual grade is the percentage of target points earned.
The final grade is the average of the individual and group grades.
FAQ
- How many tasks can I claim? What happens if someone assigns all
the tasks to themselves? What if someone joins my task but isn't working on
it?
Issue tracker will limit you to two tasks at a time. If you want to join a
task with other developers already working on it, make sure it's okay with
them. If you see someone abusing the task list, please notify the task
manager.
- What if someone sneaks into a task just before it is
completed?
You should get an email notification if anyone modifies a task. You should talk
to the person, and contact the task manager if that doesn't work.
- What if a task isn't really completed?
Managers will have three options. If the task is incomplete due to gross
negligence, the manager can reopen it, in which case the developers will lose
the points until they complete it. If the developers do not wish to continue
working on a task, the manager can leave the task closed but lower the number
of points. If the task is done but could be done better, the manager can add a
new task.
- What if someone actually completes all the "good"
tasks?
Early in the semester, you can be assured that there will be many more tasks
added to the system. Late in the semester there may be no "good" tasks (or any
at all!). The overachiever adjustment will help you in this case, making your
target points per person lower. For peace of mind, it's better to earn your
points early.
- Hey! I had a good grade and the manager added more tasks, and now
my grade is bad!
Managers add tasks for work that needs to be done. If there's more work for
the developer group, then you need to do more work yourself to maintain your
grade.
- What if a manager isn't assigning points fairly? What if a
manager isn't closing issues?
You can tell the manager why you think the points should be adjusted. If you
are unhappy with their decision, you can appeal to the professor. Decisions the
professor makes are final.
If you have any problems with a manager not managing tasks properly, you can
notify the professor. You'll also have a chance to evaluate all the managers at
the end of the project.
- Another student joined my task late. It's okay by me, but do I
really have to give them half the points? How about we both get 60% of the
points?
Yes. Sorry, but the system is complicated enough already.
- I need points, but all the remaining tasks are taken or
are too hard.
- Don't procrastinate next time. With a 2-3 task maximum, this means that
there should be plenty of tasks for everyone to sign up for.
- Work in teams. Work with an overachiever and share the points.
- Make it not a zero-sum game by adding more tasks and doing the work. You
can create your own points!
- The slackers in the class seem okay with their low grade, but my
grade is suffering because of the bad group grade. Can you protect me from
slackers?
This is by design. Everyone must share in the success (or failure) of the
project. If you want a better group grade, you will need to get the slackers
more involved, or you will need to cooperate with others to "pick up the
slack".
- Another student created a task, and I did the task. But the
manager lowered the points before closing it. That's not fair! I wouldn't have
done the work for so few points.
Managers always try to make the points consistently fair. If you do a task
before the manager has had a chance to make the points appropriate, the manager
can't do anything about that. If you are worried that this might happen,
contact a manager to make sure that the points are correct before starting the
work.
Back to CSci 435/535 Homepage.
