Difference between revisions of "Online Judging"
From zFairs Contest Management
| Line 6: | Line 6: | ||
== Judge Registration == | == Judge Registration == | ||
| − | Easily find out what the availability, contact information, category preferences, etc. are for your judges. If you are using Online Judging, your judges will need to register as a judge and use their username and password to login to judge projects on their tablet, I-Pad or mobile device on the day of the fair. | + | : Easily find out what the availability, contact information, category preferences, etc. are for your judges. If you are using Online Judging, your judges will need to register as a judge and use their username and password to login to judge projects on their tablet, I-Pad or mobile device on the day of the fair. |
== How do we ensure that each student is judged the same? == | == How do we ensure that each student is judged the same? == | ||
| − | Every judge is different, a perfect project for one judge may be an okay project for another; some judges tend to assign a high amount of points while others tend to assign just a few. To combat these inconsistencies we normalize how judges score and a project scores. This ensures that one judge does not have more pull then any other; it also addresses the issue of judges that score high vs low. | + | : Every judge is different, a perfect project for one judge may be an okay project for another; some judges tend to assign a high amount of points while others tend to assign just a few. To combat these inconsistencies we normalize how judges score and a project scores. This ensures that one judge does not have more pull then any other; it also addresses the issue of judges that score high vs low. |
| − | To do this we find the standard deviation between a judge's average score, we assign a z-score for each judge, and then we are able to adjust scores accordingly. Effectively, we are able to normalize how a judge scores. For example, if you where judged by Judge A and his average score was low so he was assigned a z-score of 1.3 then the score he gave me would be adjusted by {1.3 * (standard deviation) + (my score)}. | + | : To do this we find the standard deviation between a judge's average score, we assign a z-score for each judge, and then we are able to adjust scores accordingly. Effectively, we are able to normalize how a judge scores. For example, if you where judged by Judge A and his average score was low so he was assigned a z-score of 1.3 then the score he gave me would be adjusted by {1.3 * (standard deviation) + (my score)}. |
| − | If you would like to use the Auto Assign Judges Projects to Judge Feature, please see the [[Judge auto assign]] page. There you will find more information regarding this feature and instructions on how to use it. | + | : If you would like to use the Auto Assign Judges Projects to Judge Feature, please see the [[Judge auto assign]] page. There you will find more information regarding this feature and instructions on how to use it. |
== Online Judging Organization == | == Online Judging Organization == | ||
| − | When organizing your online judging one important question is how you are going to organize your contest. Meaning are you wanting to make judge assignments before hand, right before the contest after judges have signed in, or not make assignments at all. | + | : When organizing your online judging one important question is how you are going to organize your contest. Meaning are you wanting to make judge assignments before hand, right before the contest after judges have signed in, or not make assignments at all. |
| − | + | : {| | |
| − | * | + | | |
| − | * | + | === Questions to answer === |
| − | * | + | * Which Format are you wanting to use for Judging? Assigned Interview Times or Free form / Judge Managed |
| + | * How many times do you want each entry judged at a minimum? | ||
| + | * Which scoring format will be best for you? [[Scoring#Adjusted Score | Adjusted Score]] / [[Scoring#z-Score | Z-Score]] / [[Scoring#Raw Score | Raw Average]] | ||
* Are you going to have a finals round (using online judging or not) | * Are you going to have a finals round (using online judging or not) | ||
=== Assignments === | === Assignments === | ||
| − | You can make judging assignments once your judges have registered, depending on how you make assignments this can be super fast if or very time consuming. If you use our auto assignment feature it will take just a few seconds. | + | : You can make judging assignments once your judges have registered, depending on how you make assignments this can be super fast if or very time consuming. If you use our auto assignment feature it will take just a few seconds. |
| − | Pros: Judges have assignments, more control | + | : Pros: Judges have assignments, more control |
| − | Cons: Time consuming, No show judges, extra unplanned for judges | + | : Cons: Time consuming, No show judges, extra unplanned for judges |
| − | + | === Before Hand === | |
| − | You can make your judging assignments days before your event, once your judges and participants have registered. You can always make tweaks. The con with this option is a judge may not show and you will have to move this assignments to another judge. | + | : You can make your judging assignments days before your event, once your judges and participants have registered. You can always make tweaks. The con with this option is a judge may not show and you will have to move this assignments to another judge. |
| − | + | === After Sign In === | |
| − | On the day of your event once judges have signed in for the day you can make assignments, this allows you to avoid the pit fall of a judge not showing up, however you lose a bit of control because you are going to need to do the auto judge assign. | + | : On the day of your event once judges have signed in for the day you can make assignments, this allows you to avoid the pit fall of a judge not showing up, however you lose a bit of control because you are going to need to do the auto judge assign. |
=== Free Form === | === Free Form === | ||
| − | Don't make assignments just relax and let your judges and judge captains handle the worry. (This may be hard for some.) You can direct judges to simply select the '''category''' or '''grade division''' they want to judge and then ensure that all the projects are judged the minimum number of times. The system will push projects that have not been judged to the top of the list so no one is missed and making it easy to know which projects still need to be judged. You have little control, judges and judge captains feel empowered. | + | : Don't make assignments just relax and let your judges and judge captains handle the worry. (This may be hard for some.) You can direct judges to simply select the '''category''' or '''grade division''' they want to judge and then ensure that all the projects are judged the minimum number of times. The system will push projects that have not been judged to the top of the list so no one is missed and making it easy to know which projects still need to be judged. You have little control, judges and judge captains feel empowered. |
| + | |||
| + | |} | ||
| + | |||
| + | |||
| + | == Running a Hybrid Event == | ||
| + | : Are you running a hybrid event this year? If so here are some helpful steps you can take to make things easier. It should be noted that this is a generic list and you should feel free to tailor it to your specific needs. Let us know if there are any questions. | ||
| + | :# Create a "Virtual" project tag and an "In Person" project tag | ||
| + | :#: Staff -> Tags -> New Tag | ||
| + | :# Create a "Virtual" person tag and an "In Person" person tag for judges | ||
| + | :# Create a "Virtual" round and an "In Person" round and set each to filter judges by the appropriate tag and projects by the appropriate tag | ||
| + | :#: Judging -> Setup -> Rounds -> Create New <u>or</u> Pencil Icon -> Judge Filters -> Judge Tags | ||
| + | :#: Judging -> Setup -> Rounds -> Create New <u>or</u> Pencil Icon -> Project Filters -> Tags | ||
| + | :# Then all you need to do is tag which projects and judges will be in each round. | ||
| + | :#: People -> Click on a person's icon -> Tags | ||
| + | :#Then you'd want to go back to your rounds and walk through the normal Auto Assigning process for each round. | ||
| + | :#: It is typically very helpful to have scheduled interview times for at least the Virtual side of things. If one wasn't already set you can go back to the Judging -> Setup -> Rounds -> Pencil Icon -> Interviews are Scheduled and set the desired time frames. | ||
| + | |||
== Online Judging Videos == | == Online Judging Videos == | ||
| − | |||
* [https://www.youtube.com/watch?v=THDPsU62XXc&t=3s Virtual Contest Judging] | * [https://www.youtube.com/watch?v=THDPsU62XXc&t=3s Virtual Contest Judging] | ||
* [https://www.youtube.com/watch?v=78yK1ZbDtTc Online Judging Feedback] | * [https://www.youtube.com/watch?v=78yK1ZbDtTc Online Judging Feedback] | ||
Latest revision as of 10:03, 27 January 2022
Online Judging can refer to both the ability for Judges to judge with electronic rubrics, that they can access anywhere they have an internet connection, as well as the feature group that surrounds this capability. However, one distinction to make is that Online Judging is different from Virtual Judging / a Virtual or Remote event in that the Virtual piece is only one sub feature of Online Judging and most of the Online Judging features are available independent of the virtual aspect. zFairs uses Online Judging to support in person and remote events equally.
Judge Registration
- Easily find out what the availability, contact information, category preferences, etc. are for your judges. If you are using Online Judging, your judges will need to register as a judge and use their username and password to login to judge projects on their tablet, I-Pad or mobile device on the day of the fair.
How do we ensure that each student is judged the same?
- Every judge is different, a perfect project for one judge may be an okay project for another; some judges tend to assign a high amount of points while others tend to assign just a few. To combat these inconsistencies we normalize how judges score and a project scores. This ensures that one judge does not have more pull then any other; it also addresses the issue of judges that score high vs low.
- To do this we find the standard deviation between a judge's average score, we assign a z-score for each judge, and then we are able to adjust scores accordingly. Effectively, we are able to normalize how a judge scores. For example, if you where judged by Judge A and his average score was low so he was assigned a z-score of 1.3 then the score he gave me would be adjusted by {1.3 * (standard deviation) + (my score)}.
- If you would like to use the Auto Assign Judges Projects to Judge Feature, please see the Judge auto assign page. There you will find more information regarding this feature and instructions on how to use it.
Online Judging Organization
- When organizing your online judging one important question is how you are going to organize your contest. Meaning are you wanting to make judge assignments before hand, right before the contest after judges have signed in, or not make assignments at all.
Questions to answer
- Which Format are you wanting to use for Judging? Assigned Interview Times or Free form / Judge Managed
- How many times do you want each entry judged at a minimum?
- Which scoring format will be best for you? Adjusted Score / Z-Score / Raw Average
- Are you going to have a finals round (using online judging or not)
Assignments
- You can make judging assignments once your judges have registered, depending on how you make assignments this can be super fast if or very time consuming. If you use our auto assignment feature it will take just a few seconds.
- Pros: Judges have assignments, more control
- Cons: Time consuming, No show judges, extra unplanned for judges
Before Hand
- You can make your judging assignments days before your event, once your judges and participants have registered. You can always make tweaks. The con with this option is a judge may not show and you will have to move this assignments to another judge.
After Sign In
- On the day of your event once judges have signed in for the day you can make assignments, this allows you to avoid the pit fall of a judge not showing up, however you lose a bit of control because you are going to need to do the auto judge assign.
Free Form
- Don't make assignments just relax and let your judges and judge captains handle the worry. (This may be hard for some.) You can direct judges to simply select the category or grade division they want to judge and then ensure that all the projects are judged the minimum number of times. The system will push projects that have not been judged to the top of the list so no one is missed and making it easy to know which projects still need to be judged. You have little control, judges and judge captains feel empowered.
Running a Hybrid Event
- Are you running a hybrid event this year? If so here are some helpful steps you can take to make things easier. It should be noted that this is a generic list and you should feel free to tailor it to your specific needs. Let us know if there are any questions.
- Create a "Virtual" project tag and an "In Person" project tag
- Staff -> Tags -> New Tag
- Create a "Virtual" person tag and an "In Person" person tag for judges
- Create a "Virtual" round and an "In Person" round and set each to filter judges by the appropriate tag and projects by the appropriate tag
- Judging -> Setup -> Rounds -> Create New or Pencil Icon -> Judge Filters -> Judge Tags
- Judging -> Setup -> Rounds -> Create New or Pencil Icon -> Project Filters -> Tags
- Then all you need to do is tag which projects and judges will be in each round.
- People -> Click on a person's icon -> Tags
- Then you'd want to go back to your rounds and walk through the normal Auto Assigning process for each round.
- It is typically very helpful to have scheduled interview times for at least the Virtual side of things. If one wasn't already set you can go back to the Judging -> Setup -> Rounds -> Pencil Icon -> Interviews are Scheduled and set the desired time frames.
- Create a "Virtual" project tag and an "In Person" project tag
