Web Submission and Review, Chair Documentation

Version 0.6 - September 2007

Installing the software

Web-based installation
Non-web-based installation

First-time customization
The administration page
Managing the submission site

Tweaking Email Settings

Managing the review site
Managing the final-version submission site
1. Adding invited talks to the program

1. Installing the software

One of the goals of this software was to make administration as easy as possible. In particular, you never need to run scripts etc. In principle, to install this software you just unpack the zip file in some directory in your web tree (call it the BASE directory), point your browser at that directory and off you go.

However, that method of installation requires the web-server to be able to write files in the web-tree, which in some cases is considered a security risk. I therefore provide also a non-web-based installation method, as explained below.

Prerequisites. This is a PHP/MySQL software, which means that to use it you must have a MySQL server and a PHP engine (and your web-server must supports PHP and MySQL). I tested the software with PHP 4.3.9 and MySQL 4.1.20 (and earlier versions of it were tested with PHP 4.3.2 and MySQL 3.23). It may work even with older versions but I cannot test it. Also, proper functioning of this software depends on "proper" functioning of you web-server. In particular, some issues that I came across are:

The web-server (and the PHP engine) must be willing to upload files. For example, in one system that I used to test it, the PHP engine was configured to only allow uploads of files upto 500KB (and I was wrecking my brains as to why I can only upload pdf and not postscript files..) In my case it turned out that I had to change the parameter LimitRequestBody in php.conf (which on that system was somewhere under /etc/httpd/).
The web-server must be able to send email. For example, if this is a Windows machine then the variable SMTP in php.ini should be set correctly.
If the server is set-up with PHP in safe mode, then everything under the BASE directory must have same owner. (Also everything under the UPLOAD directory -- see below.) If safe_mode_gid is set in php.ini then instead of having the same owner they need to have the same group.

Experimenting with the software. I wrote a small testing package for the purpose of experimenting with the software. To use this package, copy the contents of the "testOnly" subdirectory to the BASE directory where you installed the software itself before you customize the installation. This testing package adds 53 dummy submissions and 17 dummy PC members to the conference database.

Troubleshooting. Most of the parameters of the software are stored in the "parameters" table in the database. The details of the database itself are stored in the file init/confParams.php under the BASE directory. If you have problems with the software, looking at these parameters may provide some hints as to the source of the problem.

1a. Web-based installation

To use this method, simply copy the contents of the webtree directory to somewhere in your web tree (call it the BASE directory), making sure that the web-server has permission to write in the "init" sub-directory of the BASE directory. After copying the files to the BASE directory (and setting the permissions), point your browser to the page chair/initialize.php under the BASE directory. In this form you need to specify various details that are neeed for initializing the new site:

Web Server. The DNS name or IP address of the web server, as accessed by the users of this software. This parameter is used when the software sends emails and needs to specify the address where some pages can be found. (Note that in some cases the administrator may use a different address. For example, the administrator uses "http://localhost/confName/" while everybody else uses "http://www.severName.org/confName/". In this case the chair needs to enter the latter address on the customization page.)

MySQL Server. This parameter tells the PHP scripts where to look for the MySQL server. In common installations the same host runs both the web-server and the MySQL server. If this is not the case (i.e., the web server communicates with a back-end database that is hosted on a different machine) then you need to check the radio-button for "The MySQL server runs on a different host" and provide the DNS name or IP address of the machine that hosts the MySQL server.

MySQL Administrator or MySQL Database and User. There are two ways to create a database for use with the submission and review software:

If you have administrative access to the MySQL server, you can simply enter the administrator username and password and the software will automatically create a new database. The "administrator" whose details are thus specified should have sufficient privileges to DROP and CREATE databases and to GRANT access privileges on these databases.
If you do not know the username/password of the MySQL administrator, you should create the database for this conference by some other means. (For example, some hosted sites have a "control panel" that lets users create new databases. In other sites one must ask the administrator to create new databases.)

Once you created the database, enter in the form the database name and the MySQL username/password that are needed to access it.

UPLOAD directory. This is the local directory where the uploaded submissions would be stored. If you do not specify anything in this field, the UPLOAD directory defaults to the subdirectory "subs" under the BASE directory. The UPLOAD directory need not be in the web-tree, but the web server must be able to read and write in that directory.

Program Chair email. In this field you can specify either just an email address or also the name (in the form Name <email>). If you do not specify the chair's name, it will default to something like "<conf-name> Chair". The email addresses that you specify here will be used as the program chair's username for the purpose of accessing the review site.

The chair's email will also be CC'ed on every email that is sent to authors of submitted papers, including submission/revision confirmations, the final decision notifications and the comments for authors. The address that you specify here can be modified from the customization page, and also from the PC membership page.

Administrator email. The administrator's email is rarly used by the software: it's main purpose is to serve as a backup address for the chai'r email. For example, if the need arise to reset the chair's password, then an email with the new password will be sent to both the chair and the adminsitator email addresses. Also, there are some rare error situations where the software would sends an email to the administrator. After it is set, the administrator email can only be changed by manually editing the file init/confParams.php under the BASE directory.

After Initialization

Once you hit the Submit button at the bottom of the initialization form, you are taken directly to the customization page in order to customize the site for the conference (see below). At this point the web server no longer needs to write to the "init" subdirectory, so you can change the permissions to make that directory read-only.

1b. Non-web-based installation

An alternative method of installation (in Unix systems) is to change directory to where you unzipped this software (above the "webtree" and "tools" directories) and from the command prompt do

  % ./tools/mkNewConf.sh confname

where confname is a short-name for the conference (e.g., xyz2007). It is recommended that you read the script "mkNewConf.sh" before running it, and you can optionally modify it to set some default values.

This shell script will ask you for the locations of the BASE and UPLOAD directories (and other parameters) and then it will install the software in the BASE directory, setting the permissions as needed. The script will then print out the username and password that are needed in order to access the customization page.

2. First-time customization

Before it can be used, the site still needs to be customized for the conference. This is done from the page chair/customize.php under the BASE directory. This form lets the program chair specify the conference name, deadlines, program-committee members and many other parameters. Links are provided from this form to this documentation page, where you can find some additional explanations about the role of different parameters. All the parameters that are specified in this form can be modified later from the administration page. Some fields in the customization form that require more explanation are described below:

Supported Formats. The software is very permissive in terms of handling formats other than the ones that you specify here. If a submitter uploads a file in an unsupported format, the only thing that happens is that the extension unsupported is added to the file name. (For example, uploading a file xyz.txt, the file will be stored as xyz.txt.unsupported.
Program Committee. In this section you must enter the PC chair's email address, and you can optionally also specify the names and email addresses of the program committee members. For each PC member (or chair) you can specify either just an email address or also the name (in the form Name <email>). If you do not specify the chair's name, it will default to something like "<conf-name> Chair". For the other members, if you do not specify the name then it will remain blank.

The email addresses that you specify here will be used as the username for the purpose of accessing the review site. The names will be displayed on the reviews and discussion boards. All the names and addresses that you supply here can later be modified (or added or removed) from the administration page.
Reviewer Preferences. If you check this box, PC members will see a link on the review page asking them to specify their reviewing preferences. The chair will then be able to see all these preferences and can take them into account in assigning submissions to reviewers. Moreover, the chair will be given the option to let the software automatically compute an assignment or submissions to reviewers based on these preferences. UNchecking this box will turn off reviewer preferences, and also disable the automatic assignment option.
Reviewer Attachments. Check this box to allow the reviewers to upload a file as an attachment to their review. This file is considered part of the comments to authors, and will be attached to the comments that are emailed to the authors at the end of the review process. Click here for some considerations for and against allowing this option.

After Customization

Once you hit the Submit button at the bottom of the customization page, you are taken to a confirmation page where you can once again verify all the parameters. Then the software is customized with these parameters and you are presented with a receipt page. That page includes the chair's password and links to the submission page and the administration page. The submission site is now ready to receive submissions, and you can use the administration page to manage it.

3. The Administration Page

The administration page is what you use the manage the site and the entire submission and review process. The links on that page change according to the different phases of the process. You have one set of links during the submission phase, another during the review phase, and yet others during and after the final-version submission phase. You also have some "general purpose" links there for changing your password, looking at the error-log file, reading the documentation, etc.

You also have an "Undo/Redo" link that lets you either undo the last change that you did from the administration page, or redo the last change that you undid. For example, if you closed the submission site and the decided to reopen it, you can use the Undo link to do that.

4. Managing the submission site

The administration page for the submission site is fairly simple. The main things that you can do from that page are:

You can see a list of all the (non-withdrawn) submissions.
You can review and modify many of the submission-site parameters that you specified during customization, including the conference submission deadline.
You can tweak some parameters of the web-site (mainly email settings) as described below.
Finally, when the submission deadline passes you can close the submission site and activate the review site. Clicking this last link will take you to a confirmation page, where you can confirm that you really want to close the submissions. The same page also lets you modify some of the review parameters that you specified during customization.

You will also see on the administration phase links to some pages that deal with review-period activities, such as setting the review guidelines, managing PC membership and setting up PC votes. These are described in more details below.

4a. Tweaking Email Settings

This form is meant to provide some workarounds common bugs and configuration errors in email systems. It currently consists of the following options:

Email header-line separation. RFC 2822 specifies that different lines in the email header should be separated using the two characters \r\n (CRLF). However, the documentation for the PHP mail function contains the following comment:

Note:If messages are not received, try using a LF (\n) only. Some poor quality Unix mail transfer agents replace LF by CRLF automatically (which leads to doubling CR if CRLF is used). This should be a last resort, as it does not comply with RFC 2822.

Accordingly, the default setting that is used in the software is CRLF, but you have the option to override that default.

X-Mailer Header Line. Sending this header line is considered standard net etiquette, but some spam-filters out there mark as spam anything that has this header line. UNcheck this box if this causes email loss.

Sender Header Line. If you know how the web-server is calling itself on email messages that it sends, you can specify it here. This is often something like userName@machineName where userName is the username of the web-server (e.g., www, apache, admin, etc.) and machineName is either the DNS name of the machine (e.g., www.mit.edu) or sometimes just localhost.

When the web-server sends email, the name of the web-server itself is set as the "envelope sender" of this message. It was observed that some ISPs filter as spam email messages where the From: header line does not match the "envelope sender". Hopefully specifying a matching "Sender:" line would prevent these emails from being discarded.

Specify "envelope sender". Sometimes you can explicitly specify the "envelope sender" to be used, by supplying the parameter "-f name-to-use" to the PHP mail function. This only works on (some) Unix systems, and does not work when PHP is in "safe mode". Also, using this option typically adds an "X-Warning:" header line to the message, which can again increase the odds of this message being discarded en-route. The PHP documentation has the following comment about this method:

The user that the web server runs as should be added as a trusted user to the sendmail configuration to prevent a 'X-Warning' header from being added to the message when the envelope sender (-f) is set using this method. For sendmail users, this file is /etc/mail/trusted-users.

5. Managing the review site

Once you close the submission site, you are taken back to the administration page, where you can now administer the review site. At the top-right corner you still have a "Submission-site" box, now consisting only of a few links that let you list the submissions is various orders (and also revise or withdraw submissions if need be). You can even submit a new papers (just in case you want to accept for consideration a submission that arrived after the deadline.)

5a. Initial set-up

One of the first things that you would want to do after the deadline (if not before) is letting the PC members access the review site. This is done from the "Manage PC membership" page. On that page you can add or remove PC members, reset the password of a PC member and modify the email address or the name that is recorded in the database for each member. The PC members can login to the site using their email address and password. When you reset the password of a PC member, an email message is sent to that member with the new password. (Also, changing the email address of a PC member automatically resets his/her password.) Note that it is possible for several people to use the same email address as long as they use different passwords (but all the passwords will be sent to that one email address).

Other things that you can do from the administration page is to create an archive file with all the submissions (so that reviewers do not need to download each submission separately), and edit the review-guideline page. The review guidelines page consists of the grade semantics, important dates in the committee work (such as the individual review deadline), and other review policies.

5b. Paper assignments and blocking access to papers

The software also lets you assign some submissions to each reviewer or block reviewers from accessing specific submissions (e.g., blocking PC members from accessing the reviews of their own papers).

If you did not disable that option, then PC members are given the option to access a Reviewing-Preferences page where they can specify what submissions they want or do not want to review, and they can also specify if they feel that they have conflict of interest with some submissions. All these preferences are made available to the chair, should you want to use them in assigning or blocking access to submissions.

On the Blocking-Access page you are presented with a list of all the PC members, and for each member you are given a list of the submission-ID's that that member specified as having conflict-of-interests with. For each PC member you can then specify a list of submissions that this member should be barred from seeing. You are then asked again to confirm these choices, and once you do the submissions that you specified are removed from the view of that PC member. (You can access the Blocking-Access page at any time during the review process, should you need to modify these decisions.)
For the task of assigning submissions to reviewers, you are given the choice to either assign everything manually, or to let the software compute an assignment (using the reviewer's preferences) that you can later adjust.

Automatically Computing Paper Assignments

If you did not disable that option, you will find a link off the administration page that points to the auto-assignment page, where you can have the software compute an assignment of submissions to reviewers. The software uses a Stable-Marriage-like algorithm to generate this assignment, as described next.

Recall that in a (generalized) stable-marriage problem, you have a bipartite graph (which in our case represent reviewers and submissions) where each node on the left can potentially have preferences about nodes on the right and vise versa. Also, there are parameters that specify for each node X in the graph the maximum number of nodes from the other side that can be matched to X (i.e., the maximum degree of X). A solution to this problem is some subset of the edges of the complete bipartite graph that satisfies all the maximum-degree constraints. A solution is maximal if you cannot add any edges to it without violating the maximum-degree constraint of one of the nodes. A solution is called unstable is there exist two nodes X₁, X₂ on the left and two nodes Y₁, Y₂ on the right such that:

X₁ is matched to Y₁ but not to Y₂, and X₂ is matched to Y₂ but not to Y₁; and
X₁ prefers Y₂ to Y₁ and Y₂ does not prefers X₂ over X₁.

(This is unstable since X₁ and Y₂ are motivated to dump Y₁ and X₂ respectively and elope together.) A solution is stable is there are no two such pairs. It is well known [1] that one can employ a greedy algorithm to compute a stable maximal solution in time proportional to the number of edges in the complete bipartite graph.

In our case, the PC members specify their preferences for the submissions that they want to review, thus establishing preferences from left to right. In the other direction, the chair is given the opportunity on the Auto-Assign page to specify preferences as to who should (or shouldn't) review what submission, thus establishing preferences from right to left. Note that specifying that "I prefer that Joe is assigned to submission 53" is different than manually assigning submission 53 to Joe. Specifically, this is just a hint to the software and is not binding, which means that you can specify as many preferences as you want without having to worry about load balancing and such like.

Once you got the preferences from all the PC members and also specified your own preferences, you are ready to let the software compute the initial assignments for you. Before the algorithm is run, however, you have to specify some parameters. In particular, you can specify a list of excluded reviewers that will be hidden from the view of the algorithm. This means that the algorithm will completely ignore these reviewers: they will not be assigned any submissions and existing assignments that they have will not be removed. Also, such existing assignments (if any) will not be counted against the number of required reviews per submission. You can use this option, for example, if one of the "PC members" on your list is the PC chair (to whom you do not want to assign any submissions).

Other parameters that you need to specify are how many reviewers are assigned for each submission and whether or not you want to keep existing assignments (if any) or start from scratch. Once you specify all the parameters, the software automatically computes an assignment of submissions to reviewers, and you are taken to the manual-assignment page where you can review and modify this assignment.

Manually Assigning Submissions to Reviewers

The manual-assignment page provides you with two different interfaces that you can use to specify the reviewing assignments. You can use either of these interfaces as you see fit, and once you "save the assignments" with one interface, the other interface is automatically updated as well.

The matrix interface: This interface consists of a matrix of check-boxes, with one column for each PC member and one row for each submission. At the bottom of each column you can see the number of submissions that are assigned to that PC member, and at the right end of each row you can see how many reviewers are assigned to that submission. Note, however, that these counters are only updated when you click the "save assignments" button.

In addition, at the bottom of each column you can find a heiristic indication of "how happy" the PC member will be with the assignment. Roughly, the software computes the fraction of requested papers from all the assigned ones and the faction of assigned papers from all the requested ones, and the "happiness level" is taken as the largets of these two fractions. A level below 25% is "Angry" , 25-49% is "Sad" , 50-74% is "Satisfied" , and from 75% and up is "Happy" .

Every entry in the matrix also displays the preferences that the PC member specified for that submission, encoded as a number between 0 and 5 and colored from red to green (if nothing appears in the entry, it means that the PC member specified the default number 3). Also, if you specified some preferences from the auto-assign page, then the checkbox itself will be colored either red (should not review) or green (should review).
The submission-list interface: This interface consists of a list of submissions, where for each submission you can specify a list of PC members that are assigned to that submission.

Scratch Assignments vs. Visible Assignments

The assignments that are automatically computed by the software or set from the manual assignment page are stored as a "scratch copy" that is only visible to the chair but not to the reviewers. This allows the chair to play around with the assignments before making them publically available.

To make the assignments visible to the reviewers, the chair must check the box to "Make these assignments visible to the reviewers" next to the "Save Assignments" button on either the matrix or the list interface. Hitting the "Save Assignments" button with this box checked will copy the scratch assignments to the visible assignments, thus letting the reviewers see their assigned papers.

Note: As opposed to paper assignments, setting or clearing conflict-of-interests status for papers will immediately be visible to the reviewers.

5c. Tracking progress

After you assigned the reviewing duties, you can track the progress of the reviewing process off the administration page. The Overview page shows a matrix of all PC members and submissions, and each entry in that matrix shows whether a PC member is assigned to a submissions and whether it entered a review of it. If the PC member entered a review for the submission, you can click on the entry to see that review. At the bottom of the page you can manage the "discuss flags" that let PC members see each other's comments and participate in the discussions.

At any time you can go to the Set Submission Status page where you can modify the status of any submission. The available status values are the usual ones, namely "No status", "Reject", "Maybe Reject", "Discuss", "Maybe accept", and "Accept". At the top of the Overview and Set-Status pages you can see how many submissions are currently assigned any of the status values.

Also, at any time you can also go from the Administration page to the Review page (via the links at the top and bottom of the page), where you can participate in the review and discussions just like any other PC member.

5d. Program-committee votes

The software includes support for setting up and managing program-committee votes. You can have the committee vote on things such as "what papers to move from 'Discuss' to 'Maybe Accept'", or "which of the accepted papers should receive the best paper award", or even "who will give the invited talk at the conference". Managing the votes is done from the Voting page. On that page you can set the parameters for a new ballot, edit the paremeters of the ballots that are currently in progress, and see a list of all the ballots (in progrress and otherwise) with a link to the result page for each one.

You can det up a ballot to vote either on submissions or on "other things" (that you need to specify explicitly). Note that PC members can only vote on submissions when they are in the public-discussion phase of the review. Before that they can only vote on "other things" (e.g., invited speakers).

When voting on submissions, you can either include all the submissions in the vote, include only submissions from specific categories (e.g., only submissions in the "Discuss" category), or explicitly list the submissions that you want to include in the vote. Note that if you choose to include in the vote, say, all the submissions in the "Discuss" category, then the submissions that the PC members see when they vote are the ones in the "Discuss" category at the time that they are voting. Put in other words: adding or removing submissions to the "Discuss" category in mid-vote will add or remove these submissions from the ballot.

When you set up ballots, the PC members will see a link from their review page that lets them participate in the current ballots, and you will be able to track the voting results off the Voting page. You can also modify the parameters of a vote, even when this vote is already in progress. Note that if you modify the list of submission-IDs or the list of "other things to vote on" in mid-vote, you must make sure that you do not modify the order of items, since the software identifies vote-items with their position in the list. For example, swapping the order of two items will result in each of them being assigned the tally of the other.

5e. Wrapping up the PC work

After all the decisions are made, the software lets you send to the authors the notification of decisions and the comments-to-authors that were entered during the review period. Just follow the appropriate link off the administration page, and you will be given the opportunity to edit these emails before they are sent. In addition to the contact authors, the chair is also CCed on all these emails.

Note that the Notification and Comments pages only send emails to contact authors of submissions whose status is either "Accept" or "Reject". These pages give you the choice of either sending emails to the contact authors of all the accepted and rejected submissions, or only to specific submissions that you explicitly list. This can be used to hold off on sending emails to some submissions (by setting their status to something other than "Accept" or "Reject") and later assigning the correct status and specifying these submission explicitly.

If you want to keep all the reviews and discussions for future reference, you can find a link off the administration page that packs all of them in one text ascii page that you can then save for your files.

Finally, you have a link to close the review site and activate the site for final-version submission. This link takes you to a confirmation page, where you can also edit the camera-version submission instructions that will be sent to the contact authors of accepted papers. (Once the review site is closed, PC members can still access the site and read the comments and discussions, but they can no longer modify them.)

6. Managing the final-version submission site

After you activated the final-version submission site, you still see a "Review site" box at the top right corner of the administration page, that lets you send emails to authors or change the status of submissions if need be. Other than that, you can see a list of accepted papers (with links to the final version when it is uploaded), create (and then download) an archive file containing the final versions of all the accepted papers, or send email to the authors of all the accepted papers (e.g., to remind everyone that the deadline is in three days).

When the camera-ready deadline arrives, you can shutdown the final-version submission site. After you do that, you are given yet another link that automates the creation of the various indexes for the proceedings. Specifically, you are given the option of specifying the order in which the papers will appear in the proceedings and the length of each paper, and then the software generates for you a LaTeX file (using Springer llncs style) that includes the external-reviewer list, the list of program committee members, the table-of-contents and the author index.

6.a Adding invited talks to the program

During the final-version submission phase you can add to the program invited talks. Doing so will allow the invited speakers to upload their talk to the server, to be included in the conference proceedings just like any other accepted paper.

7. Feedback

I will very much appreciate any comments that you can give me regarding this software. Please contact me at shaih@alum.mit.edu.

[1] Dan Gusfield , Robert W. Irving, The stable marriage problem: structure and algorithms, MIT Press, Cambridge, MA, 1989.