The PyCon review process consistently produces a conference that I find both interesting and entertaining. Unfortunately, it's also clear that the process of reviewing talks has been a brutal amount of work.
During the process last year, daily IRC meetings slowly ground through the huge field of proposals, winnowing them down in a process that consumed prodigious quantities of volunteer hours. It was a difficult process to join as a new member, and not very forgiving for those who couldn't figure out a way to devote an hour of their afternoon every day.
Last year, work began to make a portion of this process asynchronous. It was a solid first step! Still, I realized a custom tool to review talk proposals was really necessary. I want to preserve as much of the feel and quality of the synchronous process as I can, while opening things up to more folks by making the process itself asynchronous.
Committee Purpose
It's worth explicitly stating that the purpose of the program committee is to evaluate proposals, not coach those making the proposal. Early in the review process, before the deadline, we might reach out to someone with a promising proposal that needs just a little bit more work or guidance. Still, we can only judge a talk based on the proposal in front of us.
The sheer size of the task makes it hard for us to work with individuals putting together proposals. There are a lot of groups interested in helping folks put together solid PyCon proposals, and we encourage more of those groups to form! Local Python meetups are a great way to find past speakers and committee members eager and willing to help you put together a proposal.
Screening
The talk review process happens in two steps. The first step, "screening," is an attempt to separate out talks that don't really have a place at PyCon. In this stage, we ask talk reviewers to consider a talk proposal as objectively as possible.
A talk can move from the screening stage either by meeting the six minimum screening guidelines according to a certain percentage of reviewers, or because enough reviewers nominated the talk to move into the next round anyway despite not quite meeting all six criteria.
The UI looks something like this:
Minimum Screening Guidelines
1. This proposal follows PyCon's Code of Conduct
All talks must follow the Code of Conduct. There can not and will not be any exceptions. If you are unsure about a talk, reach out to me, or my co-chair Karen, and ask for feedback.
2. This proposal is complete, clearly written, and articulate.
All sections of the proposal are filled out; the proposal indicates that it was put together with some care and thought. Reviewers can basically understand what the talk is about, even if they're not experts in the talk topic.
3. This proposal has a thorough and achievable outline.
The outline is one of the best tools the committee has to understand what your talk has to offer. Please take the time to outline your talk in a fair amount of detail. Put timing on sections so we can understand the talk's emphasis. We consider the outline in the SpacePug sample proposal the minimum standard for a successful PyCon proposal's outline.
4. This proposal has a coherent subject.
Talks should be about one thing, and should cover that one thing well. There should be a central idea or concept the audience should learn.
5. This proposal is about the Python ecosystem.
This conference is, after all, PyCon!
Talks do not need to be 100% about Python to pass this criterion. Proposals should be clear how their material is relevant to Python and Pythonistas.
6. This proposal has a sufficient audience among attendees.
Talks need audiences. Talks about things which don't appear to be popular in the wider Python community probably won't fare much better at PyCon. Stating explicitly in your proposal who the talk is targeted towards and what attendees will get out of seeing the talk will help convince us there is a sufficiently large and/or passionate audience.
Batch Review
In batch review, the second step, we take the talks that made it through screening, either via nomination or by meeting all the screening criteria, and group them into small clusters of proposals with similar topics.
Reviewers then review batches and nominate zero, one, or two talks to move on to PyCon, comparing a talk to its peers. Reviewers will also comment on the batches and attempt to convince one another as to which talk or talks they should vote for.
The voting UI looks something like this:
Schedule Creation
As program chair, I then spend a lot of time with slips of paper with the accepted talks, arranging and rearranging things until I'm happy. There are a few other things that happen at this stage as well. For example, talks which ask for 45 minutes get considered for the limited 45 minute slots. If someone has had multiple proposals accepted, we'll either accept the proposal that seems to best fill out the schedule, or we'll ask them which talk they prefer.
Unlike previous years, this year speakers at PyCon will not be given more than one talk slot. This does not include other speaking opportunities at PyCon, such as tutorials or lightning talks.
We also ask the program committee to nominate talks that they think were overlooked or unfairly eliminated in the process, bringing them back into consideration to fill any gaps that might arise in the schedule.
Once all of this is done, I upload the results. We publish the schedule, and start notifying speakers!
Questions?
If you have any questions about the process, feel free to send me an email at njl@njl.us.