Poorly-Explained NeurIPS 2020 Desk-Rejects Peeve ML Researchers

Many researchers awaiting a decision on whether their papers would make it to this year’s NeurIPS conference received word this week — but the notifications have provoked more questions than they answered. NeurIPS desk-rejects (papers passed over without review) has become “a super noisy process,” complains a post in the Reddit Machine Learning group.

Over the past three weeks, NeurIPS 2020 program chairs have tasked area chairs (AC) and senior area chairs (SAC) with the job of summary rejection (aka desk-rejects): ACs did a light read of the papers and identified those they believed would not be accepted, and SACs cross-checked the selections, according to a NeurIPS blog post.

“We wish that every paper could be fully reviewed,” the program chairs wrote. “However, the growth of the field has made it difficult to do so, and we have chosen to explore the use of summary rejections to limit reviewer load.”

Sinead Williamson, an assistant professor at the University of Texas at Austin Department of Statistics and Data Sciences and a machine learning researcher, tweeted disappointment at having her work rejected with only a check-box response: “It’s the lack of feedback that rankles,” she wrote, “I want to improve the paper.”

Oh, great. Neurips desk reject. If the results are “known results” (I’m pretty confident they aren’t), perhaps the area chair could have made the effort to leave a note telling what they are? 1/n

— Sinead Williamson (@sineadwilliamso) July 13, 2020

Another researcher whose paper received a check-box style rejection shared a screenshot of the notification under Williamson’s tweet:

Toyota Technological Institute at Chicago research assistant professor D.J. Sutherland opined that the paper comment format is the problem. Conference area chairs are asked to check boxes corresponding to one-sentence reasons for a paper’s rejection, and can only provide more detailed explanations by selecting the “other reasons” box down the list, he explains.

Williamson is herself a NeurIPS 2020 area chair, according to one of her tweets. She says she desk-rejected two papers and left them as long a review as the space allowed. “Feels like common courtesy,” she said.

Other ML researchers echoed her disappointment in the lack of useful feedback. Suvrit Sra, associate professor of MIT Electrical Engineering & Computer Science, tweeted: “This is a complete fail of the @NeurIPSConf desk-rejection business (which I felt was already unfair to begin with).”

Sra shared the story of desperate students who asked him what to do next, as “the desk reject provides them with absolutely no information about what went wrong.”

ETH Zurich researcher Aurelien Lucchi, who’s also a NeurIPS area chair, tweeted he’d tried to explain why as best as possible in his desk-rejections, adding “I’m very disappointed to see that some ACs think they can just reject a paper by ticking a box with a generic (mostly non-informative) explanation.”

Lucchi proposed that NeurIPS make the optional explanation box mandatory and increase the word count limit.

The 34th annual gathering of the Conference and Workshop on Neural Information Processing Systems, NeurIPS 2020 will be a virtual-only event running December 5 through 12, the organizers announced last month. The paper submission deadline was June 5.

With 38 percent more paper submissions than in 2019, this has been another record-breaking year for NeurIPS. About 11 percent of the submissions received desk-rejects and will not be reviewed. The remaining 8,186 papers have now been assigned to the reviewers — beginning a new and no-less-anxious waiting period for involved researchers.

---

Journalist: Yuan Yuan | Editor: Michael Sarazen

---

Synced Report | A Survey of China’s Artificial Intelligence Solutions in Response to the COVID-19 Pandemic — 87 Case Studies from 700+ AI Vendors

This report offers a look at how the Chinese government and business owners have leveraged artificial intelligence technologies in the battle against COVID-19. It is also available on Amazon Kindle.

Click here to find more reports from us.

---

We know you don’t want to miss any story. Subscribe to our popular Synced Global AI Weekly to get weekly AI updates.