A preliminary analysis of AAAI-99 submissions

1
A preliminary analysis of AAAI-99 submissions

• Devika Subramanian
• Rice University

2
Distribution of 400 submitted papers by area
3
Trends in submissions

• Multi-agent/agents, KR, reasoning/UAI, planning,
  learning are the largest constituencies.
• Robotics papers are coming into AAAI however
  these are mobile robot papers.
• Neural computation papers are not a large part
  of the submissions.

4
Overall acceptance statistics

• Before SPC meeting
• Accepts (105)
• Undecided (50)
• Rejects (255)
• After SPC meeting
• Accepts (104)
• Conditional accepts (5)
• Rejects (290)
• 15 papers accepted with 1 A/2M or R.
• 3 papers rejected with 2 A/1 strong R.
• Every other accepted paper had two As.
• 32 accepted papers had 3 As! (just under 30)

5
Distribution of 109 accepted papers by area
6
Trends in accepted papers

• Search, planning, neural computation, and
  information extraction/information retrieval have
  higher than average acceptance rates.
• Data on information extraction/information
  retrieval and neural computation unreliable
  because of small sample size.

7
Other trends in papers

• AAAI still attracts the best work in planning,
  constraint satisfaction, search, multi-agent
  systems and KR.

8
Other trends in papers

• The best work in machine learning, uncertainty,
  KDD, neural computation, natural language and Web
  agents (not including multi-agent systems) is not
  being submitted to the conference.
• The work in mobile robotics submitted to the
  conference is not competitive with the work
  represented at the top vision/robotics
  conferences..

9
Any new ideas in the papers?

• Proverb a system that solves NY Times
  crosswords. A tour-de-force integration of ideas
  in AI.
• Hybrid approaches to collaborative filtering.
• New extensions to Graphplan and Satplan.
• Active learning analysis and implementation.
• Integrating the fields of constraint satisfaction
  and classical planning.

10
Remarks gathered from SPC

• Number of NLP papers submitted to AAAI is
  increasing they are of much better quality than
  in years past. Still not the very best papers
  (which tend to go to ACL), but solid work at the
  intersection of statistics/machine learning and
  NLP.
• Neural computation papers should be reviewed on a
  special track as they were this year (with two
  special SPC members Giles and Sun) even though
  the actual number of submissions from that
  community was small this year.

11
More remarks from the SPC

• Need to develop mechanisms to get the best work
  in machine learning, UAI, agents, KDD, robotics
  and neural computation to be submitted to
  conference.
• Many were concerned about the fact that most
  accepted papers were incremental advances.
• signs of a maturing field? Or self-selection
  among submissions because of reputation as
  archival conference?
• Mechanisms to detect and encourage revolutionary
  work among the submissions not working well.
• reflects fundamental split in community on what a
  significant result is, and about the extent of
  evaluation needed to prove that an idea works.

12
Some SPC statistics
Acceptance rates vary widely among
SPC. Partially explains variation in acceptance
rates across sub-areas. Mean26.37 stdev10.22
13
110 PC members
14
SPC statistics for AAAI-98
Acceptance rates vary widely. Mean30.95 stdev12
.33
15
204 PC members
16
Issues to consider for AAAI-2000 and beyond

• Size of SPC and PC to get consistent reviewing
  standards.
• Ways of attracting work-in-progress to AAAI and
  to set good evaluation guidelines for them.
• Reconsider need for an SPC meeting.
• Paper assignment to reviewers benefits from a
  manual component. Consider providing electronic
  access to paper abstracts and to assignment
  software so chairs can teleconference and do
  reviewer assignment.
• Consider accepting few papers for plenary
  presentation (say 20) and have all papers
  presented at poster sessions.
• How to exist and cooperate/compete with
  speciality conferences.