Problem and System Ratings

Problems and Data

• Use TPTP problems
• Exclude biased problems
• Divide into Specialist Problem Classes
• Use TSTP data within each class
  (resource limits beyond the systems' PPPs)

System Ranking by Subsumption

• Superset systems subsume subset systems
• Provides a partial ranking (sample data)
• SOTA contributors form a SOTA system

Problem Rating by SOTA

• Use SOTA contributors to avoid weak systems
• Rating is the fraction of SOTA contributors that fail (sample data)
• Discrete ratings: Easy - Difficult - Unsolved

System Rating by SOTA

• Use difficult problems to differentiate between systems
• Rating is the fraction of difficult problems solved (sample data)

Reports available online at www.tptp.org/cgi-bin/SystemOnTPTP

---