ProbabilityOfSuperiority.hpp File Reference

Plans regarding the method for comparing two solvers according to [Mladen Nikolic, 2010]. More...

Go to the source code of this file.

Detailed Description

Plans regarding the method for comparing two solvers according to [Mladen Nikolic, 2010].

[Mladen Nikolic, 2010, Statistical Methodology for Comparison of SAT Solvers] is to appear in SAT 2010 proceedings.

The wider framework
Specifications for ProbabilityOfSuperiority.R
  • All functions need to be specified.
  • gwtest
  • gehanw
  • rcor
  • jackknifevariance
  • bootstrapvariance
  • probsup_solvcmp
  • probsup_solvcmp_files
  • read_probsup_solvcmp
Write docus
  • How is the main function to be used? Examples?
  • Input specification:
    1. DONE (yes, this is the case) It appears that the input is just a data-frame file, which per row contains the times for the two solvers A, B to be compared.
    2. DONE (yes, needed) Is the knowledge of the cutoff-time needed for the evaluation?
    3. DONE (one row per instance As discussed in the paper, there are actually dependencies between the rows, in that a block of rows belongs to a single instances and its shuffled variations --- however it seems for the computations performed this is irrelevant?
  • Output specification:
    1. Besides the precise specification in computational terms, also the "meaning" needs to be specified.
    2. Best if a precise probabilistic semantics can be given.
    3. In any case, we need also a good discussion of the "intuitive meaning".
    4. What shall be achieved by the comparison? Is the underlying (precise) model, that one solver is better than the other iff the expected run-time over the space of sample inputs is lower?

Definition in file ProbabilityOfSuperiority.hpp.