CEC'2004 Prisoner's Dilemma competition

During last Conference on Evolutionary Computation a competition, celebrating the 20th anniversary of Robert Axelrod The evolution of Cooperation book publication, has been organized by a team of people headed by Graham Kendall from University of Nottingham, UK.

Partial results of those competition have been made available through the web site of the competition.

As some other people having participated to this competition we have studied those results in order to be able to understand how they have been produced. We thus noticed that those results have absolutely no signification due to the way the competition have been produced.

One of the participants, from University of Southampton, UK, have however pretented its team have finally beated the tit_for_tat strategy. Even more its team published and began to spread those results.

A new competition based on approximatively the same rules and methods is planned to be done for another conference next year.

As others, we feel that it was important to react to such non scientific behavior, by replying to the call for particiption for the next competition with the mail reproduced here. If you participated to the first competition (at CEC'2004) and agree with us, do not hesitate to send us a note so that you could be added to the list of signatories.

Following a mail sent by Elpida Tzafestas just after the CEC'2004 Iterated
Prisoner's Dilemma competition results publication and after some public announcement
made those last week by Southampton University, relayed in web media such as Wired and
Slashdot and some others blogs, we think it is very important for us
to react in 3 short points about this competition:

* the first prisoner's dilemma competition at CEC'2004 has been made without
   any scientific spirit

* results made by those public announcements are definitively not new
   information and are known since a long time

* new rules established for the new prisoner's dilemma competition does not
   fix problems introduced by the first competition

Let us detail those 3 points :

* The first competition has been made without any scientific approach or
   spirit. As it has been suggested by Elpida in its reaction no computation
   methods are normalised. Even more serious, no results may be accessible and
   thus reproducible before the end of 2005 while serious doubts have been
   emitted on this first competition results.

   Some participants have proposed more than one strategy, while for almost all
   other (and in any case the vast majority of participants) it was implicit
   that only one could be proposed by players.

   Some players have even proposed several times the SAME strategy, while it is
   well known that it completely changes the meaning of such a tournament (see
   for instance Axelrod's original book).

   Just for information, and using only given data (that is name of strategy,
   and of participant and obtained scores) : 26 strategies have been used more
   than once (23 used 2 times, 2 used 3 times, and one used 4 times) and
   approximatively 9 of the 43 participants proposed more than 1 strategy with
   the winner proposing more than 107 strategies on the 223 finally used.

   If people had only submitted one strategy this competition would have
   counted only 43 participants which is small. For instance, more than 10
   years ago, we organised an open competition, in association with the french
   edition of the Scientific American, which involved more than 90 differents

* Results and their interpretation publicly announced by the University of
   Southampton which has been spread, and are still spreading, are definitively
   no news.

   Defeating tit_for_tat, or "beating it" as it is often written, in the
   classical iterated prisoner's dilemma has been done since a long time by
   different people (including us). The quality of tit_for_tat has been call
   into question in a lot of not so recent references (see for instance [1] and
   [2]). We even have studied the idea of master/slaves strategies, today
   presented as NEW result, for instance in [3].

   Almost the same quantity of references on the iterated prisoner's dilemma
   with noise may be found (with another difficulty in the definition of what
   noise should be).

   We can ask ourselves if people making those announcement and those spreading
   those *new results* have study a little bit the bibliography on the subject
   (see for instance here and here).

* New rules established for the next competition do not fix problems
   introduced by the first competition :

   + only one competition is changed to accept only one strategy by
     participants, which means that almost nothing will changed in the results

   + this is still a one-shot tournament. Results are not averaged on
     differents runs of simulations. We all know problems which may arised in
     such situation due for instance to random number generator uses.

   + there is still no normalisation in the computation of score, which means
     that all strategies are not treated in the same way as it is done in all
     serious game theory simulations.

For all those reasons we symbolically refuse to participe to, and ask for the
substraction of our proposition from, the CIG'05 prisoner's dilemma
competition. We publish this statement on our web site and invite all people
agreeing with it to co-sign it by sending us an email asking to do so.

We are, however, ready to evaluate any strategies which will be send to us. We
have published methods and implemented all the tools necessary to evaluate the
*quality* of a strategy. Some of them are even available since a long time on
our project web site http://www.lifl.fr/ipd.

Pr Philippe MATHIEU

References :

[1] "A strategy of win-stay, lose-shift that outperforms tit-for-tat in the
     Prisoner's Dilemma game" by Martin Nowak and Karl Sigmund in Nature
     volume 364 in 1993
[2] "Our Meeting with Gradual, A Good Strategy for the Iterated Prisoner's
     Dilemma" by Bruno Beaufils, Jean-Paul Delahaye and Philippe Mathieu in the
     Proceedings of the Artificial Life V conference in 1996.
[3] "L'altruisme perfectionné" by Jean-Paul Delahaye and Philippe Mathieu in
     Pour La Science (French Edition of Scientific American, volume 187 in 1993.

Copyright © 2004 by LIFL, <prison@lifl.fr>