[ecoop-info] Call for Contributions: Evaluate'12 @ PLDI - Improving Experimental Evaluation through Education

[Matthias Hauswirth]

=====================================================================
                       Evaluate 2012

            Workshop on Experimental Evaluation
        of Software and Systems in Computer Science

  Theme: Improving Experimental Evaluation through Education

                  CALL FOR CONTRIBUTIONS

          http://evaluate.inf.usi.ch/evaluate2012

 Co-located with PLDI/ECOOP'12 in Beijing, China, June 15, 2012.
=====================================================================


What should students learn about experimental evaluation?
How should we teach proper experimental evaluation?

Contribute your position -- submit a short, unformatted position statement
on "Improving Experimental Evaluation through Education" to Evaluate'12.

Submit now (deadline this Friday, April 6).



Location

Evaluate 2012 is co-located with PLDI/ECOOP 2012 in Beijing, China.


Important Dates

Submission deadline: Friday, April 6, 2012
Notification deadline: Thursday, April 19, 2012
Workshop date: Friday, June 15, 2012


Context

Our effort originated at Evaluate 2010, the first workshop on
Experimental Evaluation of Software and Systems in Computer Science,
at SPLASH/OOPSLA 2010. The main outcome of that workshop was our open
Letter to PC Chairs, which continues to gain support from a growing
group of strong researchers.

Many of the Collaboratory members met again at Evaluate 2011
(co-located with PLDI 2011, which that year was part of FCRC 2011) on
Sunday, June 5, in San Jose. The goal of this second instance of the
Evaluate workshop was the development of a coherent set of
experimental evaluation anti-patterns, with the aim of allowing the
community to learn from our past mistakes. The resulting paper is
available as Evaluate Collaboratory Technical Report #1 and under
review for publication.


Call for Contributions

We are organizing this workshop out of concern for the health of
empirical science within the software and systems area of computer
science.

Classic empirical sciences (such as biology) have a deeply ingrained
culture of the scientific method and rigorous evaluation. On the other
hand, computer science struggles with the role of evaluation. This
ambivalence or confusion is most likely rooted in the reality that our
discipline encompasses both formal and empirical science. The former
is based on formal systems, and has no need for empirical observation.

Evaluation is fundamental to empirical science. Weak evaluation
retards science: it limits our creativity by failing to reveal
opportunities, and it exposes us to derailment by incorrect findings
due to invalid results.

The goal of this workshop is to raise the quality of empirical
evaluation in the area of computer systems and software through
education. We will create a set of effective teaching resources for
strengthening the education on experimental evaluation in existing
courses of the CS curriculum. The workshop will bring together expert
researchers in our field, collect their best ideas and experiences for
how to educate students about evaluation, and structure and organize
the resulting teaching material into a coherent set of educational
resources.


Workshop Format

The workshop will not be a mini conference, but a true "work" shop
with problem solving and brainstorming sessions. We solicit
submissions consisting of brief statements. The workshop organizers
will evaluate the submissions. Like we have done at Evaluate 2011, we
plan to ask each participant to present a 1-slide 90-second summary of
their statement, and we will use the remainder of the day for
structured discussions and collaboration sessions with the goal of
organizing the contributions into a coherent whole.


Publication/Dissemination of Contributions

We do not intend to publish workshop proceedings. The goal of our
workshop is to put together a coherent set of effective teaching
materials on experimental evaluation, and to make that material
publicly available on the Evaluate Collaboratory. Moreover, to inform
the community about our resources, we will discuss that material in a
submission to a venue such as the SIGCSE conference or the
Communications of the ACM.

We will make the submitted statements available on the Evaluate
Collaboratory for all participants before the workshop. We will ask
the participants to read all statements before the workshop. During
the workshop we will collaborate on reorganizing and integrating ideas
and material. We will continue that collaboration after the workshop
to make the resources publicly available and to write and submit the
common paper.


Submission Information

We solicit submissions consisting of statements addressing one or more
of the following three questions. We grouped these questions into two
perspectives:

Consumer perspective:

(1) You are a PhD advisor and get a new PhD student. What would you
like that PhD student to know about experimental evaluation?
(2) You are an employer and hire a new graduate (BS, MS, or PhD in
CS). What would you like that graduate to know about experimental
evaluation?

Producer perspective:

(3) You are a teacher and are using creative means for teaching
students about experimental evaluation. Could you share your ideas?

Please submit your statement on easychair in the form of an
abstract-only submission (just paste the text into the abstract
field).


Organizers

Steve Blackburn (Australian National University)
Amer Diwan (University of Colorado / Google)
Matthias Hauswirth (University of Lugano, Switzerland)
Peter F. Sweeney (IBM Research)


Past Workshops

Evaluate 2012 is the third workshop of the Evaluate workshop series.
The first Evaluate workshop, Evaluate 2010, was held at SPLASH/OOPSLA
2010 in Reno/Tahoe, Nevada, USA, and in 2011 we held Evaluate 2011 at
FCRC/PLDI in San Jose, California.