Miscellaneous ideas

[rougier]

• Associate editors could bring useful testimony on the practice in their field regarding reproducibility problem and the extent of the problem. How are replication considered ? valuable ? Is there any place where such reproduction/replication can be published, etc.
• Active reviewers could explain the whole review process and how different it is from regular review, how different is the public interaction with the author, how hard it is to review code, etc.
• Authors could explain what is the initial motivation for replicating results of a paper. Did they already replicated some results before ReScience and what did they do with such implementation? Was the review useful at all ? Did they get new interaction following the publication? etc.

[tpoisot]

Could we also interview, informally, editors of computational/theoretical-heavy journals and ask for their input? Reproducing is much more intense scrutiny than peer-review.

[rougier]

Oh yes, if you have some contact that would bring useful information

[khinsen]

@tpoisot Nice idea, but I suspect it will be difficult to find people who accept.

[oliviaguest]

I can think of a couple of people who might accept. Besides you don't lose anything if you have questions handy, just send them out and see who replies. To what end though? Is the idea to just include excerpts in the article? Or inform ourselves?

[rougier]

I think it would reinforce our position if we were able to explain our view on the (bad) state of reproducibility is shared by several people/editor in many domains but I'm not sure what's the best way to include such statement. If they're ok to be cited, excerpts as you suggest @oliviaguest might be one solution.

[oliviaguest]

If there really is a problem there won't be much convincing needed for the paper since the reviewers will be already on-board.

[jsta]

I am going to post my response to @rougier 's prompt for authors below. Maybe it will spark other ideas.

I was motivated to replicate the results of the original paper because I feel that working through code supplements to blog posts has really helped me learn how to science. I could have published my replication as a blog post but I wanted the exposure and permanency that goes along with journal articles. This was my first experience with formal replication. I think the review was useful because it forced me to consider how the replication would be used by people other than myself. I have not yet experienced any new interactions following publication. However, I did notify the author of the original implementation about the replication's publication. I think this may lead to future correspondence. The original author suggested that he would consider submitting his own replications to ReScience in the future.

[oliviaguest]

It's nice to know your experience was positive. Mine have been overwhelmingly negative. So much so, that even though I'm privately willing to give details, publicly going through the actual words said by some is out of the question.

[oliviaguest]

PS: I linked my SSI co-fellows, and general SSI people, etc., to this repo. I bet they will have some more useful feedback/ideas. :)

[rougier]

As authors, our initial motivation for replicating a model was that we just needed it in order to collaborate with our neurobiologist colleagues. When we arrive in our new lab, the model has just been published (2013) but the original author (postdoc) left the lab a few month before we arrive. And of course, there were no public repository, no version control and the paper describing the model was incomplete and partly inaccurate. In the end, we get our hands on the original sources (6,000 lines of Delphi) only to realize we could not compile them. It took us three months to replicate it using 250 lines of Python. But at this time, there were no place to publish this kind of replication and share the new code with colleagues. Since then, we have refined the model and made new predictions that have been confirmed. Our initial replication effort really gave the model a second life.

[benoit-girard]

Replicating previous work is a relatively routine task every time we want to build a new model: either because we want to build on it, or because we want to compare to it. We also give replication tasks to M.Sc. students every year, as projects. In all these cases, we are confronted to incomplete or inaccurate model descriptions, as well as to the impossibility to obtain the original results. Contacting the original authors sometimes solves the problem, but not so often (the "the dog ate my hard drive" syndrom). We thus accumulate knowledge, internal to the lab, about which model work, which doesn't, and how a given model has to be parameterized to really work. Without any place to publish it, this knowledge is wasted. Publishing it in ReScience, opening the discussion publicly, will be a progress for all of us.