Reviewing process should take the following steps, imho, to progressively come closer to the ideal:

  1. The paper under the review must be publicly available online (at least some version of the paper) -- even before the review, not only after acceptance.
  2. The review request should arrive as a personal email to the reviewer.
  3. All reviewers should see all reviews.
  4. The reviewers should know each others names.
  5. The reviews should be signed.
  6. Signed reviews should be available online together with the paper.
  7. The review should take the form of an open discussion.

Currently I insist only on the first item and will take the liberty of declining to review if the paper (at least an abstract with the results) is not publicly available.
When submitting the review, I will not sign any Terms and Conditions, Code of Reviewer, etc. -- if you can't accept the review by email, please check that your review management system doesn't ask me to sign anything.


If I ask you to kindly do a review for me: Please do not worry, I will not disclose your identity to the authors and the other reviewers (unless asked), won’t put your reviews online, etc.; this is my Reviewer's manifesto that applies to me as a reviewer, but doesn't have to apply to you if you agree to review for me.


Ruminations below attempt to argue in favor of the items above.

Online availability makes the progress go faster. Argument of the kind “We can’t make the paper available before publication b/c the results may be stolen” looks flawed to me b/c putting the paper on smth. like arxiv (with a timestamp) only helps claiming the authorship in any dispute (which I believe actually rarely arise). In addition, I have enough secrets to keep and don’t want to go around with more in the form of “I know such and such result has been obtained and I’m eager to discuss it but can’t do it with you before it’s published.” (I also hate to be asked to destroy the paper after reviewing; in any case I can’t promise to delete it from my memory). Last but not least, this promotes presentation of results at conferences/workshops.
I’m happy to be in good company demanding paper availability: FoCS strongly encourages the authors to post the submissions (submissions back then were non-anonymous); one of their arguments is that papers not written well enough for public dissemination are probably not ready for the submission. "The Nature journals support the posting of submitted manuscripts on community preprint servers such as arXiv and bioRxiv."

Email contact. More and more scientific publishing scam floats around and it’s getting hard to distinguish it from valid review requests coming, e.g., from smth. like taes@msubmit.net (hello, Trans on Aerospace and Electronic Systems) asking to create an account on some cite that the respected journal is using for editorial management; in extreme cases, the account creation is a multistep procedure with requests to sign some kind of “Reviewer’s code” confirming that the reviewer won’t harm the authors (I shouldn’t have been asked to do the review if there’s a worry I might do the harm).
An added value of personal email is that the editor may introduce his/herself to the reviewer in the not-so-rare case that they don’t know each other (pardon my ignorance, but I often don’t know the person asking me to review, in which case I think it would be polite to, e.g., mention who took the risk of recommending me as the reviewer, why I’m being kindly called an expert on the topic, etc.).

Seeing the other reviews (after completing one’s own) would help getting more thoughts on the topic and (who knows?) maybe coming up with new results.

Knowing the other reviewers would help finding the best experts (after the authors) on the subject of the paper, to possibly discuss the topic and get new results.

Review signing would make the reviewer more responsible and help the authors finding experts on their paper. (I sign almost all my reviews and know other, prominent researchers who do so).

Availability of signed reviews would let everyone know what somebody else thought of the paper. Note that reviews of published papers is an existing practice at the AMS Mathematical Reviews (MathSciNet); comments on papers are also available, e.g., at JOSIS. Some researchers have the courage to post reviews of their papers online -- very cool! .

Open-discussion review is a step towards ScientificPublishing 2.0 where papers would be crowdsourced, with the collective effort allowing to produce results faster. In particular, it will increase the amount and frequency of communication between the authors and the reviewers (in the current submit-review-revise-resubmit scheme it’s generally limited to 2-3 instances with month-long breaks), letting, e.g., clear up Definition 1 fast, before the rest of the paper is read with the ambiguous definition in mind. OpenReview.net is great!

I’m aware of drawbacks in the proposed steps, but not commenting on the downsides here because many of them were revealed in conversations with people who might not want their opinions to be disclosed; if you conveyed your thoughts to me and would like them to be added here, please let me know. Other forms of discussion are welcome too.


Related (imho) pages: Reproducibility (also here), Academic spring, Till Miltzow's modificaiton to The Cost of Knowledge

Update: A colleague pointed out that what I'm promoting is Interactive Public Peer Review

Points against signing reviews: reviews may be weighted by who wrote them (is it good or bad?)

JOAS seems to do what I want