Does your paper really suck?

By A. Sina Booeshaghi · June 27, 2026

Oded Rechavi, at QED Science, believes that if your paper is not in the
top 1% of their QED score then it
“sucks”. But what is this QED score and what is its purpose? Does it really
measure scientific quality? If a paper is not in the 1% does it really
suck?

These are important questions because scientists are increasingly
overwhelmed with the volume of new work posted on preprint servers and
published in journals. As a result, traditional quality signals used
for triaging papers, such as journal, conference venue, and institution,
are becoming less reliable. AI further compounds this problem by making
it easy to produce plausible scientific writing at scale. Papers are
longer, figures are denser, and the existence of a paper is no longer
sufficient evidence that it represents substantial scientific work.

In response, companies like QED Science are building AI tools to help
scientists identify quality work. QED uses Large Language Models (LLMs)
to review scientific papers and provide AI feedback. Many scientists
report that the feedback is useful and often resembles comments received
during human peer review.

QED recently released a
white paper
that goes one step further and describes the “QED Score”, a single
number that is intended to measure a paper’s quality. The QED score is
generated by prompting a collection of LLMs to review a paper for
“originality” and “validity”. The resulting evaluations are combined
into a single score, the QED score. In their white paper, the authors
claim that the QED score is a “more accurate, faster, and less biased
estimate of paper quality than journal rank.” The authors present three
validation studies, all of which compare the QED score against the
SCImago Journal Rank (SJR), a journal-level metric based on citation data. The first study
compares QED and SJR against a corpus of expert-assigned labels
(“Limited”, “Satisfactory”, and “Strong”). The second compares QED
scores for 2,879 bioRxiv preprints with the SJR of the journals in which
those papers were eventually published. The third asks experts to choose
between pairs of papers where QED and SJR disagree most strongly.

In this review, I evaluate the evidence supporting the QED score as a
measure of scientific quality. While QED clearly provides a much faster
review than traditional peer review, I find that the evidence presented
does not support the authors’ claims that the QED score is a more
accurate or less biased measure of scientific quality.

Case study 1 is methodologically opaque and does not effectively
demonstrate that the QED score measures quality

In case study 1, the authors obtain a curated dataset of 975 published
papers labelled “Limited”, “Satisfactory”, or “Strong” by a panel of
expert reviewers whose identities are not disclosed. Each paper received
a label based on validity and originality, the same criteria used to
generate the QED score. The authors then asked whether the QED or the
SJR score better predicted these labels. QED achieved an AUC of 0.863
versus SJR’s 0.804 for distinguishing “Limited” from “Satisfactory +
Strong” papers, and 0.782 versus 0.774 for distinguishing “Strong” from
“Satisfactory + Limited” papers.

These values cannot be meaningfully interpreted without the underlying
data and methodology. The paper does not report the distribution of
labels, whether the expert reviewers who generated the benchmark labels
were blinded to journal, author, or institutional identity, nor do they
provide any data or code to reproduce the analysis. The authors also
provide no guarantee that these papers were excluded from the training
data of the LLMs used to evaluate them. Therefore, case study 1 does not
establish that the QED score accurately measures scientific quality.

Case study 2 provides inconsistent evidence that the QED score measures
quality

The second case study compares QED scores for 2,879 bioRxiv preprints
with the SJR score of the journals where those preprints were
eventually published. Across all fields, the authors report a Spearman
correlation of 0.63. Within individual fields, however, the correlations
ranged from 0.78 (Genetics) to 0.39 (Systems Biology).

The authors describe the overall agreement as “substantial”, but
explain weaker agreement in some fields by arguing that the SJR score is
a noisy proxy for quality. This argument is internally inconsistent. If
the SJR score is a reasonable proxy for scientific quality, then the
weaker agreement across fields suggests that the QED score is a weak
proxy for quality. If the SJR score is a noisy proxy for scientific
quality, then agreement with the SJR score cannot be used to validate
the QED score. Either way, by the authors’ own admission, this analysis
does not establish the QED score as an accurate measure of quality.

Case study 3 contains several uncontrolled and unexplained sources of
variation that may bias the QED score’s validation

The third study asks 15 domain experts to compare papers where the QED
and SJR score disagree most strongly. For each paper the authors
subtract log(SJR + 1) from the QED score, compute pairwise
contradictions, and retain the 100 strongest disagreements. Only 70 of
these pairs were reported with “confident” expert judgments; the
remaining 30 were discarded. Among the retained pairs, experts preferred
the higher QED-scored paper roughly three times as often as the higher
SJR-scored paper.

This experiment introduces several uncontrolled and unexplained sources
of variation. First, the QED score is a paper-level metric assigned to a
preprint, whereas the SJR score is a journal-level metric assigned after
peer review. Second, comparisons are made between two different papers
where expert preference may depend on writing style, topic, or
familiarity with the field rather than scientific quality. Finally, the
authors do not explain how “confidence” was defined or why 30% of
comparisons were excluded. Consequently, case study 3 does not provide
sufficient evidence for the superiority of the QED score.

The QED score exhibits geographical bias

The QED score is not just an internal metric. QED publicly released
rankings of the top 1% of bioRxiv preprints and this public release
reveals substantial geographic bias against African and South American
scientists. (Side note that although the white paper states that QED
scored 57,455 bioRxiv preprints, the publicly accessible website
contains 53,938 domain-assigned preprints (571 in the top 1% and 53,367
in the remaining 99%). The discrepancy is not explained.)

The QED website assigns papers geographic regions based on author
affiliations. A paper may belong to multiple regions (e.g. North
America, Europe, Asia, Australia, South America, Africa), meaning a
single author is sufficient for a paper to be classified as African.
Filtering papers by geographical region on the QED website produces a
striking result: only three papers in the top 1% are classified as
African. Yet none is led primarily by African institutions.

The first paper in the top 1%,
TENM4 is an essential transduction component for touch, has 20 authors with primary affiliations in Germany; it is classified
as African because one author has a secondary affiliation in Egypt. The
second paper,
Memory Regulatory T Cells Reprogram into Protective Tfh-like
Effectors in Recurrent Malaria, has ten authors, only one of whom has an African affiliation. The
third,
Modular and redundant genomic architecture underlies combinatorial
mechanism of speciation and adaptive radiation, has eleven authors, again with only one African-affiliated author. In
other words, the top 1% contains no paper led primarily by African
institutions.

In contrast,
Inflammatory Biomarkers of Asymptomatic and Symptomatic
Tuberculosis
addresses a disease that disproportionately affects sub-Saharan Africa
and includes 28 authors with primary African affiliations and only six
with primary European or North American affiliations. Despite being far
more representative of African science, it was ranked in the bottom 99%.

Taking this a step further, the regional classifications exhibit
significant biases. Using the regional classifications reported by QED,
African classifications (3 vs. 933; p = 0.004) and South American
classifications (11 vs. 2,204; p = 0.00055) are significantly
underrepresented among papers in the top 1% relative to the remaining
99%.

Note: Ran Blekhman published
a complementary analysis
demonstrating that the QED top 1% reproduces familiar institutional
biases.

An important sanity check

As an informal experiment, I submitted the QED white paper to QED
itself. The system assigned it a QED score of 46 and identified several
methodological concerns. While this is not a formal validation – the
system was not designed to evaluate methodological white papers – it is
an interesting observation that QED itself identified some, but not all,
of the methodological concerns discussed throughout this review. I’ve
included a link to the review report generated by QED here:

QED review report for the QED Score white paper (PDF)

How do we effectively triage papers?

The rapid growth of scientific publishing is a real problem. AI has
lowered the cost of producing convincing scientific writing, making it
increasingly difficult to identify work worth reading. My concerns with
QED are not with its use of AI, but rather that the evidence presented
does not justify the claims about the score. We scientists need better
systems for organizing, evaluating, and consuming scientific literature.

I believe AI can be part of that solution. Many researchers, myself
included, have found LLMs useful for
indexing scientific papers and providing structured feedback. But assigning every paper a single number is a much stronger claim
than generating useful feedback. Compressing years of scientific work
into a single number inevitably discards too much information that
scientists care about. Such a score should not be treated as a measure
of scientific quality without transparent methodology and rigorous
independent validation.

As this critique explains, the QED score has not been rigorously
validated. The white paper’s three case studies do not demonstrate that
it is a more accurate or less biased measure of scientific quality, and
the released 1% rankings exhibit significant biases. Moreover, the
authors explicitly acknowledge that “precision of ranking within the top
1% has not been independently validated,” despite presenting the top 1%
as the central output of the system. By the standards the authors apply
to other scientific papers, the QED white paper apparently “sucks.”
Should scientists trust a score whose own validation does not meet the
standards it claims to enforce?