R-GroundBench
/

RESEARCH PROJECT PAGE

R-GroundBench: A Diagnostic Benchmark for R-Group Grounding in Markush Molecular Editing

Xin Wang*Zichuan Ying*Xinna LinJunqi ZhangHanyi XiongTianyu GaoHairong ZhangQixiang HuaBotian ShiZhenhailong WangKaicheng Yu

Westlake University · The University of Hong Kong · Shanghai Innovation Institute · Zhejiang University · Sichuan University · Shanghai AI Laboratory · HKUST (Guangzhou) · UIUC

Paper PDFGitHub ↗Hugging Face ↗Try the benchmark ↓
R-GroundBench overview: the R-group grounding challenge, benchmark tracks, and the recognition-to-generation gap

Figure 1. R-GroundBench measures the gap between candidate recognition and autonomous molecular instantiation.

Abstract

Markush structures encode molecular families through variable R-group placeholders such as R1, R2, and X. We introduce R-GroundBench, a diagnostic benchmark for multimodal R-group grounding in Markush molecular editing. Built from real patent and literature structures, it combines a controlled multiple-choice VQA track with an open-ended Generation track across four image/SMILES modalities. Models exceed 90% on Easy VQA, but fall to 56–66% on Hard VQA when same-scaffold distractors and descriptive instructions remove shortcut cues. Generation Exact Match remains below 20% for most models, and visual input can push it below 8%. The benchmark makes a specific capability gap measurable: recognizing a plausible candidate is easier than grounding and executing the edit.

01WHAT THE BENCHMARK MAKES VISIBLE

A clean-looking answer can still miss the atom.

R-GroundBench separates candidate recognition from molecular instantiation, then removes the shortcuts that make an Easy split feel deceptively solved.

/ 0146,727

Markush-containing records

Real patent and literature structures from MolParser-7M before filtering and instruction construction.

/ 024 ×

input-output modalities

Image→image, image→SMILES, E-SMILES→image, and E-SMILES→SMILES isolate alignment failures.

/ 032

complementary tracks

VQA asks “which candidate?” Generation asks the model to build the edited SMILES itself.

55,982 editing operations across 12,962 scaffolds and 56,500 instruction records. Counts refer to different construction stages.

Benchmark design

The template’s method block is used here to show what each split is testing.

R-GroundBench construction pipeline and task formulation
Easy

Cross-scaffold + direct name

Useful for checking whether a model can follow a simple edit when the backbone gives away the answer.

Medium

Same scaffold + direct name

Removes the cross-scaffold cue while keeping the substituent name explicit.

Hard

Same scaffold + description

Uses high-similarity distractors and descriptive instructions that require structure-level grounding.

NOTA

Reject invalid candidates

Tests whether a model can say that no candidate is chemically correct rather than forcing a choice.

02TRY THE TASK

Name the substituent. Find its position. Don’t trust the vibe.

A small authored slice of the benchmark logic. Candidates are real atom-and-bond graphs rendered from SMILES; this is still a teaching interaction, not a live RDKit validator.

INPUT → CANDIDATE

Symbolic input and symbolic candidates: the cleanest view of editing logic.

SMILES → MOLECULAR GRAPH

Turn a SMILES string into a molecular graph.

Ready to render a real structure.

This renders the actual atom-and-bond graph from the SMILES string. It is a visual aid, not an RDKit validity proof.

BASIC R-GROUP EDITINGE-SMILES → SMILES · 01 / 02

MARKUSH STRUCTURE + INSTRUCTION

03THE PERFORMANCE CLIFF

Easy accuracy is not the same thing as grounded editing.

Switch conditions. Keep the model in view. Notice where candidate selection stops translating into executable molecular edits.

FROM RECOGNITION TO GENERATION

When the candidates disappear, performance falls.

Open full-size figure ↗

Read each panel from left to right: Basic VQA → Advanced VQA → Generation. Selecting a plausible molecule does not guarantee that a model can construct the correct one.

Basic VQAAdvanced VQAGeneration · EMPAPER FIGURE · EARLIER SNAPSHOT
Twelve model panels show the decline from Basic and Advanced VQA to Generation Exact Match under the s2s setting.
Original s2s figure from the paper appendix. This figure reflects an earlier evaluation snapshot; some model names and scores differ from the current manuscript tables below.

PRIMARY READOUT

98.3%

Basic VQA · Easy

WHAT IT HIDES

66.1%

Basic VQA · Hard

THE GAP

32.2pp

Easy → Hard

14.7%

Average VLM Exact Match on Hard Generation (s2s)The generation track removes the candidate set. In image→SMILES it falls further to 5.2%.

THE COMPLETE RESULTS

Every model. Every modality. Every split.

Download CSV ↓

Full results transcribed from the current manuscript tables. Filter the rows or scroll horizontally to compare all metrics.

45 / 45 rows

VQA: full paper-reported results from the current manuscript
ModelModel groupModality Basic Advanced ΔEH (pp) Overall
Easy Med Hard Avg. Easy Med Hard Avg. Basic Advanced Avg.
Qwen3-VL-8BGeneral VLMsi2i 25.5 24.9 25.1 25.2 21.3 20.4 21.1 20.9 0.4 0.2 23.0
Qwen3-VL-8BGeneral VLMsi2s 93.6 74.8 55.7 74.7 55.2 54.7 55.7 55.2 37.9 -0.5 64.9
Qwen3-VL-8BGeneral VLMss2i 78.4 68.7 49.6 65.6 37.7 38.2 37.2 37.7 28.8 0.5 51.6
Qwen3-VL-8BGeneral VLMss2s 93.4 72.7 55.9 74.0 49.4 48.7 49.2 49.1 37.5 0.2 61.6
Qwen3-VL-32BGeneral VLMsi2i 98.1 90.2 55.5 81.3 44.6 44.2 47.0 45.3 42.6 -2.4 63.3
Qwen3-VL-32BGeneral VLMsi2s 97.7 91.3 60.9 83.3 60.4 60.3 61.0 60.6 36.8 -0.6 71.9
Qwen3-VL-32BGeneral VLMss2i 95.2 90.6 55.6 80.5 46.4 47.2 44.0 45.9 39.6 2.4 63.2
Qwen3-VL-32BGeneral VLMss2s 97.5 91.4 61.9 83.6 50.3 48.9 49.1 49.4 35.6 1.2 66.5
Claude Sonnet 4.6General VLMsi2i 98.5 94.6 61.6 84.9 48.4 46.4 51.6 48.8 36.9 -3.2 66.8
Claude Sonnet 4.6General VLMsi2s 98.3 94.8 63.8 85.6 59.2 52.5 54.3 55.3 34.5 4.9 70.5
Claude Sonnet 4.6General VLMss2i 97.9 86.0 59.7 81.2 53.3 50.8 52.4 52.2 38.2 0.9 66.7
Claude Sonnet 4.6General VLMss2s 98.9 89.0 61.5 83.1 57.3 52.5 54.9 54.9 37.4 2.4 69.0
GPT-4oGeneral VLMsi2i 97.5 83.8 58.2 79.8 43.9 43.9 42.9 43.6 39.3 1.0 61.7
GPT-4oGeneral VLMsi2s 96.8 90.6 60.6 82.7 57.3 58.2 52.1 55.9 36.2 5.2 69.3
GPT-4oGeneral VLMss2i 97.5 83.1 57.7 79.4 47.3 49.2 47.6 48.0 39.8 -0.3 63.7
GPT-4oGeneral VLMss2s 97.7 89.8 61.1 82.9 35.2 37.9 42.7 38.6 36.6 -7.5 60.7
GPT-4.1General VLMsi2i 97.8 87.3 60.4 81.8 53.5 50.8 50.5 51.6 37.4 3.0 66.7
GPT-4.1General VLMsi2s 97.5 90.4 62.8 83.6 59.0 60.7 54.1 57.9 34.7 4.9 70.8
GPT-4.1General VLMss2i 96.4 84.9 60.3 80.5 53.3 53.5 50.2 52.3 36.1 3.1 66.4
GPT-4.1General VLMss2s 97.3 90.5 61.9 83.2 39.1 41.6 42.1 40.9 35.4 -3.0 62.1
GPT-5.5General VLMsi2i 96.8 95.5 64.3 85.5 54.9 57.7 50.5 54.4 32.5 4.4 70.0
GPT-5.5General VLMsi2s 97.9 98.5 66.1 87.5 60.3 59.5 58.4 59.4 31.8 1.9 73.5
GPT-5.5General VLMss2i 95.3 94.0 64.4 84.6 54.9 55.4 55.8 55.4 30.9 -0.9 70.0
GPT-5.5General VLMss2s 98.3 98.5 66.1 87.6 58.8 48.1 53.8 53.6 32.2 5.0 70.6
Gemini 2.5 FlashGeneral VLMsi2i 95.3 93.8 62.6 83.9 59.8 58.2 62.2 60.1 32.7 -2.4 72.0
Gemini 2.5 FlashGeneral VLMsi2s 97.5 94.8 65.4 85.9 62.5 60.3 60.1 61.0 32.1 2.4 73.4
Gemini 2.5 FlashGeneral VLMss2i 96.8 93.4 62.0 84.1 61.4 58.7 61.4 60.5 34.8 0.0 72.3
Gemini 2.5 FlashGeneral VLMss2s 98.1 95.1 63.8 85.7 54.7 53.5 52.5 53.6 34.3 2.2 69.6
Llama 4 MaverickGeneral VLMsi2i 92.7 86.2 59.5 79.5 35.2 36.0 33.3 34.8 33.2 1.9 57.2
Llama 4 MaverickGeneral VLMsi2s 93.6 82.0 62.6 79.4 56.3 54.4 55.8 55.5 31.0 0.5 67.5
Llama 4 MaverickGeneral VLMss2i 89.7 81.2 59.9 76.9 36.0 38.2 37.5 37.2 29.8 -1.5 57.1
Llama 4 MaverickGeneral VLMss2s 90.6 82.3 63.7 78.9 33.6 36.3 37.7 35.9 26.9 -4.1 57.4
DeepSeek-V4-ProGeneral LLMss2s 97.3 98.1 68.7 88.0 52.5 55.1 54.6 54.1 28.6 13.4† 71.0
DeepSeek-R1General LLMss2s 94.0 95.0 62.6 83.9 47.6 54.7 42.0 48.1 31.4 5.6 66.0
LLaMA-3.3-70B-InstructGeneral LLMss2s 96.7 79.6 60.4 78.9 42.6 42.9 44.2 43.2 36.3 -1.6 61.1
o3General LLMss2s 97.7 97.7 69.3 88.2 44.5 43.1 52.5 46.7 28.4 -8.0 67.5
Qwen3-Max-2025-09-23General LLMss2s 98.6 95.5 64.2 86.1 47.9 49.2 50.9 49.3 34.4 -3.0 67.7
ChemVLM-8BChemical-domain Modelsi2i 34.2 36.0 28.9 33.0 23.7 21.8 22.9 22.8 11.3† 0.8 27.9
ChemVLM-8BChemical-domain Modelsi2s 96.8 85.9 56.7 79.8 63.0 63.4 63.8 63.4 33.0† -0.4† 71.6
ChemVLM-8BChemical-domain Modelss2i 62.2 53.9 40.2 52.1 35.8 38.5 37.4 37.2 24.8† -0.5† 44.7
ChemVLM-8BChemical-domain Modelss2s 96.3 84.2 58.8 79.7 61.2 60.4 61.5 61.0 34.8† 0.7† 70.4
MolVL-7BChemical-domain Modelsi2i 22.0 20.5 25.5 22.7 22.1 29.4 24.0 25.2 -2.0† -3.9† 23.9
MolVL-7BChemical-domain Modelsi2s 20.5 19.6 25.7 21.9 22.0 29.7 23.9 25.2 -3.4† -4.0† 23.6
MolVL-7BChemical-domain Modelss2i 21.7 19.5 25.2 22.1 21.9 30.0 23.8 25.2 -2.1† -3.9† 23.7
MolVL-7BChemical-domain Modelss2s 22.5 22.8 25.7 23.7 21.9 29.8 23.8 25.2 -1.3† -4.0† 24.4
Generation: full paper-reported results from the current manuscript
ModelModel groupModality Easy Hard
EM Scaff. Tani. EM Scaff. Tani.
Qwen3-VL-8BGeneral VLMss2s 4.2 49.2 44.0 3.0 52.2 45.6
Qwen3-VL-32BGeneral VLMss2s 12.8 38.0 57.7 10.0 41.4 52.9
Claude Sonnet 4.6General VLMss2s 24.0 31.2 88.6 16.2 23.0 87.5
GPT-4oGeneral VLMss2s 19.4 53.0 59.9 10.4 50.8 54.0
GPT-4.1General VLMss2s 15.4 48.2 57.3 9.8 51.0 53.1
GPT-5.5General VLMss2s 44.6 62.2 78.4 38.4 64.0 76.7
Gemini 2.5 FlashGeneral VLMss2s 20.4 55.8 62.5 20.8 56.4 62.6
Llama 4 MaverickGeneral VLMss2s 12.2 48.6 52.0 9.0 51.2 48.9
Qwen3-VL-8BGeneral VLMsi2s 1.2 25.0 22.3 0.1 25.6 23.3
Qwen3-VL-32BGeneral VLMsi2s 2.6 31.0 34.1 2.6 33.0 32.6
Claude Sonnet 4.6General VLMsi2s 10.2 23.8 66.5 7.8 17.4 72.9
GPT-4oGeneral VLMsi2s 4.4 31.4 36.5 1.8 23.2 34.4
GPT-4.1General VLMsi2s 4.2 31.4 33.8 1.8 27.6 35.4
GPT-5.5General VLMsi2s 16.2 51.8 59.8 17.0 54.0 61.3
Gemini 2.5 FlashGeneral VLMsi2s 8.2 42.2 51.1 5.6 40.4 48.2
Llama 4 MaverickGeneral VLMsi2s 7.0 45.0 43.1 5.2 44.4 40.8
DeepSeek-V4-ProGeneral LLMss2s 50.7 72.7 82.6 29.7 54.6 71.7
DeepSeek-R1General LLMss2s 24.5 45.5 73.1 20.3 45.9 64.0
LLaMA-3.3-70BGeneral LLMss2s 4.8 35.6 47.1 1.6 38.2 39.5
o3General LLMss2s 37.1 64.7 76.6 28.7 64.1 72.2
Qwen3-MaxGeneral LLMss2s 15.6 50.0 58.1 13.8 48.6 56.2
MarkushGrapher-2Chemical-domain Modelsi2s 5.4 14.8 68.6 0.0 0.0 0.0
ChemVLM-8BChemical-domain Modelsi2s 0.6 37.8 32.8 0.6 41.6 30.9
MolVL-7BChemical-domain Modelsi2s 0.0 28.4 18.0 0.0 24.6 19.1
ChemVLM-8BChemical-domain Modelss2s 2.8 34.0 41.4 1.8 36.2 36.9
MolVL-7BChemical-domain Modelss2s 0.0 18.2 19.5 0.0 15.2 16.8

Scores are percentages; ΔEH is Easy − Hard in percentage points. EM = Exact Match; Scaff. = Scaffold Match; Tani. = Tanimoto Similarity. Values and emphasis are reproduced as reported in the manuscript.

† A source-reported Δ differs from Easy − Hard for the displayed scores; the original value is retained. Ranking marks are also transcribed, not recomputed.

i = image; s = E-SMILES input / SMILES output. Bold and underlined values reproduce the paper’s best / second-best annotations within each model group and modality.

04WHEN THE SYSTEM BREAKS

Four ways a plausible molecule can still be wrong.

These are the failure patterns the benchmark is designed to expose.

F–01

Semantic disconnect

The model identifies a chlorophenyl family, then puts chlorine at the wrong position.

↗
F–02

Pseudo-reasoning

A fluent BRAF explanation still picks the weaker candidate (6.14) over the higher-activity option.

↗
F–03

Structural error

The generated SMILES is valid and moderately similar, but its local connectivity is wrong.

↗
F–04

Backbone alteration

Instead of replacing one R-group, the model changes the scaffold it was meant to preserve.

↗
Four representative R-GroundBench failure cases
Representative GPT-4.1 failures from the paper’s case study.
05READ THE LABEL

A diagnostic benchmark is a lens, not a chemistry lab.

R-GroundBench focuses on controlled R-group substitutions with candidate and generation tracks. It does not certify full-patent reasoning, open-ended drug discovery, or chemical safety.

01

Patent-level clauses can be longer, conditional, and more layout-dependent than these controlled instances.

02

Multiple choice can still leave limited structure-level shortcuts, even after Hard distractor control.

03

Generation tests guided substitution, not unconstrained molecular design.

REPRODUCE THE CLAIM

Use the real data and evaluator.

The public repository contains evaluation code. Benchmark data is released separately on Hugging Face.

↗GitHubevaluation + prompts↗Hugging Facedatasets + models↓Paper PDFfull methods + results↓Checklistreproducibility notes

BibTeX

@inproceedings{wang2027rgroundbench,
  title={R-GroundBench: A Diagnostic Benchmark for R-Group Grounding in Markush Molecular Editing},
  author={Wang, Xin and Ying, Zichuan and Lin, Xinna and others},
  year={2027}
}

This page uses the Academic Project Page Template, adapted from the Nerfies project page.