Skip to content

Qwen3 VLM: handle scenario specifc latency setting and dataset sample count to submission checker#2613

Open
zihaok wants to merge 1 commit into
mlcommons:masterfrom
zihaok:fix-qwen3-vl-submission-checker
Open

Qwen3 VLM: handle scenario specifc latency setting and dataset sample count to submission checker#2613
zihaok wants to merge 1 commit into
mlcommons:masterfrom
zihaok:fix-qwen3-vl-submission-checker

Conversation

@zihaok

@zihaok zihaok commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Background:
Qwen3 Offline and Server scenario uses 48289 samples, but Interactive scenario use 8000 samples due to runtime cost.

Hence, the difference between Server and Interactive scenario are

  • Latency constraint: 12s vs 1.5s
  • dataset sample count: 48289 vs 8000
  • Accuracy threshold: 0.7903 * 0.99 vs 0.78777 * 0.99

This PR adds this logic in submission checker to address these two scenario differences in Qwen3.

@zihaok zihaok requested review from a team as code owners July 2, 2026 06:26
@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

MLCommons CLA bot:
Thank you very much for your submission; we really appreciate it. Before we can accept your contribution,
we ask that you sign the MLCommons CLA (Apache 2). Please submit your GitHub ID to our onboarding form to initiate
authorization. If you are from a MLCommons member organization, we will request that you be added to the CLA.
If you are not from a member organization, we will email you a CLA to sign. For any questions, please contact
support@mlcommons.org.
0 out of 1 committers have signed the MLCommons CLA.
@zihao Kong
Zihao Kong seems not to be a GitHub user. You need a GitHub account after you become MLCommons member. If you have already a GitHub account, please add the email address used for this commit to your account.
You can retrigger this bot by commenting recheck in this Pull Request

@zihaok zihaok force-pushed the fix-qwen3-vl-submission-checker branch from 9f5e1eb to d42745e Compare July 2, 2026 21:50
@zihaok

zihaok commented Jul 2, 2026

Copy link
Copy Markdown
Contributor Author

recheck

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant