What is the testing setup?
The README.md says this about the benchmark
For speed, we compared different OCR methods under identical hardware and testing conditions (single replica, single concurrency), evaluating their performance in parsing and exporting Markdown files from both image and PDF inputs. Results show GLM-OCR achieves a throughput of 1.86 pages/second for PDF documents and 0.67 images/second for images, significantly outperforming comparable models.
How is this measured? The supplied Docker image for vLLM does not have the PP-DocLayout enabled, so how much do you batch? How many concurrent requests?
Could you please explain?
would love to know myself
@wamreyaz Hi, thanks for your interest in our work.
For speed benchmarking, we use the SDK to parse PDF files one at a time in a for loop, measuring the total time to process all PDFs.
Throughput is calculated as total pages divided by total time.
To fully leverage the SDK's asynchronous parsing performance, the batch size for PP-DocLayoutV3 is set to 1.
All tests are conducted on the same hardware platform.