colbert qps
0
10
20
30
40
throughput, qps
quantization config, colbert rescorer (hybrid prefetch, k=20, pf=100, concurrency=8)
37
no-quant
(f32)
18
no-quant
(f16)
28
no-quant
(turbo4)
40
scalar
int8
17
product
x16
28
binary
(f32)
17
binary
(f16)
34
turbo4
tq1