p50
p99
0
1000
2000
3000
4000
latency, ms
quantization config, colbert rescorer (hybrid prefetch, k=20, pf=100)
161
1331
no-quant
(f32)
141
1639
no-quant
(f16)
148
3232
no-quant
(turbo4)
146
2774
scalar
int8
156
2021
product
x16
137
332
binary
(f32)
142
1734
binary
(f16)
173
3480
turbo4
tq1