p50
p99
0
500
1000
1500
2000
latency, ms
quantization config, colbert rescorer (hybrid prefetch, k=20, pf=100)
136
246
no-quant
(f32)
158
1770
no-quant
(f16)
165
1346
no-quant
(turbo4)
131
1409
scalar
int8
158
539
product
x16
151
351
binary
(f32)
161
974
binary
(f16)
257
399
turbo4
tq1