p50
p99
0
500
1000
1500
2000
latency, ms
quantization config, colbert rescorer (hybrid prefetch, k=20, pf=100)
136
274
no-quant
(f32)
138
288
no-quant
(f16)
141
269
no-quant
(turbo4)
134
329
scalar
int8
147
297
product
x16
147
1236
binary
(f32)
139
292
binary
(f16)
155
297
turbo4
tq1