Skip to content

Commit bbbbfc8

Browse files
committed
Improve b64url benchmarking
Use erlperf as recommended in https://www.erlang.org/doc/system/benchmarking.html - Generate data in a more determinstic way and outside the main benchmarking lop - Benchmark encoding and decoding separately - Benchmark a wide range of sizes Issue: #5801
1 parent baf4e0c commit bbbbfc8

3 files changed

Lines changed: 58 additions & 180 deletions

File tree

src/b64url/README.md

Lines changed: 36 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -14,24 +14,45 @@ decoding Base64 URL values:
1414

1515
## Performance
1616

17-
This implementation is significantly faster than the Erlang version it replaced
18-
in CouchDB. The `benchmark.escript` file contains the original implementation
19-
(using regular expressions to replace unsafe characters in the output of the
20-
`base64` module) and can be used to compare the two for strings of various
21-
lengths. For example:
17+
This implementation is faster than the Erlang version in OTP 26-28,
18+
especially for larger binaries (1000+ bytes). To benchmark clone
19+
erlperf repo and run `./benchmark.sh` script. In the future, it's
20+
plausible Erlang OTP's base64 module may become faster than the NIF,
21+
due to improvements in the JIT capabilities but it's not there yet.
2222

2323
```
24-
ERL_LIBS=_build/default/lib/b64url/ ./test/benchmark.escript 4 10 100 30
25-
erl : 75491270 bytes / 30 seconds = 2516375.67 bps
26-
nif : 672299342 bytes / 30 seconds = 22409978.07 bps
27-
```
24+
./benchmark.sh
25+
26+
[...]
27+
28+
--- bytes: 100 -----
29+
Code || QPS Time Rel
30+
encode_otp_100 1 1613 Ki 620 ns 100%
31+
encode_nif_100 1 1391 Ki 719 ns 86%
32+
Code || QPS Time Rel
33+
decode_nif_100 1 1453 Ki 688 ns 100%
34+
decode_otp_100 1 1395 Ki 716 ns 96%
35+
36+
[...]
2837
29-
This test invocation spawns four workers that generate random strings between 10
30-
and 100 bytes in length and then perform an encode/decode on them in a tight
31-
loop for 30 seconds, and then reports the aggregate encoded data volume. Note
32-
that the generator overhead (`crypto:strong_rand_bytes/1`) is included in these
33-
results, so the relative difference in encoder throughput is rather larger than
34-
what's reported here.
38+
--- bytes: 1000 -----
39+
Code || QPS Time Rel
40+
encode_nif_1000 1 369 Ki 2711 ns 100%
41+
encode_otp_1000 1 204 Ki 4904 ns 55%
42+
Code || QPS Time Rel
43+
decode_nif_1000 1 455 Ki 2196 ns 100%
44+
decode_otp_1000 1 178 Ki 5612 ns 39%
45+
46+
[...]
47+
48+
--- bytes: 10000000 -----
49+
Code || QPS Time Rel
50+
encode_nif_10000000 1 45 22388 us 100%
51+
encode_otp_10000000 1 19 51724 us 43%
52+
Code || QPS Time Rel
53+
decode_nif_10000000 1 55 18078 us 100%
54+
decode_otp_10000000 1 17 60020 us 30%
55+
```
3556

3657
## Timeslice Consumption
3758

src/b64url/benchmark.sh

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
#!/bin/bash
2+
3+
# Expects erlperf to be installed
4+
#
5+
# $ git clone https://github.com/max-au/erlperf.git
6+
# $ cd erlperf
7+
# $ rebar3 as prod escriptize
8+
# $ cd ..
9+
10+
for i in 50 100 150 200 500 1000 5000 10000 50000 1000000 10000000; do
11+
echo ""
12+
echo "--- bytes: ${i} -----"
13+
ERL_LIBS="." erlperf/erlperf -w 2 \
14+
'runner(Bin) -> b64url:encode(Bin).' --label "encode_nif_${i}" \
15+
'runner(Bin) -> base64:encode(Bin, #{mode => urlsafe, padding => false}).' --label "encode_otp_${i}" \
16+
--init_runner_all "rand:seed(default,{1,2,3}), rand:bytes(${i})."
17+
18+
ERL_LIBS="." erlperf/erlperf -w 2 \
19+
'runner(Enc) -> b64url:decode(Enc).' --label "decode_nif_${i}" \
20+
'runner(Enc) -> base64:decode(Enc, #{mode => urlsafe, padding => false}).' --label "decode_otp_${i}" \
21+
--init_runner_all "rand:seed(default,{1,2,3}), b64url:encode(rand:bytes(round(${i} * (3/4))))."
22+
done

src/b64url/test/benchmark.escript

Lines changed: 0 additions & 165 deletions
This file was deleted.

0 commit comments

Comments
 (0)