Latency

Latency Per Batch

GPT2 Latency BS=1 GPT2 Latency BS=2 GPT2 Latency BS=4 GPT2 Latency BS=8 GPT2 Latency BS=16 GPT2 Latency BS=32 GPT2 Latency BS=64 GPT2 Latency BS=128

Throughput

Throughput Per Batch

GPT2 Throughput BS=1 GPT2 Throughput BS=2 GPT2 Throughput BS=4 GPT2 Throughput BS=8 GPT2 Throughput BS=16 GPT2 Throughput BS=32 GPT2 Throughput BS=64 GPT2 Throughput BS=128

Throughput Per Allocator

(Allocator = default)

GPT2 Throughput per BS

(Allocator = tcmalloc)

GPT2 Throughput per BS