Comparing and Benchmarking Go, Rust, C++ Java, Python, PHP, Javascript Frameworks on Latency and…

Comparing and Benchmarking Go, Rust, C++ Java, Python, PHP, Javascript Frameworks on Latency and…

Play this article

So on one Bright Sunny Beautiful day, I was thinking what Language and Framework should I chose to build a Product that I was planning to build.

So first of all deciding on a programming language and framework for a project, there are several factors to consider :

  1. Performance: execution speed, memory usage, and concurrency support.
  2. Scalability
  3. Ecosystem and libraries
  4. Community and support
  5. Developer expertise

But but but wait, the Metrics which can be computed irrespective of the project setup are metrics like :

P99.999 , P99, P95, Average Latency, Request Per Second

Let’s Start

First things to decide( In Order ) :

  • What Programming Languages to compare and for each language what Frameworks to Benchmark?
    - Go
    : Fiber, Fasthttp, Gin, Beego, Echo, Httprouter, Gorilla
    - Rust : Rocket, Iron, Warp, Actix, Nickel
    - Java : Spring, ActiveJ, Quarkus, Vert.x
    - C++ : Evhtp, Drogon
    - Python : Pyramid, FastAPI, CherryPy, Django, Flask
    - Javascript : Sifrr, Fastify, Koa, Express.js
    - PHP : Laravel, CodeIgniter, Symfony, Hyperf
  • What Metrics to compute for Benchmarking?
    -
    Number of Requests/Second**- Average Latency
    - P99 Latency
    - P90 Latency
  • Concurrency Levels
    *
    65,256,512*
  • What Hardware to be used for Benchmarking?
    -
    CPU: 8 Cores (AMD FX-8320E Eight-Core Processor)
    - RAM: 16 GB
    - OS: Linux

How did I Benchmark?

Minions on work for Benchmarking

  • wrk, results are collected using wrk : wrk is a HTTP benchmarking tool capable of generating significant load when run on a single multi-core CPU. It combines a multithreaded design with scalable event notification systems such as epoll and kqueue.

wrk -t12 -c400 -d30s 127.0.0.1:8080/index.html

// This runs a benchmark for 30 seconds, using 12 threads, and keeping 400 HTTP connections open.

Output:

Running 30s test @ 127.0.0.1:8080/index.html
12 threads and 400 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 635.91us 0.89ms 12.92ms 93.69%
Req/Sec 56.20k 8.07k 62.00k 86.54%
22464657 requests in 30.00s, 17.76GB read
Requests/sec: 748868.53
Transfer/sec: 606.33MB

  • postgresql, results are stored in postgresql
  • docker, each implementation is implemented in an isolated container
  • jq, processing docker metadata

Results :

Requests per Second

Requests per Second ( 64 Concurrency , 512 Concurrency)

  • ActiveJ framework of Java wins the War, with 183K RPS at 512 Concurrency
  • Family of C++ & Rust Frameworks seems to be the Consistent performer for this metric
  • It seems that Go-based frameworks like Fiber, Fasthttp, and Sifrr perform well, along with some C++ frameworks like Drogon and Evhtp. Among the Java-based frameworks, ActiveJ and Vertx perform well.
  • PHP frameworks like Symfony, Codeigniter4, and Laravel seem to be the weakest performers in terms of requests per second.

Average Latency (ms)

Average Latency (ms) ( 64 Concurrency, 512 Concurrency )

  • Nickel and Iron framework of Rust wins this one with leaps and bounds, 0.14 ms and 0.67 ms.
  • Rust-based frameworks show outstanding performance: Rust-based frameworks (Actix, Rocket, Iron, Warp, and Nickel) consistently demonstrate low latency across all concurrency levels. This can be attributed to Rust’s focus on performance and safety, making it an excellent choice for high-performance web applications.
  • Lightweight Java frameworks are competitive: While Java is often perceived as heavy and slow, some lightweight frameworks like ActiveJ, Quarkus, and Vert.x demonstrate competitive performance, particularly at higher concurrency levels.
  • Scalability concerns: Some frameworks (e.g., Express, Laravel, Django) show significantly higher latency as the concurrency level increases, indicating potential scalability issues.

P90 Latency (ms)

P90 Latency (ms) ( 64 Concurrency, 512 Concurrency )

  • C++ with evhtp, Java with ActiveJ and Vert.x, and Rust with Nickel show the lowest latencies across all concurrency levels.
  • PHP with CodeIgniter4 and Laravel, and Python with Django and Flask have higher latencies, especially when concurrency increases.

P99 Latency (ms)

P99 Latency (ms) ( 64 Concurrency, 512 Concurrency )

  • C++ with evhtp, Java with ActiveJ and Vert.x, and Rust with Nickel continue to demonstrate low latencies.
  • PHP with CodeIgniter4 and Laravel, and Python with Django and Flask have even higher latencies at P99 than at P90, which could indicate poor performance under heavy load.

Concluding :

Rust wins the War

  • Rust frameworks (Actix, Rocket, Iron, Warp, and Nickel) consistently perform well across different concurrency levels, especially Nickel.
  • Java frameworks (ActiveJ, Spring, Quarkus, and Vert.x) also show relatively good performance, with ActiveJ and Vert.x having the lowest latencies.
  • PHP and Python frameworks generally exhibit higher latencies, with PHP’s CodeIgniter4 and Laravel, and Python’s Django and Flask having the highest latencies.
  • Go frameworks (Gin, Gorilla-Mux, Fiber, Beego, Fasthttp, Httprouter, and Echo) have mid-range latencies compared to other languages.