Fangtooth Mellanox Connect-x3 speed issue

You have two IP addresses on the same network on the same server, 192.168.1.5 (enp2s0) and 192.168.1.35 (bond0). That means there are two routes to/from your server on the same network.

This setup is invalid and is liable to cause at least some of the discrepancy you are seeing where you were limited to around 111 MBytes/s in one direction.

The solution is to not have two IPs on the same subnet on the same device.