Im having kinda same problem. Client(s) sends packets (every 100ms), server receives them normally and sends back to the client(s) normally but client(s) receives them with delay (there are no timers on server send / client receive).
Packets are not being missed, they arrive but with a great delay.. Even if some packets were lost, it wouldnt be a big deal. Main goal is to receive packets from the server as fast as possible.
In that pic he posted u can see some clients and a server exchanging packets. Problem is the one I described previously - server is already receiving/sending ~134th packet to clients but the last packet clients received is ~80-90th.
speedtest.net results are ping 2ms, dl 65.51Mb, ul 75.86Mb :