I got the same problem, it is surely the NAT: it blocks every connection from the outside world that seems unfamiliar. You're trying to send data to the client but the client's NAT sees it and says "I don't know you! Go away!". And your data just gets rejected. To avoid such rudeness, you should use the UDP/TCP hole punching methods, or forward ports of your server.
To make it simple, hole punching is sending something to the other contact, so that your NAT knows that you were trying to make a connection with this contact. Then, the other computer can send you data, and your NAT will say "Hey, that's the guy you were trying to connect to! You can pass." and the data will reach your computer. The problem with this method is that both computers have to know each other's address and port.
On the other hand, we have port forwarding: port forwarding is allowing connections on a specific port of your computer. I don't know how to do this though, but it should not be too complicated. Forwarding ports on the server's side is enough to establish a connection. The only requirement to this method is the user having the address of the server , and the port that is forwarded. Therefore, the client just has to try to connect to the server, the server's NAT will say "The port is forwarded, you're lucky." and will do nothing. Then, with the new connection formed between the client and the server, the server knows the address of the client, and can send data without trouble (because the client has already sent data to the server, so the client's NAT will be cool on the data the server is going to send).
Edit: oooops, I didn't see that you already knew port forwarding... My bad. Well, I'll keep this comment in case if someone didn't know that...
If you did not understand this book, feel free to insult me in a polite way.