No, the packet loss has nothing to do with forgetting to call receive, OS will buffer the packets and wait for you to handle them but the buffer that OS has is limited.
Yes, server -> client and client -> server can both lose packets. Also UDP knows nothing of 'server' and 'client' so neither sending direction is special in any way.
You should probably receive all you can from socket as soon as you can.
Also, packets can arrive out of order or duplicated too.
UDP isn't reliable but people get away with it with 'tricks' that make it kind of reliable (in the sense that all data gets transported to the other end, not in the sense that all packets get there - this is impossible).
You might want to read this too:
https://github.com/SFML/SFML/wiki/FAQ#should-i-use-tcp-or-udp-sockets