It's obvious you are not yet comfortable with C++.
Learn the language a bit more before you try to create a graphical game.
First and foremost, avoid using pointers and new. It's enough just having a vector like this:
std::vector<Object> objects;
Second you can remove them by providing an index like this:
objects.erase( objects.begin() + index );
This index can be seen as an ID for each object.
Also, you had objects inherit from ObjectHandler and ObjectHandler creates new objects? That's not where you want to use inheritance. Object should be it's own base-class.
Yeah I'm not definitely comfortable with C++ yet since I have been doing this for only for a month.
I'm aware of erase, but it doesn't free much memory at all, because my objects have pointers inside of them. Perhaps I can just remove them for now and just pass them as references. I'm just scared of memory leaks and stuff like that, so I thought I should improve my design. Perhaps I'm just trying to over-complicate it too much...
About inheritance, I thought it might be good idea to have inheritance because ObjectHandler holds a pointer to Game class, which has b2World and RenderWindow etc. and I obviously need them for drawing and syncing, however when I think about what you suggested, you're absolutely right. It might be much simpler to just have Object as a base class.