So I fixed the problem a while back and forgot to post back, but to fix the problem I stored a pointer to my b2world object. Which I believe was not working previously (maybe you can confirm my understanding) because I was passing by reference and then using the copy constructor to copy over my b2world to this other class, which was incorrect. For example:
LoadMap::LoadMap(b2world& box2dWorld) : m_world(box2dWorld){
}
So I switched to using a unique_ptr and will probably use pointers for such objects.