You misunderstood. POD has nothing to do with struct or class. These two keywords are absolutely equivalent, except for the default visibility (as stated in my last post). Please lookup the definition of Plain Old Data for more information.
Inheritance with structs works perfectly fine, but of course you have to downcast a base class pointer if you want to access members of the derived class. Note that dynamic_cast is mostly a design error, as it violates the principle of dynamic polymorphism. You should try to provide an interface using virtual functions. If you know the dynamic type for sure, use static_cast to downcast.
You should replace std::shared_ptr with std::unique_ptr, because you don't use shared ownership. Like this you can avoid the large overhead of shared pointers.