Note that std::shared_ptr which I use is only appropriate if you have true shared ownership, i.e. the manager/cache does not store the resources. Instead, the resources live (at least) as long as they are being referred to. Every user of a resource keeps a shared pointer to it.
For simpler cases, you might just have a centralized class that stores the resources and ensures their lifetime. You can then give away raw pointers to resources, since they are not owning.