You have to manage the texture lifetime yourself. The sprites has a non-owner pointer to the texture. So whenever the texture is deleted, those sprites will have a dangling pointer to the texture.
In your situation, you have to make sure your caching system remain alive as long as some sprite used the textures stored within. The lifetime of your sprites won't affect the lifetime of your textures.