Sprite is a light object, so there won't be a performance impact.
But don't just do it out of laziness. Think about the design and consider in which situation it makes sense to store the sprite and in which it doesn't. Ideally you'll want to have some sort of entity class at one point anyways that you'll have to store somewhere, which then can derive from sf::Drawable and hold an sf::Sprite for example.
