I think SFML is ideal for you. It covers a huge amount of basic functionality, so that you can abstract from low-level mechanism you usually need in engines. For more elaborated tasks, there are also some extensions that can be found on the Wiki. Especially if you already have some experience with other frameworks, SFML shouldn't be a problem.
Even if it might be more difficult in the beginning, I recommend to start with SFML 2 instead of SFML 1.6. Many parts of the API have changed, and a lot of bugfixes and new features are available.