What is slow? In a graphics framework, it is often the drawcalls which limit performance. There is usually no point in worrying about single divisions or object copies
And in ProcessVertex(), divisions are of the kind
x / 255.f with a constant operand. The compiler can even optimize it to a multiplication.
sf::Color would be user-unfriendly as a 32 bit number, because it would make the use of bitwise operators necessary for component-wise access. I guess SFML itself accesses the single components a lot. And even if there are four 1 byte POD members, the class as a whole is a 4 byte type and can be treated as such, when it comes to copies.
The new graphics API is discussed
here.