Takes about a second for me:
[jj@firebreather ~]$ time wget http://www.sfml-dev.org/download/sfml/2.1/SFML-2.1-linux-gcc-64bits.tar.bz2
--2014-06-19 22:17:24-- http://www.sfml-dev.org/download/sfml/2.1/SFML-2.1-linux-gcc-64bits.tar.bz2
Resolving www.sfml-dev.org... 213.186.33.2
Connecting to www.sfml-dev.org|213.186.33.2|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1414179 (1.3M) [application/x-bzip2]
Saving to: “SFML-2.1-linux-gcc-64bits.tar.bz2.1”
100%[=============================================================================================>] 1,414,179 1.38M/s in 1.0s
2014-06-19 22:17:25 (1.38 MB/s) - “SFML-2.1-linux-gcc-64bits.tar.bz2.1” saved [1414179/1414179]
real 0m1.105s
user 0m0.004s
sys 0m0.045s
[jj@firebreather ~]$
So yeah, looks like it's something on your end.
Maybe you'll have better luck just cloning the git repository and building that - will probably be slower, but who knows, maybe your connection to github is better..
[jj@firebreather ~]$ time git clone https://github.com/SFML/SFML.git
Cloning into 'SFML'...
remote: Reusing existing pack: 21647, done.
remote: Total 21647 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (21647/21647), 54.50 MiB | 1.71 MiB/s, done.
Resolving deltas: 100% (15007/15007), done.
Checking connectivity... done.
real 0m55.684s
user 0m5.798s
sys 0m1.816s
[jj@firebreather ~]$