From my point of view you only get "faster" once you've collected experience in the given area.
I can write you a basic SFML example with state machine in 30min - 1h, but I can only do that, because I've done it at least a dozen times by now.
As soon as I then try to do any movement code, I keep slowing, because I don't remember a good/clean way to do movement code, even though I've done a bunch of times as well, but apparently not enough often.
So it really seems like a mix of deeply understanding a given concept and having it done over and over again.
That you end up spending multiple days on something you thought was easily done in a couple of hours, is quite normal, if you've never done it before. Things seem easy on the outside, but there are usually a lot of detail that you need to understand and implement. That's why it's also hard to estimate how long it takes to implement something.