I'm using View.Center property instead of MoveTo and camera moving across the map just perfectly with no any issues. It works in windowed and fullscreen mode and even with switch between these modes.
It works even if you go beyond the border of the window? Care to share that piece of code? 
Yes, it works even if you move mouse outside window. I recalculate point on the map according to the mouse current position and in such case I get a point on the map which is actually outside of visible area. Then I move center of camera to the obtained point with damping filter. At render loop I just assign camera center to View.Center and thats it. So, I get smoothness and inertial camera moving.
Actually I'm also using camera zoom. I'm implemented it by assigning zoom factor to View.Zoom.
And also I assign RenderWindow.Size to View.Size to eliminate dependency of scale from window size.
here is how I implemented it:
public class Game
{
private RenderWindow _window;
void UpdateMouse()
{
var mousePos = (Vector2f)Mouse.GetPosition(_window);
var worldPos = _camera.ViewToWorld(mousePos, (Vector2f)_window.Size);
_camera.MoveTo(worldPos);
}
//...
public void Render()
{
_rendererWorld.Render(_window, _camera, ...);
}
}
public class Camera
{
public Vector2f Center { get; private set; }
public float Scale { get; private set; }
public Vector2f ViewToWorld(Vector2f point, Vector2f viewSize)
{
return Center + (point - viewSize / 2F) / Scale;
}
//...
}
public class RendererWorld
{
public void Render(
RenderTarget target,
Camera camera,
...)
{
_view.Size = (Vector2f)target.Size;
_view.Zoom(1F / camera.Scale);
_view.Center = camera.Center;
target.SetView(_view);
// ...
}
}
