I too avoid smart pointers. Part of the reason is something Jonathan Blow said—one of very few things I agree with him about—namely that smart pointers changes the type of the object and that's not good. Illustration:
auto ptr = new Object{};
This is very easy to understand. `ptr` is simply the address of my object. The possible operations are also very limited (which is a good thing for stupid programmers like me). Passing it around is straight forward, and so is dereferencing it and performing arithmetic on it in those special cases. I feel like the object I'm handling is small and easy to understand.
auto ptr = std::unique_ptr<Object>(new Object{});
Now the type of the object has changed. It's no longer merely an address to my object, or even my object, but now I have an object of type "std::unique_ptr<..." which is something else than what I wanted. The new object I actually want sort of hidden somewhere inside it, both in terms of how unique_ptr works, and literally by how the declaration reads.
This increases the complexity of the program, because now I have to reason about how std::unique_ptr will act in addition to everything else. Mind you, this doesn't even absolve me from thinking about the lifetime of the object, it merely hides deallocation into the rules of scope instead of having me handle it explicitly. The reason why I use C++ is because I want to do memory management manually, if I wanted memory to be handled in automatic and hidden ways I would just a GC'd language instead.
I enjoy doing things explicitly, because that makes it easier for me to understand the meaning of my code. Someone may argue that smart pointers are actually super simple to understand, and if that's how you feel, then go ahead. However, if you are double freeing your pointers or forgetting to free them, or often trying to access freed memory, well I believe you have problems that smart pointers aren't going to solve. And like Jabberwocky said, if something like allocation fails then smart pointers aren't likely going to save you either.