You generally don't include external libraries in your git repository. Your repo is reserved for your code, build scripts and maybe asset files only.
For external libraries you could create SDK packages for each platform. They will contain binaries and header files and if everything is setup nicely in your build scripts, you should be able to extract the SDK and immediately start building.
You can use a separate git repo for this, however I'd probably go more with a simple file download, since git isn't really designed for large file storage.
On platforms that have a central package manager (like any Linux distro), it's recommended to use the libraries provided through the package manager. That way you don't have to deal with building them. It might help to keep track of all the dependencies and maybe write a script that will automatically install the wanted libs.
For asset files, if you're writing a small game, it might be okay to keep them in the same repository. But once things grow larger, you will have to either create another git repo, or probably better use some sort of cloud storage, since you'll be store quite a few MB or even GB of data, plus your artist should be able to easily make use of it.
I don't really recommend using submodules, since library repos are pretty much never laid out to be used as submodule. If you want to spend a lot of time, you could run a sort of "distribution" repository where you provide the needed binaries, but I'd argue it's a lot easier to create SDKs at that point.
Hope that helped.
