save/load a huge tilemap

[Geheim]

Hey guys,
I want to save and load a pretty huge tilemap (5000 x 3000 tiles). I only have one idea and I don't know if it is a good one, so I ask here
Of course I can't save/load everything at once, because it is too much (15000000 integers).
So I thought I have to split the hole map into chunks und save/load them if I enter/leave them.
So that I have several files, one for each chunk. Is this a good solution or does there exist better ones?
If I would do so, I would load the chunk where I am at and lets say the 9 chunks surrounded by it at the beginning of the game.
In-game if I leave one chunk I save it and load the chunk I entered, in threads.
And if I generate the map for the first time I should save the hole map once, or?
Greetings, Geheim!

[Nexus]

How is your tilemap represented? Does one tile only store one integer (its type)?

If so, there are several ideas to reduce the amount of data:

• Use small data types. sf::Uint8 (or unsigned char) can store 256 different tile types.
  
• Instead of a plain linear format where every tile is stored at a certain position, you could try sparse representations, especially if a most tiles are empty. A map (x,y) -> type would take up 2+2+1 bytes (index is 16 bit, type 8 bit).
• Use compression. You can implement a simple one like run-length encoding yourself. For cases where you really have to reduce the data, you might consider using an existing format like ZIP or 7z.
• Instead of several files, you can have one file for the data, and only load a part of it. You could store the addresses of the different chunks at the beginning of a file, and the actual chunks following after. But different files are easier to handle.

Some of these thoughts can be applied to RAM as well. But your map size isn't that huge: 15M 8-bit integers take up only 15MB. This at least for the case where you don't store other information per tile.

[Geheim]

Thanks for your fast response!
Yes, every tile is represented by one integer.
The size of the file is not the problem, but the speed of saving/loading.
Right, I will use sf::Uint8 instead of int, thanks for that

Which possibilities are there to load/save faster?
All at once takes over a minute on my computer, that is way too much.
Is it better to store the address and have one file, or to split them into chunks, or something else?

[FRex]

It might make it slower if you're saving it as text and not binary, but with binary there come endianess issues..
If you're ok with shaders you could just save them in chunks 1024x1024 or 2048x2048 each in png and use my/dbug's shader tilemap but I don't know how much faster that might be. And it depends a lot about what you're doing with it(like you'd have to work more to get my shader tilemap to animate or something).

[Nexus]

Which possibilities are there to load/save faster?

Didn't I mention some...

I can't get more precise unless you explain how a typical map looks and what format you use to serialize it (i.e. in what layout the values are written to the file). Of course you should use binary and not text format, I thought that was clear. Endianess is rather a theoretical issue, most of today's systems use Little Endian, including ARM architectures.

[Geheim]

Which possibilities are there to load/save faster?

Didn't I mention some...

Sorry, I meant which are better

Of course I use binary
I just try to implement some ideas and see what happens.