Author Topic: SFSL - SFML File System Library (Read 15027 times)

Disch · « **on:** June 13, 2011, 05:07:49 am »

I figured I would mention this now that I'm more or less done with my other SFML related project.

I'm starting work on an add-on library to SFML... SFSL (SFML File System Library).

As the name implies, one of the primary goals is to provide cross-platform way to interact with the file system.

It'll have the expected goodies:
- File I/O
- File/Folder creation, renaming, deletion
- Directory scanning (iterating over files/folders)

Plus some perks. The biggest one being .zip file reading/writing in a sane manner (most libs I've seen/used expect you to bend over backwards to use zips)

Benefits over stdio/iostream for normal files
- 64-bit offsets
- Ability to truncate/shorten files without having to completely wipe them
- Unicode filenames (windows' standard libs choke on this hard)
- Utility functions for endian-safe read/write of integral types

Benefits over boost::filesystem
- Seamlessly integrates with SFML
- Abstracted interface. Not strictly a singleton.

Use it to...
- Manage user profiles
- Manipulate easily distributable "packages" to allow for additional/customized levels
- ...and more!

Imagine if i/o with files in .zips were as simple as i/o with normal files. There's no reason it can't be!

I did something similar to this with another lib I wrote a long while back. Some of the code should be adaptable, but most will need to be rewritten to make it thread safe and exception safe.

Nexus · « **Reply #1 on:** June 13, 2011, 11:13:40 am »

Interesting, especially the ability to write/read from zip archives! Are you going to provide a stream interface similar to the standard library? Maybe you could even use std::ostream and std::istream, so that existing << and >> operators also work with your library...

By the way, don't you want to reuse portable code like Boost.FileSystem internally? I don't think it makes much sense to reinvent the wheel for every platform... You can still provide an interface that abstracts from the details

Xander314 · « **Reply #2 on:** June 13, 2011, 01:02:17 pm »

Looks good - as Nexus said archive handling will be useful. Do you intend just to use zip, or might you use something like the LZMA SDK to support .7z files, etc as well?

Quote

Unicode filenames

Unicode friendliness! From you? Who'd have guessed

Seriously, though, it's a useful feature

I assume you won't be providing encrypted file handling, whether due to annoying laws on export or whatever...

OniLinkPlus · « **Reply #3 on:** June 13, 2011, 03:40:19 pm »

Happiness. This is making me feel that. I don't need this right now, but if I do, it's already done.

Disch · « **Reply #4 on:** June 13, 2011, 04:21:05 pm »

Quote

Are you going to provide a stream interface similar to the standard library? Maybe you could even use std::ostream and std::istream, so that existing << and >> operators also work with your library...

Deriving from ostream/istream is definitely on my todo list. The problem is it's ridiculously complicated to do. I don't know why they made it so hard. I mean you'd think all you'd have to do is provide seek/tell/read functions...

It won't have that support immediately, but it is planned. Hopefully I can make it work.

Quote

By the way, don't you want to reuse portable code like Boost.FileSystem internally? I don't think it makes much sense to reinvent the wheel for every platform... You can still provide an interface that abstracts from the details

For systems like *nix where additional dependencies are trivial, yes. You're pretty much expected to have boost installed if you're on Linux so it's not really an issue.

For Windows users though, the extra dependency will be a hurdle, so I would rather stick to WinAPI. I already have plenty of dependencies with SFML and zlib.

Quote

Do you intend just to use zip, or might you use something like the LZMA SDK to support .7z files, etc as well?

.zip is just the most common, so I was going to start with that. I see no reason why future archives couldn't be supported, though. In fact I'm designing the framework to be easily expandable to allow for new formats (or at least, making it as easy as I can).

Quote

I assume you won't be providing encrypted file handling, whether due to annoying laws on export or whatever

I hadn't really thought about it. Would there be any point?

Xander314 · « **Reply #5 on:** June 13, 2011, 04:36:55 pm »

Quote

I hadn't really thought about it. Would there be any point?

Well when I was planning an archive manager, I considered putting it in just for the sake of it - after having waded into the largely undocumented LZMA SDK to compression, it would seem to make sense to go the whole way. Then I discovered all the export laws :/

Perhaps these reasons:
1) preventing users from easily cheating by editing game files (I guess it would also discourage modding...)
2) something like Valve's preloading whereby users could download the software in readiness for it's release, but it couldn't be run until then (this one is probably a fairly contrived reason though).

Quote

.zip is just the most common

But not as good compression ratio IIRC. Glad to hear it'll be extensible then

ActionBoy · « **Reply #6 on:** June 13, 2011, 07:55:15 pm »

Sounds like a great project. Good luck.

Laurent · « **Reply #7 on:** June 13, 2011, 08:39:44 pm »

I guess it will not depend on SFML, so why do you call it "SFML file system library"?

Disch · « **Reply #8 on:** June 14, 2011, 01:10:25 am »

Quote

I guess it will not depend on SFML, so why do you call it "SFML file system library"?

I actually was planning on making it depend on SFML and act as sort of an add-on library.

But you're right, it doesn't have to be dependant.

As for the name, I'm open to better suggestions. I honestly didn't put a whole lot of thought into the name XD

EDIT:

Actually, I remember now...

I need SFML for threading stuff. Archives will need to be threadsafe and since this lib is designed to be used with SFML it makes more sense for me to use SFML threads.

So yeah, SFML is a dependency. The system package only though.

fekmon · « **Reply #9 on:** June 15, 2011, 10:39:52 am »

Did you consider PhysicsFS ? http://icculus.org/physfs/

Thanks a lot for your previous work on the io stuff.

Disch · « **Reply #10 on:** June 15, 2011, 04:15:45 pm »

I hadn't, but I will review it when I get some time. It does seem like it would help, but it also seems to be lacking some features I want.

But I didn't look in detail.

Laurent · « **Reply #11 on:** June 15, 2011, 04:40:24 pm »

A little bit off-topic, but... a PhysFSInputStream for SFML could be a cool (and useful!) example. It would be so straight-forward that I could even include it as an example in the doc of the InputStream class.

fekmon · « **Reply #12 on:** June 16, 2011, 01:10:38 pm »

You might be interested by this wrapper of std i/ostreams for PhysicsFS: https://svn6.assembla.com/svn/tbt/branches/test-l10n/src/physfs_stream.hh

Disch · « **Reply #13 on:** June 16, 2011, 11:04:11 pm »

Thanks, fekmon. That actually is very useful.

I'll definitely check this out in more detail when I get the time.

gsaurus · « **Reply #14 on:** June 21, 2011, 09:26:16 pm »

Don't know weather this may be useful or not, but somehow it's related, so I though on pointing it.
A few months ago I did a small framework for data codification using a pipeline architecture, called Codex. It's not a compression/cryptographic library, just a way to stream data through such kind of filters (but it comes with zlib and Rijndael as default compression and encryption filters).

http://data-codex.sourceforge.net/

I created my own thread classes, they are pretty much the same as SFML ones plus another synchronization mechanism.

I also though on deriving from ostream/istream but I hadn't the time to fully understand it so I moved on with my own classes. But one can still stream using the << and >> operators for the most common types (basic types, std::string, etc), and from instances of classes deriving from cx::Resource.

I didn't implement seek, it's mainly streaming really (what would seek do if you're streaming through the network instead of to/from a file? and there may be filters not supporting it), but using standard file io to seek resources in a file and then streaming them is still a reasonable alternative.

I don't know but maybe it can be of some use to you? I would combine Codex and your file system anyway if I need it someday.