Possible to build a "virtual" file system?

+1 vote
asked Nov 17, 2015 by Uwe Keim (860 points)

Currently evaluating the File Server component to provide my own SFTP server, I dug through the API documentation and feel a bit lost.

Would the following be possible with your component?:

Create a file system that is completely independent of the actual file system on disk

e.g. I'm thinking about having handlers (or interface methods) for all kind of events like:

  • Get directory list
  • Get (download) files
  • Change directory
  • Upload files
  • etc.

My scenario is that I have a "file system" stored inside database tables and want users to access it via your server component.

Is this possible in some way?

Applies to: File Server, Rebex SFTP

2 Answers

0 votes
answered Aug 11, 2017 by Lukas Pokorny (96,250 points)
 
Best answer

We added a powerful and simple-to-use custom file system API to Rebex File Server 2017 R4.

For additional information and sample code, check out the Virtual file systems page.

The File Server package also includes a sample app that shows how to implement custom file system providers.

commented Sep 28 by GrahameHorner (170 points)
it would IMHO be good to add & expose/support for async Virtual file systems methods, so none blocking operations many be used inside the implementation including
commented Oct 1 by Lukas Pokorny (96,250 points)
We agree, and if we have been designing a new API today, we would make it a Task-based asynchronous API. However, virtual file systems are a rather niche feature, which means this most likely won't materialize in the short-term.
commented Oct 1 by GrahameHorner (170 points)
what would your recommendation be for implementing a AzureStorage Blob virtual FileSystem as the performance with regards to sync calls that are blocking should be avoided; I'd be contribute to code base if the interfaces, abstractions where opened/made public
commented Oct 2 by renestein (2,310 points)
edited Oct 2 by renestein
Thanks for your interest, Grahame!

1) I agree with you that sync VFS API over async AzureStorage API is far from ideal. We are aware of the limitations (e. g. sync over async in general, blocked thread in particular), but many our customers implemented the VFS provider for Azure blob and despite the limitations mentioned above they are satisified with the results.

2) That being said, support for the async operarations would be nice addition. As you probably know, async/await might with only slight exaggeration be characterized as an infection, which spreads through the codebase. Deceptively simple changes in VFS API would have to be mirrored in the code of the file server and may cause breaking changes in the public behavior. We are committed to backwards compatibility and we try to maintain the components as majority our customers prefer. Features should be carefully designed and implemented and we regret that we don't have unlimited time/resources to implement all nice-to-have features. But we would love to hear from our customers what they prefer.

3) We really appreciate your interest, Grahame, but we are not convinced that there is a business model that would allow us to open our source code and still generate enough revenue to stay operational and improve our products.
commented Oct 2 by GrahameHorner (170 points)
In reply

1)  my investigations have indicated with a relative low level of concurrent users 10 using AzureStorage blob filesystems in a async manor gives a performance increase of between 14% - 18% over the sync model; this performance would increase as the number of blocking operations would be reduced freeing up resources such as disk/network.


2. backwards compatibility is always a consideration, and while this is very important it shouldn't prevent improvement and/or implementations that become available in modern development techniques; introducing/extending interfaces in preparation is a good way to allow developers to allow others to develop/extend your product and submit to for a review.

3) I think you misunderstand, at present a number of interfaces and abstract classes are internal and a developer needs to use these as a base class rather then use the interface; IMHO if you made these public the above could be achieved with minimal impact.

I/We do a lot of Microsoft Open Source development opening source doesn't lose you ownership or revenue if fact it has been proven to increase revenue.
commented Oct 3 by Lukas Pokorny (96,250 points)
1) Internally, Rebex File Server is only partially asynchronous, and if we just plugged an asynchronous VFS API into its SFTP subsystem, it would still be used synchronously, mostly eliminating the performance benefits. Updating the internals to fully asynchronous model is near the top of our list of future enhancements, but it won't be done in a month. A year looks like a more reasonable estimate, but don't take my word for it.

2) What you say here makes a lot of sense - but it only applies to an open-source development model, which is not where we are today.

3) Unfortunately, our experience has been somewhat different. Any API we make public essentially becomes "a part of the contract" - our customers start using it, we have to maintain and support it and we cannot easily get rid of it because lot of third-party code relies on it. We have learned the hard way that publishing rarely-used, internal or prototype APIs usually results in them becoming a burden that hampers further development.

I/ We believe that open-source makes a lot of sense for some projects, but it's not a magic bullet that increases revenue for every kind of project. For example, there is only one viable open-source alternative to Rebex SFTP/SSH (https://github.com/sshnet/SSH.NET/) and with just one active developer, the project doesn't seem to be thriving. Of course, we cannot rule out the possibility that they are just doing something wrong and that the first software vendor to open-source their established SSH component for .NET is going to become a huge success. But we doubt it, and we are certainly not ready to take that risk yet. Sorry!
+1 vote
answered Nov 17, 2015 by Lukas Pokorny (96,250 points)
edited Aug 11, 2017 by Lukas Pokorny

Update: Rebex File Server 2017 R4 adds virtual file systems support. See the latest answer for details.


Although virtual file systems are not supported yet, Rebex File Server internals were designed with this scenario in mind and we plan to add a simple-to-use high-level API for this soon.

However, if you can't wait, you can try the Rebex File Server Beta with proof-of-concept IFilesystem interface and the related objects:

interface IFilesystem
{
    IHandle OpenDirectory(string path, string mask);
    IHandle OpenFile(string path, FileOpenMode openMode, FileAccessMode accessMode);
    void Close(IHandle handle);
    ItemInfo ReadNextItem(IHandle handle);
    int Read(IHandle handle, byte[] buffer, int offset, int count);
    void Write(IHandle handle, byte[] buffer, int offset, int count);
    long Seek(IHandle handle, FileSeekOrigin origin, long offset);
    void CreateDirectory(string path);
    void DeleteDirectory(string path);
    void DeleteFile(string path);
    ItemInfo GetItemInfo(string path);
    void SetItemInfo(IHandle handle, ItemInfo info);
    void SetItemInfo(string path, ItemInfo info);
    void Rename(string sourcePath, string targetPath);
    FilesystemInfo GetFilesystemInfo();
    string Normalize(string absolutePath);
    string ResolvePath(string path, string relativePath);
}

This is a bit more low-level that we would like, but on the other hand it makes it possible to implement a virtual file system that is completely independent of the actual file system on disk. I will mail you a download link.

To get started, try the following code. It implements a FileServerEx class that inherits from FileServer, but wraps the original file system object in a custom class that can be extended to do something else. For example, the OpenFile method can be modified to return a custom IHandle instead of calling _inner.OpenFile, and other methods that accept IHandle can be enhanced to handle both custom and built-in IHandles. Depending on your scenario, you will probably want to ignore the built-in IFilesystem implementations and write a complete custom filesystem from scratch.

   public class FileServerExt : FileServer
   {
          private class FilesystemWrapper : IFilesystem
          {
                 private readonly IFilesystem _inner;
                 public FilesystemWrapper(IFilesystem inner)
                 {
                       _inner = inner;
                 }

                 void IFilesystem.Close(IHandle handle)
                 {
                       _inner.Close(handle);
                 }

                 void IFilesystem.CreateDirectory(string path)
                 {
                       _inner.CreateDirectory(path);
                 }

                 void IFilesystem.DeleteDirectory(string path)
                 {
                       _inner.DeleteDirectory(path);
                 }

                 void IFilesystem.DeleteFile(string path)
                 {
                       _inner.DeleteFile(path);
                 }

                 FilesystemInfo IFilesystem.GetFilesystemInfo()
                 {
                       return _inner.GetFilesystemInfo();
                 }

                 ItemInfo IFilesystem.GetItemInfo(string path)
                 {
                       return _inner.GetItemInfo(path);
                 }

                 string IFilesystem.Normalize(string absolutePath)
                 {
                       return _inner.Normalize(absolutePath);
                 }

                 IHandle IFilesystem.OpenDirectory(string path, string mask)
                 {
                       return _inner.OpenDirectory(path, mask);
                 }

                 IHandle IFilesystem.OpenFile(string path, FileOpenMode openMode, FileAccessMode accessMode)
                 {
                       return _inner.OpenFile(path, openMode, accessMode);
                 }

                 int IFilesystem.Read(IHandle handle, byte[] buffer, int offset, int count)
                 {
                       return _inner.Read(handle, buffer, offset, count);
                 }

                 ItemInfo IFilesystem.ReadNextItem(IHandle handle)
                 {
                       return _inner.ReadNextItem(handle);
                 }

                 void IFilesystem.Rename(string sourcePath, string targetPath)
                 {
                       _inner.Rename(sourcePath, targetPath);
                 }

                 string IFilesystem.ResolvePath(string path, string relativePath)
                 {
                       return _inner.ResolvePath(path, relativePath);
                 }

                 long IFilesystem.Seek(IHandle handle, FileSeekOrigin origin, long offset)
                 {
                       return _inner.Seek(handle, origin, offset);
                 }

                 void IFilesystem.SetItemInfo(string path, ItemInfo info)
                 {
                       _inner.SetItemInfo(path, info);
                 }

                 void IFilesystem.SetItemInfo(IHandle handle, ItemInfo info)
                 {
                       _inner.SetItemInfo(handle, info);
                 }

                 void IFilesystem.Write(IHandle handle, byte[] buffer, int offset, int count)
                 {
                       _inner.Write(handle, buffer, offset, count);
                 }
          }

          protected override IFilesystem GetFilesystem(ServerUser user)
          {
                 IFilesystem inner = base.GetFilesystem(user);
                 IFilesystem outer = new FilesystemWrapper(inner);
                 return outer;
          }

   }

Please note that when using the FileServerExt class intead of FileServer class, user’s virtualRootPath still applies, making it possible for different users to access different parts of the custom filesystem.

Other users that want to test beta version, write us to support@rebex.net.

commented Dec 11, 2015 by Uwe Keim (860 points)
Is it possible inside my IFilesystem-derived class to somehow access information about the current connection?

E.g. from which IP address the request comes from, etc. I.e. something like the "Request" object in an HTTP web application.
commented Dec 11, 2015 by Lukas Pokorny (96,250 points)
This is not possible at the moment, but it surely looks like a must-have feature. I think the `FileServer` object's `GetFilesystem` virtual method should get a "Context" object instead of `ServerUser` that would at least contain the user and request info. I will look into this next week.

Alternatively, it should be possible to create a custom ServerUser object that inherits from FileServerUser and provides additional information about the current connection. If you created insteances of these objects in FileServer's Authentication event (see http://www.rebex.net/file-server/features/authentication.aspx#custom-authentication for details), you could copy the information about the current connection from the event arguments.
commented Jun 15, 2016 by s4ber7 (140 points)
I don't see a FilesystemException in the current beta binaries. Is there a new way to respond with "Not Supported"?
commented Jun 15, 2016 by Tomas Knopp (58,890 points)
edited Aug 3, 2016 by Lukas Pokorny
Thanks for pointing it out s4ber7! We actually forgot to make the FilesystemException public in the old beta. I have prepared a new beta that makes the FilesystemException public so that you can handle it properly in your code.
commented Aug 11, 2017 by Lukas Pokorny (96,250 points)
Rebex File Server 2017 R4 adds virtual file systems support. See the latest answer for details: http://forum.rebex.net/5789/possible-to-build-a-virtual-file-system?show=7281#a7281
...