+1 vote
by (1.2k points)

Currently evaluating the File Server component to provide my own SFTP server, I dug through the API documentation and feel a bit lost.

Would the following be possible with your component?:

Create a file system that is completely independent of the actual file system on disk

e.g. I'm thinking about having handlers (or interface methods) for all kind of events like:

  • Get directory list
  • Get (download) files
  • Change directory
  • Upload files
  • etc.

My scenario is that I have a "file system" stored inside database tables and want users to access it via your server component.

Is this possible in some way?

Applies to: File Server, Rebex SFTP

2 Answers

0 votes
by (148k points)
 
Best answer

We added a powerful and simple-to-use custom file system API to Rebex File Server 2017 R4.

For additional information and sample code, check out the Virtual file systems page.

The File Server package also includes a sample app that shows how to implement custom file system providers.

by (170 points)
it would IMHO be good to add & expose/support for async Virtual file systems methods, so none blocking operations many be used inside the implementation including
by (148k points)
We agree, and if we have been designing a new API today, we would make it a Task-based asynchronous API. However, virtual file systems are a rather niche feature, which means this most likely won't materialize in the short-term.
by (170 points)
what would your recommendation be for implementing a AzureStorage Blob virtual FileSystem as the performance with regards to sync calls that are blocking should be avoided; I'd be contribute to code base if the interfaces, abstractions where opened/made public
by (5.3k points)
edited by
Thanks for your interest, Grahame!

1) I agree with you that sync VFS API over async AzureStorage API is far from ideal. We are aware of the limitations (e. g. sync over async in general, blocked thread in particular), but many our customers implemented the VFS provider for Azure blob and despite the limitations mentioned above they are satisified with the results.

2) That being said, support for the async operarations would be nice addition. As you probably know, async/await might with only slight exaggeration be characterized as an infection, which spreads through the codebase. Deceptively simple changes in VFS API would have to be mirrored in the code of the file server and may cause breaking changes in the public behavior. We are committed to backwards compatibility and we try to maintain the components as majority our customers prefer. Features should be carefully designed and implemented and we regret that we don't have unlimited time/resources to implement all nice-to-have features. But we would love to hear from our customers what they prefer.

3) We really appreciate your interest, Grahame, but we are not convinced that there is a business model that would allow us to open our source code and still generate enough revenue to stay operational and improve our products.
by (170 points)
In reply

1)  my investigations have indicated with a relative low level of concurrent users 10 using AzureStorage blob filesystems in a async manor gives a performance increase of between 14% - 18% over the sync model; this performance would increase as the number of blocking operations would be reduced freeing up resources such as disk/network.


2. backwards compatibility is always a consideration, and while this is very important it shouldn't prevent improvement and/or implementations that become available in modern development techniques; introducing/extending interfaces in preparation is a good way to allow developers to allow others to develop/extend your product and submit to for a review.

3) I think you misunderstand, at present a number of interfaces and abstract classes are internal and a developer needs to use these as a base class rather then use the interface; IMHO if you made these public the above could be achieved with minimal impact.

I/We do a lot of Microsoft Open Source development opening source doesn't lose you ownership or revenue if fact it has been proven to increase revenue.
by (148k points)
1) Internally, Rebex File Server is only partially asynchronous, and if we just plugged an asynchronous VFS API into its SFTP subsystem, it would still be used synchronously, mostly eliminating the performance benefits. Updating the internals to fully asynchronous model is near the top of our list of future enhancements, but it won't be done in a month. A year looks like a more reasonable estimate, but don't take my word for it.

2) What you say here makes a lot of sense - but it only applies to an open-source development model, which is not where we are today.

3) Unfortunately, our experience has been somewhat different. Any API we make public essentially becomes "a part of the contract" - our customers start using it, we have to maintain and support it and we cannot easily get rid of it because lot of third-party code relies on it. We have learned the hard way that publishing rarely-used, internal or prototype APIs usually results in them becoming a burden that hampers further development.

I/ We believe that open-source makes a lot of sense for some projects, but it's not a magic bullet that increases revenue for every kind of project. For example, there is only one viable open-source alternative to Rebex SFTP/SSH (https://github.com/sshnet/SSH.NET/) and with just one active developer, the project doesn't seem to be thriving. Of course, we cannot rule out the possibility that they are just doing something wrong and that the first software vendor to open-source their established SSH component for .NET is going to become a huge success. But we doubt it, and we are certainly not ready to take that risk yet. Sorry!
+1 vote
by (148k points)
edited by

Update: Rebex File Server 2017 R4 adds virtual file systems support. See the latest answer for details.


Although virtual file systems are not supported yet, Rebex File Server internals were designed with this scenario in mind and we plan to add a simple-to-use high-level API for this soon.

However, if you can't wait, you can try the Rebex File Server Beta with proof-of-concept IFilesystem interface and the related objects:

interface IFilesystem
{
    IHandle OpenDirectory(string path, string mask);
    IHandle OpenFile(string path, FileOpenMode openMode, FileAccessMode accessMode);
    void Close(IHandle handle);
    ItemInfo ReadNextItem(IHandle handle);
    int Read(IHandle handle, byte[] buffer, int offset, int count);
    void Write(IHandle handle, byte[] buffer, int offset, int count);
    long Seek(IHandle handle, FileSeekOrigin origin, long offset);
    void CreateDirectory(string path);
    void DeleteDirectory(string path);
    void DeleteFile(string path);
    ItemInfo GetItemInfo(string path);
    void SetItemInfo(IHandle handle, ItemInfo info);
    void SetItemInfo(string path, ItemInfo info);
    void Rename(string sourcePath, string targetPath);
    FilesystemInfo GetFilesystemInfo();
    string Normalize(string absolutePath);
    string ResolvePath(string path, string relativePath);
}

This is a bit more low-level that we would like, but on the other hand it makes it possible to implement a virtual file system that is completely independent of the actual file system on disk. I will mail you a download link.

To get started, try the following code. It implements a FileServerEx class that inherits from FileServer, but wraps the original file system object in a custom class that can be extended to do something else. For example, the OpenFile method can be modified to return a custom IHandle instead of calling _inner.OpenFile, and other methods that accept IHandle can be enhanced to handle both custom and built-in IHandles. Depending on your scenario, you will probably want to ignore the built-in IFilesystem implementations and write a complete custom filesystem from scratch.

   public class FileServerExt : FileServer
   {
          private class FilesystemWrapper : IFilesystem
          {
                 private readonly IFilesystem _inner;
                 public FilesystemWrapper(IFilesystem inner)
                 {
                       _inner = inner;
                 }

                 void IFilesystem.Close(IHandle handle)
                 {
                       _inner.Close(handle);
                 }

                 void IFilesystem.CreateDirectory(string path)
                 {
                       _inner.CreateDirectory(path);
                 }

                 void IFilesystem.DeleteDirectory(string path)
                 {
                       _inner.DeleteDirectory(path);
                 }

                 void IFilesystem.DeleteFile(string path)
                 {
                       _inner.DeleteFile(path);
                 }

                 FilesystemInfo IFilesystem.GetFilesystemInfo()
                 {
                       return _inner.GetFilesystemInfo();
                 }

                 ItemInfo IFilesystem.GetItemInfo(string path)
                 {
                       return _inner.GetItemInfo(path);
                 }

                 string IFilesystem.Normalize(string absolutePath)
                 {
                       return _inner.Normalize(absolutePath);
                 }

                 IHandle IFilesystem.OpenDirectory(string path, string mask)
                 {
                       return _inner.OpenDirectory(path, mask);
                 }

                 IHandle IFilesystem.OpenFile(string path, FileOpenMode openMode, FileAccessMode accessMode)
                 {
                       return _inner.OpenFile(path, openMode, accessMode);
                 }

                 int IFilesystem.Read(IHandle handle, byte[] buffer, int offset, int count)
                 {
                       return _inner.Read(handle, buffer, offset, count);
                 }

                 ItemInfo IFilesystem.ReadNextItem(IHandle handle)
                 {
                       return _inner.ReadNextItem(handle);
                 }

                 void IFilesystem.Rename(string sourcePath, string targetPath)
                 {
                       _inner.Rename(sourcePath, targetPath);
                 }

                 string IFilesystem.ResolvePath(string path, string relativePath)
                 {
                       return _inner.ResolvePath(path, relativePath);
                 }

                 long IFilesystem.Seek(IHandle handle, FileSeekOrigin origin, long offset)
                 {
                       return _inner.Seek(handle, origin, offset);
                 }

                 void IFilesystem.SetItemInfo(string path, ItemInfo info)
                 {
                       _inner.SetItemInfo(path, info);
                 }

                 void IFilesystem.SetItemInfo(IHandle handle, ItemInfo info)
                 {
                       _inner.SetItemInfo(handle, info);
                 }

                 void IFilesystem.Write(IHandle handle, byte[] buffer, int offset, int count)
                 {
                       _inner.Write(handle, buffer, offset, count);
                 }
          }

          protected override IFilesystem GetFilesystem(ServerUser user)
          {
                 IFilesystem inner = base.GetFilesystem(user);
                 IFilesystem outer = new FilesystemWrapper(inner);
                 return outer;
          }

   }

Please note that when using the FileServerExt class intead of FileServer class, user’s virtualRootPath still applies, making it possible for different users to access different parts of the custom filesystem.

Other users that want to test beta version, write us to support@rebex.net.

by (1.2k points)
Incredibly awesome, Lukas! Thanks a ton! I'll just go on and buy File Server now. (I'm already a long-term customer of the (S)FTP(S) client library).
by (1.2k points)
Oops. Seems that my "Rebex File Transfer Pack" subscription already contains the File Server. Even more awesome, no purchases to do!
by (148k points)
I sent a link to a custom built of Rebex File Server to your e-mail address. Please give it a try and let us know if you run into any issues.
by (1.2k points)
edited by
Very first feedback: It seems that if I do not pass a physical folder path when adding a user, it fails to even instantiate my own IFileServer class. I'm using Path.GetTempPath() now to fulfil this requirement.
by (1.2k points)
Just a quick update: I'm currently working on implementing all the interface methods. I'm half way through — browsing directories and subdirectories and downloading files already works like a charm. Now on to the other functions…
by (1.2k points)
One question: Could you please shade some light onto the idea behind the SetItemInfo() method, especially together with the Length property of the ItemInfo class?

I do see in the ILSpy disassembly of your code that you change/cut the length of a file. I don't get this one: In which use case would one ever want to cut a file?
by (148k points)
Thanks for your feedback! The physical folder path check is annoying in this scenario - Path.GetTempPath() is a nice workaround, but we will try to come up with a more elegant solution. Thanks for bringing this to our attention.
by (148k points)
SetItemInfo represents SFTP protocol's SSH_FXP_SETSTAT and SSH_FXP_FSETSTAT requests that make it possible to change file attributes including file length (see https://tools.ietf.org/html/draft-ietf-secsh-filexfer-02#section-6.9 for details). Cutting a file is only one of the purposes this servers (and common Windows and Unix file systems support it) -  another, much more useful purpose is enlarging a file without having to write all the data - this makes it possible to easily create a long empty file.

In SFTP, in .NET, in Win32 API and in POSIX, the call to enlarge and cut a file is actually one and the same, and I can imagine scenarios where cutting a file might be useful - Rebex ZIP does this to shrink a ZIP file after items have been removed, for example, and it can be used over Rebex SFTP. An alternative would be to copy all data to a new file and replace the current file with that, but that would consume too much disk space for very large ZIP files.

However, if you are absolutely sure that none of your SFTP clients will ever need to cut files, feel free to refuse "cut file" requests by throwing a FilesystemException if your underlying storage doesn't support this feature.
by (1.2k points)
Thanks for your detailed and insightful answers, Lukas!
by (1.2k points)
Is it possible inside my IFilesystem-derived class to somehow access information about the current connection?

E.g. from which IP address the request comes from, etc. I.e. something like the "Request" object in an HTTP web application.
by (148k points)
This is not possible at the moment, but it surely looks like a must-have feature. I think the `FileServer` object's `GetFilesystem` virtual method should get a "Context" object instead of `ServerUser` that would at least contain the user and request info. I will look into this next week.

Alternatively, it should be possible to create a custom ServerUser object that inherits from FileServerUser and provides additional information about the current connection. If you created insteances of these objects in FileServer's Authentication event (see http://www.rebex.net/file-server/features/authentication.aspx#custom-authentication for details), you could copy the information about the current connection from the event arguments.
by (140 points)
I don't see a FilesystemException in the current beta binaries. Is there a new way to respond with "Not Supported"?
by (58.9k points)
edited by
Thanks for pointing it out s4ber7! We actually forgot to make the FilesystemException public in the old beta. I have prepared a new beta that makes the FilesystemException public so that you can handle it properly in your code.
by (148k points)
Rebex File Server 2017 R4 adds virtual file systems support. See the latest answer for details: http://forum.rebex.net/5789/possible-to-build-a-virtual-file-system?show=7281#a7281
...