0 votes
by (150 points)

Hi Team,
I need to create a virtual folder/file hierarchy based on JSON data from the backend. I also need to implement custom authentication; I have implemented custom authentication logic in the below event.

          var server = new FileServer();
        server.LogWriter = new ConsoleLogWriter(LogLevel.Debug);
        server.Bind(22, FileServerProtocol.Sftp);
        server.Keys.Add(new SshPrivateKey(@"C:\SFTPRebex\sftpppk.ppk"));
        server.PathAccessAuthorization += Server_PathAccessAuthorization;
        server.Authentication += Server_Authentication;

After authentication, subscribe to file system notifier to represent virtual directory/file structure as follows..

              string virtualRoot = @"D:\SFTPRebex\Test"; // a dummy directory.
        var localFS = new LocalFileSystemProvider(@"D:\SFTPRebex\Test", FileSystemType.ReadWrite);

        FileSystemNotifier notifier = localFS.GetFileSystemNotifier();
        notifier.GetAttributesCompleted += Notifier_GetAttributesCompleted;
        notifier.GetChildrenPreview += Notifier_GetChildrenPreview;
        notifier.GetChildrenSurrogate += Notifier_GetChildrenSurrogate;

In GetChildrenSurrogate event, created virtual structure as follows.

         private static void Notifier_GetChildrenSurrogate(object sender, GetChildrenEventArgs e)
    {
        // throw new NotImplementedException();
        List<NodeBase> basenode = new List<NodeBase>();
        NodeTimeInfo nodeTimeInfo = new NodeTimeInfo(DateTime.Now, DateTime.Now, DateTime.Now);
        var fileAttribute = new System.IO.FileAttributes();
        NodeAttributes nodeAttributes = new NodeAttributes(fileAttribute);
        NodeBase nodeBase = new DirectoryNode("VirtualDirectory", (DirectoryNode)e.Node, nodeTimeInfo, new NodeAttributes(fileAttribute));
        FileNode fileNode = new FileNode("VirtualFile.txt", (DirectoryNode)e.Node, nodeTimeInfo, nodeAttributes);

        basenode.Add(nodeBase);
        basenode.Add(fileNode);
        e.ResultChildren = basenode;
    }

The above structure worked and I can see the folder (“VirtualDirectory”) & file (“VirtualFile.txt”) in the SFTP Client.
which event will be triggered at server side when the client tries to download the file? I need to send the file stream to the client.

Which event will be triggered at server side when the client clicks on the directory? I need to represent file & directories inside the folder.

Thanks,
Srikar.

Applies to: Rebex SFTP

4 Answers

0 votes
by (70.2k points)
selected by
 
Best answer

To implement your own Virtual File System, I suggest you to implement your own custom File System Provider. You can find inspiration at FileServerCustomFS sample, which contains two custom File System Providers.


If you still rather prefer event based API, you need to rewrite also:

  • GetNodeSurrogate - to resolve your custom nodes.
  • GetContentSurrogate - to provide your custom file content.
  • GetLengthSurrogate - to display correct content length.

Please note that items assigned in the GetChildrenSurrogate are not stored anywhere. If you want to keep them and use them later, you have to do it on our own.

The sample implementation using notifier can look like this:

private class MyDirectory : DirectoryNode
{
    public IEnumerable<NodeBase> Children { get; set; }

    public MyDirectory(string nodeName, DirectoryNode parent, NodeTimeInfo nodeTimeInfo = null, NodeAttributes attributes = null) : base(nodeName, parent, nodeTimeInfo, attributes)
    {
    }
}

private class MyFile : FileNode
{
    public MyFile(string nodeName, DirectoryNode parent, NodeTimeInfo nodeTimeInfo = null, NodeAttributes attributes = null) : base(nodeName, parent, nodeTimeInfo, attributes)
    {
    }

    public long GetLength()
    {
        return this.Path.StringPath.Length;
    }

    public NodeContent GetReadContent()
    {
        // generate file content on-the-fly or read it from DB
        var ms = new MemoryStream(Encoding.ASCII.GetBytes(this.Path.ToString()));
        return NodeContent.CreateReadOnlyContent(ms);

        // or open associated file on disk
        return NodeContent.CreateReadOnlyContent(File.OpenRead("..." + this.Path));
    }
}

private static IEnumerable<NodeBase> GenerateChildren(DirectoryNode parent, Dictionary<string, NodeBase> nodes)
{
    // generate some children for this node
    var d = new MyDirectory("level-" + parent.Path.PathPartsCount, parent);
    var f = new MyFile("level-" + parent.Path.PathPartsCount + ".txt", parent);

    // add them to global list
    nodes[d.Path.StringPath] = d;
    nodes[f.Path.StringPath] = f;

    // return as collection
    var children = new List<NodeBase>();
    children.Add(d);
    children.Add(f);
    return children;
}

...

var nodes = new Dictionary<string, NodeBase>(StringComparer.OrdinalIgnoreCase);

var vfs = new MountCapableFileSystemProvider();
var notifier = vfs.GetFileSystemNotifier();

notifier.GetChildrenSurrogate += (s, e) =>
{
    if (e.Node.IsRootDirectory)
    {
        if (e.Node.Context == null)
            e.Node.Context = GenerateChildren(e.Node as DirectoryNode, nodes);
        e.ResultChildren = e.Node.Context as IEnumerable<NodeBase>;
        return;
    }

    var mydir = e.Node as MyDirectory;
    if (mydir == null)
        return;

    if (mydir.Children == null)
        mydir.Children = GenerateChildren(mydir, nodes);

    e.ResultChildren = mydir.Children;
};

notifier.GetNodeSurrogate += (s, e) =>
{
    NodeBase node;
    if (nodes.TryGetValue(e.Path.StringPath, out node))
        e.ResultNode = node;
};

notifier.GetContentSurrogate += (s, e) =>
{
    var myfile = e.Node as MyFile;
    if (myfile == null)
        return;

    e.ResultContent = myfile.GetReadContent();
};

notifier.GetLengthSurrogate += (s, e) =>
{
    var myfile = e.Node as MyFile;
    if (myfile == null)
        return;

    e.ResultLength = myfile.GetLength();
};
by (150 points)
Hi Team,
Is it possible to read the stream data directly as soon as chunk of data arrives at the server side?
We don’t want to save the uploaded file on disk. As and when SFTP clients sends a file in chunks, at server side we would like to send the chunk to our backend system.
At present, we use “Server_FileUploaded” to identify the file is uploaded but it will be fired when the file is completely uploaded into server’s hard disk.
by (70.2k points)
Yes, it is possible.
Which type of Virtual system are you using?
Did you write your own File System Provider as I suggested, or did you continue to use event-based API using FileSystemNotifier?
I will send you sample code for your case.
by (150 points)
Hi,
As suggested above, we are planning to use "DriveFileSystemProvider".
Can you send a sample code?
by (130 points)
Hi - Thanks for this code, I managed to get things working. However, I'm trying to set up a situation whereby I can request a filename via SFTP and then it dynamically looks up the information from a database and sends it back as a JSON file.

From what I can see, you need to set up all of the Nodes via the GetChildrenSurrogate method before they will be accepted as a valid node when GetNodeSurrogate is called when the download is requested. Is there any way to allow a download even if the filename wasn't known beofre the download request was called?
by (70.2k points)
Please see my latest answer https://forum.rebex.net/19646/need-to-create-virtual-file-system-based-on-data-from-backend?show=22384#a22384

I wanted to use code formatting, which is not possible in this forum comments unfortunately.
0 votes
by (150 points)
Hi,
We are okay with either event-based or custom File System Provider. Our requirement is as follows..
1) Read file chunks from the socket.
2) We need metadata related to the chunk like (chunk offset, chunk request-id/file info, identify the user details).

Please let us know which approach will suite to the above requirement?
0 votes
by (70.2k points)
edited by

The suggested way is to write your custom provider - similar to mentioned "DriveFileSystemProvider".

The content requested by the SFTP client is handled using the GetContent() method.
To transfer data directly to your backed (without storing it at the server), you need to use the NodeContent.CreateImmediateWriteContent(yourStream) method.
Then Read() and Write() methods are callend on the given yourStream instance depending whether the SFTP client is performing download or upload.

I have prepared "academic" sample of such functionality. Please download the modified sample.

The TcpFileSystemProvider.GetContent() shows how uploaded/downloaded data can be provided on-the-fly from your backend system using standard NetworkStream.

In your case, replace NetworkStream with your own stream with desired functionality ('chunk offset').
The 'chunk request-id/file info' can be identified by node.Path - the parameter of the GetContent() method.
To identify 'user details', you can use ServerSession.Current.

The modified sample also contains the Backend project, which is very simple application, which provides data for the presented TcpFileSystemProvider.

The modified sample works like this:
- The server has access to "c:/data/PC1", which contains only empty files.
- The Backend app has access to "c:/data/PC2", which contains data of the files.
- When a SFTP client requests download/upload of a file, the server requests data from Backend app using very simple protocol over TCP/IP.

+1 vote
by (70.2k points)

It is not necessary to construct nodes before a download attempt. The presented code was just an example of GetFileSystemNotifier() usage.

If you add logging into each notifier event, you will have better idea, how it all works.

For example, see this server logging:

DEBUG FileServer(1)[5] Server: Session 1: Starting SftpModule(1) subsystem.
    GetNodeSurrogate: /
    GetLengthSurrogate: /
DEBUG SftpModule(1)[5] SFTP: Getting item info on '/': success.
    GetNodeSurrogate: /a/b/c.txt
    GetContentSurrogate: /a/b/c.txt
DEBUG SftpModule(1)[7] SFTP: Opening file '/a/b/c.txt' (Open, Read): success.
DEBUG SftpModule(1)[7] SFTP: Closing file '/a/b/c.txt': success.
    GetNodeSurrogate: /x/y/z
    GetChildrenSurrogate: /x/y/z
DEBUG SftpModule(1)[10] SFTP: Opening directory '/x/y/z': success.
    GetLengthSurrogate: /x/y/z/level-3
    GetLengthSurrogate: /x/y/z/level-3.txt
DEBUG SftpModule(1)[10] SFTP: Closing directory '/x/y/z': success (4 items enumerated).
INFO FileServer(1)[7] SSH: Session 1: Connection closed by the remote host.

Every operation starts with GetNodeSurrogate.
Then depending on the requested operation it is followed by:

  • GetContentSurrogate = download of a file
  • GetChildrenSurrogate = listing of a directory
  • GetLengthSurrogate = getting info about the item

If you want to enable download of any file on the fly at any time, you need to provide appropriate GetNodeSurrogate and GetContentSurrogate implementation.
In my presented code, just change the GetNodeSurrogate event handler like this:

notifier.GetNodeSurrogate += (s, e) =>
{
    e.ResultNode = ConstructNode(e.Path, vfs.Root);
};

...

private static NodeBase ConstructNode(NodePath path, DirectoryNode root)
{
    if (path.IsRootPath)
        return root;

    // first construct "parent directory structure"
    int i;
    DirectoryNode parent = root;
    for (i = 0; i < path.PathPartsCount - 1; i++)
    {
        parent = new MyDirectory(path[i], parent);
    }

    if (path.HasExtension)
    {
        // consider any path with extension as file
        return new MyFile(path[i], parent);
    }
    else
    {
        // consider any path without extension as directory
        return new MyDirectory(path[i], parent);
    }
}

Please note that this will always create the required "parent directory structure", wasting memory of the server and triggering .NET garbage collection more often.
It would be better to cache created MyDirectory and MyFile nodes similarly as shown in my previous answer, but it would bring more complexity to the code. This code is just to demonstrate the possibilities in the simplest possible way.

by (130 points)
edited by
Hi - thanks, this is great and works when I'm attempting to connect via FileZilla.

However for some reason when I try to download a file using Rebex's GetFileAsync method, it fails with 'Internal Server Error'.

Do you know why this might be? Is it because the file isn't passing all of the attributes, e.g. last save time, filesize etc, or another reason?

I'm simply using

long transferredFileSize = await sftp.GetFileAsync(remotePath, tempDownloadedFile);

Also, calls to sftpClient.FileExists(serverFilename) fail in the same way.

Thanks for your help.


Further note:
On the server, the only main difference between calls appears to be that for successful FileZilla calls, GetNodeSurrogate is called twice before GetContentSurrogate, however for unsuccessful calls from Rebex's SFTP client, it seems to go GetNodeSurrogate -> GetContentSurrogate -> GetNodeSurrogate.

I'm not sure if this is significant or not
by (144k points)
The  'Internal Server Error' means that an unexpected exception occurred at the server. In this case, error details are not shared with the client, but they are written to a server-side log (if enabled). Would it be possible to enable logging at the server at LogLevel.Debug level (as described at https://www.rebex.net/file-server/features/logging.aspx#logwriter) and see what's going on? Additionally, please create a corresponding client-side log as well. Either post the relevant part of the logs here or mail them to support@rebex.net
by (130 points)
Hey thank you, yes I identified it, you need to call GetTimeInfoSurrogate and GetAttributesSurrogate. If either aren't set, it will fail.
by (70.2k points)
Great that you were able to solve the issue.

I was trying to reproduce the reported 'Internal Server Error' with Rebex's GetFileAsync() method, but without a success.

Is it possible that you are using some older version of Rebex assemblies? Or is it possible that an error was raised somewhere in your code (it would lead to 'Internal Server Error' as well)?

If you want to further diagnose the issue, can you please send us a small project to support@rebex.net, which will reproduce the issue?
by (130 points)
Hi - thanks for your reply. I have subsequently found that I think it was because I was requesting that the Rebex SFTP client sets the date and time of the file to that of the original server file time, which requires implementation of the GetTimeInfoSurrogate etc.
by (70.2k points)
Thank you for the info. I tried this as well, but still I was not able to reproduce the error.

If GetTimeInfoSurrogate is not implemented, the current date-time should be reported.

As I wrote, If you want to further diagnose the issue, please send us a small project to support@rebex.net, which will reproduce the issue.
...