How can I stream an Azure cloud blob when downloading through SFTP server?

Question

Hello,

I need to deploy an SFTP server as a service backed by Azure Blob storage. For this purpose, using the Rebex FileServer component, I'm implementing a virtual file system which is bridging the blob storage (currently using development storage provided by Azure Storage Emulator).
The streaming works fine while uploading to the server, but I receive a "Seek fault" error when downloading; the BlobReadStream is beeing seeked beyond its length (ArgumentOutOfRangeException: The argument 'offset' is larger than maximum of '119166'). Both the stream's length and GetLength(NodeBase node) return the correct value. If I download the whole blob(e.g. in a MemoryStream), it works fine, but this approach is not feasible as we are dealing with large files, of several gigabytes.

Here are some lines of code:

protected override NodeContent GetContent(NodeBase node, NodeContentParameters contentParameters)
    {
        Stream stream;
        NodeContent nodeContent;

        switch (contentParameters.AccessType)
        {
            case NodeContentAccess.Read:
                stream = _azureFS.OpenFileRead(node.Path.StringPath);
                nodeContent = NodeContent.CreateReadOnlyContent(stream);
                break;
            default:
                //....
                break;
        }

        return nodeContent;
    }
public Stream OpenFileRead(string path)
    {
        CloudBlockBlob blob = GetCloudBlob(path);

        if (blob == null)
            return null;

        Stream stream = blob.OpenRead();
        return stream;

        //MemoryStream ms = new MemoryStream();
        //blob.DownloadToStream(ms);
        //ms.Seek(0, SeekOrigin.Begin);
        //return ms;
    }

And the call stack:

Connecting through either FileZilla or WinSCP produces the same outcome.

What am I missing?

Thanks a lot,
Paula

Answer 1

Hi Paula,
thanks for the logs.

As you already know, the Seek method of the BlobReadStream class throws ArgumentOutOfRangeException when a sftp client tries to seek beyond the end of the stream.

Behavior of the BlobReadStream is radically different from the behavior of other commonly used and well known streams.

See the doc for the method Stream.Seek.
https://msdn.microsoft.com/en-us/library/system.io.stream.seek%28v=vs.110%29.aspx?f=255&MSPPError=-2147217396

Excerpt relevant for our discussion.
"Seeking to any location beyond the length of the stream is supported."

I have made a fix for you. When you use the method,

return NodeContent.CreateReadOnlyContent(...)

then our inner stream does not allow seek beyond the length of the BlobReadStream (or any other stream).
If you can drop me a mail to support@rebex.net. I will be glad to prepare hotfix for you.

Answer 2

Hi Paula,
I apologize for the delayed answer.
We are investigating the issue.
In the meantime, could you please send us communication log related to the issue?
It can be created like this.

  server.LogWriter = new TeeLogWriter(
          new FileLogWriter("c:/data/sftp.debug.log.txt", LogLevel.Debug),
          new FileLogWriter("c:/data/sftp.verbose.log.txt", LogLevel.Verbose)
  );

Thank you!

Comments

Thanks for following-up.

Debug log: http://bit.ly/2z1OnSU
Verbose log: http://bit.ly/2z6Q6WK