FileServer / VirtualFileSystem / Additional methods

Question

Hi,

when implementing a virtual filesystem it would be really beneficial for performance and scalability reasons to have an alternative method to handling file content. At the moment it is always a stream which contains the full content.

It would help to have something similar to:

int Read(IHandle handle, byte[] buffer, int offset, int count);
void Write(IHandle handle, byte[] buffer, int offset, int count);
long Seek(IHandle handle, FileSeekOrigin origin, long offset);

As was described in https://forum.rebex.net/5789/possible-to-build-a-virtual-file-system?show=5792#a5792 .

I know that since then the virtual file system became available and I really like it alot... but it does not scale well in cases where the backend storage does not have a stream interface and the data is really big (> 1 GB).

Is there any option in the current version to help with this? Or perhaps some update in future?

Thanks in advance.

Best Regards
Yahia

Answer 1

HI Yahia2020,
thanks for the question and welcome to the Rebex forum.
I think that custom "partial download/upload stream" will satisfy your needs.
Please see my answer.

https://forum.rebex.net/12951/can-i-intercept-resumed-downloads-with-virtual-file-system?show=12958#a12958

Comments

Hi Rene,

Thank you - this helps... if I understand it correctly we need to implement a "Stream interface" for our backend to achieve that, right?

BUT there is another part to the question:
In the above scenario we have uploads too. I would like to keep the uploaded data somewhere (like a temporary file, not in memory since it is big) and only write it to the backend on "Close" event...

Is that somehow possible?

Thanks.

Best Regards
Yahia

---

Hi Yahia,
yes, you have to implement a simple stream-like adapter for your backend.

Uploads can be handled similarly.

For example:

In your FileSystem provider.

        protected override NodeContent GetContent(NodeBase node, NodeContentParameters contentParameters)
        {
           
            //Handle read-only streams if needed.
           

            //Create writeStream
            var tempFilePath = ... //create temp file path (used for "local" upload);
            var mySpecialStream = new MySpecialStream(tempFilePath); //Your special stream for partial download/upload
            NodeContent content = NodeContent.CreateDelayedWriteContent(mySpecialStream);
            return content;
        }
       
       
        protected override NodeBase SaveContent(NodeBase node, NodeContent content)
        {
            //Upload error occured
            if (content.WasStreamClosedForcefully)
            {
                //handle error;
            }

            var myStream = (MySpecialStream) content.GetStream();
            var tempFileStream = myStream.LocalStream;
           
            //Upload data from tempFileStream now or offload the upload of the file to another task, use an upload queue in another process...
           
        }

       
       
where MySpecialStream handles write operation.

public MySpecialStream : Stream
{
    Stream _localStream;
   public MySpecialStream(string tempUploadFilePath)
   {
        _localStream = File.Open(tempUploadFilePath, FileMode.CreateNew);
   }
   
   public override void Write (byte[] buffer, int offset, int count)
   {
        _myLocalStream.Write(buffer, offset, count);
   }
   
   protected virtual void Dispose (bool disposing)
   {
       
        _myLocalStream.Dispose();
   }
   
   public Stream LocalStream
   {
        get
        {
            return _myLocalStream;
        }
   }
   //Implement other Stream members.
}

It is possible to use NodeContent.CreateImmediateWriteContent (method SaveContent method is then not called) and initiate upload of the "temp file" to the "backend" in the MySpecialStream.Dispose(bool) method, but this solution has the following drawbacks.

1) You don't know if the upload finished successfully (Stream does not expose property WasStreamClosedForcefully).
2) Using the delayed upload masked by the Create*Immediate*WriteContent method leads to a dirty and opaque code.
3) Hijacking Dispose method (typically used for clean-up) for upload leads to an opaque code.

---

Hi Rene,  thank you very much.

This sounds good - I will give it a try :-)