0 votes
by (120 points)

Hello

I have been downloading files using FileTransferClient.Download(FileSet, String, TransferMethod, ActionOnExistingFiles) for some time.

Recently, I added date filtering, so that if a file was created before a certain date, it would be ignored. As such, I wrote the following code:

For Each item As FtpItem In _client.GetItems(FileSet) …..
If (item.LastWriteTime > filterFromDate.Date) Then
_client.Download(item.Path, LocalPath, TraversalMode.MatchFilesShallow, TransMethod, ActionType)

This approach iterates through every file that is found for the FileSet and then calls FileTransferClient.Download(..) for each file.

Unfortunataly, as this process is run on a regular basis the additional overhead of calling Download(..) for every file is turning a once 12-15 second process into many minutes that is longer than the restart interval for the process.

What I ideally need is a way to send the FileSet definition to Download(..) but include my date filter on each FtpItem.

Is there a way to do this by overriding a method etc?

With thanks

Paul

Applies to: Rebex FTP/SSL

1 Answer

0 votes
by (70.2k points)

Unfortunately, this kind of filtering in single Download() call is not possible.

However, there are two ways, how to do this:

  1. The most effective solution is to call _client.GetItems(FileSet) and then call in loop _client.GetFile() for each file.
    Please note that calling _client.Download() for single file has additional overhead.

  2. Unfortunately, the FileSet class can filter files based on relative paths only (not dates). However, this logic can be achieved, if you execute one additional GetItems() to create "path-->item cache". Once the FileSet has associated "path-->item cache", it can filter by any criteria.

The file set for second solution can look like this:

public class MyFileSet : Rebex.IO.FileSet
{
    public Dictionary<string, FileSystemItem> ItemCache { get; private set; }
    public bool IsFiltering { get; set; }

    private bool _failOnNewlyDiscoveredFiles;

    public MyFileSet(string basePath) : base(basePath.TrimEnd('/'))
    {
        ItemCache = new Dictionary<string, FileSystemItem>();
    }

    public void Reset()
    {
        ItemCache.Clear();
        IsFiltering = false;
    }

    public override bool IsMatch(string relativePath, FileSetMatchMode mode)
    {
        if (IsFiltering && mode == FileSetMatchMode.MatchFile)
        {
            // prepare absolute path - used in cache
            string absolutePath = string.Format("{0}/{1}", BasePath, relativePath);

            FileSystemItem item;
            if (ItemCache.TryGetValue(absolutePath, out item))
            {
                return Filter(item);
            }

            // the file was not found in cache - this is probably newly added file
            if (_failOnNewlyDiscoveredFiles)
                throw new InvalidOperationException(string.Format("Newly discovered file encountered '{0}'.", absolutePath));
            else
                return false; // skip it here, it can be transferred in future
        }

        return base.IsMatch(relativePath, mode);
    }

    private bool Filter(FileSystemItem item)
    {
        // implement filter logic here
        return item.LastWriteTime >= new DateTime(2007, 03, 19, 20, 53, 00);
    }
}

You can just rewrite the Filter(FileSystemItem) method to include your filtering logic.

This file set can be used like this (working example, which connects to test.rebex.net site):

// connect to your server
var client = new FileTransferClient();
client.Connect("test.rebex.net", FileTransferMode.Sftp);
client.Login("demo", "password");

// prepare file set
var set = new MyFileSet("/pub/example"); // use absolute path here
set.Include("*.*", TraversalMode.MatchFilesDeep); // add some includes

// prepare item cache for later filtering
var items = client.GetItems(set);
foreach (var item in items.OrderBy(fsi => fsi.LastWriteTime))
{
    // display info of all discovered items
    Console.WriteLine("Item: {0}  {1}", item.Path, item.LastWriteTime);
    set.ItemCache.Add(item.Path, item);
}

// register event to display info of all transferred files
client.TransferProgressChanged += (s, e) =>
{
    if (e.TransferState == TransferProgressState.FileTransferred)
        Console.WriteLine("Trasnferred: {0}  {1}", e.SourceItem.Path, e.SourceItem.LastWriteTime);
};

// download items with advanced filtering
set.IsFiltering = true;
client.Download(set, "c:/data");

The only overhead compared to client.Download(set) call is the initial execution of client.GetItems(set).

by (120 points)
Hi Lukas

This is really helpful - thank you!

I am just about to implement your suggested approach with "MyFileSet" to perform a filter for FileTransferClient.Download(FileSet, String, TransferMethod, ActionOnExistingFiles)

In reviewing our existing code further, I have noticed we also use FileTransferClient.Download(remotePath As String, localDirectoryPath As String, TransferMethod, ActionOnExistingFiles) - this does NOT use FileSet.  Is there a similar approach available for filtering when taking this approach?

Sorry for widening the question.
by (70.2k points)
No, advanced filtering can be achieved only when using FileSet. However, you can transform (String + TraversalMode) -> FileSet.

It can be done like this:

    string remotePath = "/pub/example";
    string localDirectoryPath = "c:/data";
    var mode = TraversalMode.Recursive;
    var method = TransferMethod.Copy;
    var action = ActionOnExistingFiles.OverwriteAll;

    remotePath = remotePath.TrimEnd('/');
    var remoteName = Path.GetFileName(remotePath);
    var basePath = remotePath.Substring(0, remotePath.Length - remoteName.Length);
    if (basePath.Length == 0)
        basePath = ".";
    else if (basePath != "/")
        basePath = basePath.TrimEnd('/');

    var fs = new FileSet(basePath, remoteName, mode);

Then, instead of using:
  client.Download(remotePath, localDirectoryPath, mode, method, action);
Use this:
  client.Download(fs, localDirectoryPath, method, action);


Note:
There is no overload: FileTransferClient.Download(String, String, TransferMethod, ActionOnExistingFiles)
You probably mean: FileTransferClient.Download(String, String, TraversalMode, TransferMethod, ActionOnExistingFiles)
...