"Rebex.Net.SftpException: Permission denied" and "... Failure" errors: meaning

Question

Hi everyone, first time poster so please forgive me for naive questions.

We suddenly started receiving the following errors from our content delivery platform (Telestream Vantage 8.0) on SFTP transfers w/o any changes on our side:

Rebex.Net.SftpException: Permission denied; Permission denied.
Rebex.Net.SftpException: Failure; Failure.

This was a complete outage, with the total inability of the entire cluster to initiate SFTP transfers to any destination - local or remote. It lasted from 8:50am to 2:50pm PT or about 6 hours total, and went away without any action on our side.

The errors appeared to come from Rebex SFTP .Net library that the application is using. Telestream Support suggested the firewall or an issue with the endpoint however none of those were the culprits:

• manual SFTP transfers - e.g. via Filezilla - from the same server where we're seeing these errors - go through w/o any issues
• as mentioned, no changes on our side
• the errors are unusual - we've never seen them before despite a number of prior incidents with network connectivity and bad credentials.

In addition, when setting up or validating an existing SFTP configuration in that application (Telestream Vantage 8.0), we were getting this error:

"An error occurred while establishing a connection to the SFTP server [some.sftp.server.net]: The source was not found, but some or all event logs could not be searched. To create the source, you need permission to read all event logs to make sure that the new source name is unique. Inaccessible logs: Security: The source was not found, but some or all event logs could not be searched. To create the s..."

These errors certainly don't look like bad credentials - and we don't see them anywhere in the logs - only in the UI during validation. Anyone knows what these mean?

Some other unusual bits of info:

Errors such as this happen sporadically (about 10-20 a month across our cluster) since January 2019 - but do not appear to be fatal:

Error: Info: Rebex.Net.SftpException: No such file; No such file.
   at vaml.amsm(manq are, Type arf)
   at vaml.amrx(String api, Boolean apj, Boolean apk, Boolean apl, vamm apm, vamr& apn, valv apo)
   at Rebex.Net.Sftp.dvkw(String uf, Boolean ug, Boolean uh, Boolean ui, valv uj)
Info: Info: Assembly: Rebex.Sftp 2019 R3 for .NET 4.6-4.8
Info: Info: Platform: Windows 6.2.9200 64-bit; CLR: 4.0.30319.42000
Debug: Info: Culture: en; Windows-1252
Info: Info: Assembly: Rebex.Sftp 2019 R3 for .NET 4.6-4.8
Info: Info: Platform: Windows 6.2.9200 64-bit; CLR: 4.0.30319.42000
Debug: Info: Culture: en; Windows-1252

Informational messages such as this started to happen suddenly at a very high rate of 1-2 per second across our 16-strong cluster, since 11am on March 30, 2020 - while very rarely (once a day) occurring prior to that:

Info: Assembly: Rebex.Sftp 2019 R3 for .NET 4.6-4.8
Info: Platform: Windows 6.2.9200 32-bit; CLR: 4.0.30319.42000
Info: Culture: en; Windows-1252

(There's no known correlation of these to anything happening in our systems - not application, OS or any other upgrade or changes.)

P.S. It'd be understandable if you tell me to contact my application support - already have, with zero effect: "Check your firewall" What firewall if Filezilla is working from the same server? "Change the password", etc. This is why I am hoping to get some clarity here.

Thanks!

Answer 1

The errors you are getting actually come from the SFTP server and represent server-side SFTP error codes (SSH_FX_PERMISSION_DENIED and SSH_FX_FAILURE; see the SFTP specification for details) and both were reported by the server in response to client’s SFTP requests.

This actually indicates that Rebex SFTP client was able to connect to the server, negotiate an SSH session, authenticate, and successfully establish an SFTP session. So apparently, there is nothing wrong with your firewall or your password. The issue is at the SFTP protocol level for sure.

In order to be able to tell more about what's going on and which kinds of SFTP requests actually failed (and perhaps speculate on the reason or recommend a workaround), we would need a chunk of the log that shows these error responses in context of the whole communication. They come from the server's SSHFXPSTATUS responses, but based on the information above, we don't know which SFTP requests triggered them. In other words, we have no idea yet what Telestream's application was actually attempting when these errors occurred.

Perhaps Telestream's application can produce such logs? If not, I would suggest downloading and installing the trial version of Rebex SFTP that comes with a sample SFTP client (SftpWinFormClient.exe) where logging at 'Debug' level can be enabled. Then, use this sample app to achieve the same task (like with FileZilla). If it consistently works, this would indicate that Telestream's app is doing something wrong. If it fails as well, we would have a debug log to analyze.

Comments

Thank you so much for the response!

> Perhaps Telestream's application can produce such logs?

I wish. There are no UI controls or props files where we could enable this, regretfully. I supposed we could ask Telestream - but similar requests haven't gotten very far in the past.

> If not, I would suggest downloading and installing the trial version of Rebex SFTP that comes with a sample SFTP client (SftpWinFormClient.exe) where logging at 'Debug' level can be enabled.

Will do!