The Sftp class offers GetFile method which can transfer the file from the SFTP server into a Stream. If you are able to process the video from the Stream then, you could use the GetFile(string, Stream) overload to download the file and process the Stream then (i.e. play the video/audio as it gets downloaded).
Please bear in mind that the SFTP protocol is request based which means it was not designed specifically for the task of video/audio streaming, but still the task can be acomplished as suggested above.