logstash patterns for RebexSFTP

Question

We are likely migrating to RebexSFTP soon. I would like to use filebeat/logstash to parse the logfiles into elasticsearch, has anyone created grok patterns? Would be great to reuse them if possible.

Answer 1

You can use this pattern to parse server log (without stacktraces):

^(?<timestamp>[\d\-]+ [\d\:\.]+) (?<tz>[\+\-]?[\d\:]+) \[(?<level>\w+)\] (?<message>.+)

For access log:

^(?<timestamp>[\d_.]+),"(?<ip>[^"]+)",%{NUMBER:session_id},"(?<user>[^"]+)","(?<action>[^"]+)",(?<args>.*)

As an alternative, you can configure your own format, even connect directly to ElasticSearch, using custom logging configuration

Comments

Thank you, appreciate the detail!

Answer 2

Unfortunately, we do not have grok patters for Rebex logs. And we are not aware of anybody who created it for themself neither.

However, Rebex logs are very easy to parse. It has this structure:

Date Time LogLevel ObjectType(ObjectId)[ThreadId] Area: Message

Where:

• Date format is YYYY-MM-DD.
• Time format is hh:mm:ss.fff.
• LogLevel can be one of: VERBOSE, DEBUG, INFO, ERROR.

Example:

2023-02-24 15:01:24.083 INFO HttpRequest(1)[1] HTTP: Connecting to 'http://test.rebex.net'...

In case of ERROR the Message can span over multiple lines (in case of writing whole exception's stack trace).
In case of VERBOSE the Message can be followed by multiple lines of raw data in format:

 oooo |xx-xx-xx-xx-xx-xx-xx-xx xx-xx-xx-xx-xx-xx-xx-xx| aaaaaaaaaaaaaaaa

Where:

• oooo is 4-digit hexadecimal offset (within the packet).
• xx is 2-digit hexadecimal value of the byte.
• a is ASCII representation of the byte or dot (.) if the byte is non-printable ASCII character.

Example:

2023-02-24 15:01:24.271 VERBOSE HttpRequest(1)[1] HTTP: Received data:
 0000 |48-54-54-50-2F-31-2E-31 20-32-30-30-20-4F-4B-0D| HTTP/1.1 200 OK.
 0010 |0A-43-6F-6E-6E-65-63-74 69-6F-6E-3A-20-63-6C-6F| .Connection: clo
 0020 |73-65-0D-0A-0D-0A                              | se....

---

UPDATE:

You can also write your own ILogWriter implementation and format log messages as you want.

Comments

Thank you, appreciate the comprehensive answer. Upon further inspection, this is not the formatting I'm seeing in my RebexSFTP server logs?

I'm seeing this:
20230224_152000.451,"::1",1,"username","login",[]
20230224_152000.489,"::1",1,"username","access",["/mount", "Read"]
20230224_152000.551,"::1",1,"username","access",["/mount", "Read"]
20230224_152000.630,"::1",1,"username","access",["/mount", "List"]
20230224_152023.823,"::1",1,"username","access",["/mount\README.md", "Write, Create"]
20230224_152023.825,"::1",1,"username","access",["/mount\README.md", "Read"]
20230224_152024.868,"::1",1,"username","access",["/mount\README.md", "Write, Create"]
20230224_152024.892,"::1",1,"username","upload",["/mount\README.md", 985]
20230224_152024.893,"::1",1,"username","access",["/mount\README.md", "Write"]
20230224_152024.903,"::1",1,"username","access",["/mount", "List"]
20230224_152029.022,"::1",1,"username","access",["/mount\README.md", "Read"]
20230224_152029.024,"::1",1,"username","access",["/mount\README.md", "Read"]
20230224_152030.770,"::1",1,"username","download",["/mount\README.md", 985]
20230224_152037.013,"::1",1,"username","access",["/mount\README.md", "Delete"]
20230224_152037.021,"::1",1,"username","access",["/mount", "List"]

---

Oh, this kind of log is produced by the Rebex Buru SFTP server product (https://www.rebex.net/buru-sftp-server) - I was talking about format of Rebex SFTP Server component for .NET.
I will assign this question to Rebex Buru. They will answer ASAP.