Collect VanDyke VShell SFTP logs

Supported in:

Google secops SIEM

This document explains how to ingest VanDyke VShell SFTP logs to Google Security Operations using Bindplane with Syslog or Amazon S3 with a log shipper. The parser transforms raw logs into a structured UDM format. It handles both JSON and SYSLOG formats, extracts relevant fields like IP addresses, ports, and event details, and enriches the data with contextual information like platform details and security severity.

Before you begin

Make sure you have the following prerequisites:

A Google SecOps instance
Privileged access to the VanDyke VShell server or management console
For Option 1: A Windows 2016 or later or Linux host with systemd for the Bindplane agent
For Option 2: AWS account with S3 access and NXLog/Fluent Bit on the VShell server

Option 1: Integration via Bindplane and Syslog

This option provides real-time log streaming with minimal latency and is recommended for most deployments.

Sign in to the Google SecOps console.
Go to SIEM Settings > Collection Agents.
Download the Ingestion Authentication File. Save the file securely on the system where Bindplane will be installed or where you'll configure the feed.

Get Google SecOps customer ID

Sign in to the Google SecOps console.
Go to SIEM Settings > Profile.
Copy and save the Customer IDfrom the Organization Detailssection.

Install the Bindplane agent

Install the Bindplane agent on your Windows or Linux operating system according to the following instructions.

Windows installation

Open the Command Promptor PowerShellas an administrator.

Run the following command:

  msiexec 
  
 / 
 i 
  
 "https://github.com/observIQ/bindplane-agent/releases/latest/download/observiq-otel-collector.msi" 
  
 / 
 quiet

Linux installation

Open a terminal with root or sudo privileges.

Run the following command:

 sudo  
sh  
-c  
 " 
 $( 
curl  
-fsSlL  
https://github.com/observiq/bindplane-agent/releases/latest/download/install_unix.sh ) 
 " 
  
install_unix.sh

Additional installation resources

For additional installation options, consult this installation guide .

Configure the Bindplane agent to ingest Syslog and send to Google SecOps

Access the configuration file:
1. Locate the config.yaml file. Typically, it's in the /etc/bindplane-agent/ directory on Linux or in the installation directory on Windows.
2. Open the file using a text editor (for example, nano , vi , or Notepad).

Edit the config.yaml file as follows:

  receivers 
 : 
  
 udplog 
 : 
  
 # Replace the port and IP address as required 
  
 listen_address 
 : 
  
 "0.0.0.0:514" 
 exporters 
 : 
  
 chronicle/chronicle_w_labels 
 : 
  
 compression 
 : 
  
 gzip 
  
 # Adjust the path to the credentials file you downloaded in Step 1 
  
 creds_file_path 
 : 
  
 '/path/to/ingestion-authentication-file.json' 
  
 # Replace with your actual customer ID from Step 2 
  
 customer_id 
 : 
  
 YOUR_CUSTOMER_ID 
  
 endpoint 
 : 
  
 malachiteingestion-pa.googleapis.com 
  
 # Custom log type - requires parser extension 
  
 log_type 
 : 
  
 'VANDYKE_SFTP' 
  
 raw_log_field 
 : 
  
 body 
  
 ingestion_labels 
 : 
 service 
 : 
  
 pipelines 
 : 
  
 logs/source0__chronicle_w_labels-0 
 : 
  
 receivers 
 : 
  
 - 
  
 udplog 
  
 exporters 
 : 
  
 - 
  
 chronicle/chronicle_w_labels

Replace the port and IP address as required in your infrastructure.
Replace <customer_id> with the actual customer ID.
Update /path/to/ingestion-authentication-file.json to the path where the authentication file was saved in the Get Google SecOps ingestion authentication file section.
For TCP instead of UDP, replace udplog with tcplog .

Restart the Bindplane agent to apply the changes

To restart the Bindplane agent in Linux, run the following command:
```
 sudo  
systemctl  
restart  
bindplane-agent 
```
To restart the Bindplane agent in Windows, you can either use the Servicesconsole or enter the following command:
```
 net stop BindPlaneAgent && net start BindPlaneAgent 
```

Configure Syslog forwarding on VanDyke VShell

For VShell on Windows

Sign in to the VShell Control Panel.
Go to Loggingsettings.
Configure remote syslog logging:
- Enable logging to remote syslog server.
- Server address: Enter the Bindplane agent IP address.
- Port: Enter 514 (or your configured port).
- Protocol: Select UDP(or TCP if configured).
- Message groups: Select Connection, Authentication, SFTP, FTPS, HTTPS, Errors, Warnings, and Informational.
Click Apply > OK.

For VShell on Linux/macOS

Edit the vshelld_configfile (typically /etc/vshell/vshelld_config ).

Configure the following parameters:

 SyslogFacility LOG_LOCAL3
LogLevel INFO

Configure your system's syslogto forward LOG_LOCAL3 to the Bindplane agent:
- Edit /etc/rsyslog.conf or /etc/syslog-ng/syslog-ng.conf .
- Add: local3.* @bindplane-agent-ip:514 (for UDP) or local3.* @@bindplane-agent-ip:514 (for TCP).

Restart the VShell service and syslog service:

 sudo  
systemctl  
restart  
vshelld
sudo  
systemctl  
restart  
rsyslog

Option 2: Integration via AWS S3

This option is useful for environments that require log archival or where direct syslog forwarding is not feasible. Note that this requires a log shipper on the VShell server since AWS Lambda cannot access on-premise files.

Configure AWS S3 bucket and IAM for Google SecOps

Create Amazon S3 bucketfollowing this user guide: Creating a bucket .
Save bucket Nameand Regionfor future reference (for example, vandyke-sftp-logs ).
Create a Userfollowing this user guide: Creating an IAM user .
Select the created User.
Select Security credentialstab.
Click Create Access Keyin section Access Keys.
Select Third-party serviceas Use case.
Click Next.
Optional: Add a description tag.
Click Create access key.
Click Download CSV fileto save the Access Keyand Secret Access Keyfor future reference.
Click Done.
Select Permissionstab.
Click Add permissionsin section Permissions policies.
Select Add permissions.
Select Attach policies directly.
Search for AmazonS3FullAccesspolicy.
Select the policy.
Click Next.
Click Add permissions.

Install and Configure Log Shipper on VShell Server

Choose one of the following options based on your operating system:

Option A: Using NXLog (Windows)

Download and install NXLog Community Editionfrom nxlog.co .

Edit C:\Program Files\nxlog\conf\nxlog.conf :

 <Extension  
json>  
Module  
xm_json
</Extension>

<Extension  
syslog>  
Module  
xm_syslog
</Extension>

<Input  
vshell_log>  
Module  
im_file  
File  
"C:\\Program  
Files\\VanDyke  
Software\\VShell\\Log\\VShell.log"  
SavePos  
TRUE  
<Exec>  
#  
Parse  
VShell  
log  
format  
if  
$raw_event  
=~  
/^(\d{4}-\d{2}-\d{2}  
\d{2}:\d{2}:\d{2})  
(\S+)  
(\S+)  
(\S+)  
(\S+)  
(\S+)  
(\S+)  
(\S+)  
(\S+)  
(\S+)  
(\S+)  
(\S+)  
(\S+)  
(\S+)  
(\S+)  
"(.*)"$/  
{  
$EventTime  
=  
$1;  
$Protocol  
=  
$2;  
$EventType  
=  
$3;  
$SessionID  
=  
$4;  
$ClientIP  
=  
$5;  
$ClientPort  
=  
$6;  
$Username  
=  
$7;  
$Filename  
=  
$9;  
$BytesDown  
=  
$10;  
$BytesUp  
=  
$11;  
$ServerIP  
=  
$14;  
$ServerPort  
=  
$15;  
$EventMessage  
=  
$16;  
#  
Convert  
to  
JSON  
to_json();  
}  
</Exec>
</Input>

<Output  
s3>  
Module  
om_exec  
Command  
C:\scripts\upload_to_s3.ps1  
Args  
%FILEPATH%
</Output>

<Route  
vshell_to_s3>  
Path  
vshell_log  
=>  
s3
</Route>

Create PowerShell script C:\scripts\upload_to_s3.ps1 :

  param 
 ( 
 [string] 
 $FilePath 
 ) 
 $bucket 
 = 
 "vandyke-sftp-logs" 
 $key 
 = 
 "vshell/ 
 $( 
 Get-Date 
 -Format 
 'yyyy/MM/dd/HH' 
 ) 
 / 
 $( 
 Get-Date 
 -Format 
 'yyyyMMddHHmmss' 
 ) 
 .json" 
 # Batch logs 
 $logs 
 = 
 Get-Content 
 $FilePath 
 | 
 ConvertFrom-Json 
 $jsonLines 
 = 
 $logs 
 | 
 ForEach 
 -Object 
 { 
 $_ 
 | 
 ConvertTo-Json 
 -Compress 
 } 
 $content 
 = 
 $jsonLines 
 -join 
 " 
 `n 
 " 
 # Upload to S3 
 Write-S3Object 
 -BucketName 
 $bucket 
 -Key 
 $key 
 -Content 
 $content 
 -ProfileName 
 default

Create AWS credentials profile:

  Set-AWSCredential 
 -AccessKey 
 YOUR_ACCESS_KEY 
 -SecretKey 
 YOUR_SECRET_KEY 
 -StoreAs 
 default

Schedule the PowerShell script to run every 5 minutes using Task Scheduler.

Option B: Using Fluent Bit (Linux)

Install Fluent Bit:

 curl  
https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh  
 | 
  
sh

Configure /etc/fluent-bit/fluent-bit.conf :

  [SERVICE] 
  
 Flush        5 
  
 Daemon       On 
  
 Log_Level    info 
 [INPUT] 
  
 Name              tail 
  
 Path              /var/log/vshell/vshell.log 
  
 Parser            vshell_parser 
  
 Tag               vshell.* 
  
 Refresh_Interval  5 
  
 Mem_Buf_Limit     10MB 
 [PARSER] 
  
 Name        vshell_parser 
  
 Format      regex 
  
 Regex       ^(?<timestamp>\d{4}-\d{2}-\d{2} \d{2} 
 : 
 \d{2}:\d{2}) (?<protocol>\S+) (?<event_type>\S+) (?<session_id>\S+) (?<client_ip>\S+) (?<client_port>\S+) (?<username>\S+) (?<dash>\S+) (?<filename>\S+) (?<bytes_down>\S+) (?<bytes_up>\S+) (?<field1>\S+) (?<field2>\S+) (?<server_ip>\S+) (?<server_port>\S+) "(?<event_message>[^"]*)" 
 [OUTPUT] 
  
 Name                  s3 
  
 Match                 vshell.* 
  
 bucket                vandyke-sftp-logs 
  
 region                us-east-1 
  
 use_put_object        On 
  
 total_file_size       5M 
  
 upload_timeout        10s 
  
 compression           gzip 
  
 s3_key_format         /vshell/%Y/%m/%d/%H/%{hostname}_%{uuid}.json.gz

Configure AWS credentials:

  export 
  
 AWS_ACCESS_KEY_ID 
 = 
YOUR_ACCESS_KEY export 
  
 AWS_SECRET_ACCESS_KEY 
 = 
YOUR_SECRET_KEY

Start Fluent Bit:

 sudo  
systemctl  
 enable 
  
fluent-bit
sudo  
systemctl  
start  
fluent-bit

Configure a feed in Google SecOps to ingest VanDyke VShell logs

Go to SIEM Settings > Feeds.
Click Add New Feed.
In the Feed namefield, enter a name for the feed (for example, VanDyke VShell SFTP logs ).
Select Amazon S3 V2as the Source type.
Select VanDyke SFTPas the Log type(custom).
Click Next.
Specify values for the following input parameters:
- S3 URI: s3://vandyke-sftp-logs/vshell/
- Source deletion options: Select Retain(recommended) or the deletion option according to your preference.
- Maximum File Age: Include files modified in the last number of days. Default is 180 days.
- Access Key ID: User access key with access to the S3 bucket.
- Secret Access Key: User secret key with access to the S3 bucket.
- Asset namespace: The asset namespace .
- Ingestion labels: The label applied to the events from this feed.
Click Next.
Review your new feed configuration in the Finalizescreen, and then click Submit.

UDM Mapping Table

Log field	UDM mapping	Logic
agent.id	read_only_udm.observer.asset_id	Concatenates "filebeat:" with the value of agent.id field
agent.type	read_only_udm.observer.application	Directly maps the value of agent.type field
agent.version	read_only_udm.observer.platform_version	Directly maps the value of agent.version field
ecs.version	read_only_udm.metadata.product_version	Directly maps the value of ecs.version field
host.architecture	read_only_udm.target.asset.hardware.cpu_platform	Directly maps the value of host.architecture field
host.hostname	read_only_udm.target.hostname	Directly maps the value of host.hostname field
host.id	read_only_udm.principal.asset_id	Concatenates "VANDYKE_SFTP:" with the value of host.id field
host.ip	read_only_udm.target.ip	Directly maps each IP address in the host.ip array to a separate read_only_udm.target.ip field
host.mac	read_only_udm.target.mac	Directly maps each MAC address in the host.mac array to a separate read_only_udm.target.mac field
host.os.kernel	read_only_udm.target.platform_patch_level	Directly maps the value of host.os.kernel field
host.os.platform	read_only_udm.target.platform	Maps the value of host.os.platform to WINDOWS, LINUX, MAC, or UNKNOWN_PLATFORM based on the value
host.os.version	read_only_udm.target.platform_version	Directly maps the value of host.os.version field
log.file.path	read_only_udm.principal.process.file.full_path	Directly maps the value of log.file.path field
logstash.collect.timestamp	read_only_udm.metadata.collected_timestamp	Parses the timestamp from logstash.collect.timestamp field and converts it to a timestamp object
logstash.irm_environment	read_only_udm.additional.fields.value.string_value	Directly maps the value of logstash.irm_environment field. The key is set to "irm_environment"
logstash.irm_region	read_only_udm.additional.fields.value.string_value	Directly maps the value of logstash.irm_region field. The key is set to "irm_region"
logstash.irm_site	read_only_udm.additional.fields.value.string_value	Directly maps the value of logstash.irm_site field. The key is set to "irm_site"
logstash.process.host	read_only_udm.observer.hostname	Directly maps the value of logstash.process.host field
message		Used for extracting various fields using grok patterns and regular expressions
	read_only_udm.metadata.event_type	Set to "NETWORK_FTP"
	read_only_udm.metadata.log_type	Set to "VANDYKE_SFTP"
	read_only_udm.metadata.product_event_type	Extracted from the message field using a grok pattern
	read_only_udm.metadata.product_name	Set to "VANDYKE_SFTP"
	read_only_udm.metadata.vendor_name	Set to "VANDYKE SOFTWARE"
	read_only_udm.network.application_protocol	Set to "SSH" if the description field contains "SSH2" or "SSH", otherwise set to "HTTP" if the method field matches HTTP methods
	read_only_udm.network.http.method	Extracted from the message field using a grok pattern, only if it matches common HTTP methods
	read_only_udm.network.http.referral_url	Extracted from the message field using a grok pattern
	read_only_udm.network.http.response_code	Extracted from the message field using a grok pattern and converted to an integer
	read_only_udm.network.ip_protocol	Set to "TCP" if the description field contains "TCP"
	read_only_udm.principal.ip	Extracted from the message field using a grok pattern
	read_only_udm.principal.port	Extracted from the message field using a grok pattern and converted to an integer
	read_only_udm.security_result.description	Extracted from the message field using a grok pattern
	read_only_udm.security_result.severity	Set to "HIGH" if syslog_severity is "error" or "warning", "MEDIUM" if it's "notice", and "LOW" if it's "information" or "info"
	read_only_udm.security_result.severity_details	Directly maps the value of syslog_severity field
	read_only_udm.target.ip	Extracted from the message field using a grok pattern
	read_only_udm.target.port	Extracted from the message field using a grok pattern and converted to an integer
	read_only_udm.target.process.pid	Extracted from the message field using a grok pattern
syslog_severity		Used for determining the severity of the security_result

Need more help? Get answers from Community members and Google SecOps professionals.