Send Local Logs to S3

If you are interested in streaming logs to an S3 bucket, the Log Stream functionality is a more streamlined solution for that need.

Scenario: You want to save gateway/relay logs to an Amazon S3 bucket. This guide presents a simple method to send all gateway/relay logs to S3.

As with all gateway/relay logs, the logs stored on the gateway/relay do not include Admin UI activities, which can be accessed via the sdm audit activities command. The following script includes an additional step to run that command and export those logs concurrently.

Set up the Export

1. Enable relay logging in the Admin UI under Settings > Security within the Log Encryption & Storage tab. Ensure logging is set to STDOUT.

2. Create an admin token with only the Audit > Activities permission. Save this token to add to the script in step 5.

3. From the AWS IAM Dashboard, go to Access management > Users. On the Security credentials tab for the user, generate and AWS access key and AWS secret access key.

4. Ensure the gateway or relay has the aws-cli tools installed.

5. Save the following script as s3export.sh. This script exports in 15-minute intervals; if you prefer to do it more or less frequently change the FROMTIME and TOTIME variables.

   #!/bin/bash
   
   # day, hour, minute timestamp
   TIMESTAMP=`date +'%Y%m%d%H%M'`
   # to prevent overlapping records, do 16 min ago to 1 min ago
   FROMTIME=`date --date="16 minutes ago" +'%Y-%m-%d %H:%M:%S'`
   TOTIME=`date --date="1 minutes ago" +'%Y-%m-%d %H:%M:%S'`
   # this token needs only audit/activities permission
   export SDM_ADMIN_TOKEN=[token]
   S3NAME=strongdm-log-$TIMESTAMP.gz
   S3ACTIVITIESNAME=strongdm-activities-$TIMESTAMP.gz
   S3PATH=s3://bucket/path/to/logs # no trailing slash
   export AWS_ACCESS_KEY_ID=[token]
   export AWS_SECRET_ACCESS_KEY=[token]
   
   # ensure AWS environment variables are in place
   
   journalctl -q -o cat --since "$FROMTIME" --until "$TOTIME" -u sdm-proxy | \
   gzip | aws s3 cp - $S3PATH/$S3NAME
   
   sdm audit activities --from "$FROMTIME" --to "$TOTIME" | \
   gzip | aws s3 cp - $S3PATH/$S3ACTIVITIESNAME

6. Add the following line to /etc/crontab. If you changed the export interval above, change the cron interval here to match.

   0,15,30,45 * * * * root /home/ubuntu/s3export.sh

7. Verify that files are being generated every 15 minutes in your S3 bucket.

Extract SSH Captures From Exported Logs

If your organization requires or is interested in extracting captured SSH sessions from your exported activity logs, see section View Logs from the CLI for more information.