update_insights.sh

Utility to copy audit logs to Apache Iceberg.

The update_insights.sh file is a utility to copy audit logs generated by Data Fabric onto Apache Iceberg (Iceberg) to be able to query the data stored in the Data Fabric audit logs.

This utility can be found in the /opt/mapr/server/tools folder.

Iceberg uses Hive metastore and MySQL to store the Iceberg catalog. Data Fabric is used for storing Iceberg metadata and data.

In the Hive Metastore, the default namespace is used by Data Fabric and type of audit log, that is, mfs/cldb/s3server/auth is the table name identifier. These table name identifiers can be used to query the table data that has been added to Iceberg from the Data Fabric audit logs.

Prerequisites

MySQL must be installed for proper working of Iceberg. The connection parameters must have been configured for Iceberg to connect to it.
To be able to add, drop, or print an audit log file to the Iceberg table, you must have installed Hive and the mapr-hivemetastore service that is downloadable from the site that hosts the HPE Ezmeral Data Fabric packages.
NOTE The update_insights.sh utility connects to Hive metastore using the thrift protocol(thrift://localhost:9083).

Auditing must be enabled on the cluster or fabric and audit logs must be available before running the update_insights.sh utility.

Audit Log location

The update_insights.sh utility requires the audit log file location.

The following table lists the audit logs along with their respective locations.

Audit Log File	Location
MFS audit log	`/var/mapr/local/<hostname>/audit/5660/FS-Audit*` NOTE The above statement denotes the absolute file path for file names beginning with FS-Audit
S3 audit log	`/var/mapr/local/mapr.s3.audit/<hostname>`
CLDB audit log	`/opt/mapr/logs/cldbaudit.log`
Authentication audit log	`/opt/mapr/logs/authaudit.log`

Syntax

To view the usage help for update_insights.sh, change directory to /opt/mapr/server/tools and run the script with the --help argument.

cd /opt/mapr/server/tools
$ ./update_insights.sh --help

Following is the usage help.

  usage: update_insights
   -action <arg>     {add|drop|print}
   -auditfile <arg>  Path to Audit log file      
   -endline <arg>    End line number to add from audit log file to insights. default is EOF
   -startline <arg>  Start line number to add from audit log file to insights. default is 1. Line numbers start at 1.
   -type {mfs|auth|cldb|s3server} component

TIP If values for startline and endline are not specified while running update_insights.sh, the specified operation such as add, print is performed on the entire content of the audit log in question.

Parameter	Description
`action`	This is a mandatory parameter and denotes the action to perform on the audit log. Action can have the value `add`, `drop` or `print`. Use the value `add` to add the specified audit file to Iceberg. Use the value `drop` to drop or remove the Iceberg table. Use the value `print` to print the Iceberg table contents.
`auditfile`	The absolute path of the audit file to copy to Iceberg. For the `add` action, the auditfile is a required field. For the `drop` and `print` actions, the value is not required. IMPORTANT If the audit file to add is on a local volume, you must mount the file system before running the utility, and then, provide the absolute path.
`endline`	The end line number to add from audit log file to Iceberg. The default value is EOF.
`startline`	The start line number to add from audit log file to Iceberg. The default value is 1.
`type`	The type of audit log file to add to, drop from or print to Iceberg. The type parameter can accept the value, `mfs`, `auth`, `cldb` or `s3server`. Use the value, `mfs` for mfs audit log, `auth` for authentication log, `cldb` for cldb log and `s3server` for s3 server log. Operation on only a single audit log/component can be performed at a given time.

Examples

Add MFS audit log starting from line 1 to line 5 to the Iceberg table.

#/opt/mapr/server/tools/update_insights.sh -type mfs -action add -endline 5 -auditfile FS-Audit.log

Drop or delete authentication audit log from a fabric to the Iceberg table.

#/opt/mapr/server/tools/update_insights.sh -type auth -action drop

Print the MFS audit log file contents from Iceberg to the console.

#/opt/mapr/server/tools/update_insights.sh -type mfs -action print

See Configuring Data Fabric to Track User Behavior for more information about user behavior tracking configuration in Data Fabric.