update_insights.sh

Utility to manage Insight service related Iceberg tables.

The update_insights.sh file is a utility to manage the Insight service-related Iceberg Tables. These Iceberg tables are designed to store the audit records generated by Data Fabric in a more structured manner.

The update_insights.sh utility is available as a part of mapr-insight package.

The insight-related Iceberg tables are stored on Data Fabric. The tables use the Iceberg Catalog as Hive via the Hive Metastore service, which must be configured with RDBMS for production or could use the default Derby DB for trial purposes.

Insight related Iceberg tables are placed in the default namespace of Hive Metastore.

Prerequisites

  • To be able to manage Insight related Iceberg tables, you must have the Hive Meta store service running on cluster (installable via mapr-hivemetastore package. The package is downloadable from the site that hosts the HPE Ezmeral Data Fabric packages).
    NOTE
    The update_insights.sh utility connects to Hive metastore using the thrift protocol(thrift://localhost:9083).
  • Auditing must be enabled on the cluster or fabric and audit logs must be available before running the update_insights.sh utility.

Steps to configure Data Fabric before running the update_insights.sh utility

  1. Install a basic Data Fabric cluster with mapr-insight package.
  2. Once the cluster is up and running, install the mapr-hivemetastore package from EEP and configure mapr-hivemetastore onto the cluster/fabric.
  3. Run update_insights.sh with it's options to manage the insight related Iceberg Tables
    TIP
    If you wish to add the MFS audit logs to Iceberg tables, Run the expandaudit utility on MFS audit records before updating Iceberg on the audit logs (There can be different FIDs that belong to the same file. Running expandaudit ensures that the filename is the same for different audit log entries that refer to different fids of a given file. The expandaudit utility makes the audit log contents more user-friendly by replacing ids with names).

Audit Log location

The update_insights.sh utility requires the audit log file location.

The following table lists the audit logs along with their respective locations.
Audit Log File Location
MFS audit log /var/mapr/local/<hostname>/audit/5660/FS-Audit*
NOTE
The above statement denotes the absolute file path for file names beginning with FS-Audit
S3 audit log /var/mapr/local/mapr.s3.audit/<hostname>
CLDB audit log /opt/mapr/logs/cldbaudit.log
Authentication audit log /opt/mapr/logs/authaudit.log

Syntax

To view the usage help for update_insights.sh, change directory to /opt/mapr/server/tools and run the script with the --help argument.
cd /opt/mapr/server/tools
$ ./update_insights.sh

Following is the usage help.

 
                            
usage: update_insights:

-a,--action <arg> {add|drop|print|count}
-b,--brief Brief snapshot listing.
-d,--delete-files Delete files when expiring snapshot
-da,--delete-added-files Whene expiring snapshot, delete files that were added when snapshot was created
-e,--endline <arg> End line number to add from audit log file to insights, default is EOF
-es,--expire-snap <arg> Expire snapshot with given snap-id
-et,--expire-ts <arg> Expire snapshots older than given ms timestamp
-f,--auditfile <arg> Path to Audit log file
-fpc,--fidPathDBCheck <arg> retrieve give fidPath entry
-fps,--fidPathDBScan dump the fidPathDB
-l,--list-snaps List snapshots
-lo,--list-orphans List orphan metadata files (does not handle data files)
-n,--table-name <arg> Use the given table, rather than the default for the component
-ps,--print-schema Print schema
-s,--startline <arg> Start line number to add from audit log file to insights, default is 1. Line numbers start at 1.
-t,--type <arg> {mfs|auth|cldb|s3server} component
-tm,--trial-mode Choose tables for trial mode.
-tn,--tail-n <arg> Number of records to be displayed from the last.
-v,--verbose Verbose snapshot listing.⁠
TIP
If values for startline and endline are not specified while running update_insights.sh, the specified operation such as add, print is performed on the entire content of the audit log in question.
Option Description
action

Denotes the action to perform on the audit log. Action can have the value add, drop, print count. Use the value add to add the specified audit file to Iceberg. A new file is added as a new snapshot to Iceberg. Use the value drop to drop or remove the Iceberg table. Use the value print to print the Iceberg table contents. Use the value count to count the number of rows in the Iceberg table. Must be used with the mandatory type option.
brief Used, along with list-snaps option, to obtain brief list of Iceberg snapshots for the specified type of Iceberg table. Must be used with the mandatory type option.
delete-files Used to delete data files associated with Iceberg snapshot, when the snapshot is expired. Used along with expire-snap or expire-ts option.
delete-added-files Used to delete data files added in an Iceberg snapshot, when the snapshot is expired. Used along with expire-snap or expire-ts option.
endline Used along with the action option of add (end line number to add audit log to specified type of Iceberg table) or print (end record number to have console output of records present in the specified type of Iceberg table). The default value is EOF. Must be used with the mandatory type option.
expire-snap Used to expire Iceberg snapshot for the specified type of Iceberg table. Takes snapshot ID as parameter. Must be used with the mandatory type option.
expire-ts Used to expire Iceberg snapshots that are older than the specified timestamp for the specified type of Iceberg table. The timestamp is Epoch Time in milliseconds. Specify a timestamp as an argument for this option. Must be used with the mandatory type option.
auditfile The absolute path of the audit file to copy to Iceberg. For the add action, the auditfile is a required field. For the drop and print actions, the value is not required. Must be used with the mandatory type option.
IMPORTANT
If the audit file to add is on a local volume, you must mount the file system before running the utility, and then, provide the absolute path.
list-snaps Used to list all snapshots for the specified type of Iceberg table. Must be used with the mandatory type option.
list-orphans Used to parameter to list all orphan Iceberg metadata files that do not point to a Iceberg data file or snapshot, for the specified type of Iceberg table. Must be used with the mandatory type option.
table-name Used to specify a custom Iceberg table name, else the type name will be used for Iceberg table. Must be used with the mandatory type option.
startline used along with the action option of add (start line number to add audit log to specified type of Iceberg table) or print (start record number to have console output of records present in the specified type of Iceberg table). The default value is 1. Must be used with the mandatory type option.
type type is a mandatory field which needs to used along with other options. The type of audit log file to add to, drop from or print to Iceberg tables. Type can accept the value, mfs, auth, cldb or s3server. Use the value, mfs for mfs audit log, auth for authentication log, cldb for cldb log and s3server for s3 server log. Operation on only a single audit log/component can be performed at a given time. Must be used with the mandatory type option.
tail-n Used to display the last n records present in the Iceberg table. Must be used with the mandatory type option and action option. Must not be used with startline or endline options.
verbose Used to get more detailed output when used along with options like list-snaps and list-orphan metadata files for the specified type of Iceberg table. verbose cannot be used in combination with brief option. Must be used with the mandatory type option.

Examples

Add cldb audit log to the Iceberg table.


#/opt/mapr/server/tools/update_insights.sh -t cldb -a add -f /opt/mapr/logs/cldbaudit.log.json
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.mapr.fs.ShimLoader (file:/opt/mapr/lib/maprfs-7.10.0.0-mapr.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.mapr.fs.ShimLoader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Iceberg table: cldb_is

Add cldb audit log contents from line 5 to line 20, to the Iceberg table.

/opt/mapr/server/tools/update_insights.sh -t cldb -a add -f /opt/mapr/logs/cldbaudit.log.json -s 5 -e 20

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.mapr.fs.ShimLoader (file:/opt/mapr/lib/maprfs-7.10.0.0-mapr.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.mapr.fs.ShimLoader 
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations 
WARNING: All illegal access operations will be denied in a future release 
Iceberg table: cldb_is⁠

Drop or delete cldb table from Iceberg.

#/opt/mapr/server/tools/update_insights.sh -t cldb -a drop

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.mapr.fs.ShimLoader (file:/opt/mapr/lib/maprfs-7.10.0.0-mapr.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.mapr.fs.ShimLoader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
cldb Table deleted successfully

Print contents of cldb Iceberg table to the console.

#/opt/mapr/server/tools/update_insights.sh -t cldb -a print

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.mapr.fs.ShimLoader (file:/opt/mapr/lib/maprfs-7.10.0.0-mapr-SNAPSHOT.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.mapr.fs.ShimLoader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Iceberg table: cldb_is
1. Record(6460176864726381504, 2025-04-14T08:02:33.608Z, configRead, cluster, mapr, 10.163.167.202, 5000, null, null, null, null, null, null, null, null, null, null, 0, null, null, null, null, null, null)
2. Record(6460176864726381504, 2025-04-14T08:02:33.610Z, configRead, cluster, mapr, 10.163.167.202, 5000, null, null, null, null, null, null, null, null, null, null, 0, null, null, null, null, null, null)
3. Record(6460176864726381504, 2025-04-14T08:27:33.609Z, configRead, cluster, mapr, 10.163.167.202, 5000, null, null, null, null, null, null, null, null, null, null, 0, null, null, null, null, null, null)
4. Record(6460176864726381504, 2025-04-14T08:27:33.610Z, configRead, cluster, mapr, 10.163.167.202, 5000, null, null, null, null, null, null, null, null, null, null, 0, null, null, null, null, null, null)
5. Record(6460176864726381504, 2025-04-14T08:11:33.608Z, configRead, cluster, mapr, 10.163.167.202, 5000, null, null, null, null, null, null, null, null, null, null, 0, null, null, null, null, null, null)
6. Record(6460176864726381504, 2025-04-14T08:11:33.610Z, configRead, cluster, mapr, 10.163.167.202, 5000, null, null, null, null, null, null, null, null, null, null, 0, null, null, null, null, null, null)
7. Record(6460176864726381504, 2025-04-14T08:41:33.608Z, configRead, cluster, mapr, 10.163.167.202, 5000, null, null, null, null, null, null, null, null, null, null, 0, null, null, null, null, null, null)
8. Record(6460176864726381504, 2025-04-14T08:41:33.610Z, configRead, cluster, mapr, 10.163.167.202, 5000, null, null, null, null, null, null, null, null, null, null, 0, null, null, null, null, null, null)⁠

Print records from 5 to 15 present in the cldb Iceberg table to the console.

/opt/mapr/server/tools/update_insights.sh -t cldb -a print -s 5 -e 15

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.mapr.fs.ShimLoader (file:/opt/mapr/lib/maprfs-7.10.0.0-mapr.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.mapr.fs.ShimLoader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
5. Record(9, 2024-07-30T07:04:20.848Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)
6. Record(10, 2024-07-30T07:04:20.850Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)
7. Record(11, 2024-07-30T07:05:20.848Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)
8. Record(12, 2024-07-30T07:05:20.849Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)
9. Record(13, 2024-07-30T07:06:20.848Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)
10. Record(14, 2024-07-30T07:06:20.850Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)
11. Record(15, 2024-07-30T07:07:20.848Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)
12. Record(16, 2024-07-30T07:07:20.850Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)
13. Record(17, 2024-07-30T07:08:20.849Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)
14. Record(18, 2024-07-30T07:08:20.850Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)
15. Record(19, 2024-07-30T07:09:20.848Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)⁠

Brief list of snapshots for cldb Iceberg table

/opt/mapr/server/tools/update_insights.sh -t cldb -l -b

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.mapr.fs.ShimLoader (file:/opt/mapr/lib/maprfs-7.10.0.0-mapr.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.mapr.fs.ShimLoader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
s-id: 7721073411112484179, timeStr: 2024-07-30 03:37:55.000 GMT-0700, time: 1722335875242, par-id: null
s-id: 4886192842798080540, timeStr: 2024-07-30 03:38:24.000 GMT-0700, time: 1722335904308, par-id: 7721073411112484179

curr snap id: 4886192842798080540⁠

Use custom name for Iceberg table instead of using type name for Iceberg table 

/opt/mapr/server/tools/update_insights.sh -t cldb -a add -f /opt/mapr/logs/cldbaudit.log.json -n cldbtest


WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.mapr.fs.ShimLoader (file:/opt/mapr/lib/maprfs-7.10.0.0-mapr.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.mapr.fs.ShimLoader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

List snapshot for cldb Iceberg table in verbose mode

update_insights.sh -t cldb -l -vWARNING: An illegal reflective access operation has occurred

WARNING: Illegal reflective access by com.mapr.fs.ShimLoader (file:/opt/mapr/lib/maprfs-7.10.0.0-mapr.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.mapr.fs.ShimLoader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
s-id: 7721073411112484179, timeStr: 2024-07-30 03:37:55.000 GMT-0700, time: 1722335875242, par-id: null
num manifests: 1
path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/efbc75cc-3a41-4559-b539-f0b38f7b48bd-m0.avro, DATA, (*)
added files : 1, added rows: 660, len: 6943
added files:
path: hdfs://CLUSTER1/var/mapr/insights/cldb/013426bf-1d32-4cf2-806a-52969cd7ec3f
id lower: 1, id upper: 660
added file count: 1
removed data files:
removed data file count: 0
added delete files:
added delete file count: 0
all files:
path: hdfs://CLUSTER1/var/mapr/insights/cldb/013426bf-1d32-4cf2-806a-52969cd7ec3f
all file count: 1

s-id: 4886192842798080540, timeStr: 2024-07-30 03:38:24.000 GMT-0700, time: 1722335904308, par-id: 7721073411112484179
num manifests: 2
path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/d414d29b-b912-42e5-9848-684d7280a56c-m0.avro, DATA, (*)
added files : 1, added rows: 660, len: 6946
path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/efbc75cc-3a41-4559-b539-f0b38f7b48bd-m0.avro, DATA
added files : 1, added rows: 660, len: 6943
added files:
path: hdfs://CLUSTER1/var/mapr/insights/cldb/ff7d2233-b795-4f12-8c89-abc78b0f9c76
id lower: 1, id upper: 660
added file count: 1
removed data files:
removed data file count: 0
added delete files:
added delete file count: 0
all files:
path: hdfs://CLUSTER1/var/mapr/insights/cldb/ff7d2233-b795-4f12-8c89-abc78b0f9c76
path: hdfs://CLUSTER1/var/mapr/insights/cldb/013426bf-1d32-4cf2-806a-52969cd7ec3f
all file count: 2

curr snap id: 4886192842798080540
table properties:

Expire snapshot of cldb Iceberg table using Snapshot ID and delete the files added as part of the snapshot

/opt/mapr/server/tools/update_insights.sh -t cldb -es 4886192842798080540 -da


WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.mapr.fs.ShimLoader (file:/opt/mapr/lib/maprfs-7.10.0.0-mapr.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.mapr.fs.ShimLoader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Got expire-snap, snapId: 4886192842798080540
Snapshots before expiring snapshot 4886192842798080540:

s-id: 4886192842798080540, timeStr: 2024-07-30 03:38:24.000 GMT-0700, time: 1722335904308, par-id: 7721073411112484179
num manifests: 2
path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/d414d29b-b912-42e5-9848-684d7280a56c-m0.avro, DATA, (*)
path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/efbc75cc-3a41-4559-b539-f0b38f7b48bd-m0.avro, DATA
added files:
path: hdfs://CLUSTER1/var/mapr/insights/cldb/ff7d2233-b795-4f12-8c89-abc78b0f9c76
added file count: 1
removed data files:
removed data file count: 0
added delete files:
added delete file count: 0
all files:
path: hdfs://CLUSTER1/var/mapr/insights/cldb/ff7d2233-b795-4f12-8c89-abc78b0f9c76
path: hdfs://CLUSTER1/var/mapr/insights/cldb/013426bf-1d32-4cf2-806a-52969cd7ec3f
all file count: 2

s-id: 3544046758310960189, timeStr: 2024-07-30 03:48:54.000 GMT-0700, time: 1722336534517, par-id: 4886192842798080540
num manifests: 2
path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/d414d29b-b912-42e5-9848-684d7280a56c-m0.avro, DATA
path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/efbc75cc-3a41-4559-b539-f0b38f7b48bd-m0.avro, DATA
added files:
added file count: 0
removed data files:
removed data file count: 0
added delete files:
added delete file count: 0
all files:
path: hdfs://CLUSTER1/var/mapr/insights/cldb/ff7d2233-b795-4f12-8c89-abc78b0f9c76
path: hdfs://CLUSTER1/var/mapr/insights/cldb/013426bf-1d32-4cf2-806a-52969cd7ec3f
all file count: 2


curr snap id: 3544046758310960189
Expiring snapshot: 4886192842798080540
2024-07-30 03:51:56,779 INFO iceberg.RemoveSnapshots: Expiring snapshot with id: 4886192842798080540
2024-07-30 03:51:56,788 INFO iceberg.RemoveSnapshots: Committed snapshot changes
2024-07-30 03:51:56,791 INFO iceberg.RemoveSnapshots: Expired snapshot: BaseSnapshot{id=4886192842798080540, timestamp_ms=1722335904308, operation=append, summary={added-data-files=1, added-records=660, added-files-size=9163, changed-partition-count=1, total-records=1320, total-files-size=18326, total-data-files=2, total-delete-files=0, total-position-deletes=0, total-equality-deletes=0}, manifest-list=hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/snap-4886192842798080540-1-d414d29b-b912-42e5-9848-684d7280a56c.avro, schema-id=0}
2024-07-30 03:51:56,791 INFO iceberg.RemoveSnapshots: Committed snapshot changes; cleaning up expired manifests and data files.
2024-07-30 03:51:56,801 WARN iceberg.RemoveSnapshots: Manifests to delete:
2024-07-30 03:51:56,801 WARN iceberg.RemoveSnapshots: Manifests Lists to delete: hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/snap-4886192842798080540-1-d414d29b-b912-42e5-9848-684d7280a56c.avro
deleting data file: hdfs://CLUSTER1/var/mapr/insights/cldb/ff7d2233-b795-4f12-8c89-abc78b0f9c76
About to delete nsnaps: 1, delAddedFiles: true, delFiles: false, nFiles: 1
2024-07-30 03:51:57,015 INFO iceberg.SnapshotProducer: Committed snapshot 2831938722446196530 (StreamingDelete)
2024-07-30 03:51:57,214 INFO hive.HiveTableOperations: Committed to table hive.default.cldb with the new metadata location hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00004-569c881d-ca7d-4320-afba-21bd83ad4e7c.metadata.json
2024-07-30 03:51:57,214 INFO iceberg.BaseMetastoreTableOperations: Successfully committed to table hive.default.cldb in 179 ms
2024-07-30 03:51:57,227 INFO iceberg.BaseMetastoreTableOperations: Refreshing table metadata from new version: hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00004-569c881d-ca7d-4320-afba-21bd83ad4e7c.metadata.json
Snapshots after expiring snapshot 4886192842798080540:
s-id: 3544046758310960189, timeStr: 2024-07-30 03:48:54.000 GMT-0700, time: 1722336534517, par-id: 4886192842798080540
num manifests: 2
path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/d414d29b-b912-42e5-9848-684d7280a56c-m0.avro, DATA
path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/efbc75cc-3a41-4559-b539-f0b38f7b48bd-m0.avro, DATA
added files:
added file count: 0
removed data files:
removed data file count: 0
added delete files:
added delete file count: 0
all files:
path: hdfs://CLUSTER1/var/mapr/insights/cldb/ff7d2233-b795-4f12-8c89-abc78b0f9c76
path: hdfs://CLUSTER1/var/mapr/insights/cldb/013426bf-1d32-4cf2-806a-52969cd7ec3f
all file count: 2

s-id: 2831938722446196530, timeStr: 2024-07-30 03:51:57.000 GMT-0700, time: 1722336717012, par-id: 3544046758310960189
num manifests: 2
path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/ffac5034-b430-468b-a2b7-b11e4f2b37c4-m0.avro, DATA, (*)
path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/efbc75cc-3a41-4559-b539-f0b38f7b48bd-m0.avro, DATA
added files:
added file count: 0
removed data files:
path: hdfs://CLUSTER1/var/mapr/insights/cldb/ff7d2233-b795-4f12-8c89-abc78b0f9c76
removed data file count: 1
added delete files:
added delete file count: 0
all files:
path: hdfs://CLUSTER1/var/mapr/insights/cldb/013426bf-1d32-4cf2-806a-52969cd7ec3f
all file count: 1


curr snap id: 2831938722446196530

Expire only the snapshot for the specified Snapshot ID of cldb Iceberg table

/opt/mapr/server/tools/update_insights.sh -t cldb -es 7721073411112484179

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.mapr.fs.ShimLoader (file:/opt/mapr/lib/maprfs-7.10.0.0-mapr.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.mapr.fs.ShimLoader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Got expire-snap, snapId: 7721073411112484179
Snapshots before expiring snapshot 7721073411112484179:
s-id: 7721073411112484179, timeStr: 2024-07-30 03:37:55.000 GMT-0700, time: 1722335875242, par-id: null
num manifests: 1
path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/efbc75cc-3a41-4559-b539-f0b38f7b48bd-m0.avro, DATA, (*)
added files:
path: hdfs://CLUSTER1/var/mapr/insights/cldb/013426bf-1d32-4cf2-806a-52969cd7ec3f
added file count: 1
removed data files:
removed data file count: 0
added delete files:
added delete file count: 0
all files:
path: hdfs://CLUSTER1/var/mapr/insights/cldb/013426bf-1d32-4cf2-806a-52969cd7ec3f
all file count: 1

s-id: 4886192842798080540, timeStr: 2024-07-30 03:38:24.000 GMT-0700, time: 1722335904308, par-id: 7721073411112484179
num manifests: 2
path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/d414d29b-b912-42e5-9848-684d7280a56c-m0.avro, DATA, (*)
path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/efbc75cc-3a41-4559-b539-f0b38f7b48bd-m0.avro, DATA
added files:
path: hdfs://CLUSTER1/var/mapr/insights/cldb/ff7d2233-b795-4f12-8c89-abc78b0f9c76
added file count: 1
removed data files:
removed data file count: 0
added delete files:
added delete file count: 0
all files:
path: hdfs://CLUSTER1/var/mapr/insights/cldb/ff7d2233-b795-4f12-8c89-abc78b0f9c76
path: hdfs://CLUSTER1/var/mapr/insights/cldb/013426bf-1d32-4cf2-806a-52969cd7ec3f
all file count: 2


curr snap id: 4886192842798080540
Expiring snapshot: 7721073411112484179
2024-07-30 03:48:54,348 INFO iceberg.RemoveSnapshots: Expiring snapshot with id: 7721073411112484179
2024-07-30 03:48:54,356 INFO iceberg.RemoveSnapshots: Committed snapshot changes
2024-07-30 03:48:54,359 INFO iceberg.RemoveSnapshots: Expired snapshot: BaseSnapshot{id=7721073411112484179, timestamp_ms=1722335875242, operation=append, summary={added-data-files=1, added-records=660, added-files-size=9163, changed-partition-count=1, total-records=660, total-files-size=9163, total-data-files=1, total-delete-files=0, total-position-deletes=0, total-equality-deletes=0}, manifest-list=hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/snap-7721073411112484179-1-efbc75cc-3a41-4559-b539-f0b38f7b48bd.avro, schema-id=0}
2024-07-30 03:48:54,360 INFO iceberg.RemoveSnapshots: Committed snapshot changes; cleaning up expired manifests and data files.
2024-07-30 03:48:54,369 WARN iceberg.RemoveSnapshots: Manifests to delete:
2024-07-30 03:48:54,369 WARN iceberg.RemoveSnapshots: Manifests Lists to delete: hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/snap-7721073411112484179-1-efbc75cc-3a41-4559-b539-f0b38f7b48bd.avro
About to delete nsnaps: 0, delAddedFiles: false, delFiles: false, nFiles: 0
2024-07-30 03:48:54,519 INFO iceberg.SnapshotProducer: Committed snapshot 3544046758310960189 (StreamingDelete)
2024-07-30 03:48:54,702 INFO hive.HiveTableOperations: Committed to table hive.default.cldb with the new metadata location hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00003-1fe48d4e-7be1-4b09-8133-9c8d79419a37.metadata.json
2024-07-30 03:48:54,702 INFO iceberg.BaseMetastoreTableOperations: Successfully committed to table hive.default.cldb in 163 ms
2024-07-30 03:48:54,714 INFO iceberg.BaseMetastoreTableOperations: Refreshing table metadata from new version: hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00003-1fe48d4e-7be1-4b09-8133-9c8d79419a37.metadata.json
Snapshots after expiring snapshot 7721073411112484179:
s-id: 4886192842798080540, timeStr: 2024-07-30 03:38:24.000 GMT-0700, time: 1722335904308, par-id: 7721073411112484179
num manifests: 2
path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/d414d29b-b912-42e5-9848-684d7280a56c-m0.avro, DATA, (*)
path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/efbc75cc-3a41-4559-b539-f0b38f7b48bd-m0.avro, DATA
added files:
path: hdfs://CLUSTER1/var/mapr/insights/cldb/ff7d2233-b795-4f12-8c89-abc78b0f9c76
added file count: 1
removed data files:
removed data file count: 0
added delete files:
added delete file count: 0
all files:
path: hdfs://CLUSTER1/var/mapr/insights/cldb/ff7d2233-b795-4f12-8c89-abc78b0f9c76
path: hdfs://CLUSTER1/var/mapr/insights/cldb/013426bf-1d32-4cf2-806a-52969cd7ec3f
all file count: 2

s-id: 3544046758310960189, timeStr: 2024-07-30 03:48:54.000 GMT-0700, time: 1722336534517, par-id: 4886192842798080540
num manifests: 2
path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/d414d29b-b912-42e5-9848-684d7280a56c-m0.avro, DATA
path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/efbc75cc-3a41-4559-b539-f0b38f7b48bd-m0.avro, DATA
added files:
added file count: 0
removed data files:
removed data file count: 0
added delete files:
added delete file count: 0
all files:
path: hdfs://CLUSTER1/var/mapr/insights/cldb/ff7d2233-b795-4f12-8c89-abc78b0f9c76
path: hdfs://CLUSTER1/var/mapr/insights/cldb/013426bf-1d32-4cf2-806a-52969cd7ec3f
all file count: 2


curr snap id: 3544046758310960189
------------------------------------------------------------

Expire snapshots older than given ms timestamp (Epoch time) and delete the data files for cldb Iceberg table

# /opt/mapr/server/tools/update_insights.sh -t cldb -et 1722336908008 -d

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.mapr.fs.ShimLoader (file:/opt/mapr/lib/maprfs-7.10.0.0-mapr.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.mapr.fs.ShimLoader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Got expire-ts, expire-ms: 1722336908008, long: 1722336908008
Snapshots before expiry, time: 2024-07-30 03:55:08.000 GMT-0700, timems: 1722336908008
CurrTime: 2024-07-30 03:56:17.000 GMT-0700, timems: 1722336977925
s-id: 3544046758310960189, timeStr: 2024-07-30 03:48:54.000 GMT-0700, time: 1722336534517, par-id: 4886192842798080540
num manifests: 2
path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/d414d29b-b912-42e5-9848-684d7280a56c-m0.avro, DATA
path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/efbc75cc-3a41-4559-b539-f0b38f7b48bd-m0.avro, DATA
added files:
added file count: 0
removed data files:
removed data file count: 0
added delete files:
added delete file count: 0
all files:
path: hdfs://CLUSTER1/var/mapr/insights/cldb/ff7d2233-b795-4f12-8c89-abc78b0f9c76
path: hdfs://CLUSTER1/var/mapr/insights/cldb/013426bf-1d32-4cf2-806a-52969cd7ec3f
all file count: 2

s-id: 2831938722446196530, timeStr: 2024-07-30 03:51:57.000 GMT-0700, time: 1722336717012, par-id: 3544046758310960189
num manifests: 2
path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/ffac5034-b430-468b-a2b7-b11e4f2b37c4-m0.avro, DATA, (*)
path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/efbc75cc-3a41-4559-b539-f0b38f7b48bd-m0.avro, DATA
added files:
added file count: 0
removed data files:
path: hdfs://CLUSTER1/var/mapr/insights/cldb/ff7d2233-b795-4f12-8c89-abc78b0f9c76
removed data file count: 1
added delete files:
added delete file count: 0
all files:
path: hdfs://CLUSTER1/var/mapr/insights/cldb/013426bf-1d32-4cf2-806a-52969cd7ec3f
all file count: 1

s-id: 2297561737173859503, timeStr: 2024-07-30 03:55:08.000 GMT-0700, time: 1722336908008, par-id: 2831938722446196530
num manifests: 2
path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/30ea15da-e532-4eae-b3f7-a2fab2b30623-m0.avro, DATA, (*)
path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/efbc75cc-3a41-4559-b539-f0b38f7b48bd-m0.avro, DATA
added files:
path: hdfs://CLUSTER1/var/mapr/insights/cldb/179b3702-db89-49ac-9d42-dd3dc68af0cc
added file count: 1
removed data files:
removed data file count: 0
added delete files:
added delete file count: 0
all files:
path: hdfs://CLUSTER1/var/mapr/insights/cldb/179b3702-db89-49ac-9d42-dd3dc68af0cc
path: hdfs://CLUSTER1/var/mapr/insights/cldb/013426bf-1d32-4cf2-806a-52969cd7ec3f
all file count: 2

curr snap id: 2297561737173859503
Expiring snapshots before time: 2024-07-30 03:55:08.000 GMT-0700
About to delete nsnaps: 2, delAddedFiles: false, delFiles: true, nFiles: 0
2024-07-30 03:56:18,663 INFO iceberg.SnapshotProducer: Committed snapshot 3527657542738416278 (StreamingDelete)
2024-07-30 03:56:18,828 INFO hive.HiveTableOperations: Committed to table hive.default.cldb with the new metadata location hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00006-89188307-09c7-4f16-b8b1-eabef5d38356.metadata.json
2024-07-30 03:56:18,828 INFO iceberg.BaseMetastoreTableOperations: Successfully committed to table hive.default.cldb in 149 ms
2024-07-30 03:56:18,838 INFO iceberg.BaseMetastoreTableOperations: Refreshing table metadata from new version: hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00006-89188307-09c7-4f16-b8b1-eabef5d38356.metadata.json
Snapshots after expiring snaps:

s-id: 2297561737173859503, timeStr: 2024-07-30 03:55:08.000 GMT-0700, time: 1722336908008, par-id: 2831938722446196530
num manifests: 2
path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/30ea15da-e532-4eae-b3f7-a2fab2b30623-m0.avro, DATA, (*)
path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/efbc75cc-3a41-4559-b539-f0b38f7b48bd-m0.avro, DATA
added files:
path: hdfs://CLUSTER1/var/mapr/insights/cldb/179b3702-db89-49ac-9d42-dd3dc68af0cc
added file count: 1
removed data files:
removed data file count: 0
added delete files:
added delete file count: 0
all files:
path: hdfs://CLUSTER1/var/mapr/insights/cldb/179b3702-db89-49ac-9d42-dd3dc68af0cc
path: hdfs://CLUSTER1/var/mapr/insights/cldb/013426bf-1d32-4cf2-806a-52969cd7ec3f
all file count: 2

s-id: 3527657542738416278, timeStr: 2024-07-30 03:56:18.000 GMT-0700, time: 1722336978660, par-id: 2297561737173859503
num manifests: 2
path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/30ea15da-e532-4eae-b3f7-a2fab2b30623-m0.avro, DATA
path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/6ef85c57-4556-43ea-9a0c-e59ce65d6eb0-m0.avro, DATA, (*)
added files:
added file count: 0
removed data files:
path: hdfs://CLUSTER1/var/mapr/insights/cldb/013426bf-1d32-4cf2-806a-52969cd7ec3f
removed data file count: 1
added delete files:
added delete file count: 0
all files:
path: hdfs://CLUSTER1/var/mapr/insights/cldb/179b3702-db89-49ac-9d42-dd3dc68af0cc
all file count: 1

List orphan metadata files associated with cldb Iceberg table

# /opt/mapr/server/tools/update_insights.sh -t cldb -lo

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.mapr.fs.ShimLoader (file:/opt/mapr/lib/maprfs-7.10.0.0-mapr.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.mapr.fs.ShimLoader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Orphan files:
/mapr/CLUSTER1/var/mapr/insights/cldb/metadata/00001-9d65df8c-e9cb-40cf-8a5f-d256999fac83.metadata.json
/mapr/CLUSTER1/var/mapr/insights/cldb/metadata/00003-1fe48d4e-7be1-4b09-8133-9c8d79419a37.metadata.json
/mapr/CLUSTER1/var/mapr/insights/cldb/metadata/00000-9dbcee30-f022-4abf-ba91-75e01c77a406.metadata.json
/mapr/CLUSTER1/var/mapr/insights/cldb/metadata/00002-7304f0cc-d517-4c8d-929b-c28f4afbd392.metadata.json
/mapr/CLUSTER1/var/mapr/insights/cldb/metadata/00004-569c881d-ca7d-4320-afba-21bd83ad4e7c.metadata.json
/mapr/CLUSTER1/var/mapr/insights/cldb/metadata/00005-586c0de1-4cdb-45c8-813c-834b3202d115.metadata.json
------------------------------

List last 4 insight records

# ./update_insights.sh --action print -t mfs --tail-n 4 
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.mapr.fs.ShimLoader (file:/opt/mapr/lib/maprfs-7.10.0.0-mapr-SNAPSHOT.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.mapr.fs.ShimLoader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Iceberg table: mfs_is
1103. Record(6460176864726381504, 2025-04-10T22:49:38.532Z, RENAME, root, 0, 10.163.167.202, null, null, null, 2150.16.2, null, 2150.16.2, null, null, null, null, null, null, null, login.defs, null, 76125909, null, null, null, null, 0, login.defs._COPYING_, /vol1/, vol1, null, /vol1/, null, null, null, null, null, null, null, null, null, null)
1104. Record(6460176864726381504, 2025-04-10T22:49:46.537Z, GETPATHFORFID, null, 5000, 10.163.167.202, null, null, null, 2150.81.131424, null, null, null, null, null, null, null, null, null, null, null, 76125909, null, null, null, null, 0, null, /vol1/login.defs, vol1, null, null, null, null, null, null, null, null, null, null, null, null)
1105. Record(6460176864726381504, 2025-04-10T22:49:46.549Z, LOOKUP, null, 5000, 2620:0000:0a07:6115:0010:0163:0167:0202, null, null, null, 2150.16.2, null, 2150.81.131424, null, null, null, null, null, null, null, null, null, 76125909, null, null, null, null, 0, login.defs, /vol1/, vol1, null, /vol1/login.defs, null, null, null, null, null, null, null, null, null, null)
1106. Record(6460176864726381504, 2025-04-10T22:50:09.733Z, READDIR, root, 0, 2620:0000:0a07:6115:0010:0163:0167:0202, null, null, null, 2150.16.2, null, null, null, null, null, null, null, null, null, null, null, 76125909, null, null, null, null, 0, null, /vol1/, vol1, null, null, null, null, null, null, null, null, null, null, null, null)

See Configuring Data Fabric to Track User Behavior for more information about user behavior tracking configuration in Data Fabric.