update_insights.sh

Utility to copy audit logs to Apache Iceberg.

The update_insights.sh file is a utility to copy audit logs generated by Data Fabric onto Apache Iceberg (Iceberg) to be able to query the data stored in the Data Fabric audit logs.

The update_insights.sh utility is available as a part of mapr-core or mapr-core-internal package. This utility can be found in the /opt/mapr/server/tools folder. The utility stores the audit log data in Iceberg tables.

Data Fabric is used for storing Iceberg metadata and data. Iceberg uses Hive Metastore to store the Iceberg catalog.

In the Hive Metastore, the default namespace is used by Data Fabric and type of audit log, that is, mfs or cldb or s3server or auth is the table name identifier. These table name identifiers can be used to query the table data that has been added to Iceberg from the Data Fabric audit logs.

Prerequisites

  • MySQL must be installed for proper working of Iceberg. The connection parameters must have been configured for Iceberg to connect to it.
  • To be able to add, drop, or print an audit log file to the Iceberg table, you must have the mapr-hivemetastore packages installed on your cluster. The package is downloadable from the site that hosts the HPE Ezmeral Data Fabric packages.
    NOTE
    The update_insights.sh utility connects to Hive metastore using the thrift protocol(thrift://localhost:9083).
  • Auditing must be enabled on the cluster or fabric and audit logs must be available before running the update_insights.sh utility.

Steps to configure Data Fabric before running the update_insights.sh utility

  1. Install a basic Data Fabric cluster.
  2. Once the cluster is up and running, install the mapr-hivemetastore package from EEP and configure mapr-hivemetastore.
  3. Run the expandaudit utility before updating Iceberg. This is because there can be different FIDs that belong to the same file. Running expandaudit ensures that the filename is the same for different audit log entries that refer to different fids of a given file. The expandaudit utility makes the audit log contents more user-friendly by replacing ids with names.

  4. Run update_insights.sh to copy audit logs onto Apache Iceberg (Iceberg) tables, so that the data that is copied or added to Iceberg tables can be queried.

Audit Log location

The update_insights.sh utility requires the audit log file location.

The following table lists the audit logs along with their respective locations.
Audit Log File Location
MFS audit log /var/mapr/local/<hostname>/audit/5660/FS-Audit*
NOTE
The above statement denotes the absolute file path for file names beginning with FS-Audit
S3 audit log /var/mapr/local/mapr.s3.audit/<hostname>
CLDB audit log /opt/mapr/logs/cldbaudit.log
Authentication audit log /opt/mapr/logs/authaudit.log

Syntax

To view the usage help for update_insights.sh, change directory to /opt/mapr/server/tools and run the script with the --help argument.
cd /opt/mapr/server/tools
$ ./update_insights.sh

Following is the usage help.

 usage: update_insights
                            -action <arg>     {add|drop|print}
   -auditfile <arg>  Path to Audit log file      
   -endline <arg>    End line number to add from audit log file to insights. default is EOF
   -startline <arg>  Start line number to add from audit log file to insights. default is 1. Line numbers start at 1.
   -type {mfs|auth|cldb|s3server} component   
 -a,--action <arg>          {add|drop|print}
 -b,--brief                 Brief snapshot listing.
 -d,--delete-files          Delete files when expiring snapshot
 -da,--delete-added-files   Whene expiring snapshot, delete files that
                            were added when snapshot was created
 -e,--endline <arg>         End line number to add from audit log file to
                            insights, default is EOF
 -es,--expire-snap <arg>    Expire snapshot with given snap-id
 -et,--expire-ts <arg>      Expire snapshots older than given ms timestamp
 -f,--auditfile <arg>       Path to Audit log file
 -l,--list-snaps            List snapshots
 -lo,--list-orphans         List orphan metadata files (does not handle
                            data files)
 -n,--table-name <arg>      Use the given table, rather than the default
                            for the component
 -s,--startline <arg>       Start line number to add from audit log file
                            to insights, default is 1.  Line numbers start
                            at 1.
 -t,--type <arg>            {mfs|auth|cldb|s3server} component
 -v,--verbose               Verbose snapshot listing.
TIP
If values for startline and endline are not specified while running update_insights.sh, the specified operation such as add, print is performed on the entire content of the audit log in question.
Option Description
action Denotes the action to perform on the audit log. Action can have the value add, drop or print. Use the value add to add the specified audit file to Iceberg. A new file is added as a new snapshot to Iceberg. Use the value drop to drop or remove the Iceberg table. Use the value print to print the Iceberg table contents. Must be used with the mandatory type option.
brief Used, along with list-snaps option, to obtain brief list of Iceberg snapshots for the specified type of Iceberg table. Must be used with the mandatory type option.
delete-files Used to delete data files associated with Iceberg snapshot, when the snapshot is expired. Used along with expire-snap or expire-ts option.
delete-added-files Used to delete data files added in an Iceberg snapshot, when the snapshot is expired. Used along with expire-snap or expire-ts option.
endline Used along with the action option of add (end line number to add audit log to specified type of Iceberg table) or print (end record number to have console output of records present in the specified type of Iceberg table). The default value is EOF. Must be used with the mandatory type option.
expire-snap Used to expire Iceberg snapshot for the specified type of Iceberg table. Takes snapshot ID as parameter. Must be used with the mandatory type option.
expire-ts Used to expire Iceberg snapshots that are older than the specified timestamp for the specified type of Iceberg table. The timestamp is Epoch Time in milliseconds. Specify a timestamp as an argument for this option. Must be used with the mandatory type option.
auditfile The absolute path of the audit file to copy to Iceberg. For the add action, the auditfile is a required field. For the drop and print actions, the value is not required. Must be used with the mandatory type option.
IMPORTANT
If the audit file to add is on a local volume, you must mount the file system before running the utility, and then, provide the absolute path.
list-snaps Used to list all snapshots for the specified type of Iceberg table. Must be used with the mandatory type option.
list-orphans Used to parameter to list all orphan Iceberg metadata files that do not point to a Iceberg data file or snapshot, for the specified type of Iceberg table. Must be used with the mandatory type option.
table-name Used to specify a custom Iceberg table name, else the type name will be used for Iceberg table. Must be used with the mandatory type option.
startline used along with the action option of add (start line number to add audit log to specified type of Iceberg table) or print (start record number to have console output of records present in the specified type of Iceberg table). The default value is 1. Must be used with the mandatory type option.
type type is a mandatory field which needs to used along with other options. The type of audit log file to add to, drop from or print to Iceberg tables. Type can accept the value, mfs, auth, cldb or s3server. Use the value, mfs for mfs audit log, auth for authentication log, cldb for cldb log and s3server for s3 server log. Operation on only a single audit log/component can be performed at a given time. Must be used with the mandatory type option.
verbose Used to get more detailed output when used along with options like list-snaps and list-orphan metadata files for the specified type of Iceberg table. verbose cannot be used in combination with brief option. Must be used with the mandatory type option.

Examples

Add cldb audit log to the Iceberg table.

#/opt/mapr/server/tools/update_insights.sh -t cldb -a add -f /opt/mapr/logs/cldbaudit.log.json
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.mapr.fs.ShimLoader (file:/opt/mapr/lib/maprfs-7.8.0.0-mapr.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.mapr.fs.ShimLoader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2024-07-30 22:17:41,524 INFO util.log: Logging initialized @3455ms to org.eclipse.jetty.util.log.Slf4jLog
2024-07-30 22:17:42,314 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:17:42,314 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:17:42,317 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:17:42,318 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:17:42,318 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:17:42,318 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:17:42,649 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:17:42,649 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:17:42,649 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:17:42,650 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:17:42,650 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:17:42,650 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:17:42,650 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:17:42,650 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:17:42,650 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:17:42,777 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:17:42,777 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:17:42,777 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:17:42,797 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:17:42,797 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:17:42,797 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:17:42,864 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2024-07-30 22:17:42,979 INFO metastore.HiveMetaStoreClient: HMS client filtering is enabled.
2024-07-30 22:17:42,985 INFO metastore.HiveMetaStoreClient: Trying to connect to metastore with URI thrift://localhost:9083
2024-07-30 22:17:42,998 INFO metastore.HiveMetaStoreClient: HMSC::open(): Creating MapR-SASL-based thrift connection
2024-07-30 22:17:43,012 INFO security.ThriftTransportHelper: User authentication with method: MAPR-SECURITY
2024-07-30 22:17:43,075 INFO metastore.HiveMetaStoreClient: Opened a connection to metastore, current connections: 1
2024-07-30 22:17:43,076 INFO metastore.HiveMetaStoreClient: Connected to metastore.
2024-07-30 22:17:43,084 INFO metastore.RetryingMetaStoreClient: RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.metastore.HiveMetaStoreClient ugi=root (auth:CUSTOM) retries=1 delay=1 lifetime=0
2024-07-30 22:17:43,286 INFO iceberg.BaseMetastoreCatalog: Table properties set at catalog level through catalog properties: {}
2024-07-30 22:17:43,290 INFO iceberg.BaseMetastoreCatalog: Table properties enforced at catalog level through catalog properties: {}
2024-07-30 22:17:43,854 INFO hive.HiveTableOperations: Committed to table hive.default.cldb with the new metadata location hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00000-cefd1e2a-2008-4f9d-a7a1-adf3bc17aa1a.metadata.json
2024-07-30 22:17:43,854 INFO iceberg.BaseMetastoreTableOperations: Successfully committed to table hive.default.cldb in 533 ms
2024-07-30 22:17:44,780 INFO iceberg.BaseMetastoreTableOperations: Refreshing table metadata from new version: hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00000-cefd1e2a-2008-4f9d-a7a1-adf3bc17aa1a.metadata.json
2024-07-30 22:17:45,000 INFO compress.CodecPool: Got brand-new compressor [.gz]
2024-07-30 22:17:46,237 INFO hive.HiveTableOperations: Committed to table hive.default.cldb with the new metadata location hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00001-4970dd4c-c9ed-40b0-8bc1-cb008ae607f1.metadata.json
2024-07-30 22:17:46,237 INFO iceberg.BaseMetastoreTableOperations: Successfully committed to table hive.default.cldb in 119 ms
2024-07-30 22:17:46,238 INFO iceberg.SnapshotProducer: Committed snapshot 1816103760980415386 (MergeAppend)
2024-07-30 22:17:46,247 INFO iceberg.BaseMetastoreTableOperations: Refreshing table metadata from new version: hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00001-4970dd4c-c9ed-40b0-8bc1-cb008ae607f1.metadata.json

Add cldb audit log contents from line 5 to line 20, to the Iceberg table.

/opt/mapr/server/tools/update_insights.sh -t cldb -a add -f /opt/mapr/logs/cldbaudit.log.json -s 5 -e 20
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.mapr.fs.ShimLoader (file:/opt/mapr/lib/maprfs-7.8.0.0-mapr.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.mapr.fs.ShimLoader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2024-07-30 22:25:29,983 INFO util.log: Logging initialized @3499ms to org.eclipse.jetty.util.log.Slf4jLog
2024-07-30 22:25:30,779 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:25:30,780 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:25:30,783 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:25:30,783 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:25:30,783 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:25:30,783 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:25:31,118 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:25:31,118 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:25:31,118 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:25:31,118 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:25:31,118 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:25:31,118 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:25:31,118 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:25:31,118 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:25:31,118 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:25:31,246 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:25:31,246 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:25:31,246 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:25:31,265 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:25:31,265 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:25:31,265 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:25:31,326 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2024-07-30 22:25:31,441 INFO metastore.HiveMetaStoreClient: HMS client filtering is enabled.
2024-07-30 22:25:31,447 INFO metastore.HiveMetaStoreClient: Trying to connect to metastore with URI thrift://localhost:9083
2024-07-30 22:25:31,462 INFO metastore.HiveMetaStoreClient: HMSC::open(): Creating MapR-SASL-based thrift connection
2024-07-30 22:25:31,473 INFO security.ThriftTransportHelper: User authentication with method: MAPR-SECURITY
2024-07-30 22:25:31,532 INFO metastore.HiveMetaStoreClient: Opened a connection to metastore, current connections: 1
2024-07-30 22:25:31,533 INFO metastore.HiveMetaStoreClient: Connected to metastore.
2024-07-30 22:25:31,540 INFO metastore.RetryingMetaStoreClient: RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.metastore.HiveMetaStoreClient ugi=root (auth:CUSTOM) retries=1 delay=1 lifetime=0
2024-07-30 22:25:31,743 INFO iceberg.BaseMetastoreCatalog: Table properties set at catalog level through catalog properties: {}
2024-07-30 22:25:31,747 INFO iceberg.BaseMetastoreCatalog: Table properties enforced at catalog level through catalog properties: {}
2024-07-30 22:25:32,285 INFO hive.HiveTableOperations: Committed to table hive.default.cldb with the new metadata location hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00000-0e0a1743-3f12-45d2-a97b-18808ed4ab93.metadata.json
2024-07-30 22:25:32,285 INFO iceberg.BaseMetastoreTableOperations: Successfully committed to table hive.default.cldb in 507 ms
2024-07-30 22:25:32,487 INFO iceberg.BaseMetastoreTableOperations: Refreshing table metadata from new version: hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00000-0e0a1743-3f12-45d2-a97b-18808ed4ab93.metadata.json
2024-07-30 22:25:32,634 INFO compress.CodecPool: Got brand-new compressor [.gz]
2024-07-30 22:25:33,680 INFO hive.HiveTableOperations: Committed to table hive.default.cldb with the new metadata location hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00001-cc6b72f1-07d3-4d28-a7a1-1a9dd4c08a8a.metadata.json
2024-07-30 22:25:33,680 INFO iceberg.BaseMetastoreTableOperations: Successfully committed to table hive.default.cldb in 120 ms
2024-07-30 22:25:33,680 INFO iceberg.SnapshotProducer: Committed snapshot 7326414452571699455 (MergeAppend)
2024-07-30 22:25:33,689 INFO iceberg.BaseMetastoreTableOperations: Refreshing table metadata from new version: hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00001-cc6b72f1-07d3-4d28-a7a1-1a9dd4c08a8a.metadata.json

Drop or delete cldb table from Iceberg.

#/opt/mapr/server/tools/update_insights.sh -t cldb -a drop
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.mapr.fs.ShimLoader (file:/opt/mapr/lib/maprfs-7.8.0.0-mapr.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.mapr.fs.ShimLoader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2024-07-30 22:14:32,784 INFO util.log: Logging initialized @3479ms to org.eclipse.jetty.util.log.Slf4jLog
2024-07-30 22:14:33,560 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:14:33,560 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:14:33,563 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:14:33,564 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:14:33,564 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:14:33,564 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:14:33,887 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:14:33,887 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:14:33,887 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:14:33,887 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:14:33,887 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:14:33,887 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:14:33,887 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:14:33,887 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:14:33,887 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:14:34,015 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:14:34,016 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:14:34,016 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:14:34,035 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:14:34,035 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:14:34,035 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:14:34,098 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2024-07-30 22:14:34,209 INFO metastore.HiveMetaStoreClient: HMS client filtering is enabled.
2024-07-30 22:14:34,215 INFO metastore.HiveMetaStoreClient: Trying to connect to metastore with URI thrift://localhost:9083
2024-07-30 22:14:34,228 INFO metastore.HiveMetaStoreClient: HMSC::open(): Creating MapR-SASL-based thrift connection
2024-07-30 22:14:34,239 INFO security.ThriftTransportHelper: User authentication with method: MAPR-SECURITY
2024-07-30 22:14:34,295 INFO metastore.HiveMetaStoreClient: Opened a connection to metastore, current connections: 1
2024-07-30 22:14:34,296 INFO metastore.HiveMetaStoreClient: Connected to metastore.
2024-07-30 22:14:34,302 INFO metastore.RetryingMetaStoreClient: RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.metastore.HiveMetaStoreClient ugi=root (auth:CUSTOM) retries=1 delay=1 lifetime=0
2024-07-30 22:14:34,520 INFO iceberg.BaseMetastoreTableOperations: Refreshing table metadata from new version: hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00006-89188307-09c7-4f16-b8b1-eabef5d38356.metadata.json
2024-07-30 22:14:34,959 INFO iceberg.BaseMetastoreCatalog: Table loaded by catalog: hive.default.cldb
2024-07-30 22:14:34,985 INFO iceberg.BaseMetastoreTableOperations: Refreshing table metadata from new version: hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00006-89188307-09c7-4f16-b8b1-eabef5d38356.metadata.json
2024-07-30 22:14:35,537 INFO iceberg.CatalogUtil: Manifests to delete: GenericManifestFile{content=DATA, path=hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/6ef85c57-4556-43ea-9a0c-e59ce65d6eb0-m0.avro, length=6947, partition_spec_id=0, added_snapshot_id=3527657542738416278, added_data_files_count=0, added_rows_count=0, existing_data_files_count=0, existing_rows_count=0, deleted_data_files_count=1, deleted_rows_count=660, partitions=[], key_metadata=null, sequence_number=0, min_sequence_number=0}, GenericManifestFile{content=DATA, path=hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/efbc75cc-3a41-4559-b539-f0b38f7b48bd-m0.avro, length=6943, partition_spec_id=0, added_snapshot_id=7721073411112484179, added_data_files_count=1, added_rows_count=660, existing_data_files_count=0, existing_rows_count=0, deleted_data_files_count=0, deleted_rows_count=0, partitions=[], key_metadata=null, sequence_number=0, min_sequence_number=0}, GenericManifestFile{content=DATA, path=hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/30ea15da-e532-4eae-b3f7-a2fab2b30623-m0.avro, length=6951, partition_spec_id=0, added_snapshot_id=2297561737173859503, added_data_files_count=1, added_rows_count=696, existing_data_files_count=0, existing_rows_count=0, deleted_data_files_count=0, deleted_rows_count=0, partitions=[], key_metadata=null, sequence_number=0, min_sequence_number=0}
2024-07-30 22:14:35,673 INFO hive.HiveCatalog: Dropped table: default.cldb
cldb Table deleted successfully

Print contents of cldb Iceberg table to the console.

#/opt/mapr/server/tools/update_insights.sh -t cldb -a print
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.mapr.fs.ShimLoader (file:/opt/mapr/lib/maprfs-7.8.0.0-mapr.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.mapr.fs.ShimLoader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2024-07-30 22:28:47,673 INFO util.log: Logging initialized @3455ms to org.eclipse.jetty.util.log.Slf4jLog
2024-07-30 22:28:48,439 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:28:48,439 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:28:48,442 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:28:48,443 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:28:48,443 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:28:48,443 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:28:48,767 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:28:48,767 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:28:48,767 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:28:48,767 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:28:48,767 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:28:48,767 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:28:48,767 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:28:48,767 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:28:48,768 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:28:48,895 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:28:48,895 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:28:48,895 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:28:48,914 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:28:48,915 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:28:48,915 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:28:48,978 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2024-07-30 22:28:49,090 INFO metastore.HiveMetaStoreClient: HMS client filtering is enabled.
2024-07-30 22:28:49,096 INFO metastore.HiveMetaStoreClient: Trying to connect to metastore with URI thrift://localhost:9083
2024-07-30 22:28:49,109 INFO metastore.HiveMetaStoreClient: HMSC::open(): Creating MapR-SASL-based thrift connection
2024-07-30 22:28:49,120 INFO security.ThriftTransportHelper: User authentication with method: MAPR-SECURITY
2024-07-30 22:28:49,178 INFO metastore.HiveMetaStoreClient: Opened a connection to metastore, current connections: 1
2024-07-30 22:28:49,179 INFO metastore.HiveMetaStoreClient: Connected to metastore.
2024-07-30 22:28:49,185 INFO metastore.RetryingMetaStoreClient: RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.metastore.HiveMetaStoreClient ugi=root (auth:CUSTOM) retries=1 delay=1 lifetime=0
2024-07-30 22:28:49,400 INFO iceberg.BaseMetastoreTableOperations: Refreshing table metadata from new version: hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00001-cc6b72f1-07d3-4d28-a7a1-1a9dd4c08a8a.metadata.json
2024-07-30 22:28:49,832 INFO iceberg.BaseMetastoreCatalog: Table loaded by catalog: hive.default.cldb
2024-07-30 22:28:49,858 INFO iceberg.BaseMetastoreTableOperations: Refreshing table metadata from new version: hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00001-cc6b72f1-07d3-4d28-a7a1-1a9dd4c08a8a.metadata.json
2024-07-30 22:28:49,862 INFO iceberg.BaseMetastoreCatalog: Table loaded by catalog: hive.default.cldb
2024-07-30 22:28:49,892 INFO iceberg.BaseTableScan: Scanning table hive.default.cldb snapshot 7326414452571699455 created at 2024-07-31 05:25:33.555 with filter true
2024-07-30 22:28:50,743 INFO compress.CodecPool: Got brand-new decompressor [.gz]
1. Record(5, 2024-07-30T07:02:20.848Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)
2. Record(6, 2024-07-30T07:02:20.849Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)
3. Record(7, 2024-07-30T07:03:20.848Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)
4. Record(8, 2024-07-30T07:03:20.849Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)
5. Record(9, 2024-07-30T07:04:20.848Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)
6. Record(10, 2024-07-30T07:04:20.850Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)
7. Record(11, 2024-07-30T07:05:20.848Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)
8. Record(12, 2024-07-30T07:05:20.849Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)
9. Record(13, 2024-07-30T07:06:20.848Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)
10. Record(14, 2024-07-30T07:06:20.850Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)
11. Record(15, 2024-07-30T07:07:20.848Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)
12. Record(16, 2024-07-30T07:07:20.850Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)
13. Record(17, 2024-07-30T07:08:20.849Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)
14. Record(18, 2024-07-30T07:08:20.850Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)
15. Record(19, 2024-07-30T07:09:20.848Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)
16. Record(20, 2024-07-30T07:09:20.849Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)

Print records from 5 to 15 present in the cldb Iceberg table to the console.

/opt/mapr/server/tools/update_insights.sh -t cldb -a print -s 5 -e 15
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.mapr.fs.ShimLoader (file:/opt/mapr/lib/maprfs-7.8.0.0-mapr.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.mapr.fs.ShimLoader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2024-07-30 22:29:51,426 INFO util.log: Logging initialized @3489ms to org.eclipse.jetty.util.log.Slf4jLog
2024-07-30 22:29:52,199 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:29:52,199 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:29:52,202 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:29:52,202 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:29:52,202 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:29:52,202 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:29:52,530 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:29:52,530 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:29:52,530 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:29:52,530 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:29:52,530 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:29:52,530 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:29:52,530 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:29:52,530 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:29:52,530 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:29:52,658 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:29:52,659 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:29:52,659 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:29:52,677 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:29:52,677 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:29:52,677 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:29:52,740 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2024-07-30 22:29:52,856 INFO metastore.HiveMetaStoreClient: HMS client filtering is enabled.
2024-07-30 22:29:52,861 INFO metastore.HiveMetaStoreClient: Trying to connect to metastore with URI thrift://localhost:9083
2024-07-30 22:29:52,876 INFO metastore.HiveMetaStoreClient: HMSC::open(): Creating MapR-SASL-based thrift connection
2024-07-30 22:29:52,887 INFO security.ThriftTransportHelper: User authentication with method: MAPR-SECURITY
2024-07-30 22:29:52,948 INFO metastore.HiveMetaStoreClient: Opened a connection to metastore, current connections: 1
2024-07-30 22:29:52,950 INFO metastore.HiveMetaStoreClient: Connected to metastore.
2024-07-30 22:29:52,956 INFO metastore.RetryingMetaStoreClient: RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.metastore.HiveMetaStoreClient ugi=root (auth:CUSTOM) retries=1 delay=1 lifetime=0
2024-07-30 22:29:53,177 INFO iceberg.BaseMetastoreTableOperations: Refreshing table metadata from new version: hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00001-cc6b72f1-07d3-4d28-a7a1-1a9dd4c08a8a.metadata.json
2024-07-30 22:29:53,612 INFO iceberg.BaseMetastoreCatalog: Table loaded by catalog: hive.default.cldb
2024-07-30 22:29:53,638 INFO iceberg.BaseMetastoreTableOperations: Refreshing table metadata from new version: hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00001-cc6b72f1-07d3-4d28-a7a1-1a9dd4c08a8a.metadata.json
2024-07-30 22:29:53,642 INFO iceberg.BaseMetastoreCatalog: Table loaded by catalog: hive.default.cldb
2024-07-30 22:29:53,667 INFO iceberg.BaseTableScan: Scanning table hive.default.cldb snapshot 7326414452571699455 created at 2024-07-31 05:25:33.555 with filter true
2024-07-30 22:29:54,514 INFO compress.CodecPool: Got brand-new decompressor [.gz]
5. Record(9, 2024-07-30T07:04:20.848Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)
6. Record(10, 2024-07-30T07:04:20.850Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)
7. Record(11, 2024-07-30T07:05:20.848Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)
8. Record(12, 2024-07-30T07:05:20.849Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)
9. Record(13, 2024-07-30T07:06:20.848Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)
10. Record(14, 2024-07-30T07:06:20.850Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)
11. Record(15, 2024-07-30T07:07:20.848Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)
12. Record(16, 2024-07-30T07:07:20.850Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)
13. Record(17, 2024-07-30T07:08:20.849Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)
14. Record(18, 2024-07-30T07:08:20.850Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)
15. Record(19, 2024-07-30T07:09:20.848Z, configRead, cluster, mapr, 127.0.0.1, 5000, null, null, null, null, null, null, null, null, null, null, 0)

Brief list of snapshots for cldb Iceberg table

/opt/mapr/server/tools/update_insights.sh -t cldb -l -b
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.mapr.fs.ShimLoader (file:/opt/mapr/lib/maprfs-7.8.0.0-mapr.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.mapr.fs.ShimLoader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2024-07-30 03:40:38,843 INFO util.log: Logging initialized @3475ms to org.eclipse.jetty.util.log.Slf4jLog
2024-07-30 03:40:39,649 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:40:39,650 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:40:39,653 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:40:39,653 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:40:39,653 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:40:39,653 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:40:39,984 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:40:39,984 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:40:39,984 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:40:39,984 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:40:39,984 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:40:39,984 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:40:39,984 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:40:39,984 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:40:39,984 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:40:40,113 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:40:40,113 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:40:40,113 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:40:40,133 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:40:40,133 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:40:40,133 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:40:40,198 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2024-07-30 03:40:40,311 INFO metastore.HiveMetaStoreClient: HMS client filtering is enabled.
2024-07-30 03:40:40,316 INFO metastore.HiveMetaStoreClient: Trying to connect to metastore with URI thrift://localhost:9083
2024-07-30 03:40:40,330 INFO metastore.HiveMetaStoreClient: HMSC::open(): Creating MapR-SASL-based thrift connection
2024-07-30 03:40:40,342 INFO security.ThriftTransportHelper: User authentication with method: MAPR-SECURITY
2024-07-30 03:40:40,401 INFO metastore.HiveMetaStoreClient: Opened a connection to metastore, current connections: 1
2024-07-30 03:40:40,402 INFO metastore.HiveMetaStoreClient: Connected to metastore.
2024-07-30 03:40:40,409 INFO metastore.RetryingMetaStoreClient: RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.metastore.HiveMetaStoreClient ugi=root (auth:CUSTOM) retries=1 delay=1 lifetime=0
2024-07-30 03:40:40,649 INFO iceberg.BaseMetastoreTableOperations: Refreshing table metadata from new version: hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00002-7304f0cc-d517-4c8d-929b-c28f4afbd392.metadata.json
2024-07-30 03:40:41,097 INFO iceberg.BaseMetastoreCatalog: Table loaded by catalog: hive.default.cldb
2024-07-30 03:40:41,129 INFO iceberg.BaseMetastoreTableOperations: Refreshing table metadata from new version: hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00002-7304f0cc-d517-4c8d-929b-c28f4afbd392.metadata.json
2024-07-30 03:40:41,133 INFO iceberg.BaseMetastoreCatalog: Table loaded by catalog: hive.default.cldb
s-id: 7721073411112484179, timeStr: 2024-07-30 03:37:55.000 GMT-0700, time: 1722335875242, par-id: null
s-id: 4886192842798080540, timeStr: 2024-07-30 03:38:24.000 GMT-0700, time: 1722335904308, par-id: 7721073411112484179

curr snap id: 4886192842798080540

Use custom name for Iceberg table instead of using type name for Iceberg table 

/opt/mapr/server/tools/update_insights.sh -t cldb -a add -f /opt/mapr/logs/cldbaudit.log.json -n cldbtest
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.mapr.fs.ShimLoader (file:/opt/mapr/lib/maprfs-7.8.0.0-mapr.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.mapr.fs.ShimLoader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2024-07-30 03:45:07,264 INFO util.log: Logging initialized @3487ms to org.eclipse.jetty.util.log.Slf4jLog
2024-07-30 03:45:08,083 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:45:08,083 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:45:08,086 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:45:08,086 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:45:08,086 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:45:08,086 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:45:08,420 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:45:08,420 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:45:08,420 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:45:08,421 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:45:08,421 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:45:08,421 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:45:08,421 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:45:08,421 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:45:08,421 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:45:08,549 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:45:08,549 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:45:08,549 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:45:08,568 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:45:08,568 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:45:08,569 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:45:08,632 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2024-07-30 03:45:08,746 INFO metastore.HiveMetaStoreClient: HMS client filtering is enabled.
2024-07-30 03:45:08,752 INFO metastore.HiveMetaStoreClient: Trying to connect to metastore with URI thrift://localhost:9083
2024-07-30 03:45:08,766 INFO metastore.HiveMetaStoreClient: HMSC::open(): Creating MapR-SASL-based thrift connection
2024-07-30 03:45:08,777 INFO security.ThriftTransportHelper: User authentication with method: MAPR-SECURITY
2024-07-30 03:45:08,834 INFO metastore.HiveMetaStoreClient: Opened a connection to metastore, current connections: 1
2024-07-30 03:45:08,835 INFO metastore.HiveMetaStoreClient: Connected to metastore.
2024-07-30 03:45:08,842 INFO metastore.RetryingMetaStoreClient: RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.metastore.HiveMetaStoreClient ugi=root (auth:CUSTOM) retries=1 delay=1 lifetime=0
2024-07-30 03:45:09,047 INFO iceberg.BaseMetastoreCatalog: Table properties set at catalog level through catalog properties: {}
2024-07-30 03:45:09,052 INFO iceberg.BaseMetastoreCatalog: Table properties enforced at catalog level through catalog properties: {}
2024-07-30 03:45:09,623 INFO hive.HiveTableOperations: Committed to table hive.default.cldbtest with the new metadata location hdfs://CLUSTER1/var/mapr/insights/cldbtest/metadata/00000-77a49924-b658-43f6-9a9a-4d84de9aa4ec.metadata.json
2024-07-30 03:45:09,624 INFO iceberg.BaseMetastoreTableOperations: Successfully committed to table hive.default.cldbtest in 542 ms
2024-07-30 03:45:10,011 INFO iceberg.BaseMetastoreTableOperations: Refreshing table metadata from new version: hdfs://CLUSTER1/var/mapr/insights/cldbtest/metadata/00000-77a49924-b658-43f6-9a9a-4d84de9aa4ec.metadata.json
2024-07-30 03:45:10,187 INFO compress.CodecPool: Got brand-new compressor [.gz]
2024-07-30 03:45:11,275 INFO hive.HiveTableOperations: Committed to table hive.default.cldbtest with the new metadata location hdfs://CLUSTER1/var/mapr/insights/cldbtest/metadata/00001-d0ee8a9d-fbec-467b-a1e3-8898a92c6fae.metadata.json
2024-07-30 03:45:11,275 INFO iceberg.BaseMetastoreTableOperations: Successfully committed to table hive.default.cldbtest in 145 ms
2024-07-30 03:45:11,276 INFO iceberg.SnapshotProducer: Committed snapshot 126577181692791339 (MergeAppend)
2024-07-30 03:45:11,288 INFO iceberg.BaseMetastoreTableOperations: Refreshing table metadata from new version: hdfs://CLUSTER1/var/mapr/insights/cldbtest/metadata/00001-d0ee8a9d-fbec-467b-a1e3-8898a92c6fae.metadata.json

List snapshot for cldb Iceberg table in verbose mode

update_insights.sh -t cldb -l -v
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.mapr.fs.ShimLoader (file:/opt/mapr/lib/maprfs-7.8.0.0-mapr.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.mapr.fs.ShimLoader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2024-07-30 03:41:53,108 INFO util.log: Logging initialized @3492ms to org.eclipse.jetty.util.log.Slf4jLog
2024-07-30 03:41:53,897 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:41:53,897 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:41:53,900 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:41:53,900 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:41:53,900 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:41:53,900 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:41:54,235 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:41:54,235 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:41:54,235 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:41:54,235 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:41:54,235 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:41:54,235 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:41:54,235 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:41:54,235 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:41:54,235 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:41:54,364 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:41:54,364 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:41:54,365 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:41:54,384 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:41:54,384 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:41:54,384 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:41:54,450 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2024-07-30 03:41:54,568 INFO metastore.HiveMetaStoreClient: HMS client filtering is enabled.
2024-07-30 03:41:54,574 INFO metastore.HiveMetaStoreClient: Trying to connect to metastore with URI thrift://localhost:9083
2024-07-30 03:41:54,589 INFO metastore.HiveMetaStoreClient: HMSC::open(): Creating MapR-SASL-based thrift connection
2024-07-30 03:41:54,601 INFO security.ThriftTransportHelper: User authentication with method: MAPR-SECURITY
2024-07-30 03:41:54,663 INFO metastore.HiveMetaStoreClient: Opened a connection to metastore, current connections: 1
2024-07-30 03:41:54,665 INFO metastore.HiveMetaStoreClient: Connected to metastore.
2024-07-30 03:41:54,671 INFO metastore.RetryingMetaStoreClient: RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.metastore.HiveMetaStoreClient ugi=root (auth:CUSTOM) retries=1 delay=1 lifetime=0
2024-07-30 03:41:54,902 INFO iceberg.BaseMetastoreTableOperations: Refreshing table metadata from new version: hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00002-7304f0cc-d517-4c8d-929b-c28f4afbd392.metadata.json
2024-07-30 03:41:55,364 INFO iceberg.BaseMetastoreCatalog: Table loaded by catalog: hive.default.cldb
2024-07-30 03:41:55,394 INFO iceberg.BaseMetastoreTableOperations: Refreshing table metadata from new version: hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00002-7304f0cc-d517-4c8d-929b-c28f4afbd392.metadata.json
2024-07-30 03:41:55,399 INFO iceberg.BaseMetastoreCatalog: Table loaded by catalog: hive.default.cldb
s-id: 7721073411112484179, timeStr: 2024-07-30 03:37:55.000 GMT-0700, time: 1722335875242, par-id: null
num manifests: 1
   path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/efbc75cc-3a41-4559-b539-f0b38f7b48bd-m0.avro, DATA, (*)
   added files : 1, added rows: 660, len: 6943
added files:
   path: hdfs://CLUSTER1/var/mapr/insights/cldb/013426bf-1d32-4cf2-806a-52969cd7ec3f
         id lower: 1, id upper: 660
   added file count: 1
removed data files:
   removed data file count: 0
added delete files:
   added delete file count: 0
all files:
   path: hdfs://CLUSTER1/var/mapr/insights/cldb/013426bf-1d32-4cf2-806a-52969cd7ec3f
   all file count: 1

s-id: 4886192842798080540, timeStr: 2024-07-30 03:38:24.000 GMT-0700, time: 1722335904308, par-id: 7721073411112484179
num manifests: 2
   path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/d414d29b-b912-42e5-9848-684d7280a56c-m0.avro, DATA, (*)
   added files : 1, added rows: 660, len: 6946
   path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/efbc75cc-3a41-4559-b539-f0b38f7b48bd-m0.avro, DATA
   added files : 1, added rows: 660, len: 6943
added files:
   path: hdfs://CLUSTER1/var/mapr/insights/cldb/ff7d2233-b795-4f12-8c89-abc78b0f9c76
         id lower: 1, id upper: 660
   added file count: 1
removed data files:
   removed data file count: 0
added delete files:
   added delete file count: 0
all files:
   path: hdfs://CLUSTER1/var/mapr/insights/cldb/ff7d2233-b795-4f12-8c89-abc78b0f9c76
   path: hdfs://CLUSTER1/var/mapr/insights/cldb/013426bf-1d32-4cf2-806a-52969cd7ec3f
   all file count: 2


curr snap id: 4886192842798080540
table properties:

Expire snapshot of cldb Iceberg table using Snapshot ID and delete the files added as part of the snapshot

/opt/mapr/server/tools/update_insights.sh -t cldb -es 4886192842798080540 -da
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.mapr.fs.ShimLoader (file:/opt/mapr/lib/maprfs-7.8.0.0-mapr.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.mapr.fs.ShimLoader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2024-07-30 03:51:54,113 INFO util.log: Logging initialized @3467ms to org.eclipse.jetty.util.log.Slf4jLog
2024-07-30 03:51:54,907 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:51:54,907 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:51:54,910 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:51:54,911 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:51:54,911 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:51:54,911 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:51:55,242 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:51:55,242 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:51:55,242 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:51:55,242 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:51:55,242 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:51:55,243 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:51:55,243 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:51:55,243 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:51:55,243 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:51:55,368 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:51:55,368 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:51:55,368 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:51:55,388 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:51:55,388 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:51:55,388 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:51:55,454 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2024-07-30 03:51:55,566 INFO metastore.HiveMetaStoreClient: HMS client filtering is enabled.
2024-07-30 03:51:55,571 INFO metastore.HiveMetaStoreClient: Trying to connect to metastore with URI thrift://localhost:9083
2024-07-30 03:51:55,586 INFO metastore.HiveMetaStoreClient: HMSC::open(): Creating MapR-SASL-based thrift connection
2024-07-30 03:51:55,597 INFO security.ThriftTransportHelper: User authentication with method: MAPR-SECURITY
2024-07-30 03:51:55,656 INFO metastore.HiveMetaStoreClient: Opened a connection to metastore, current connections: 1
2024-07-30 03:51:55,657 INFO metastore.HiveMetaStoreClient: Connected to metastore.
2024-07-30 03:51:55,664 INFO metastore.RetryingMetaStoreClient: RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.metastore.HiveMetaStoreClient ugi=root (auth:CUSTOM) retries=1 delay=1 lifetime=0
2024-07-30 03:51:55,887 INFO iceberg.BaseMetastoreTableOperations: Refreshing table metadata from new version: hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00003-1fe48d4e-7be1-4b09-8133-9c8d79419a37.metadata.json
2024-07-30 03:51:56,341 INFO iceberg.BaseMetastoreCatalog: Table loaded by catalog: hive.default.cldb
2024-07-30 03:51:56,370 INFO iceberg.BaseMetastoreTableOperations: Refreshing table metadata from new version: hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00003-1fe48d4e-7be1-4b09-8133-9c8d79419a37.metadata.json
2024-07-30 03:51:56,374 INFO iceberg.BaseMetastoreCatalog: Table loaded by catalog: hive.default.cldb
Got expire-snap, snapId: 4886192842798080540
Snapshots before expiring snapshot 4886192842798080540:
s-id: 4886192842798080540, timeStr: 2024-07-30 03:38:24.000 GMT-0700, time: 1722335904308, par-id: 7721073411112484179
num manifests: 2
   path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/d414d29b-b912-42e5-9848-684d7280a56c-m0.avro, DATA, (*)
   path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/efbc75cc-3a41-4559-b539-f0b38f7b48bd-m0.avro, DATA
added files:
   path: hdfs://CLUSTER1/var/mapr/insights/cldb/ff7d2233-b795-4f12-8c89-abc78b0f9c76
   added file count: 1
removed data files:
   removed data file count: 0
added delete files:
   added delete file count: 0
all files:
   path: hdfs://CLUSTER1/var/mapr/insights/cldb/ff7d2233-b795-4f12-8c89-abc78b0f9c76
   path: hdfs://CLUSTER1/var/mapr/insights/cldb/013426bf-1d32-4cf2-806a-52969cd7ec3f
   all file count: 2

s-id: 3544046758310960189, timeStr: 2024-07-30 03:48:54.000 GMT-0700, time: 1722336534517, par-id: 4886192842798080540
num manifests: 2
   path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/d414d29b-b912-42e5-9848-684d7280a56c-m0.avro, DATA
   path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/efbc75cc-3a41-4559-b539-f0b38f7b48bd-m0.avro, DATA
added files:
   added file count: 0
removed data files:
   removed data file count: 0
added delete files:
   added delete file count: 0
all files:
   path: hdfs://CLUSTER1/var/mapr/insights/cldb/ff7d2233-b795-4f12-8c89-abc78b0f9c76
   path: hdfs://CLUSTER1/var/mapr/insights/cldb/013426bf-1d32-4cf2-806a-52969cd7ec3f
   all file count: 2


curr snap id: 3544046758310960189
Expiring snapshot: 4886192842798080540
2024-07-30 03:51:56,779 INFO iceberg.RemoveSnapshots: Expiring snapshot with id: 4886192842798080540
2024-07-30 03:51:56,788 INFO iceberg.RemoveSnapshots: Committed snapshot changes
2024-07-30 03:51:56,791 INFO iceberg.RemoveSnapshots: Expired snapshot: BaseSnapshot{id=4886192842798080540, timestamp_ms=1722335904308, operation=append, summary={added-data-files=1, added-records=660, added-files-size=9163, changed-partition-count=1, total-records=1320, total-files-size=18326, total-data-files=2, total-delete-files=0, total-position-deletes=0, total-equality-deletes=0}, manifest-list=hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/snap-4886192842798080540-1-d414d29b-b912-42e5-9848-684d7280a56c.avro, schema-id=0}
2024-07-30 03:51:56,791 INFO iceberg.RemoveSnapshots: Committed snapshot changes; cleaning up expired manifests and data files.
2024-07-30 03:51:56,801 WARN iceberg.RemoveSnapshots: Manifests to delete:
2024-07-30 03:51:56,801 WARN iceberg.RemoveSnapshots: Manifests Lists to delete: hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/snap-4886192842798080540-1-d414d29b-b912-42e5-9848-684d7280a56c.avro
   deleting data file: hdfs://CLUSTER1/var/mapr/insights/cldb/ff7d2233-b795-4f12-8c89-abc78b0f9c76
About to delete nsnaps: 1, delAddedFiles: true, delFiles: false, nFiles: 1
2024-07-30 03:51:57,015 INFO iceberg.SnapshotProducer: Committed snapshot 2831938722446196530 (StreamingDelete)
2024-07-30 03:51:57,214 INFO hive.HiveTableOperations: Committed to table hive.default.cldb with the new metadata location hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00004-569c881d-ca7d-4320-afba-21bd83ad4e7c.metadata.json
2024-07-30 03:51:57,214 INFO iceberg.BaseMetastoreTableOperations: Successfully committed to table hive.default.cldb in 179 ms
2024-07-30 03:51:57,227 INFO iceberg.BaseMetastoreTableOperations: Refreshing table metadata from new version: hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00004-569c881d-ca7d-4320-afba-21bd83ad4e7c.metadata.json
Snapshots after expiring snapshot 4886192842798080540:
s-id: 3544046758310960189, timeStr: 2024-07-30 03:48:54.000 GMT-0700, time: 1722336534517, par-id: 4886192842798080540
num manifests: 2
   path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/d414d29b-b912-42e5-9848-684d7280a56c-m0.avro, DATA
   path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/efbc75cc-3a41-4559-b539-f0b38f7b48bd-m0.avro, DATA
added files:
   added file count: 0
removed data files:
   removed data file count: 0
added delete files:
   added delete file count: 0
all files:
   path: hdfs://CLUSTER1/var/mapr/insights/cldb/ff7d2233-b795-4f12-8c89-abc78b0f9c76
   path: hdfs://CLUSTER1/var/mapr/insights/cldb/013426bf-1d32-4cf2-806a-52969cd7ec3f
   all file count: 2

s-id: 2831938722446196530, timeStr: 2024-07-30 03:51:57.000 GMT-0700, time: 1722336717012, par-id: 3544046758310960189
num manifests: 2
   path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/ffac5034-b430-468b-a2b7-b11e4f2b37c4-m0.avro, DATA, (*)
   path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/efbc75cc-3a41-4559-b539-f0b38f7b48bd-m0.avro, DATA
added files:
   added file count: 0
removed data files:
   path: hdfs://CLUSTER1/var/mapr/insights/cldb/ff7d2233-b795-4f12-8c89-abc78b0f9c76
   removed data file count: 1
added delete files:
   added delete file count: 0
all files:
   path: hdfs://CLUSTER1/var/mapr/insights/cldb/013426bf-1d32-4cf2-806a-52969cd7ec3f
   all file count: 1


curr snap id: 2831938722446196530

Expire only the snapshot for the specified Snapshot ID of cldb Iceberg table

/opt/mapr/server/tools/update_insights.sh -t cldb -es 7721073411112484179
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.mapr.fs.ShimLoader (file:/opt/mapr/lib/maprfs-7.8.0.0-mapr.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.mapr.fs.ShimLoader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2024-07-30 03:48:51,677 INFO util.log: Logging initialized @3525ms to org.eclipse.jetty.util.log.Slf4jLog
2024-07-30 03:48:52,502 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:48:52,502 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:48:52,505 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:48:52,505 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:48:52,505 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:48:52,505 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:48:52,836 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:48:52,836 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:48:52,836 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:48:52,836 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:48:52,836 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:48:52,836 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:48:52,836 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:48:52,837 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:48:52,837 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:48:52,965 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:48:52,965 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:48:52,965 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:48:52,985 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:48:52,985 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:48:52,985 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:48:53,046 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2024-07-30 03:48:53,160 INFO metastore.HiveMetaStoreClient: HMS client filtering is enabled.
2024-07-30 03:48:53,166 INFO metastore.HiveMetaStoreClient: Trying to connect to metastore with URI thrift://localhost:9083
2024-07-30 03:48:53,180 INFO metastore.HiveMetaStoreClient: HMSC::open(): Creating MapR-SASL-based thrift connection
2024-07-30 03:48:53,191 INFO security.ThriftTransportHelper: User authentication with method: MAPR-SECURITY
2024-07-30 03:48:53,249 INFO metastore.HiveMetaStoreClient: Opened a connection to metastore, current connections: 1
2024-07-30 03:48:53,250 INFO metastore.HiveMetaStoreClient: Connected to metastore.
2024-07-30 03:48:53,257 INFO metastore.RetryingMetaStoreClient: RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.metastore.HiveMetaStoreClient ugi=root (auth:CUSTOM) retries=1 delay=1 lifetime=0
2024-07-30 03:48:53,475 INFO iceberg.BaseMetastoreTableOperations: Refreshing table metadata from new version: hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00002-7304f0cc-d517-4c8d-929b-c28f4afbd392.metadata.json
2024-07-30 03:48:53,916 INFO iceberg.BaseMetastoreCatalog: Table loaded by catalog: hive.default.cldb
2024-07-30 03:48:53,945 INFO iceberg.BaseMetastoreTableOperations: Refreshing table metadata from new version: hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00002-7304f0cc-d517-4c8d-929b-c28f4afbd392.metadata.json
2024-07-30 03:48:53,950 INFO iceberg.BaseMetastoreCatalog: Table loaded by catalog: hive.default.cldb
Got expire-snap, snapId: 7721073411112484179
Snapshots before expiring snapshot 7721073411112484179:
s-id: 7721073411112484179, timeStr: 2024-07-30 03:37:55.000 GMT-0700, time: 1722335875242, par-id: null
num manifests: 1
   path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/efbc75cc-3a41-4559-b539-f0b38f7b48bd-m0.avro, DATA, (*)
added files:
   path: hdfs://CLUSTER1/var/mapr/insights/cldb/013426bf-1d32-4cf2-806a-52969cd7ec3f
   added file count: 1
removed data files:
   removed data file count: 0
added delete files:
   added delete file count: 0
all files:
   path: hdfs://CLUSTER1/var/mapr/insights/cldb/013426bf-1d32-4cf2-806a-52969cd7ec3f
   all file count: 1

s-id: 4886192842798080540, timeStr: 2024-07-30 03:38:24.000 GMT-0700, time: 1722335904308, par-id: 7721073411112484179
num manifests: 2
   path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/d414d29b-b912-42e5-9848-684d7280a56c-m0.avro, DATA, (*)
   path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/efbc75cc-3a41-4559-b539-f0b38f7b48bd-m0.avro, DATA
added files:
   path: hdfs://CLUSTER1/var/mapr/insights/cldb/ff7d2233-b795-4f12-8c89-abc78b0f9c76
   added file count: 1
removed data files:
   removed data file count: 0
added delete files:
   added delete file count: 0
all files:
   path: hdfs://CLUSTER1/var/mapr/insights/cldb/ff7d2233-b795-4f12-8c89-abc78b0f9c76
   path: hdfs://CLUSTER1/var/mapr/insights/cldb/013426bf-1d32-4cf2-806a-52969cd7ec3f
   all file count: 2


curr snap id: 4886192842798080540
Expiring snapshot: 7721073411112484179
2024-07-30 03:48:54,348 INFO iceberg.RemoveSnapshots: Expiring snapshot with id: 7721073411112484179
2024-07-30 03:48:54,356 INFO iceberg.RemoveSnapshots: Committed snapshot changes
2024-07-30 03:48:54,359 INFO iceberg.RemoveSnapshots: Expired snapshot: BaseSnapshot{id=7721073411112484179, timestamp_ms=1722335875242, operation=append, summary={added-data-files=1, added-records=660, added-files-size=9163, changed-partition-count=1, total-records=660, total-files-size=9163, total-data-files=1, total-delete-files=0, total-position-deletes=0, total-equality-deletes=0}, manifest-list=hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/snap-7721073411112484179-1-efbc75cc-3a41-4559-b539-f0b38f7b48bd.avro, schema-id=0}
2024-07-30 03:48:54,360 INFO iceberg.RemoveSnapshots: Committed snapshot changes; cleaning up expired manifests and data files.
2024-07-30 03:48:54,369 WARN iceberg.RemoveSnapshots: Manifests to delete:
2024-07-30 03:48:54,369 WARN iceberg.RemoveSnapshots: Manifests Lists to delete: hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/snap-7721073411112484179-1-efbc75cc-3a41-4559-b539-f0b38f7b48bd.avro
About to delete nsnaps: 0, delAddedFiles: false, delFiles: false, nFiles: 0
2024-07-30 03:48:54,519 INFO iceberg.SnapshotProducer: Committed snapshot 3544046758310960189 (StreamingDelete)
2024-07-30 03:48:54,702 INFO hive.HiveTableOperations: Committed to table hive.default.cldb with the new metadata location hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00003-1fe48d4e-7be1-4b09-8133-9c8d79419a37.metadata.json
2024-07-30 03:48:54,702 INFO iceberg.BaseMetastoreTableOperations: Successfully committed to table hive.default.cldb in 163 ms
2024-07-30 03:48:54,714 INFO iceberg.BaseMetastoreTableOperations: Refreshing table metadata from new version: hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00003-1fe48d4e-7be1-4b09-8133-9c8d79419a37.metadata.json
Snapshots after expiring snapshot 7721073411112484179:
s-id: 4886192842798080540, timeStr: 2024-07-30 03:38:24.000 GMT-0700, time: 1722335904308, par-id: 7721073411112484179
num manifests: 2
   path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/d414d29b-b912-42e5-9848-684d7280a56c-m0.avro, DATA, (*)
   path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/efbc75cc-3a41-4559-b539-f0b38f7b48bd-m0.avro, DATA
added files:
   path: hdfs://CLUSTER1/var/mapr/insights/cldb/ff7d2233-b795-4f12-8c89-abc78b0f9c76
   added file count: 1
removed data files:
   removed data file count: 0
added delete files:
   added delete file count: 0
all files:
   path: hdfs://CLUSTER1/var/mapr/insights/cldb/ff7d2233-b795-4f12-8c89-abc78b0f9c76
   path: hdfs://CLUSTER1/var/mapr/insights/cldb/013426bf-1d32-4cf2-806a-52969cd7ec3f
   all file count: 2

s-id: 3544046758310960189, timeStr: 2024-07-30 03:48:54.000 GMT-0700, time: 1722336534517, par-id: 4886192842798080540
num manifests: 2
   path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/d414d29b-b912-42e5-9848-684d7280a56c-m0.avro, DATA
   path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/efbc75cc-3a41-4559-b539-f0b38f7b48bd-m0.avro, DATA
added files:
   added file count: 0
removed data files:
   removed data file count: 0
added delete files:
   added delete file count: 0
all files:
   path: hdfs://CLUSTER1/var/mapr/insights/cldb/ff7d2233-b795-4f12-8c89-abc78b0f9c76
   path: hdfs://CLUSTER1/var/mapr/insights/cldb/013426bf-1d32-4cf2-806a-52969cd7ec3f
   all file count: 2


curr snap id: 3544046758310960189

Expire snapshots older than given ms timestamp (Epoch time) and delete the data files for cldb Iceberg table

# /opt/mapr/server/tools/update_insights.sh -t cldb -et 1722336908008 -d
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.mapr.fs.ShimLoader (file:/opt/mapr/lib/maprfs-7.8.0.0-mapr.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.mapr.fs.ShimLoader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2024-07-30 03:56:15,617 INFO util.log: Logging initialized @3496ms to org.eclipse.jetty.util.log.Slf4jLog
2024-07-30 03:56:16,412 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:56:16,412 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:56:16,415 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:56:16,416 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:56:16,416 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:56:16,416 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:56:16,748 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:56:16,748 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:56:16,748 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:56:16,748 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:56:16,748 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:56:16,748 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:56:16,748 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:56:16,748 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:56:16,748 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:56:16,874 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:56:16,874 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:56:16,874 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:56:16,895 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 03:56:16,895 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 03:56:16,895 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 03:56:16,964 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2024-07-30 03:56:17,085 INFO metastore.HiveMetaStoreClient: HMS client filtering is enabled.
2024-07-30 03:56:17,091 INFO metastore.HiveMetaStoreClient: Trying to connect to metastore with URI thrift://localhost:9083
2024-07-30 03:56:17,107 INFO metastore.HiveMetaStoreClient: HMSC::open(): Creating MapR-SASL-based thrift connection
2024-07-30 03:56:17,119 INFO security.ThriftTransportHelper: User authentication with method: MAPR-SECURITY
2024-07-30 03:56:17,188 INFO metastore.HiveMetaStoreClient: Opened a connection to metastore, current connections: 1
2024-07-30 03:56:17,190 INFO metastore.HiveMetaStoreClient: Connected to metastore.
2024-07-30 03:56:17,197 INFO metastore.RetryingMetaStoreClient: RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.metastore.HiveMetaStoreClient ugi=root (auth:CUSTOM) retries=1 delay=1 lifetime=0
2024-07-30 03:56:17,426 INFO iceberg.BaseMetastoreTableOperations: Refreshing table metadata from new version: hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00005-586c0de1-4cdb-45c8-813c-834b3202d115.metadata.json
2024-07-30 03:56:17,889 INFO iceberg.BaseMetastoreCatalog: Table loaded by catalog: hive.default.cldb
2024-07-30 03:56:17,917 INFO iceberg.BaseMetastoreTableOperations: Refreshing table metadata from new version: hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00005-586c0de1-4cdb-45c8-813c-834b3202d115.metadata.json
2024-07-30 03:56:17,921 INFO iceberg.BaseMetastoreCatalog: Table loaded by catalog: hive.default.cldb
Got expire-ts, expire-ms: 1722336908008, long: 1722336908008
Snapshots before expiry, time: 2024-07-30 03:55:08.000 GMT-0700, timems: 1722336908008
  CurrTime: 2024-07-30 03:56:17.000 GMT-0700, timems: 1722336977925
s-id: 3544046758310960189, timeStr: 2024-07-30 03:48:54.000 GMT-0700, time: 1722336534517, par-id: 4886192842798080540
num manifests: 2
   path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/d414d29b-b912-42e5-9848-684d7280a56c-m0.avro, DATA
   path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/efbc75cc-3a41-4559-b539-f0b38f7b48bd-m0.avro, DATA
added files:
   added file count: 0
removed data files:
   removed data file count: 0
added delete files:
   added delete file count: 0
all files:
   path: hdfs://CLUSTER1/var/mapr/insights/cldb/ff7d2233-b795-4f12-8c89-abc78b0f9c76
   path: hdfs://CLUSTER1/var/mapr/insights/cldb/013426bf-1d32-4cf2-806a-52969cd7ec3f
   all file count: 2

s-id: 2831938722446196530, timeStr: 2024-07-30 03:51:57.000 GMT-0700, time: 1722336717012, par-id: 3544046758310960189
num manifests: 2
   path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/ffac5034-b430-468b-a2b7-b11e4f2b37c4-m0.avro, DATA, (*)
   path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/efbc75cc-3a41-4559-b539-f0b38f7b48bd-m0.avro, DATA
added files:
   added file count: 0
removed data files:
   path: hdfs://CLUSTER1/var/mapr/insights/cldb/ff7d2233-b795-4f12-8c89-abc78b0f9c76
   removed data file count: 1
added delete files:
   added delete file count: 0
all files:
   path: hdfs://CLUSTER1/var/mapr/insights/cldb/013426bf-1d32-4cf2-806a-52969cd7ec3f
   all file count: 1

s-id: 2297561737173859503, timeStr: 2024-07-30 03:55:08.000 GMT-0700, time: 1722336908008, par-id: 2831938722446196530
num manifests: 2
   path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/30ea15da-e532-4eae-b3f7-a2fab2b30623-m0.avro, DATA, (*)
   path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/efbc75cc-3a41-4559-b539-f0b38f7b48bd-m0.avro, DATA
added files:
   path: hdfs://CLUSTER1/var/mapr/insights/cldb/179b3702-db89-49ac-9d42-dd3dc68af0cc
   added file count: 1
removed data files:
   removed data file count: 0
added delete files:
   added delete file count: 0
all files:
   path: hdfs://CLUSTER1/var/mapr/insights/cldb/179b3702-db89-49ac-9d42-dd3dc68af0cc
   path: hdfs://CLUSTER1/var/mapr/insights/cldb/013426bf-1d32-4cf2-806a-52969cd7ec3f
   all file count: 2


curr snap id: 2297561737173859503
Expiring snapshots before time: 2024-07-30 03:55:08.000 GMT-0700
2024-07-30 03:56:18,373 INFO iceberg.RemoveSnapshots: Expiring snapshots older than: 2024-07-30 10:55:08.008 (1722336908008)
2024-07-30 03:56:18,383 INFO iceberg.RemoveSnapshots: Committed snapshot changes
2024-07-30 03:56:18,386 INFO iceberg.RemoveSnapshots: Expired snapshot: BaseSnapshot{id=3544046758310960189, timestamp_ms=1722336534517, operation=delete, summary={changed-partition-count=0, total-records=1320, total-files-size=18326, total-data-files=2, total-delete-files=0, total-position-deletes=0, total-equality-deletes=0}, manifest-list=hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/snap-3544046758310960189-1-941c0871-11ff-4c85-8c5f-751e4b1e3a1d.avro, schema-id=0}
2024-07-30 03:56:18,386 INFO iceberg.RemoveSnapshots: Expired snapshot: BaseSnapshot{id=2831938722446196530, timestamp_ms=1722336717012, operation=delete, summary={deleted-data-files=1, deleted-records=660, removed-files-size=9163, changed-partition-count=1, total-records=660, total-files-size=9163, total-data-files=1, total-delete-files=0, total-position-deletes=0, total-equality-deletes=0}, manifest-list=hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/snap-2831938722446196530-1-ffac5034-b430-468b-a2b7-b11e4f2b37c4.avro, schema-id=0}
2024-07-30 03:56:18,386 INFO iceberg.RemoveSnapshots: Committed snapshot changes; cleaning up expired manifests and data files.
2024-07-30 03:56:18,443 WARN iceberg.RemoveSnapshots: Manifests to delete: hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/ffac5034-b430-468b-a2b7-b11e4f2b37c4-m0.avro, hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/d414d29b-b912-42e5-9848-684d7280a56c-m0.avro
2024-07-30 03:56:18,443 WARN iceberg.RemoveSnapshots: Manifests Lists to delete: hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/snap-2831938722446196530-1-ffac5034-b430-468b-a2b7-b11e4f2b37c4.avro, hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/snap-3544046758310960189-1-941c0871-11ff-4c85-8c5f-751e4b1e3a1d.avro
   deleting data file: hdfs://CLUSTER1/var/mapr/insights/cldb/ff7d2233-b795-4f12-8c89-abc78b0f9c76
   deleting data file: hdfs://CLUSTER1/var/mapr/insights/cldb/013426bf-1d32-4cf2-806a-52969cd7ec3f
   deleting data file: hdfs://CLUSTER1/var/mapr/insights/cldb/013426bf-1d32-4cf2-806a-52969cd7ec3f
About to delete nsnaps: 2, delAddedFiles: false, delFiles: true, nFiles: 0
2024-07-30 03:56:18,663 INFO iceberg.SnapshotProducer: Committed snapshot 3527657542738416278 (StreamingDelete)
2024-07-30 03:56:18,828 INFO hive.HiveTableOperations: Committed to table hive.default.cldb with the new metadata location hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00006-89188307-09c7-4f16-b8b1-eabef5d38356.metadata.json
2024-07-30 03:56:18,828 INFO iceberg.BaseMetastoreTableOperations: Successfully committed to table hive.default.cldb in 149 ms
2024-07-30 03:56:18,838 INFO iceberg.BaseMetastoreTableOperations: Refreshing table metadata from new version: hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00006-89188307-09c7-4f16-b8b1-eabef5d38356.metadata.json
Snapshots after expiring snaps:
s-id: 2297561737173859503, timeStr: 2024-07-30 03:55:08.000 GMT-0700, time: 1722336908008, par-id: 2831938722446196530
num manifests: 2
   path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/30ea15da-e532-4eae-b3f7-a2fab2b30623-m0.avro, DATA, (*)
   path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/efbc75cc-3a41-4559-b539-f0b38f7b48bd-m0.avro, DATA
added files:
   path: hdfs://CLUSTER1/var/mapr/insights/cldb/179b3702-db89-49ac-9d42-dd3dc68af0cc
   added file count: 1
removed data files:
   removed data file count: 0
added delete files:
   added delete file count: 0
all files:
   path: hdfs://CLUSTER1/var/mapr/insights/cldb/179b3702-db89-49ac-9d42-dd3dc68af0cc
   path: hdfs://CLUSTER1/var/mapr/insights/cldb/013426bf-1d32-4cf2-806a-52969cd7ec3f
   all file count: 2

s-id: 3527657542738416278, timeStr: 2024-07-30 03:56:18.000 GMT-0700, time: 1722336978660, par-id: 2297561737173859503
num manifests: 2
   path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/30ea15da-e532-4eae-b3f7-a2fab2b30623-m0.avro, DATA
   path : hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/6ef85c57-4556-43ea-9a0c-e59ce65d6eb0-m0.avro, DATA, (*)
added files:
   added file count: 0
removed data files:
   path: hdfs://CLUSTER1/var/mapr/insights/cldb/013426bf-1d32-4cf2-806a-52969cd7ec3f
   removed data file count: 1
added delete files:
   added delete file count: 0
all files:
   path: hdfs://CLUSTER1/var/mapr/insights/cldb/179b3702-db89-49ac-9d42-dd3dc68af0cc
   all file count: 1


curr snap id: 3527657542738416278

List orphan metadata files associated with cldb Iceberg table

# /opt/mapr/server/tools/update_insights.sh -t cldb -lo
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.mapr.fs.ShimLoader (file:/opt/mapr/lib/maprfs-7.8.0.0-mapr.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.mapr.fs.ShimLoader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2024-07-30 22:06:33,263 INFO util.log: Logging initialized @3467ms to org.eclipse.jetty.util.log.Slf4jLog
2024-07-30 22:06:34,078 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:06:34,079 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:06:34,082 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:06:34,082 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:06:34,082 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:06:34,082 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:06:34,406 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:06:34,406 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:06:34,406 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:06:34,407 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:06:34,407 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:06:34,407 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:06:34,407 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:06:34,407 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:06:34,407 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:06:34,535 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:06:34,535 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:06:34,535 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:06:34,555 WARN util.MapRSecurityUtil: Environment variable MAPR_HOME is null
2024-07-30 22:06:34,555 WARN util.MapRSecurityUtil: System property mapr.home.dir is null
2024-07-30 22:06:34,555 WARN util.MapRSecurityUtil: Setting MapR home as /opt/mapr by default
2024-07-30 22:06:34,620 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2024-07-30 22:06:34,728 INFO metastore.HiveMetaStoreClient: HMS client filtering is enabled.
2024-07-30 22:06:34,734 INFO metastore.HiveMetaStoreClient: Trying to connect to metastore with URI thrift://localhost:9083
2024-07-30 22:06:34,747 INFO metastore.HiveMetaStoreClient: HMSC::open(): Creating MapR-SASL-based thrift connection
2024-07-30 22:06:34,758 INFO security.ThriftTransportHelper: User authentication with method: MAPR-SECURITY
2024-07-30 22:06:34,814 INFO metastore.HiveMetaStoreClient: Opened a connection to metastore, current connections: 1
2024-07-30 22:06:34,815 INFO metastore.HiveMetaStoreClient: Connected to metastore.
2024-07-30 22:06:34,823 INFO metastore.RetryingMetaStoreClient: RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.metastore.HiveMetaStoreClient ugi=root (auth:CUSTOM) retries=1 delay=1 lifetime=0
2024-07-30 22:06:35,039 INFO iceberg.BaseMetastoreTableOperations: Refreshing table metadata from new version: hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00006-89188307-09c7-4f16-b8b1-eabef5d38356.metadata.json
2024-07-30 22:06:35,477 INFO iceberg.BaseMetastoreCatalog: Table loaded by catalog: hive.default.cldb
2024-07-30 22:06:35,504 INFO iceberg.BaseMetastoreTableOperations: Refreshing table metadata from new version: hdfs://CLUSTER1/var/mapr/insights/cldb/metadata/00006-89188307-09c7-4f16-b8b1-eabef5d38356.metadata.json
2024-07-30 22:06:35,508 INFO iceberg.BaseMetastoreCatalog: Table loaded by catalog: hive.default.cldb
Num Orphan files: 6
Orphan files:
/mapr/CLUSTER1/var/mapr/insights/cldb/metadata/00001-9d65df8c-e9cb-40cf-8a5f-d256999fac83.metadata.json
/mapr/CLUSTER1/var/mapr/insights/cldb/metadata/00003-1fe48d4e-7be1-4b09-8133-9c8d79419a37.metadata.json
/mapr/CLUSTER1/var/mapr/insights/cldb/metadata/00000-9dbcee30-f022-4abf-ba91-75e01c77a406.metadata.json
/mapr/CLUSTER1/var/mapr/insights/cldb/metadata/00002-7304f0cc-d517-4c8d-929b-c28f4afbd392.metadata.json
/mapr/CLUSTER1/var/mapr/insights/cldb/metadata/00004-569c881d-ca7d-4320-afba-21bd83ad4e7c.metadata.json
/mapr/CLUSTER1/var/mapr/insights/cldb/metadata/00005-586c0de1-4cdb-45c8-813c-834b3202d115.metadata.json

See Configuring Data Fabric to Track User Behavior for more information about user behavior tracking configuration in Data Fabric.