Before You Configure Drill to Run Under YARN

Provides the steps required to repackage the Drill-on-YARN archive.

During Drill-on-YARN installation, the system fails to upload the Drill archive because JNI finds a mismatch for Java and native libraries. If you install and try to start Drill-on-YARN using the mapr-drill-yarn package, the system returns an error.

To prevent this issue from occurring, complete the following steps to repackage the Drill-on-YARN archive:
  1. Install Drill-on-YARN, as described in Installing Drill to Run Under YARN, but do not configure or start Drill-on-YARN.
  2. Create a file named recreate_archive.sh with the following information:
    TIP
    You can create and run this file in any location you choose.
    read -p "This operation will recreate drill.tar.gz in drill home directory. Continue? " -n 1 -r
echo
if [[ ! $REPLY =~ ^[Yy]$ ]]
then
        exit 1
fi

drillHome="/opt/mapr/drill/drill-$(cat /opt/mapr/drill/drillversion)"
hadoopHome="/opt/mapr/hadoop/hadoop-$(cat /opt/mapr/hadoop/hadoopversion)"

hbaseJar="$(ls /opt/mapr/lib/mapr-hbase-*-mapr.jar)"
maprWebJar="$(ls /opt/mapr/lib/mapr-security*-mapr.jar)"
maprdbJar="$(ls /opt/mapr/lib/maprdb-[0-9].[0-9].[0-9].[0-9][0-9]-mapr.jar)"
mapredJar="$(ls /opt/mapr/lib/maprdb-mapreduce-*-mapr.jar)"
maprfsJar="$(ls /opt/mapr/lib/maprfs-[0-9].[0-9].[0-9].[0-9][0-9]-mapr.jar)"
jerseyClientJar="$(ls ${hadoopHome}/share/hadoop/yarn/lib/jersey-client-*.jar)"
jerseyCoreJar="$(ls ${hadoopHome}/share/hadoop/yarn/lib/jersey-core-*.jar)"

echo "Drop old mapr jars from ${drillHome}/jars/3rdparty/"
rm -f ${drillHome}/jars/3rdparty/mapr-hbase*
rm -f ${drillHome}/jars/3rdparty/mapr-security-web-*
rm -f ${drillHome}/jars/3rdparty/maprdb-*
rm -f ${drillHome}/jars/3rdparty/maprfs-*

echo "Copy new jars from /opt/mapr/lib/"
cp ${hbaseJar} ${drillHome}/jars/3rdparty/
[ $? == 0 ] && echo "${hbaseJar} has been copied"
cp ${maprWebJar} ${drillHome}/jars/3rdparty/
[ $? == 0 ] && echo "${maprWebJar} has been copied"
cp ${maprdbJar} ${drillHome}/jars/3rdparty/
[ $? == 0 ] && echo "${maprdbJar} has been copied"
cp ${mapredJar} ${drillHome}/jars/3rdparty/
[ $? == 0 ] && echo "${mapredJar} has been copied"
cp ${maprfsJar} ${drillHome}/jars/3rdparty/
[ $? == 0 ] && echo "${maprfsJar} has been copied"

echo "Copy jersey jars from hadoop for timeline client"
cp ${jerseyClientJar} ${drillHome}/jars/3rdparty/
[ $? == 0 ] && echo "${jerseyClientJar} has been copied"
cp ${jerseyCoreJar} ${drillHome}/jars/3rdparty/
[ $? == 0 ] && echo "${jerseyCoreJar} has been copied"

if [ -f ${drillHome}/drill.tar.gz ]; then
	rm -f ${drillHome}/drill.tar.gz
	echo "${drillHome}/drill.tar.gz has been dropped"
fi

tempDir=drill-$(date +%s)

mkdir /tmp/$tempDir
[ $? == 0 ] && echo "Created temporary directory ${tempDir}"

cd /tmp/$tempDir

mkdir drill
[ $? == 0 ] && echo "Created drill directory"

cp -r ${drillHome}/* ./drill/
echo "${drillHome} copied to drill directory"

tar -czf drill.tar.gz ./drill
echo "Created new drill archive"

cp drill.tar.gz ${drillHome}/
echo "drill.tar.gz copied to ${drillHome}"

rm -rf /tmp/$tempDir
echo -e "\033[0;32mDONE.\033[0m"
  3. Issue the following command to run the script:
    
./recreate_archive.sh
    When the script runs, a prompt appears:
    This operation will recreate drill.tar.gz in drill home directory. Continue?

    Reply with y to allow the script to update the JAR files in Drill-on-YARN.

    You should see output similar to the following (versions may differ):

    ls: cannot access '/opt/mapr/hadoop/hadoop-3.3.5/share/hadoop/yarn/lib/jersey-core-*.jar': No such file or directory
Drop old mapr jars from /opt/mapr/drill/drill-1.20.3/jars/3rdparty/
Copy new jars from /opt/mapr/lib/
/opt/mapr/lib/mapr-hbase-7.8.0.0-mapr.jar has been copied
/opt/mapr/lib/mapr-security-web-7.8.0.0-mapr.jar has been copied
/opt/mapr/lib/maprdb-7.8.0.0-mapr.jar has been copied
/opt/mapr/lib/maprdb-mapreduce-7.8.0.0-mapr.jar has been copied
/opt/mapr/lib/maprfs-7.8.0.0-mapr.jar has been copied
Copy jersey jars from hadoop for timeline client
/opt/mapr/hadoop/hadoop-3.3.5/share/hadoop/yarn/lib/jersey-client-1.19.4.jar has been copied
cp: missing destination file operand after '/opt/mapr/drill/drill-1.20.3/jars/3rdparty/'
Try 'cp --help' for more information.
Created temporary directory drill-1737566437
Created drill directory
/opt/mapr/drill/drill-1.20.3 copied to drill directory
Created new drill archive
drill.tar.gz copied to /opt/mapr/drill/drill-1.20.3
DONE.
  4. Continue to Step 1: Configure Drill.