hadoop fs
The hadoop fs
command runs a generic file system user client that
interacts with the file system. Starting from EEP 7.1.0, all hadoop fs
commands support operations on symlinks.
PATH
contains the following directories: -
C:\Windows\system32
-
C:\Windows
If they are not present, the hadoop fs
command might fail
silently.
Syntax
hadoop [ Generic Options ] fs
[-cat <src>]
[-chgrp [-R] GROUP PATH...]
[-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]
[-chown [-R] [OWNER][:[GROUP]] PATH...]
[-copyFromLocal <localsrc> ... <dst>]
[-copyToLocal [-ignoreCrc] [-crc] <src> <localdst>]
[-count[-q] <path>]
[-cp <src> <dst> -p[e]]
[-df <path>]
[-du <path>]
[-dus <path>]
[-expunge]
[-get [-ignoreCrc] [-crc] <src> <localdst>]
[-getfattr [-R] {-n name | -d} [-e <encoding>] <path>]
[-getmerge <src> <localdst> [addnl]]
[-help [cmd]]
[-ls <path>]
[-lsr <path>]
[-mkdir <path>]
[-moveFromLocal <localsrc> ... <dst>]
[-moveToLocal <src> <localdst>]
[-mv <src> <dst>]
[-put <localsrc> ... <dst>]
[-rm [-skipTrash] <src>]
[-rmr [-skipTrash] <src>]
[-setfattr -n name [-v value] | -x name <path>]
[-stat [format] <path>]
[-tail [-f] <path>]
[-test -[ezd] <path>]
[-text <path>]
[-touchz <path>]
Parameters
Command Options
The following command parameters are supported for hadoop
fs
:
Parameter |
Description |
---|---|
|
Fetch all files that match the file pattern defined by the
|
|
Changes permissions of a file.
This works similar to shell's |
|
Changes owner and group of a file. This is similar to
shell's WARNING Avoid using '.' to separate user name and group though
Linux allows it. If user names have dots in them and you are using local
file system, you might see surprising results since shell command
chown is used for local files. |
|
This is equivalent to |
|
Identical to the |
|
Identical to the |
|
Count the number of directories, files and bytes under the
paths
that match the specified file pattern. The output columns
are:
|
|
Copy files that match the file pattern |
|
Shows the capacity, free and used space of the file system. If the file system has multiple partitions, and no path to a particular partition is specified, then the status of the root partitions will be shown. |
|
Show the amount of space, in bytes, used by the files
that
match the specified file pattern. Equivalent to the Unix
command |
|
Show the amount of space, in bytes, used by the files
that
match the specified file pattern. Equivalent to the Unix
command |
|
Specify the file system to use. If not specified, the current
configuration is used, taken from the following, in increasing
precedence: |
|
Copy files that match the file pattern <src> to the local name. <src> is kept. When copying multiple files, the destination must be a directory. |
getfattr [-R] -n <name> | -d [-e <encoding>]
<path> |
Retrieve all the extended attribute values (if any) for a file
or directory. Here:
|
|
Get all the files in the directories that
match the source file pattern and merge and sort them to
only
one file on local fs. |
|
Displays help for given command or all commands if none is specified. |
|
List the contents that match the specified file pattern.
If
path is not specified, the contents of
|
|
Recursively list the contents that match the specified
file pattern. Behaves very similarly to |
|
Create a directory in specified location. |
|
Same as |
|
Not implemented yet |
|
Move files that match the specified file pattern
|
|
Copy files from the local file system into fs. To copy files, user must
have write permission on the directory or the |
|
Delete all files that match the specified file
pattern.
Equivalent to the Unix command |
|
Remove all directories which match the specified file
pattern. Equivalent to the Unix command |
-setfattr -n <name> [-v <value>] | -x <name>
<path> |
Set or remove an extended attribute name and value. Here:
|
|
Print statistics about the file/directory at
|
|
Show the last 1KB of the file.
The |
|
Write a timestamp in |
|
If file { exists, has zero length, is a directory then return 0, else return 1. |
|
Takes a source file and outputs the file in text format. The allowed formats are zip and TextRecordInputStream. |
Generic Options
The following generic options are supported for the hadoop
fs
command: -conf <configuration file>
,
-D <property=value>
, -fs <local|file
system URI>
, -jt
<local|jobtracker:port>
, -files
<file1,file2,file3,...>
, -libjars
<libjar1,libjar2,libjar3,...>
, and -archives
<archive1,archive2,archive3,...>
. For more information
on generic options, see Generic
Options.
Examples
hadoop fs -setfattr -n user.key1 -v val1 /xattrs/m7user1/file1.txt
hadoop fs -setfattr -x user.key1 /xattrs/m7user1/dir1
[root@qa-node110 ~]# hadoop fs -getfattr -n user.key1 -e text /xattr/file1
# file: /xattr/file1
user.key1="value1"
[root@qa-node110 ~]# hadoop fs -getfattr -n user.key1 -e hex /xattr/file1
# file: /xattr/file1
user.key1=0x76616c756531
[root@qa-node110 ~]# hadoop fs -getfattr -n user.key1 -e base64 /xattr/file1
# file: /xattr/file1
user.key1=0sdmFsdWUx
[root@qa-node110 ~]# hadoop fs -getfattr -n user.key1 /xattr/file2
# file: /xattr/file2
user.key1="value1"
[root@qa-node110 ~]# hadoop fs -getfattr -d /xattr/file2
# file: /xattr/file2
user.key2="value2"
user.key1="value1"
[root@qa-node110 ~]# hadoop fs -getfattr -R -d /xattr/
# file: /xattr
# file: /xattr/file1
user.key2="value2"
# file: /xattr/file2
user.key2="value2"
user.key1="value1"
[root@qa-node110 ~]# hadoop fs -getfattr -R -n user.key1 /xattr/
# file: /xattr
user.key1="value1"
# file: /xattr/file1
getfattr: No such attribute
# file: /xattr/file2
user.key1="value1"