Copy files to all Hadoop DFS directories

Question

I am trying to copy files to hdfs. I need to copy all files from a specified local folder(myData) to all directories available in my hdfs. I am using wildcards for this.

Here is my code:

bin/hdfs dfs -copyFromLocal myData/* /*/

This is my error:

copyFromLocal: `/var/': No such file or directory 

Normally when i specify a single directory in hdfs, all my files are being copied successfully to that directory.
But when i use wildcards(*) to copy to all directories in my hdfs, i get the above error.

Is there any way to get around this?
Thank you. 

Answer 1

Hi @Bhavish. There is no Hadoop command to copy a local file to all/multiple directories in hdfs. But you can do it using a loop in the bash script. 

Create a new bash script, I named my file copytoallhdfs.sh:

$ nano copytoallhdfs.sh

I came up with the following script for your problem statement:

#!/bin/bash
myarray=`hdfs dfs -ls -C /path/to/directory`
for name in $myarray
do hdfs dfs -copyFromLocal /path/to/source /$name; done

Save(Ctrl+o) and close(Ctrl+x) this file

Now make this script executable:

$ sudo chmod 777 copytoallhdfs.sh

and run the script:

$./copytoallhdfs.sh

This worked for me. 

Comments

Hello @Omkar,

Your response is working perfectly for me. I just made some changes to the script.

Here is my script:

#!/bin/bash

myarray=`bin/hdfs dfs -ls -C /`

echo $myarray;
for name in $myarray
do bin/hdfs dfs -copyFromLocal myData/* $name;
done

The line  echo $myarray; gives me /folder1 /folder2 /folder3.

Since the slash notation '/' is already in the output, i have removed the slash before the variable $name. Also i removed the slash before pathToSource since it was not working.

Many thanks.

---

Happy to be of help @Bhavish!