install python hdfs API library - libpyhdfs


** install libpyhdfs $ sudo apt-get install libhdfs4-dev
$ svn co http://libpyhdfs.googlecode.com/svn/branches/bundle/ libpyhdfs
$ cd libpyhdfs/lib
$ wget http://libpyhdfs.googlecode.com/files/commons-logging-1.0.4.jar
$ wget http://libpyhdfs.googlecode.com/files/hadoop-0.20.1-core.jar
$ wget http://libpyhdfs.googlecode.com/files/libhdfs.so.0 ==> you need to copy your own installed hadoop lib. In my case, I copyed /usr/local/hadoop-1.0.4/c++/Linux-amd64-64/lib/libhdfs.so.0 to libpyhdfs/lib
$ ln -s libhdfs.so.0 libhdfs.so
$ cd ..
# python setup.py install --prefix="/usr/local"
If you see the following error:
/usr/lib/jvm/java-6-sun/include/jni.h:27:20: error: jni_md.h: No such file or directory

Edit /usr/lib/jvm/java-6-sun/include/jni.h
change - 27 #include "jni_md.h"
into + 27 #include "linux/jni_md.h"



** Run test script
$ cd test
$ python pyhdfs_test.py
If you see the following error:
ImportError:libhdfs.so.0:cannot open shared object file:No such file or directory
you need to edit/etc/ld.so.conf.
Add  mentioned above /usr/local/hadoop-1.0.4/c++/Linux-amd64-64/lib/ directory to it.
In my case, it looks like ...
$cat /etc/ld.so.conf
/usr/local/hadoop-1.0.4/c++/Linux-amd64-64/lib/
include /etc/ld.so.conf.d/*.conf
Now you meed to run below command.
$ sudo/sbin/ldconfig -v
If you see error as below...
Environment variable CLASSPATH not set!
Just add CLASSPATH to /etc/profile as below...
export CLASSPATH=$(find ${HADOOP_HOME} -name *.jar | tr '\n' ':')

Now it should work.



Reference Site https://code.google.com/p/libpyhdfs/source/browse/branches/bundle/INSTALL
http://www.opendebug.com/article/483718

댓글

이 블로그의 인기 게시물

ubuntu에서 samba로 파일 공유하기

화이트해커를 위한 암호와 해킹

Shell Program(1) 변수, 상수