Everything can be distributed

Hadoop 0.23 was released several ago. The most notable improvements of this version include two architectural changes.

1. NameNode Federation For the purpose of a better scalability, Namenode is divided into several nodes, each of which holds an independent namespace. Datanodes register with all of those namespaces, they are responsible for providing storage for data blocks.


2. Yarn Also for the purpose of a better scalability, the functionality of JobTracker is separated into two parts: resource management and job monitoring/tracking. The first part with its name of yarn is abstracted into a new component of hadoop infrastructure, like the kernel means to Linux. MapReduce framework will just be a user space application of yarn.

In order to building hadoop 0.23, you should prepare those executables or libraries list below

1. JDK1.6


2. Maven Hadoop 0.23 will no longer use ant any more, you can download maven from here and install it follow the instructions of that link.


3. Protobuf Google protobuf is used in hadoop 0.23 for new version of RPC. Protoc is needed when building hadoop. If this tool hasn't been installed, the building would got an error. So download protobuf from here and install it by ./configure && make && sudo make install

NOTE That would be better not do the building on an ubuntu system, or building would failed when compiling native code

[INFO] /bin/bash ./libtool  --tag=CC   --mode=compile gcc -DHAVE_CONFIG_H -I.  -I/home/min/software/jdk1.6.0_29/include -I/home/min/software/jdk1.6.0_29/include/linux -I/home/min/code/hadoop/hadoop/branch-0.23/hadoop-common-project/hadoop-common/target/native/src -I/home/min/code/hadoop/hadoop/branch-0.23/hadoop-common-project/hadoop-common/target/native/javah -I/usr/local/include -g -Wall -fPIC -O2 -m32 -g -O2 -MT ZlibCompressor.lo -MD -MP -MF .deps/ZlibCompressor.Tpo -c -o ZlibCompressor.lo `test -f 'src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c' || echo './'`src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c
[INFO] libtool: compile:  gcc -DHAVE_CONFIG_H -I. -I/home/min/software/jdk1.6.0_29/include -I/home/min/software/jdk1.6.0_29/include/linux -I/home/min/code/hadoop/hadoop/branch-0.23/hadoop-common-project/hadoop-common/target/native/src -I/home/min/code/hadoop/hadoop/branch-0.23/hadoop-common-project/hadoop-common/target/native/javah -I/usr/local/include -g -Wall -fPIC -O2 -m32 -g -O2 -MT ZlibCompressor.lo -MD -MP -MF .deps/ZlibCompressor.Tpo -c src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c  -fPIC -DPIC -o .libs/ZlibCompressor.o
[INFO] src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c: In function 'Java_org_apache_hadoop_io_compress_zlib_ZlibCompressor_initIDs':
[INFO] src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c:71:41: error: expected expression before ',' token
[INFO] make: *** [ZlibCompressor.lo] Error 1

Here is an explanation from an ubuntu core developer

More recent version of ubuntu only link 'as-needed' so the macro used to determine the name of the libz.so fails.

Now that you can download a tarball release of Hadoop from here, or fetch the source code by subversion.

 svn co http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.23

Enter the source directory and package it by maven.

cd branch-0.23
mvn clean install package -Pdist -Dtar -Pnative -DskipTests=true

Finally, make sure all target tarballs are built.

$ find . -name "*.tar.gz"
./hadoop-mapreduce-project/target/hadoop-mapreduce-0.23.1-SNAPSHOT.tar.gz
./hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/test.tar.gz
./hadoop-tools/hadoop-tools-dist/target/hadoop-tools-dist-0.23.1-SNAPSHOT.tar.gz
./hadoop-project-dist/target/hadoop-project-dist-0.23.1-SNAPSHOT.tar.gz
./hadoop-hdfs-project/hadoop-hdfs-httpfs/downloads/tomcat.tar.gz
./hadoop-hdfs-project/hadoop-hdfs-httpfs/target/hadoop-hdfs-httpfs-0.23.1-SNAPSHOT/share/hadoop/httpfs/tomcat/bin/tomcat-native.tar.gz
./hadoop-hdfs-project/hadoop-hdfs-httpfs/target/hadoop-hdfs-httpfs-0.23.1-SNAPSHOT/share/hadoop/httpfs/tomcat/bin/commons-daemon-native.tar.gz
./hadoop-hdfs-project/hadoop-hdfs-httpfs/target/hadoop-hdfs-httpfs-0.23.1-SNAPSHOT.tar.gz
./hadoop-hdfs-project/hadoop-hdfs/downloads/commons-daemon-1.0.3-bin-linux-i686.tar.gz
./hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-0.23.1-SNAPSHOT.tar.gz
./hadoop-dist/target/hadoop-0.23.1-SNAPSHOT/share/hadoop/httpfs/tomcat/bin/tomcat-native.tar.gz
./hadoop-dist/target/hadoop-0.23.1-SNAPSHOT/share/hadoop/httpfs/tomcat/bin/commons-daemon-native.tar.gz
./hadoop-dist/target/hadoop-0.23.1-SNAPSHOT.tar.gz
./hadoop-common-project/hadoop-common/target/hadoop-common-0.23.1-SNAPSHOT.tar.gz

Labels: build, federation, hadoop, namenode, yarn