Everything can be distributed

1. NameNode Federation For the purpose of a better scalability, Namenode is divided into several nodes, each of which holds an independent namespace. Datanodes register with all of those namespaces, they are responsible for providing storage for data blocks.


2. Yarn Also for the purpose of a better scalability, the functionality of JobTracker is separated into two parts: resource management and job monitoring/tracking. The first part with its name of yarn is abstracted into a new component of hadoop infrastructure, like the kernel means to Linux. MapReduce framework will just be a user space application of yarn.

1. JDK1.6


2. Maven Hadoop 0.23 will no longer use ant any more, you can download maven from here and install it follow the instructions of that link.


3. Protobuf Google protobuf is used in hadoop 0.23 for new version of RPC. Protoc is needed when building hadoop. If this tool hasn't been installed, the building would got an error. So download protobuf from here and install it by ./configure && make && sudo make install

[INFO] /bin/bash ./libtool  --tag=CC   --mode=compile gcc -DHAVE_CONFIG_H -I.  -I/home/min/software/jdk1.6.0_29/include -I/home/min/software/jdk1.6.0_29/include/linux -I/home/min/code/hadoop/hadoop/branch-0.23/hadoop-common-project/hadoop-common/target/native/src -I/home/min/code/hadoop/hadoop/branch-0.23/hadoop-common-project/hadoop-common/target/native/javah -I/usr/local/include -g -Wall -fPIC -O2 -m32 -g -O2 -MT ZlibCompressor.lo -MD -MP -MF .deps/ZlibCompressor.Tpo -c -o ZlibCompressor.lo `test -f 'src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c' || echo './'`src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c
[INFO] libtool: compile:  gcc -DHAVE_CONFIG_H -I. -I/home/min/software/jdk1.6.0_29/include -I/home/min/software/jdk1.6.0_29/include/linux -I/home/min/code/hadoop/hadoop/branch-0.23/hadoop-common-project/hadoop-common/target/native/src -I/home/min/code/hadoop/hadoop/branch-0.23/hadoop-common-project/hadoop-common/target/native/javah -I/usr/local/include -g -Wall -fPIC -O2 -m32 -g -O2 -MT ZlibCompressor.lo -MD -MP -MF .deps/ZlibCompressor.Tpo -c src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c  -fPIC -DPIC -o .libs/ZlibCompressor.o
[INFO] src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c: In function 'Java_org_apache_hadoop_io_compress_zlib_ZlibCompressor_initIDs':
[INFO] src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c:71:41: error: expected expression before ',' token
[INFO] make: *** [ZlibCompressor.lo] Error 1

More recent version of ubuntu only link 'as-needed' so the macro used to determine the name of the libz.so fails.

 svn co http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.23

cd branch-0.23
mvn clean install package -Pdist -Dtar -Pnative -DskipTests=true

$ find . -name "*.tar.gz"
./hadoop-mapreduce-project/target/hadoop-mapreduce-0.23.1-SNAPSHOT.tar.gz
./hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/test.tar.gz
./hadoop-tools/hadoop-tools-dist/target/hadoop-tools-dist-0.23.1-SNAPSHOT.tar.gz
./hadoop-project-dist/target/hadoop-project-dist-0.23.1-SNAPSHOT.tar.gz
./hadoop-hdfs-project/hadoop-hdfs-httpfs/downloads/tomcat.tar.gz
./hadoop-hdfs-project/hadoop-hdfs-httpfs/target/hadoop-hdfs-httpfs-0.23.1-SNAPSHOT/share/hadoop/httpfs/tomcat/bin/tomcat-native.tar.gz
./hadoop-hdfs-project/hadoop-hdfs-httpfs/target/hadoop-hdfs-httpfs-0.23.1-SNAPSHOT/share/hadoop/httpfs/tomcat/bin/commons-daemon-native.tar.gz
./hadoop-hdfs-project/hadoop-hdfs-httpfs/target/hadoop-hdfs-httpfs-0.23.1-SNAPSHOT.tar.gz
./hadoop-hdfs-project/hadoop-hdfs/downloads/commons-daemon-1.0.3-bin-linux-i686.tar.gz
./hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-0.23.1-SNAPSHOT.tar.gz
./hadoop-dist/target/hadoop-0.23.1-SNAPSHOT/share/hadoop/httpfs/tomcat/bin/tomcat-native.tar.gz
./hadoop-dist/target/hadoop-0.23.1-SNAPSHOT/share/hadoop/httpfs/tomcat/bin/commons-daemon-native.tar.gz
./hadoop-dist/target/hadoop-0.23.1-SNAPSHOT.tar.gz
./hadoop-common-project/hadoop-common/target/hadoop-common-0.23.1-SNAPSHOT.tar.gz

Labels: build, federation, hadoop, namenode, yarn

# posted by Min Zhou : 6:47 AM