Xeon Processor D-1540: 10GbE Driver in Linux (8086:15ad)

If you reach this post via some search engine, you know what title means, right! :)

Well. Today, I receive a SuperMicro server with Xeon Processor D-1540. It comes with 2 x 1GbE ports (Intel I350) and 2 x 10GbE ports (Intel, too). After some checking inside the chassis, I decided to install the operating system.

Firstly, I boot up the server via my disk-less CentOS 6.6. Opps! No 10GbE network interfaces are shown up.

# lspci -nn | grep Ethernet
03:00.0 Ethernet controller [0200]: Intel Corporation Device [8086:15ad]
03:00.1 Ethernet controller [0200]: Intel Corporation Device [8086:15ad]


I search on PCI Database and there is no such 15ad, as of May 12, 2015.

Check around, I have found some information said that such interfaces maybe X552 and the like. ixgbe kernel module should work well with such devices.

Therefore, I decided to try the just-released Debian 8 with newer kernel 3.16. But, oppps, again, no 10GbE network interfaces shown up too.

Ok! I will try the latest ixgbe driver. It should work since I have a similar experience with AWS xlarge nodes.

# apt-get install linux-headers-`uname -r`
# wget http://sourceforge.net/projects/e1000/files/ixgbe%20stable/4.0.3/ixgbe-4.0.3.tar.gz
# tar zxf ixgbe-4.0.3.tar.gz
# cd ixgbe-4.0.3/src
# make
# modprobe -r ixgbe
# make install
# modprobe ixgbe
# ifconfig -a


And I see what I want to see :).

To recompile that kernel module automatically with new installed kernels, I make a kmod.

# apt-get install dkms
# tar zxf ixgbe-4.0.3.tar.gz -C /usr/src
# cat > /usr/src/ixgbe-4.0.3/dkms.conf <<EOF
PACKAGE_NAME="ixgbe"
PACKAGE_VERSION="4.0.3"
CLEAN="cd src/; make clean"
MAKE="cd src/; make BUILD_KERNEL=\${kernelver}"
BUILT_MODULE_LOCATION[0]="src/"
BUILT_MODULE_NAME[0]="ixgbe"
DEST_MODULE_LOCATION[0]="/updates"
DEST_MODULE_NAME[0]="ixgbe"
AUTOINSTALL="yes"
EOF
# dkms add -m ixgbe -v 4.0.3
# dkms build -m ixgbe -v 4.0.3
# dkms install -m ixgbe -v 4.0.3


If you need to use virtual functions, do the similar steps with the latest stable ixgbevf.

Use Certificate-based Authentication with the Open Source Version of SoftEtherVPN

First of all, to whom that do not know what is SoftEtherVPN: It is "An Open-Source Free ​Cross-platform Multi-protocol VPN Program" released by the University of Tsukuba, Japan. The servers and clients can work on multiple platforms such as Linux, Mac OS X, FreeBSD, Windows... as well as allow you to use OS's native VPN client such as Windows' PPTP, OpenVPN, IPSec, and so on.

One of the very useful features of SoftEtherVPN Server is that it allows clients to be authenticated using certificates or active directories. However, such features are disabled in the open-source version of SoftEtherVPN Server.

In practical usages, while I am using SoftEtherVPN at several sites, I have found that I can use certificate-based at some sites and cannot at others with the same binary version of SoftEtherVPN Server downloaded from its website. Therefore, I've decided to investigate in its source code.

After a while, I have found the following function which disables the advanced features.

// Update the global server flags
void UpdateGlobalServerFlags(SERVER *s, CAPSLIST *t)
{
bool is_restricted = false;
// Validate arguments
if (s == NULL || t == NULL)
{
return;
}

is_restricted = SiIsEnterpriseFunctionsRestrictedOnOpenSource(s->Cedar);

SetGlobalServerFlag(GSF_DISABLE_PUSH_ROUTE, is_restricted);
SetGlobalServerFlag(GSF_DISABLE_RADIUS_AUTH, is_restricted);
SetGlobalServerFlag(GSF_DISABLE_CERT_AUTH, is_restricted);
SetGlobalServerFlag(GSF_DISABLE_DEEP_LOGGING, is_restricted);
SetGlobalServerFlag(GSF_DISABLE_AC, is_restricted);
SetGlobalServerFlag(GSF_DISABLE_SYSLOG, is_restricted);
}


The above code means the key here is the function "SiIsEnterpriseFunctionsRestrictedOnOpenSource", which is used to identify that whether or not to restrict advanced features.

Go up to the definition of above function, I have found the following comment.

// Check whether some enterprise functions are restricted
//
// ** Hints by Daiyuu Nobori, written on March 19, 2014 **
//
// The following 'enterprise functions' are implemented on SoftEther VPN Server
// since March 19, 2014. However, these functions are disabled on
// SoftEther VPN Servers which run in Japan and China.
//
// - RADIUS / NT Domain user authentication
// - RSA certificate authentication
// - Deep-inspect packet logging
// - Source IP address control list
// - syslog transfer
//
// The SoftEther VPN Project intentionally disables these functions for users
// in Japan and China. The reason is: Daiyuu Nobori, the chief author of
// SoftEther VPN, has been liable to observe the existing agreements and
// restrictions between him and some companies. The agreements have regulated
// the region-limited restriction to implement and distribute the above
// enterprise functions on the SoftEther VPN open-source program.
//
// Therefore, the SoftEther VPN Project distributes the binary program and
// the source code with the "SiIsEnterpriseFunctionsRestrictedOnOpenSource"
// function. This function identifies whether the SoftEther VPN Server
// program is running in either Japan or China. If the restricted region is
// detected, then the above enterprise functions will be disabled.
//
// Please note that the above restriction has been imposed only on the
// original binaries and source codes from the SoftEther VPN Project.
// Anyone, except Daiyuu Nobori, who understands and writes the C language
// program can remove this restriction at his own risk.


Wow, you are a great man with a kind hints, Daiyuu Nobori. We can disable the restrictions with our own risks.

All other steps are now based on you. If you plan to use SoftEtherVPN Server outside of Japan and China, just download the compiled version from its website. Otherwise, go on to disable the above function with your own knowledge on C programming language.

Have a nice day!

A Simple Way to Create A Windows Installation USB on Linux

Sometime you might want to create a flash drive (USB drive) for installing Windows, but... on Linux. One of the simple way is to use UNetbootin. However, the recent versions of UNetbootin do not support NTFS partition anymore, and you just cannot create such a drive in usual way.

Fortunately, UNetbootin is not support NTFS partition in GUI only, it still work like a charm with the command line options.

Now, it is time to begin.

0. I assume that you have a ISO file of Windows 7 (or 8) Installation Disk (win7.iso, for example) and a flash drive with the capacity of at least 4GB (which is recognized as /dev/sdd, for example.) I also assume that you already have UNetbootin in your system.

1. First of all, creating a partition which is big enough for storing Windows installation packages. It should be larger than 3.2 GB. You can use any tool that you are familiar with, such as gparted, Disk Utility, etc. I use parted here for example.

$ sudo parted -s /dev/sdd mklabel msdos mkpart primary 1024KiB 4GiB print
$ sudo mkfs.ntfs -L WIN /dev/sdd1

2. Mount the created partition to a point in your system. If you use Genome, it should mount that partition automatically to /run/media/${USER}/WIN. If it is not the case, use the following command.

$ sudo mount /dev/sdd1 /mnt

3. Run the following command.

$ sudo unetbootin method=diskimage isofile=win7.iso targetdrive=/dev/sdd1

The GUI will be shown with all the options you need. Click OK.

Blah, blah, blah...

CSSH on MacOS X 10.8+

I assume that you've already known what CSSH (or ClusterSSH in formal) is. In short, it is a way for control many Unix/Linux boxes in parallel in the same way.

I always use CSSH on my Fedora box at work, but this is the first time I have to use it at my home to control my cluster. And my laptop's operating system is Mac OS X 10.8. That is the problem.

Installation of CSSH on Mac OS X seems to be simple. If you want to do it in a lazy way like me, use port.

sudo port install cssh

Then, you try to connect to some host, for example.

cssh root@somehost

It should works well??? Oh no, some strange error message occur like the one as follows.

Can't connect to display `tmp/launch-P6rxkT/org.macosforge.xquartz:0': Invalid argument at /opt/local/lib/perl5/vendor_perl/5.12.4/X11/Protocol.pm line 2270

If you open file Protocol.pm and go to line number 2270, you will see that it try to use INETSocket (wrong) instead of UNIXSocket (right). That is because $host is not equal to "unix". Now you have to change something here to let it use UNIXSocket by checking whether $host contains "xquartz" or not. You change the line number 2262 (the IF command before the error line) from

if ($host eq 'unix') {

to

if ($host eq 'unix' || index($host, "xquartz") != -1) {

(Ahh, that's the error of Perl 5.12.4 distribution of  Mac Ports.)

That's all. Let's try again and see everything goes well.

Supermicro Server: Reset the IPMI Password

You might sometime forget your IPMI password or you get the server from other and want to reset it? There are some options for you. However, first of all, you should try with the default IPMI account, which is ADMIN / ADMIN.

IPMICFG

Supermicro provides a tool for you to manage your IPMI settings named IPMICFG. You can download it from the following location.

ftp://ftp.supermicro.com/utility/IPMICFG/

It contains tools for DOS, Linux, and Windows.

IPMI Tool

If you have Linux installed on that server, you can install OpenIPMI and IPMI Tool to manage the local or remote IPMI service.

To reset the password, all you have to do is as follows.

# service ipmi restart

If IPMI service could not be started, please stop here. You might not have IPMI support in your server.

First, you have to get the user id of the account "ADMIN".

# ipmitool user list

ID  Name      Callin  Link Auth IPMI Msg   Channel Priv Limit
2   ADMIN            true    false      false      Unknown (0x00)

You might see the output as above with the user id of account "ADMIN" is 2.

Now, you change the password of user with id = 2 to "MATKHAU", for example.

# ipmitool user set password 2 MATKHAU

That's all. Have a nice try!

RHEL/CentOS/Fedora: How to Disable CPU Frequency Scaling/Throttling

Attention! Be sure you REALLY want to perform this action! Do it at your own risk!

I assume that you may already know the reason before reaching my post. Run the following command as root. That's all.

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor >& /dev/null

Have fun!!!

Some post-installation steps to make Fedora 19 more comfortable

So, as you may now, Fedora 19 (Schrödinger’s Cat) has released for few days. And this time I decided to upgrade my PC immediately, and everything seems to be very smooth.

For Fedora, I have to make some minor changes to make it more comfortable, such as:

• Install Liberation Fonts (version 2).
• Fix the font's rendering problem.

Liberation Fonts

Liberation fonts are installed in Fedora by default. However, it is kept at version 1.07 with very poor quality. And to get the much better quality, I always upgrade them to version 2.

You can download compiled fonts from the Liberation Fonts page or RPM packages from Fedora's page. I choose the former method.

$ wget https://fedorahosted.org/releases/l/i/liberation-fonts/liberation-fonts-ttf-2.00.0.tar.gz
$ tar -zxvf liberation-fonts-ttf-2.00.0.tar.gz
$ sudo cp -f liberation-fonts-ttf-2.00.1/*.ttf /usr/share/fonts/liberation/
$ sudo fc-cache

Now the Liberation fonts are ready to use. We will fix the font rendering problem before change font settings for GNOME to reflect what we have done so far.

Fix Font Rendering Problem

If you've ever see an Ubuntu screen, you may see a much better font rendering. The problem here is Fedora disable RGBA sub-pixel hinting. All you have to do is to enable such feature by install package freetype-freeworld from RPM Fusion.

$ rpm -ivh ftp://mirror.switch.ch/pool/3/mirror/rpmfusion/free/fedora/releases/19/Everything/x86_64/os/freetype-freeworld-2.4.11-2.fc19.x86_64.rpm

Now, to change the default fonts and default font hinting methods for GNOME, you should install gnome-tweak-tool.

$ sudo yum install gnome-tweak-tool
$ gnome-tweak-tool &

Change the default font settings as in the below screen.

To make everything the same as Ubuntu, you should run the following command.

$ echo "Xft.lcdfilter: lcddefault" >> ~/.Xresources

You may need to log out and log in again to reflect the changes.

SOCKS Proxy over SSH with OpenSSH (Linux) or PuTTY (Windows)

Do you want to use a proxy for surfing Internet without using untrusted proxies? If you have a SSH account to a server with an Internet connection, this article will show you how to create your own SOCKS proxy for above purpose. I will also show you how to do that on both Linux and Windows.

1. Create SOCKS Proxy

Linux

On Linux, all you need is an OpenSSH client which is normally installed by default by almost distros. Open Terminal and run the following command to create a SOCKS proxy which listens for connections on port 1080.

$ ssh -D 1080 username@hostname

A more advantage command which I normally use is as follow.

$ ssh -NCf -o "ServerAliveInterval 300" -D 1080 username@hostname

Please check the man page of ssh for options.

Windows

PuTTY might be the most popular SSH client on Windows. Fortunately, it also supports SOCKS proxy. You should use the latest version which can be downloaded from here.

Open PuTTY an declare information such as host name as normal.

The main window of PuTTY where you can declare host name, port, etc.

The important part is to open a port for listening locally as a SOCKS proxy. Change to "Tunnels" category; fill the port to "Source port" and choose "Dynamic"; click on "Add" button.

Now, you can back to the "Session" category and click on "Open" button to log in to your server.

To avoid session timeout, you should tell PuTTY to send some null packets to the server after some duration. To do that, go to "Connection" category and fill out the number of seconds between null packet sending.

That is enough for proxy. You now have a SOCKS proxy listening on port 1080. It is time to configure web browser redirect every connection via newly created proxy.

2. Web Browser Configuration

Firefox

If you use Firefox, you can set the proxy information by accessing "Options" dialog on Windows or "Preferences" dialog on Linux. Then change to "Advanced" category; choose "Network" tab and click con the "Settings..." button.

Declare the proxy information as in the following screenshot.

Click on OK and OK again. That's all! You now can access to an IP check service to confirm that your web connection routed via the proxy. You can try IP2LOCATION.

Chrome and Internet Explorer

Both Google Chrome and Internet Explorer use the system settings for the proxy. You can declare the information by opening "Internet Options" dialog from Control Panel or Internet Explorer or Chrome.

Change to "Connections" tab and click on "LAN settings" button.

On the "Local Area Network (LAN) Settings" dialog, un-check "Automatically detect settings" and check on "Use a proxy server for your LAN (These settings will not apply to dial-up or VPN connections)." Then click on "Advanced" button.

Declare information as in the above screenshot and click on OK.

On Google Chrome, you can open the system proxy settings by choosing "Settings -> Show advanced settings... -> Change proxy settings..."

Hope this article is useful to you, sometime ;).

Cache Issues in the I/O Performance Test

Sometime you have to benchmark IO performance such as file read/write. If it was the first time you did such kind of task, you almost probability notice that the reading speed is very high, hundreds of MB per second, with a slow hard disk, such as mechanical one with 5400rpm. That is the effects of file caching.

I will show you here some techniques to overcome or minimize the cache effects.

1. Re-mount

The easiest way is to umount and re-mount the corresponding mount point. For example,

# umount /home
# mount /home

2. Clear Cache of the OS

On Linux, you can clear or drop cache of the OS by using the following command.

# sync && echo 3 > /proc/sys/vm/drop_caches

The former part of the above command will commit buffer cache to disk, and the latter part will tell OS to drop buffer caches immediately.

There are three levels of dropping cache with corresponding numbers.
1 - Free pagecache
2 - Free dentries and inodes.
3 - Free pagecache, dentries, and inodes.

3. Use Direct I/O for POSIX

When you open a file with the flag O_DIRECT, you bypass the I/O buffers and therefore bypass the cache effects of cache at the operating system level. Almost device drivers support POSIX compatible API also support O_DIRECT flag, except of some parallel distributed file systems such as PanFS.

Benchmark tools such as IOR (-B), IOzone (-I), dd (direct) have flag for this feature. In programming, use the following example.

open(filename, O_DIRECT);

4. Clear Cache for a Specific File

If you want to clear cache for a specific file, you can use the following program clearcache.c to tell OS not to cache the file. It actually helps you clear cache of a list of files provided via program's arguments.

/* clearcache.c - mrcuongnv */

#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <errno.h>
#include <string.h>

int
clear_file_cache(filename)
     char *filename;
{
  int fd, rs;

  printf("%s", filename);
  if ((fd = open(filename, O_RDONLY)) != -1) {
    if ((rs = posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED)) == 0) {
      printf(" --> Cleared\n");
      return 0;
    }
  }
  printf(" --> %s (%d)\n", strerror(errno), errno);
  return 1;
}

int
main(argc, argv)
     int argc;
     char *argv[];
{
  if (argc < 2) {
    fprintf(stderr, "Syntax: %s FILES\n", argv[0]);
    return 1;
  }

  int i, rs = 0;
  for (i = 1; i < argc; i++) {
    rs += clear_file_cache(argv[i]);
  }

  return rs;
}

The most important part in the above code is posix_fadvise(), which tells OS that the specified data will not be accessed in the near future.

Git: Ignore All Contents of a Directory but the Directory Itself

You sometime want to ignore all contents of a directory but the directory itself in a commitment, such as cache/ directory for example. It can both reduce the amount of data and temporary files to be stored in the repository.

In case of Git, all you need is to create a .gitignore file inside the cache/ directory with  the content as follows.

*
!.gitignore

The first line tells Git to ignore all contents, but the second line tells Git to keep the file .gitignore, and therefore, keeps it to be a non-empty directory.

Activate OpenMPI on Fedora 17/18/19

On Fedora 17/18/19, and some previous version, there is no default link to OpenMPI binaries and libraries. On 64-bit system, they are located in /usr/lib64/openmpi/{bin,lib,share}. To activate them for everyone to use, we have to make some changes as follows.

For OpenMPI's libraries

$ sudo /bin/sh -c 'echo "/usr/lib64/openmpi/lib" >> /etc/ld.so.conf.d/openmpi-64.conf'
$ sudo ldconfig

For OpenMPI's binaries

$ sudo echo 'export PATH=/usr/lib64/openmpi/bin:${PATH}' >> /etc/profile.d/openmpi.sh
$ sudo echo 'set path = ( /usr/lib64/openmpi/bin ${path} )' >> /etc/profile.d/openmpi.csh

Exit your terminal screen and open it again to activate the new settings.

Done!

@10/Jul/2013: I have made a minor change in title and change the double quote to single quote of the echo command to keep ${PATH} from auto-expansion.

How to Overcome "Nautilus cannot handle smb locations"

I recently meet a problem with Nautilus in accessing the Windows shares or Samba shares. Whenever I tried "smb://" in Nautilus, I received "Nautilus cannot handle "smb" locations". When I tried to use "Connect to Server" dialog, there was no option "Windows share" in Type field.

After looking at a series of posts on the Internet, I have found that it is the problem of GVFS without Samba support. Consequently, the solution on Fedora is very very simple, just install the gvfs-smb package.

# yum install gvfs-smb

Actually, the most time-consuming part is the problem caused by myself, perhaps. There is some conflict between samba4 and samba (3) on my Fedora 17. The _local repository contains libwbclient v4 which is conflict with gvfs-smb which uses libwbclient v3. It takes very long  time to recognize the reason and to solve it. If you meet the same problem on new Fedoras (16, 17) and cannot solve it, please leave a comment here.

Setting up RSH on RHEL/CentOS

Although SSH dominates on remote control of Linux/Unix system because of its secure, RSH still has dedicated area to work such as in the internal network of a cluster, for example. However, setting up RSH server is not as trivial as SSH server. It takes some little effort for working properly.

At first step, install RSH server using yum as usual. Do it on every machines.

# yum install rsh-server rsh

RSH service is put under xinetd service, and it is disabled by default. To enable rsh service, open file /etc/xinetd.d/rsh and change "yes" to "no" at the line "disabled".

sed -i "s/^\(.*disable.*=.*\)yes\(.*\)$/\1no\2/g" /etc/xinetd.d/rsh

Next, you have to edit file /etc/securetty to append rsh at the end of it.

# ( grep rsh /etc/securetty ) || ( echo "rsh" >> /etc/securetty )

Now, you can restart xinetd to start rsh.

# service xinetd restart

For each user who wants to use rsh to run commands on remote machines, edit files ~/.rhosts on remote machines to include the hostname or IP address of the local machine. For example, a typical .rhosts file looks like follows.

master
compute-0-0
compute-0-1
compute-0-2

The user on a machine that has such .rhosts file in the home directory allows others from hosts master, compute-0-0, compute-0-1, compute-0-2 run a command on it. Check the man page of rhosts for advanced options.

If you meet some errors related to "back connection", please turn off the firewall of the local machine or at least open ports in range 1011 to 1023, which is normally used for back connection in rsh protocol. By the way, port 514 should be opened on the remote machine too.

Decode .HQX File on Linux

A very simple solution is to use UUDeview which is available in many Linux distros. You can also use macutils on Debian-based distros. Or even easier, an online tool is here.

On Fedora:

# yum install uudeview

On Debian/Ubuntu:

# apt-get install uudeview

To decode file fxppd1110bm922ien.hqx, for example, you can use the following command.

$ uudeview -i fxppd1110bm922ien.hqx

Have a good day!

How to Set up Hadoop 1.0.0 on RHEL/CentOS

Finally, Hadoop 1.0.0 was released yesterday after six years :).

My research field is Natural Language Processing, and I have used Nutch (with Hadoop) since 2007. I have been following news and articles on Hadoop for a long time, but I have not set up any new version for about two years. However, I were very surprised when I visited Hadoop website today because of... the version 1.0.0. It is obviously that I immediately downloaded and installed it on my server.

Now, I will note steps that I used to setup Hadoop to run on single node, fix some minor errors, and test it. It is very easy to install Hadoop nowadays, much much much easier than two years ago.

Some information about my server:

• Server: DELL, Intel Xeon X3450 @ 2.67GHz (4 cores, HT enabled).
• RAM: 8.0 GB
• OS: Red Hat Enterprise Linux Server release 6.2 (Santiago)

Based on the information that I collected in the installation process, there is no difference if you use CentOS 6, or even RHEL/CentOS 5.

Requirements:

• Oracle JDK 1.6 (I am not sure whether it works well with OpenJDK or Oracle JDK 1.7.)

Now, we begin...

Step 1. Download and install Oracle JDK 1.6 from Oracle Java SE site. I used jdk-6u30-linux-x64-rpm.bin. I do the following commands in my Linux box.

# wget http://download.oracle.com/otn-pub/java/jdk/6u30-b12/jdk-6u30-linux-x64-rpm.bin
# chmod +x jdk-6u30-linux-x64-rpm.bin
# ./jdk-6u30-linux-x64-rpm.bin

Make sure that JDK was installed to directory /usr/java/default, in which default is a symbolic link to latest and latest is, in turn, a symbolic link to jdk1.6.0_30. Hadoop will use this default location to run regardless of the default version of Java on your system, such as OpenJDK.

Step 2. Download and install Hadoop. Hadoop 1.0.0 is shipped with many packages including source, general binary, rpm, and deb for both 32-bit OS (i386) and 64-bit OS (amd64). I used hadoop-1.0.0-1.amd64.rpm because my OS is RHEL 64-bit. If you use Debian, please use .deb package.

# wget http://ftp.jaist.ac.jp/pub/apache/hadoop/common/hadoop-1.0.0/hadoop-1.0.0-1.amd64.rpm
# rpm -ivh hadoop-1.0.0-1.amd64.rpm

Very easy, and very fast, huh!!!

Step 3. Set up Hadoop for using on single node. This step shows how the installation is too easy.

# hadoop-setup-single-node.sh

Answer "yes" (y) for all questions. After the setup has finished, services of Hadoop will be started automatically, including hadoop-namenode, hadoop-datanode, hadoop-jobtracker, and hadoop-tasktracker.

Step 3a. Fix a minor bug.

All four services seem to be started successfully. However, Hadoop is at 1.0.0, it should have some bugs :D. Indeed, there is at least a bug related to the mapredsystem directory on HDFS that makes hadoop-jobtracker cannot start although it said OK when you started the service. You can check the log file for more details. It is located under /var/log/hadoop/mapred/ with the name hadoop-mapred-jobtracker-$HOSTNAME.log. The log might have a portion look as follows.

WARN org.apache.hadoop.mapred.JobTracker: Failed to operate on mapred.system.dir (hdfs://localhost:8020/mapred/mapredsystem) because of permissions.
WARN org.apache.hadoop.mapred.JobTracker: Manually delete the mapred.system.dir (hdfs://localhost:8020/mapred/mapredsystem) and then start the JobTracker.
WARN org.apache.hadoop.mapred.JobTracker: Bailing out ...
org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=mapred, access=WRITE, inode="":hdfs:supergroup:rwxr-xr-x

The log said that Hadoop cannot operate the system directory of Map-Reduce /mapred/mapredsystem since user is mapred and owner is hdfs. Furthermore, the directory owner does not allow others to write on it. I check the details by using the following command.

# hadoop fs -ls /
drwxrwxrwx   - hdfs   supergroup          0 2011-12-28 20:08 /tmp
drwxr-xr-x   - hdfs   supergroup          0 2011-12-28 21:46 

What??? I did not see the directory /mapred. Hadoop might have a bug in the installation script. I fixed it by creating /mapred using hdfs user and change the owner of that directory to mapred user.

# sudo -u hdfs hadoop fs -mkdir /mapred
# sudo -u hdfs hadoop fs -chown mapred /mapred

Restart JobTracker.

# service hadoop-jobtracker restart

I checked the log file again, and everything was ok.

Step 4. Test Hadoop.

Hadoop 1.0.0 has been shipped with an example to validate the setup process. Run it as follow.

# hadoop-validate-setup.sh --user=hdfs

If you get "teragen, terasort, teravalidate passed." near the end of the output, everything is ok.

You can check the progress by accessing JobTracker website via http://localhost:50030/.

You can also check nodes by accessing NameNode website via http://localhost:50070/.

I will try to setup a new Hadoop cluster in this New Year Holiday, and put my notes on this blog ASAP.

Enjoy your New Year Holiday!!!

How to Change MAC Address on RHEL/CentOS/Fedora

On the RHEL/CentOS/Fedora boxes, there is a line named HWADDR in the interface configuration file /etc/sysconfig/network-scripts/ifcfg-eth0, for example, storing the MAC address of the corresponding ethernet interface.

Nothing will happen if you change the MAC addresss in this line, and you will probably receive the following notice when restart the network service.

Shutting down interface eth0:  Device eth0 has MAC address AA:BB:CC:DD:EE:FF, instead of configured address 00:11:22:33:44:55. Ignoring.

To change the MAC address, just keep the HWADDR as it is, and add the following line to the above file.

MACADDR=00:11:22:33:44:55

Restart the network service.

# service network restart
# ifconfig eth0

New MAC address! Congrats!!!

Note: You should change eth0 to eth1, or the Ethernet interface you want to change its MAC address.

[BlackBerry] Mapping Vendor ID to Vendor Name (Updated on 9/20/2011)

To whom that want to know the vendor name of your BB,

First of all, you need to find out the vendor id. On the QWERTY keyboard, press Alt + Shift + H and check the line Vendor ID.

I decided to build my own script to extract the list of vendor name of BB after an hour searching that information for my BB. I extracted the following list from file "C:\Program Files (x86)\Common Files\Research In Motion\AppLoader\Vendor.xml". The path may be different in your computer. I use Windows 7 64-bit.

import sys
import os
from xml.etree.ElementTree import ElementTree

VENDOR_FILE = 'C:\Program Files (x86)\Common Files\Research In Motion\AppLoader\Vendor.xml'

def main():
    tree = ElementTree()
    tree.parse(VENDOR_FILE)
    d = {}
    for vendor in tree.iter('vendor'):
        d[int(vendor.attrib['id'], 16)] = vendor.attrib['Name']
    for id in sorted(d):
        print '%d\t%x\t%s' % (id, id, d[id])

if __name__ == '__main__':
    sys.exit(main())

Last updated on September 20, 2011.

Dec	Hex	Name
100	64	T-Mobile US
101	65	Cingular Wireless
102	66	AT&T
103	67	Nextel/Nextel International - Brazil/Nextel International - Peru/Nextel International - Mexico
104	68	Sprint
105	69	Verizon Wireless
106	6a	AllTel
107	6b	Rogers
109	6d	Bell Mobility
111	6f	O2 DE
113	71	Telfort
114	72	T-Mobile Austria - AT/T-Mobile Germany - DE
115	73	TIM Italy
116	74	Hutchison
117	75	Bouygues
118	76	Vodafone SFR (France)
119	77	Orange France/Orange Senegal/Orange Ivory Coast/Orange Caraibe/Orange Madagascar (FR PRD)
120	78	Vodafone UK/Vodafone Netherlands
121	79	AMX Telcel Mexico
122	7a	Telstra
123	7b	T-Mobile UK/T-Mobile Croatia - HR
124	7c	Vodafone Germany
125	7d	O2 UK/O2 Ireland/O2 IoM (Manx Telecom)/Tesco Mobile
126	7e	Telus
127	7f	SMART Philippines
128	80	StarHub
129	81	Telefonica Spain
130	82	Vodafone Switzerland (SwissCom)
131	83	Cable & Wireless West Indies/Cable & Wireless Channel Islands/Cable & Wireless Panama/Cable & Wireless TSTT
132	84	Vodafone Italy
133	85	Vodafone Spain
134	86	T-Mobile Netherlands - NL
135	87	Cincinnati Bell Wireless
136	88	Telefonica Colombia/Telefonica Nicaragua/Telefonica Mexico/Telefonica Panama/Telefonica Argentina/Telefonica Chile/Telefonica Ecuador/Telefonica El Salvador/Telefonica Uruguay/Telefonica Guatemala/Telefonica Peru/Telefonica Venezuela
137	89	Vodafone Austria (Mobilkom)
138	8a	Vodafone Australia
139	8b	Vodafone Ireland/Vodafone Airtel (Jersey)/Vodafone Lithuania (Bite)/Vodafone Latvia (Bite)
140	8c	Telenor Sweden
141	8d	CSL
142	8e	Orange UK/Telkom/Orange Botswana/Orange Mauritius
143	8f	Vodafone New Zealand
144	90	SingTel
145	91	Globe
146	92	Optus
147	93	Orange BE (Mobistar)
148	94	Vodafone Hungary
149	95	Bharti (Airtel)/Bharti Sri Lanka/Bharti Seychelles/Airtel Bangladesh
150	96	KPN
151	97	Wind Hellas
152	98	Vodafone Belgium (Proximus)
153	99	Vodafone Portugal
154	9a	TIM Brazil
155	9b	BT Mobile
156	9c	Verizon Wireless Wholesale - EarthLink
158	9e	E-Plus
159	9f	BASE
160	a0	Dobson Communications
161	a1	Vodafone Egypt
162	a2	Orange Switzerland
164	a4	T-Mobile
165	a5	Maxis
166	a6	Vodafone Denmark (TDC)
167	a7	Vodafone M1 (Singapore)/Vodafone Dialog (Sri Lanka)/Vodafone Celcom/Vodafone XL (Indonesia)
168	a8	Vodacom South Africa
169	a9	T-Mobile Poland (PTC)
170	aa	T-Mobile Czech - CZ/T-Mobile Slovakia - SK
171	ab	T-Mobile Hungary - HU
173	ad	MTN - South Africa/MTN - Nigeria/MTN - Cyprus/MTN - Ghana/MTN - Cameroon/MTN - Uganda/MTN Rwanda/MTN - Swaziland/MTN - Benin/MTN - Congo-Brazzaville/MTN Botswana
174	ae	Entel PCS (TIM Chile)
175	af	Orange Spain/Orange Dominicana
176	b0	Vodafone SmarTone (Hong Kong)
177	b1	Verizon Wireless Wholesale - TCS/CloseCall
178	b2	AVEA
179	b3	FAST 100
180	b4	Turkcell
181	b5	Partner
183	b7	Orange Romania/Orange Poland/Orange NL/Orange Slovakia/Orange Spain Global
186	ba	Telkomsel
188	bc	Vodafone Greece
189	bd	USCC
190	be	Mobilink
191	bf	Velocita Wireless
194	c2	Vodafone Luxembourg
195	c3	Vodafone Iceland
197	c5	Vodafone Romania (Mobifon)
198	c6	Vodafone Czech
199	c7	Vodafone Bahrain
202	ca	T-Mobile Slovakia - SK
204	cc	China Mobile
205	cd	Movilnet
209	d1	Sympac
210	d2	Personal Argentina
212	d4	Etisalat
213	d5	CBeyond
214	d6	AMX Group/ Group
215	d7	Telefonica Venezuela
216	d8	Telefonica Brazil
217	d9	Orange Romania/Orange Moldova/Orange Poland/Orange Slovakia
218	da	KTPowertel
220	dc	DoCoMo
222	de	Vodafone Bulgaria (Mobiltel)
224	e0	PCCW
225	e1	Hawaiian Telcom/Credo Mobile/Sprint MVNO - Telispire/Mitel Solutions
227	e3	Mobily
228	e4	SaskTel
229	e5	O2 CZ (Eurotel)/O2 Slovakia
230	e6	Hutchison India
236	ec	Reliance
237	ed	Verizon Wireless Wholesale/Verizon Wireless Wholesale - Telispire/Verizon Wireless Wholesale - Deltacom/Verizon Wireless Wholesale - Payless/Verizon Long Distance LLC
238	ee	Vodafone Turkey
239	ef	Meditel
240	f0	Indosat/StarOne
241	f1	Alcatel Shanghai Bell
245	f5	3-Sweden/Denmark/3-Italy/3-UK/3-Austria/3-Ireland
247	f7	Vodafone Essar
248	f8	Centennial Wireless/Centennial PR
250	fa	T-Mobile Austria - AT
254	fe	Oi Brazil
255	ff	Telecom New Zealand
258	102	3-Australia
259	103	TSTT Trinidad and Tobago
269	10d	TATA DOCOMO/TATA INDICOM
271	10f	T-Mobile Croatia - HR/T-Mobile Macedonia - MC/T-Mobile Montenegro - MN
273	111	BT Italy
274	112	1&1
277	115	MTS Mobility
278	116	Virgin Mobile
280	118	Orange Slovakia
282	11a	Taiwan Mobile
285	11d	Orange Austria
286	11e	Vodafone Malta
288	120	BASE (JiM)
294	126	Cellular South
295	127	China Mobile HK
296	128	MTS Russia
297	129	VimpelCom
298	12a	Digitel
299	12b	SK Telecom
300	12c	Solo (Bell)
301	12d	Carphone Warehouse Spain/Carphone Warehouse UK/Carphone Warehouse France/Carphone Warehouse NL/Phones 4 U
302	12e	Cable & Wireless UK/20:20 UK/20:20 France/20:20 Germany/Poste Mobile/Brightpoint Germany/Brightpoint UK/Brightstar AZLAN DE/Brightstar AZLAN UK/LuxGSM/Brightstar AZLAN FR/Transatel
303	12f	Iusacell Mexico/Unefon
308	134	XL Indonesia
309	135	Fido Solutions
310	136	Wind Italy
312	138	Mobinil Egypt
313	139	Chunghwa
315	13b	Metro PCS
316	13c	Nextel International - Mexico
317	13d	Nextel International - Peru
318	13e	Nextel International - Brazil
319	13f	Nextel International - Argentina
322	142	Vodafone Qatar
324	144	Etisalat Misr
326	146	Brightpoint Australia/Ingram Micro AU/Ingram Micro NZ/Vodafone New Zealand/Telecom New Zealand
327	147	Axis
329	149	China Telecom
331	14b	Bell Mobility
332	14c	Telus
336	150	Q-Tel
337	151	Phone House France
339	153	Phone House Germany
340	154	Phone House Netherlands
352	160	3-Indonesia
353	161	Millicom (TIGO) - Honduras/Millicom (TIGO) - Paraguay/Millicom (TIGO) - Bolivia/Millicom (TIGO) - Colombia/Millicom (TIGO) - El Salvador/Millicom (TIGO) - Guatemala
369	171	3-Italy
370	172	Wind Canada
371	173	SMART Indonesia
373	175	Digicel
375	177	Orange Dominicana
376	178	DTAC
377	179	IDEA
378	17a	Brightstar Hong Kong/Hutchison/PCCW/HK Regional PRD/Vodafone SmarTone/China Mobile HK/SiS (Hong Kong)/CSL/SmarTone Macau
379	17b	Open Mobile
380	17c	Digital China
381	17d	Virgin Mobile UK
382	17e	AMX Comcel Colombia
383	17f	AMX Claro Ecuador
384	180	AMX Claro Argentina
385	181	AMX Claro Brazil
387	183	3-Sweden/Denmark
388	184	3-Austria
389	185	Boost Mobile
390	186	SaskTel
391	187	MTS Mobility
392	188	Virgin Mobile
394	18a	Leap
395	18b	BR Regional PRD
396	18c	Vodafone Ghana
397	18d	Koodo Mobile
398	18e	Koodo Mobile
399	18f	Mobilicity
415	19f	AMX Claro Panama/AMX Claro El Salvador/AMX Claro Dominican Republic/AMX Claro Honduras/AMX Claro Guatemala/AMX Claro Nicaragua
416	1a0	Brightstar Venezuela/Brightpoint Puerto Rico/Brightpoint Venezuela/Brightpoint Panama/Brightpoint Colombia
417	1a1	Virgin Mobile US
431	1af	Videotron
432	1b0	MTS India
433	1b1	AMX Claro Jamaica
434	1b2	AMX Claro Puerto Rico
435	1b3	AMX Claro Dominican Republic
436	1b4	Bakrie Telecom
437	1b5	ATNI
442	1ba	Argentina Regional PRD
451	1c3	China Unicom
462	1ce	Virgin Mobile France
477	1dd	Meteor Communication
600	258	Axiom/Etisalat Misr/GO Mobile/Emtel/Zain Zambia/Etisalat Nigeria/Orange Jordan/Telecom Serbia/Safaricom/Ufone/DU/Zain Jordan/Wataniya Kuwait/Batelco/Telenor Pakistan/Zain Nigeria/STC Saudi Arabia/Mcel/EMS/Zain Ghana/Zain Kuwait/GeoCell/Zain Malawi
601	259	Brightpoint France/Brightstar AZLAN FR/Zain Madagascar/Zain Congo/20:20 France/Togocell/Moov CDI/Telma/Sentel/Moov Benin/Orascom/Moov Gabon/Virgin Mobile France/Zain Gabon/Orange Tunisia/Moov CAR/Coriolis/WANA/Tunise Telecom/Moov Togo/Djezzy/Maroc
602	25a	AzerCell/Tcell Somoncom/Viva Armenia/Tcell Indigo/UMC/Moldcell/EMS EU
603	25b	Gibtelecom/TMN Portugal/Jersey Telecom/Mobitel Slovenia/Optimus/Cable & Wireless UK/Telenor Serbia/Telenor Hungary (Pannon)/Telenor Norway/LuxGSM/Trigcom/Globul/Polkomtel/Brightstar AZLAN UK/Brightpoint Netherlands/Virgin Mobile UK/Brightpoint UK
604	25c	20:20 Germany/Sunrise/Brightstar AZLAN DE/Brightpoint Germany/Orange Switzerland/Brightpoint Slovakia/Ingram Micro DE/Vodafone Hungary/O2 CZ (Eurotel)/O2 DE/Orange Austria
605	25d	Cosmote Greece/Wind Hellas/Vodafone Greece
606	25e	Pelephone/Partner/Cellcom Wireless
609	261	Docomo Pacific/MPulse/Brightpoint Singapore/DiGi/Grameenphone/TMIC Cambodia/Brightstar Singapore/Brightstar Vietnam/SiS (Singapore)/Brightstar Malaysia/CamGSM/Etisalat Lanka/Mobitel Srilanka/Viettel/DST Communication Brunei/SMART Philippines/Globe
610	262	AIS Thailand/True Move/SiS (Thailand)/I-Mobile PLUS/DTAC
611	263	Redington/BSNL/Loop Mobile/MTNL/Aircel/Etisalat Lanka/Grameenphone/Ingram India
612	264	Brightpoint Colombia/BTC Bahamas/Telecom Personal/Trilogy/i wireless/Telem/Viaero/Telesur/Cellular One/West Central Wireless/UTS/Comcel Haiti/Alegro Ecuador/Mobility/SETAR/Antel/Nuevatel Bolivia/Vitelcom/Corr Wireless/ICE/Long Lines/Entel/Brightstar
613	265	Brightpoint Singapore/Axis/3-Indonesia/Indosat/XL Indonesia/Telkomsel
614	266	SunCellular/Brightpoint Singapore
620	26c	Brightstar Vietnam/Viettel
624	270	Taiwan Mobile/Brightstar Taiwan/Chunghwa