Wednesday, December 28, 2011

How to Set up Hadoop 1.0.0 on RHEL/CentOS

Finally, Hadoop 1.0.0 was released yesterday after six years :).

My research field is Natural Language Processing, and I have used Nutch (with Hadoop) since 2007. I have been following news and articles on Hadoop for a long time, but I have not set up any new version for about two years. However, I were very surprised when I visited Hadoop website today because of... the version 1.0.0. It is obviously that I immediately downloaded and installed it on my server.

Now, I will note steps that I used to setup Hadoop to run on single node, fix some minor errors, and test it. It is very easy to install Hadoop nowadays, much much much easier than two years ago.

Some information about my server:
  • Server: DELL, Intel Xeon X3450 @ 2.67GHz (4 cores, HT enabled).
  • RAM: 8.0 GB
  • OS: Red Hat Enterprise Linux Server release 6.2 (Santiago)
Based on the information that I collected in the installation process, there is no difference if you use CentOS 6, or even RHEL/CentOS 5.

Requirements:
  • Oracle JDK 1.6 (I am not sure whether it works well with OpenJDK or Oracle JDK 1.7.)
Now, we begin...

Step 1. Download and install Oracle JDK 1.6 from Oracle Java SE site. I used jdk-6u30-linux-x64-rpm.bin. I do the following commands in my Linux box.

# wget http://download.oracle.com/otn-pub/java/jdk/6u30-b12/jdk-6u30-linux-x64-rpm.bin
# chmod +x jdk-6u30-linux-x64-rpm.bin
# ./jdk-6u30-linux-x64-rpm.bin

Make sure that JDK was installed to directory /usr/java/default, in which default is a symbolic link to latest and latest is, in turn, a symbolic link to jdk1.6.0_30. Hadoop will use this default location to run regardless of the default version of Java on your system, such as OpenJDK.

Step 2. Download and install Hadoop. Hadoop 1.0.0 is shipped with many packages including source, general binary, rpm, and deb for both 32-bit OS (i386) and 64-bit OS (amd64). I used hadoop-1.0.0-1.amd64.rpm because my OS is RHEL 64-bit. If you use Debian, please use .deb package.

# wget http://ftp.jaist.ac.jp/pub/apache/hadoop/common/hadoop-1.0.0/hadoop-1.0.0-1.amd64.rpm
# rpm -ivh hadoop-1.0.0-1.amd64.rpm

Very easy, and very fast, huh!!!

Step 3. Set up Hadoop for using on single node. This step shows how the installation is too easy.

# hadoop-setup-single-node.sh

Answer "yes" (y) for all questions. After the setup has finished, services of Hadoop will be started automatically, including hadoop-namenode, hadoop-datanode, hadoop-jobtracker, and hadoop-tasktracker.

Step 3a. Fix a minor bug.

All four services seem to be started successfully. However, Hadoop is at 1.0.0, it should have some bugs :D. Indeed, there is at least a bug related to the mapredsystem directory on HDFS that makes hadoop-jobtracker cannot start although it said OK when you started the service. You can check the log file for more details. It is located under /var/log/hadoop/mapred/ with the name hadoop-mapred-jobtracker-$HOSTNAME.log. The log might have a portion look as follows.

WARN org.apache.hadoop.mapred.JobTracker: Failed to operate on mapred.system.dir (hdfs://localhost:8020/mapred/mapredsystem) because of permissions.
WARN org.apache.hadoop.mapred.JobTracker: Manually delete the mapred.system.dir (hdfs://localhost:8020/mapred/mapredsystem) and then start the JobTracker.
WARN org.apache.hadoop.mapred.JobTracker: Bailing out ...
org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=mapred, access=WRITE, inode="":hdfs:supergroup:rwxr-xr-x

The log said that Hadoop cannot operate the system directory of Map-Reduce /mapred/mapredsystem since user is mapred and owner is hdfs. Furthermore, the directory owner does not allow others to write on it. I check the details by using the following command.

# hadoop fs -ls /
drwxrwxrwx   - hdfs   supergroup          0 2011-12-28 20:08 /tmp
drwxr-xr-x   - hdfs   supergroup          0 2011-12-28 21:46 

What??? I did not see the directory /mapred. Hadoop might have a bug in the installation script. I fixed it by creating /mapred using hdfs user and change the owner of that directory to mapred user.

# sudo -u hdfs hadoop fs -mkdir /mapred
# sudo -u hdfs hadoop fs -chown mapred /mapred

Restart JobTracker.

# service hadoop-jobtracker restart

I checked the log file again, and everything was ok.

Step 4. Test Hadoop.

Hadoop 1.0.0 has been shipped with an example to validate the setup process. Run it as follow.

# hadoop-validate-setup.sh --user=hdfs

If you get "teragen, terasort, teravalidate passed." near the end of the output, everything is ok.

You can check the progress by accessing JobTracker website via http://localhost:50030/.

You can also check nodes by accessing NameNode website via http://localhost:50070/.

I will try to setup a new Hadoop cluster in this New Year Holiday, and put my notes on this blog ASAP.

Enjoy your New Year Holiday!!!

Thursday, November 24, 2011

How to Change MAC Address on RHEL/CentOS/Fedora

On the RHEL/CentOS/Fedora boxes, there is a line named HWADDR in the interface configuration file /etc/sysconfig/network-scripts/ifcfg-eth0, for example, storing the MAC address of the corresponding ethernet interface.

Nothing will happen if you change the MAC addresss in this line, and you will probably receive the following notice when restart the network service.

Shutting down interface eth0:  Device eth0 has MAC address AA:BB:CC:DD:EE:FF, instead of configured address 00:11:22:33:44:55. Ignoring.

To change the MAC address, just keep the HWADDR as it is, and add the following line to the above file.

MACADDR=00:11:22:33:44:55

Restart the network service.

# service network restart
# ifconfig eth0

New MAC address! Congrats!!!

Note: You should change eth0 to eth1, or the Ethernet interface you want to change its MAC address.

Tuesday, September 20, 2011

[BlackBerry] Mapping Vendor ID to Vendor Name (Updated on 9/20/2011)

To whom that want to know the vendor name of your BB,

First of all, you need to find out the vendor id. On the QWERTY keyboard, press Alt + Shift + H and check the line Vendor ID.

I decided to build my own script to extract the list of vendor name of BB after an hour searching that information for my BB. I extracted the following list from file "C:\Program Files (x86)\Common Files\Research In Motion\AppLoader\Vendor.xml". The path may be different in your computer. I use Windows 7 64-bit.

import sys
import os
from xml.etree.ElementTree import ElementTree

VENDOR_FILE = 'C:\Program Files (x86)\Common Files\Research In Motion\AppLoader\Vendor.xml'

def main():
    tree = ElementTree()
    tree.parse(VENDOR_FILE)
    d = {}
    for vendor in tree.iter('vendor'):
        d[int(vendor.attrib['id'], 16)] = vendor.attrib['Name']
    for id in sorted(d):
        print '%d\t%x\t%s' % (id, id, d[id])

if __name__ == '__main__':
    sys.exit(main())

Last updated on September 20, 2011.

Dec
Hex
Name
100
64
T-Mobile US
101
65
Cingular Wireless
102
66
AT&T
103
67
Nextel/Nextel International - Brazil/Nextel International - Peru/Nextel International - Mexico
104
68
Sprint
105
69
Verizon Wireless
106
6a
AllTel
107
6b
Rogers
109
6d
Bell Mobility
111
6f
O2 DE
113
71
Telfort
114
72
T-Mobile Austria - AT/T-Mobile Germany - DE
115
73
TIM Italy
116
74
Hutchison
117
75
Bouygues
118
76
Vodafone SFR (France)
119
77
Orange France/Orange Senegal/Orange Ivory Coast/Orange Caraibe/Orange Madagascar (FR PRD)
120
78
Vodafone UK/Vodafone Netherlands
121
79
AMX Telcel Mexico
122
7a
Telstra
123
7b
T-Mobile UK/T-Mobile Croatia - HR
124
7c
Vodafone Germany
125
7d
O2 UK/O2 Ireland/O2 IoM (Manx Telecom)/Tesco Mobile
126
7e
Telus
127
7f
SMART Philippines
128
80
StarHub
129
81
Telefonica Spain
130
82
Vodafone Switzerland (SwissCom)
131
83
Cable & Wireless West Indies/Cable & Wireless Channel Islands/Cable & Wireless Panama/Cable & Wireless TSTT
132
84
Vodafone Italy
133
85
Vodafone Spain
134
86
T-Mobile Netherlands - NL
135
87
Cincinnati Bell Wireless
136
88
Telefonica Colombia/Telefonica Nicaragua/Telefonica Mexico/Telefonica Panama/Telefonica Argentina/Telefonica Chile/Telefonica Ecuador/Telefonica El Salvador/Telefonica Uruguay/Telefonica Guatemala/Telefonica Peru/Telefonica Venezuela
137
89
Vodafone Austria (Mobilkom)
138
8a
Vodafone Australia
139
8b
Vodafone Ireland/Vodafone Airtel (Jersey)/Vodafone Lithuania (Bite)/Vodafone Latvia (Bite)
140
8c
Telenor Sweden
141
8d
CSL
142
8e
Orange UK/Telkom/Orange Botswana/Orange Mauritius
143
8f
Vodafone New Zealand
144
90
SingTel
145
91
Globe
146
92
Optus
147
93
Orange BE (Mobistar)
148
94
Vodafone Hungary
149
95
Bharti (Airtel)/Bharti Sri Lanka/Bharti Seychelles/Airtel Bangladesh
150
96
KPN
151
97
Wind Hellas
152
98
Vodafone Belgium (Proximus)
153
99
Vodafone Portugal
154
9a
TIM Brazil
155
9b
BT Mobile
156
9c
Verizon Wireless Wholesale - EarthLink
158
9e
E-Plus
159
9f
BASE
160
a0
Dobson Communications
161
a1
Vodafone Egypt
162
a2
Orange Switzerland
164
a4
T-Mobile
165
a5
Maxis
166
a6
Vodafone Denmark (TDC)
167
a7
Vodafone M1 (Singapore)/Vodafone Dialog (Sri Lanka)/Vodafone Celcom/Vodafone XL (Indonesia)
168
a8
Vodacom South Africa
169
a9
T-Mobile Poland (PTC)
170
aa
T-Mobile Czech - CZ/T-Mobile Slovakia - SK
171
ab
T-Mobile Hungary - HU
173
ad
MTN - South Africa/MTN - Nigeria/MTN - Cyprus/MTN - Ghana/MTN - Cameroon/MTN - Uganda/MTN Rwanda/MTN - Swaziland/MTN - Benin/MTN - Congo-Brazzaville/MTN Botswana
174
ae
Entel PCS (TIM Chile)
175
af
Orange Spain/Orange Dominicana
176
b0
Vodafone SmarTone (Hong Kong)
177
b1
Verizon Wireless Wholesale - TCS/CloseCall
178
b2
AVEA
179
b3
FAST 100
180
b4
Turkcell
181
b5
Partner
183
b7
Orange Romania/Orange Poland/Orange NL/Orange Slovakia/Orange Spain Global
186
ba
Telkomsel
188
bc
Vodafone Greece
189
bd
USCC
190
be
Mobilink
191
bf
Velocita Wireless
194
c2
Vodafone Luxembourg
195
c3
Vodafone Iceland
197
c5
Vodafone Romania (Mobifon)
198
c6
Vodafone Czech
199
c7
Vodafone Bahrain
202
ca
T-Mobile Slovakia - SK
204
cc
China Mobile
205
cd
Movilnet
209
d1
Sympac
210
d2
Personal Argentina
212
d4
Etisalat
213
d5
CBeyond
214
d6
AMX Group/ Group
215
d7
Telefonica Venezuela
216
d8
Telefonica Brazil
217
d9
Orange Romania/Orange Moldova/Orange Poland/Orange Slovakia
218
da
KTPowertel
220
dc
DoCoMo
222
de
Vodafone Bulgaria (Mobiltel)
224
e0
PCCW
225
e1
Hawaiian Telcom/Credo Mobile/Sprint MVNO - Telispire/Mitel Solutions
227
e3
Mobily
228
e4
SaskTel
229
e5
O2 CZ (Eurotel)/O2 Slovakia
230
e6
Hutchison India
236
ec
Reliance
237
ed
Verizon Wireless Wholesale/Verizon Wireless Wholesale - Telispire/Verizon Wireless Wholesale - Deltacom/Verizon Wireless Wholesale - Payless/Verizon Long Distance LLC
238
ee
Vodafone Turkey
239
ef
Meditel
240
f0
Indosat/StarOne
241
f1
Alcatel Shanghai Bell
245
f5
3-Sweden/Denmark/3-Italy/3-UK/3-Austria/3-Ireland
247
f7
Vodafone Essar
248
f8
Centennial Wireless/Centennial PR
250
fa
T-Mobile Austria - AT
254
fe
Oi Brazil
255
ff
Telecom New Zealand
258
102
3-Australia
259
103
TSTT Trinidad and Tobago
269
10d
TATA DOCOMO/TATA INDICOM
271
10f
T-Mobile Croatia - HR/T-Mobile Macedonia - MC/T-Mobile Montenegro - MN
273
111
BT Italy
274
112
1&1
277
115
MTS Mobility
278
116
Virgin Mobile
280
118
Orange Slovakia
282
11a
Taiwan Mobile
285
11d
Orange Austria
286
11e
Vodafone Malta
288
120
BASE (JiM)
294
126
Cellular South
295
127
China Mobile HK
296
128
MTS Russia
297
129
VimpelCom
298
12a
Digitel
299
12b
SK Telecom
300
12c
Solo (Bell)
301
12d
Carphone Warehouse Spain/Carphone Warehouse UK/Carphone Warehouse France/Carphone Warehouse NL/Phones 4 U
302
12e
Cable & Wireless UK/20:20 UK/20:20 France/20:20 Germany/Poste Mobile/Brightpoint Germany/Brightpoint UK/Brightstar AZLAN DE/Brightstar AZLAN UK/LuxGSM/Brightstar AZLAN FR/Transatel
303
12f
Iusacell Mexico/Unefon
308
134
XL Indonesia
309
135
Fido Solutions
310
136
Wind Italy
312
138
Mobinil Egypt
313
139
Chunghwa
315
13b
Metro PCS
316
13c
Nextel International - Mexico
317
13d
Nextel International - Peru
318
13e
Nextel International - Brazil
319
13f
Nextel International - Argentina
322
142
Vodafone Qatar
324
144
Etisalat Misr
326
146
Brightpoint Australia/Ingram Micro AU/Ingram Micro NZ/Vodafone New Zealand/Telecom New Zealand
327
147
Axis
329
149
China Telecom
331
14b
Bell Mobility
332
14c
Telus
336
150
Q-Tel
337
151
Phone House France
339
153
Phone House Germany
340
154
Phone House Netherlands
352
160
3-Indonesia
353
161
Millicom (TIGO) - Honduras/Millicom (TIGO) - Paraguay/Millicom (TIGO) - Bolivia/Millicom (TIGO) - Colombia/Millicom (TIGO) - El Salvador/Millicom (TIGO) - Guatemala
369
171
3-Italy
370
172
Wind Canada
371
173
SMART Indonesia
373
175
Digicel
375
177
Orange Dominicana
376
178
DTAC
377
179
IDEA
378
17a
Brightstar Hong Kong/Hutchison/PCCW/HK Regional PRD/Vodafone SmarTone/China Mobile HK/SiS (Hong Kong)/CSL/SmarTone Macau
379
17b
Open Mobile
380
17c
Digital China
381
17d
Virgin Mobile UK
382
17e
AMX Comcel Colombia
383
17f
AMX Claro Ecuador
384
180
AMX Claro Argentina
385
181
AMX Claro Brazil
387
183
3-Sweden/Denmark
388
184
3-Austria
389
185
Boost Mobile
390
186
SaskTel
391
187
MTS Mobility
392
188
Virgin Mobile
394
18a
Leap
395
18b
BR Regional PRD
396
18c
Vodafone Ghana
397
18d
Koodo Mobile
398
18e
Koodo Mobile
399
18f
Mobilicity
415
19f
AMX Claro Panama/AMX Claro El Salvador/AMX Claro Dominican Republic/AMX Claro Honduras/AMX Claro Guatemala/AMX Claro Nicaragua
416
1a0
Brightstar Venezuela/Brightpoint Puerto Rico/Brightpoint Venezuela/Brightpoint Panama/Brightpoint Colombia
417
1a1
Virgin Mobile US
431
1af
Videotron
432
1b0
MTS India
433
1b1
AMX Claro Jamaica
434
1b2
AMX Claro Puerto Rico
435
1b3
AMX Claro Dominican Republic
436
1b4
Bakrie Telecom
437
1b5
ATNI
442
1ba
Argentina Regional PRD
451
1c3
China Unicom
462
1ce
Virgin Mobile France
477
1dd
Meteor Communication
600
258
Axiom/Etisalat Misr/GO Mobile/Emtel/Zain Zambia/Etisalat Nigeria/Orange Jordan/Telecom Serbia/Safaricom/Ufone/DU/Zain Jordan/Wataniya Kuwait/Batelco/Telenor Pakistan/Zain Nigeria/STC Saudi Arabia/Mcel/EMS/Zain Ghana/Zain Kuwait/GeoCell/Zain Malawi
601
259
Brightpoint France/Brightstar AZLAN FR/Zain Madagascar/Zain Congo/20:20 France/Togocell/Moov CDI/Telma/Sentel/Moov Benin/Orascom/Moov Gabon/Virgin Mobile France/Zain Gabon/Orange Tunisia/Moov CAR/Coriolis/WANA/Tunise Telecom/Moov Togo/Djezzy/Maroc
602
25a
AzerCell/Tcell Somoncom/Viva Armenia/Tcell Indigo/UMC/Moldcell/EMS EU
603
25b
Gibtelecom/TMN Portugal/Jersey Telecom/Mobitel Slovenia/Optimus/Cable & Wireless UK/Telenor Serbia/Telenor Hungary (Pannon)/Telenor Norway/LuxGSM/Trigcom/Globul/Polkomtel/Brightstar AZLAN UK/Brightpoint Netherlands/Virgin Mobile UK/Brightpoint UK
604
25c
20:20 Germany/Sunrise/Brightstar AZLAN DE/Brightpoint Germany/Orange Switzerland/Brightpoint Slovakia/Ingram Micro DE/Vodafone Hungary/O2 CZ (Eurotel)/O2 DE/Orange Austria
605
25d
Cosmote Greece/Wind Hellas/Vodafone Greece
606
25e
Pelephone/Partner/Cellcom Wireless
609
261
Docomo Pacific/MPulse/Brightpoint Singapore/DiGi/Grameenphone/TMIC Cambodia/Brightstar Singapore/Brightstar Vietnam/SiS (Singapore)/Brightstar Malaysia/CamGSM/Etisalat Lanka/Mobitel Srilanka/Viettel/DST Communication Brunei/SMART Philippines/Globe
610
262
AIS Thailand/True Move/SiS (Thailand)/I-Mobile PLUS/DTAC
611
263
Redington/BSNL/Loop Mobile/MTNL/Aircel/Etisalat Lanka/Grameenphone/Ingram India
612
264
Brightpoint Colombia/BTC Bahamas/Telecom Personal/Trilogy/i wireless/Telem/Viaero/Telesur/Cellular One/West Central Wireless/UTS/Comcel Haiti/Alegro Ecuador/Mobility/SETAR/Antel/Nuevatel Bolivia/Vitelcom/Corr Wireless/ICE/Long Lines/Entel/Brightstar
613
265
Brightpoint Singapore/Axis/3-Indonesia/Indosat/XL Indonesia/Telkomsel
614
266
SunCellular/Brightpoint Singapore
620
26c
Brightstar Vietnam/Viettel
624
270
Taiwan Mobile/Brightstar Taiwan/Chunghwa