Friday, January 17, 2014

Configuring Debian 6 and 7 for IBM Mid Range Storage Systems (DCS3700 and DS3500) 
Part 1

I like IBM storage and, while I don't use their storage exclusively, am rather fond of the DS3500 (traditional low-midrange integrated controller storage array) and the DCS3700 (midrange high-density storage arrays) solutions.  Both units are easily expandable with SAS expansion shelves and the 3700 series in particular receives frequent line refreshes that boost performance and scalability.
However, a minor hassle is that the IBM DS Storage Manager and Remote Management software are not made available to Linux platforms other than Red Hat and SUSE Enterprise.  I run a Debian-only shop (for historical rather than religious reasons) and was not prepared to make exceptions for our storage servers for the sake of convenience.

Here's a quick how-to for installing and configuring a Debian system for use with the aforementioned IBM storage arrays.  These steps apply to both 64-bit Debian 6 and 7 (32-bit should be the same).  It is most probable that these instructions will fare well for other IBM storage systems (both SAS and FC) that work with IBM's DS Storage Manager software.  

IBM x3650 M4 server with 2x QLogic 8Gb dual-port FC HBA for IBM System X (QLE2562)
IBM DCS3700 dual FC controllers
1x connection from each HBA to each disk controller
In-band management and out-band SNMP and mail alerts.

1) Download the latest IBM DS Storage Manager Software and, optionally, Remote Manager Software from IBM.  At the time of writing this can be found by doing a product search at -> support & downloads -> downloads -> fixed, updates & drivers
The current file is SM10.86_Linux_64bit_x86-64_SMIA-10.86.x5.43.tar

2) Some packages to install before the journey:
apt-get install multipath-tools sysfsutils qlogic-tools alien

3) Ensure that the installed HBA is properly detected by the OS. Some command variations:
$ dmesg | grep qla
[    7.712817] qla2xxx [0000:20:00.1]-00fa:10: QLogic Fibre Channed HBA Driver:

$ lspci | grep -i qlogic
11:00.0 Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02)

$ lsmod | grep qla
qla2xxx               304826  5
scsi_transport_fc      39180  1 qla2xxx

$ systool -c fc_host -v
$ systool -m qla2xxx -v

4) Untar the IBM DS Storage Manager software and begin installation (follow the prompts for the default install):
$ cd Linux_x64_10p86/Linux_x86-64
$ chmod +x SMIA-LINUXX64-10.86.0A05.0043.bin
$ ./SMIA-LINUXX64-10.86.0A05.0043.bin

5) The installation will install dump RPMs and rest of the software in /opt/IBM_DS.  Go there are convert those RPMs into DEBs:
$ alien --scripts SMagent.rpm
$ alien --scripts SMutil.rpm
$ alien --scripts SMesm.rpm
$ alien --scripts SMruntime.rpm
$ alien --scripts SMclient.rpm

Then dpkg -i each of the resulting .deb packages

SMagent and SMmonitor daemons will have init scripts created and should start automatically after installation.

6) Configure multipathing.  
Create a file named /etc/multipath.conf.  Since this is an IBM array we're going to use the RDAC driver for multipath I/O.

But first we need to get the WWN:
/lib/udev/scsi_id -g -u /dev/sdc (or whatever sd device the storage mapped to)
My output was: 360080e5000324c18000000a852d67c5e

Copy the WWN for use in /etc/multipath.conf:
defaults {
        udev_dir                /dev
        path_checker            rdac
        user_friendly_names     yes
blacklist {
        # devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
        devnode "^(ram|raw|loop|fd|md|sr|scd|st)[0-9]*"
        devnode "^hd[a-z][[0-9]*]"
        devnode "^sd[a-b]"
multipaths {
       multipath {
       wwid    360080e5000324c18000000a852d67c5e
                alias                   3700-array-1

devices {
        device {
                vendor                  "IBM"
                product                 "1818"
                path_grouping_policy    group_by_prio
                getuid_callout          "/lib/udev/scsi_id -g -u /dev/%n"
                path_checker            rdac
                prio                    rdac
                hardware_handler        "1 rdac"
                path_selector           "round-robin 0"
                failback                immediate
                features                "1 queue_if_no_path"
                rr_weight               uniform
                rr_min_io               100
                no_path_retry           queue


7) Restart the multipath-tools daemon:
$ /etc/init.d/multipath-tools restart

8) List the multipath devices:
$ multipath -ll

9) Launch the SMclient utility to configure your disk array:
$ /opt/IBM_DS/client/SMclient &

Monday, June 6, 2011

Fereyel and Debbo Engal the Witch

Just recently I remembered the African folktale "Fereyel and Debbo Engel the Witch" which was part of a old Caedmon recording named Spirits and Spooks for Hallowe'en Summoned up by William Conrad.  I recalled the story as best I could to my daughter but was disappointed to learn there was no copy of the text on the web.
Here it is, faithfully transcribed from the record.

+++ Fereyel and Debbo Engal the Witch +++
     A long time ago there lived a witch called Debbo Engal.  She had ten daughters who were beautiful girls whom all men sought after.  And from time to time youths would make the long journey to the house where they lived hidden away in the bush.  But none of these young men ever returned to their villages again.  Although nobody knew the reason why.          
     Debbo Engal knew, however.  When young men called to see her lovely daughters, she would pretend to be delighted to meet them, giving them palm wine to drink and serving them choice food until night fell.  
     And then she would say, "It is too late and the night is too dark for you to walk back to your homes through the bush.  Why not stay the night here and then go home at daybreak in safety?"
     The young men would gladly agree.  And Debbo Engal would tell them to lie down around the fire she kept burning in the biggest hut in the compound.  And soon all would be asleep.  The wicked witch would then sharpen her large knife, creep up to the lads, and kill them silently, one-by-one, with the skill of long practice.  Then in the morning she would -- eat them.  Debbo Engal did not feed on rice or corn or yams -- only human flesh satisfied her cruel appetite.
     Now in a village some miles away lived a woman who had ten sons.  And they heard of the beauty of Debbo Engal's daughters and wanted to visit them.  Their mother entreated the boys not to go.  
     "It is an evil compound; keep away my sons!" she begged.  "So many young men have gone never to return and I don't want to lose all my sons at once!"
     But the lads laughed at her fears and assured her that they could look after each other and that ten men would be a match for any woman.  Besides, the daughters were said to be so very beautiful that none of the young men could rest until they had seen the maidens.
     Early the next morning the ten brothers set off in high spirits, singing and laughing as they walked along the narrow paths which led to the bush to Debbo Engal's compound.  
No sooner had they left their mother then she gave birth to an eleventh son.  But what a strange looking child he was, being scarcely the size of his mother's little finger.  Then he stood upright straight away and spoke to her.  
     "Good mother," he said, his bright little black eyes gazing fixedly at her face.  "Where are my brothers?"
     "They have gone to Debbo Engal's compound," she replied in amazement, wondering how it was that he knew he had any brothers.
     At this the little boy gave a shout exclaiming, "Then I must go after them and save them!"  And he ran swiftly down the path which his brothers had taken.  
     Very soon he saw the ten lads in the distance and called after them, "Hey!  Hey, wait for me!"
     The brothers stopped and turned to see who was calling.  And when the tiny boy ran up to them they stared open-mouthed.  
     Presently, one of them managed to say, "Who are you and what do you want?"
     "My name is Fereyel and I'm your youngest brother," he replied.
     "Indeed you are not for there are only ten of us," they replied.  "Now go away and leave us in peace!"
     "I want to come with you to save you from harm," said Fereyel.
     At this the brothers were angry and began to beat him, saying, "Don't be so silly; how can you be our brother?  Now go away and leave us in peace!"
     They beat him so hard that he lay senseless on the ground.  And then the unkind brothers went on their way toward Debbo Engal's home. 
     Some time later, one of the brothers found a piece of beautiful cloth lying across the path.  
     "Look what I found!" he exclaimed.  "Some careless person has dropped this fine cloth!  This really is a lucky journey, isn't it?"
     He picked up the cloth, slung it over his shoulder, and continued on his way.  But somehow the cloth seemed to get heavier and heavier.  
     And presently, he said to the second brother, "Will you carry this for me?  It's so very heavy on my shoulder."
     The second brother laughed at him for a weakling.  But very soon he, too, found the cloth too heavy and passed it on to the third.  And so it went on until it reached the eldest of the ten brothers.  
     And when he complained about the weight, a shrill voice from inside the cloth called out, "I'm inside!  That's why you find the cloth so heavy.  It's Fereyel, your youngest brother!"
     The young men were furious.  And shaking Fereyel out of the cloth, they beat him again and again until once more they left him lying senseless beside the path.
     "Well that's the end of him," they said.  "Lying little scoundrel!"
     So they went on their way; for it was a long journey.  And they began to hurry since they had wasted some time in beating Fereyel.  
     Suddenly one of the brothers kicked his toe against a piece of metal.  And as he bent to pick it up, he saw that it was a silver ring.  
     "Heh-heh, what luck!" he exclaimed.  "Somebody has dropped a ring and now it's mine!"  And placing it on his finger, he swaggered happily along.  But after a few minutes, his hand hung heavily at his side.  It was all that he could do to walk, so weighty had the ring become.  And the same thing then happened with the ring as with the cloth -- each brother taking turns to wear it, but passing it on when it got too heavy until, at last, it reached the eldest.
     "There's something odd about this ring," he said.
     And was just taking it off his finger when Fereyel's voice piped up saying, "I'm inside!  Th-that's why it's so heavy!"  And he jumped out of the ring onto the ground.  
     Well, the brothers were about to beat him again when the eldest said, "He seems determined to follow us, and he's certainly been very cunning about it.  Leave him alone and let him follow us to Debbo Engal's place after all."

     So on they went until at last they reached the compound they were seeking.  And Debbo Engal came out to greet them.
     "Welcome!" she cried.  "Welcome to our home!  Come and meet my daughters."
     The ten girls were very lovely and the brothers could scarcely take their eyes away from them.  They were led away to the largest hut, and Debbo Engal brought them delicious food and drink.  
     At first, she did not see Fereyel for he was hidden behind the eldest brother's foot.  
     But suddenly she caught sight of him, picked him up, and exclaimed, "What a charming little fellow you are!  Come with me to my hut and I will see that you are properly looked after.  Never have I seen anyone so tiny - you must stay with me and be mine!"
     The brothers were surprised when Fereyel allowed himself to be led away without protest.  But they soon forgot all about him as they feasted and drank and danced with the ten beautiful girls.  

     Night came.  And the brothers talked about going home.  But Debbo Engal persuaded them to stay where they were.  
     "There is no moon," she said, "and you might lose your way.  There are many snakes and wild animals about at this season, too.  So stay with us and return to home by daylight tomorrow."
     The lads needed little persuasion and soon began another dance, while Debbo Engal brought more palm wine to refresh them.  At last, however, the ten boys and girls had to admit that they were too tired to stay awake any longer.  And Debbo Engal lent the brothers some mats and pillows on which to rest in the large hut, where the girls were already almost asleep.  The wicked witch went back to her hut and gave Fereyel a comfortable mat to sleep on and a specially soft pillow for his head.  
     "There you are!" she said.  "Go to sleep now and do not wake until the morning.  I shall sleep on the mat beside you, my little man, so you'll be quite safe."
     So saying, she lay down and closed her eyes.  And soon the compound was wrapped in silence.  
     Presently, Debbo Engal sat up and bent over Fereyel to see if he was asleep.  He closed his eyes and kept perfectly still.  She stoop up, went to the corner where she kept her big knife, but just as she was taking hold of it, Fereyel called out, "What are you doing?!"
     Hastily replacing the knife, Debbo Engal said sweetly, "Aren't you asleep yet, little man?  Let me smooth your pillow for you."  
     And she tidied his bed and shook up the pillow and begged him to sleep in peace.  
     Once again she lay down beside him and once again Fereyel pretended to sleep.  So that after an hour, the wicked witch got up for the second time and took out her knife, ready to sharpen it.
     "What are you doing?!" called Fereyel again.  
     So making some excuse, Debbo Engal came back to her bed and told him to go to sleep again.  For a long time after that, all was quiet.  But Fereyel did not sleep.  He waited until the steady breathing of the woman on the mat beside him told him she was asleep.  And then silently, he crept out of the hut, and made his way to where his brothers and the ten beautiful maidens were.  Gently and silently, he changed all their clothes, putting the white gowns the boys wore over the girls.  And covering his brothers in the blue robes of the women.  Then he returned to Debbo Engal's hut, lay down again and waited.  
     Sure enough, Debbo Engal soon woke with a start.  And for the third time, she crept to the corner of her hut, seized her knife, and began to sharpen it.  Fereyel did not interrupt her this time, and she slipped out of the door, holding the gleaming blade in her hand.  
     Stealthily, she entered the young people's hut, bent over the ten sleeping forms wrapped in white clothes, and cut their throats with practiced skill.  
     "Heh-heh!  They'll make me a splendid meal tomorrow!" she muttered to herself as she lay down contentedly and fell asleep again.
     As soon as he was sure Debbo Engal would not wake, Fereyel hurried into the big hut and shook each of his brothers by their shoulders.
     "Get up!  Get up!" he whispered.  "Debbo Engal meant to kill you all and had I not changed over your clothes she would have done so.  Look...!"
     And he pointed to the ten girls who lay with their throats cut.
     "The old witch thinks it is you she has killed!"
     The brothers needed no second bidding, but tumbled hastily out of the door, and began their journey home through the bush, anxious to get as far away from Debbo Engal as soon as possible before she woke up again.  But it was no use.
     As soon as the witch woke and discovered that Fereyel was no longer by her side, she rushed into her daughter's hut and saw that she had killed them by mistake in the darkness.  Uttering a fearful cry she called up the wind, mounted on its back, and flew towards the brothers who were as of yet scarcely half-way home.
Fereyel saw her coming.  "Look out!" he shouted to his brothers.  "Here come the old witch!"
     The brothers were panic stricken but Fereyel knew what to do.  Seizing a hen's egg from under a bush, he dashed it on the ground between them and Debbo Engal.  The egg immediately turned into a wide, deep river.  And the young men were able to continue on their way.  

     Debbo Engal was furious and turned about at once, and made for home.  But the brothers had not got rid of her so easily.  For she came back with her magic calabash and began to empty out all the water from the swiftly flowing river.  Soon there was not a drop left and she was able to continue her journey once more.  
     Fereyel saw her coming and shouted, "Look out!  Here comes the old witch again!" while he seized a large stone, flung it in her path.  Immediately it changed into a high mountain, and the brothers continued on their journey, certain that Debbo Engal could not get to them now.  
     But the witch was not defeated yet.  She went back to her home on another puff of wind and fetched her magic axe.  And then she hacked and chopped and chopped and hacked until at last the whole mountain disappeared, and she was able to continue on her way.  
     But she was too late.  Just then, Fereyel saw her coming again and gave his brothers a warning shout.  
     "Look out!" he cried as they saw their village ahead.  
     And with one final effort, they reached their house.  Debbo Engal knew that she could not touch them there, and went away defeated, muttering fearful curses under her breath.  
     But Debbo Engal did not let the matter rest there.  She was determined to get hold of the young men and kill them.  Even though she had mistakenly killed her own daughters.  So she lay in hiding and waited her chance. 

     Early next morning, the village headman told the brothers to go into the bush and collect logs.  Somewhat fearfully, they went, keeping close together, and glancing over their shoulders from time-to-time in case the witch turned up again.  They did not see her, however, for the very good reason that she had heard the headman's instructions and had immediately turned herself into a log of wood.  As the lads collected the logs, they stacked them beside the path.  
     "Come on!" one of the called to Fereyel.  "Don't be so lazy!  Why are you standing still while we do all the work?"
     "Because Debbo Engal has turned herself into a log and I do not want to be the one who picks her up," he explained.
     On hearing this, the brothers threw down the logs they were carrying and raced for home.  Debbo Engal, who was furious that she had not yet been picked up, changed herself back into a witch, and hid in the bush still longing for revenge.  

     A few days later, the brothers went off into the bush to collect wild plums.  At first, they only found trees with somewhat withered fruit.  But suddenly they came upon a bush with bright, green leaves, and luscious, juicy plums hanging from its branches.  
     "Look at this; what luck!" exclaimed the eldest brother, reaching out his hand to pluck the fruit.
     "Stop!" commanded Fereyel.  "Don't you realize it's a magic tree and Debbo Engal is inside it?  If you fill your calabashes with fruit, she'll soon have you under her spell." 
     The brothers dropped their calabashes and ran home with haste.  And once again, Debbo Engal's plans were frustrated.  

     The next morning when the brothers came out of their compound, they saw a grey donkey grazing on the communal grass at the edge of the village.  It seemed to belong to no one and the brothers thought it must have strayed from a nearby village.  
     "What luck!" said the eldest.  "Let's all have a donkey ride!"
     And one-by-one, they climbed onto the donkey's back, until all ten of them were perched up there precariously.  
     And then they turned to Fereyel standing beside them and called, "Room for one more -- jump on!"
     "There's no room at all," replied Fereyel.  "Even I, as small as I am, could not get on that donkey's back now."
     Immediately, the strangest thing happened.  The donkey began to grow longer and there was plenty of room for Fereyel.  
     "Hah hah!" he shouted.  "You won't catch me climbing on the back of such an elongated donkey!"
     Then much to everyone's surprise, the donkey shrank back to its normal size.  
     Fereyel laughed.  "You have all been tricked again!" he said.  "Donkeys don't usually understand what human beings are saying, but this one does, so it must be Debbo Engal again.  Get off if you value your lives!"
     The brothers tumbled off the donkey's back and the animal went braying back to the bush where it changed into Debbo Engal.  

     And now the witch was desperate.  She had tried all her magic tricks, save one.  And she was determined to make this a success.  
     "If I can only catch Fereyel, I shall be sure of the others," she said to herself, and sat in deep contemplation planning another wicked scheme.

     The next morning, a beautiful maiden walked into the village.  The villagers crowded round her and asked, why she had come?
     "I want to see Fereyel," she replied in a clear, bell-like voice.  "Will you lead me to his house?"
     Fereyel was amazed to see such an attractive girl and asked her to come into the visitor's hut.  And then he went out and killed a young goat and told his mother to cook the meat for his beautiful guest.  All day long her entertained the maiden, giving her delicious food to eat and talking to her all the while.  The villagers, who had never seen such beauty before, came peeping into the hut from time-to-time, and went away exclaiming loudly at the wonderful sight.  

     When evening came, the maiden said she must go back to her home.   
     "Will you lead me through the bush, Fereyel?" she asked.  "It is too dark for me to go alone."
     Fereyel willingly agreed and the whole village turned out to bid them goodbye.  
     It was very dark.  And Fereyel led the way along the little winding path that the maiden had told him led to her home.  And then suddenly, she disappeared behind a thick tree trunk and was completely hidden.  Fereyel stood still, alert and waiting, straining his eyes in the dark.  And then, out slithered a horrible fat python which made straight for Fereyel and would have coiled itself round him and crushed him to death had he not been waiting for this moment.  
     "Hah-hah!  Debbo Engal!" he laughed, and changed himself into a roaring fire.  
     The python had no time to turn around.  It could not stop its huge rippling body from dashing straight into the fire where it immediately perished.  

     Great was the joy in Fereyel's village when he went home and told his brothers the tale.  
     And great was the feasting and dancing they had that night to celebrate the death of the wicked witch, Debbo Engal.

Wednesday, January 26, 2011

OSX PDF Readers - Search Functionality - Acrobat vs. Skim vs. Preview

Since I'm a heavy reader of technical PDF documents, here is a quick survey of the search functionality in Preview (v.5.0.3), Skim (v.1.3.10), and Adobe Acrobat Reader (v.10.0.0).

For starters, if you use only default Apple apps and expect the most seamless integration, then Preview is your best bet as it integrates nicely with Safari.  However, since I use Chrome, I really don't care about that.

For this comparison, I won't be discussing the full feature-set offered by these PDF viewers but rather only those features that are of interest to me - specifically search accuracy and functionality.

My test doc will be "Brocade FastIron_07200_ConfigGuide.pdf", a technical document of 1,854 pages.

Test 1 (search for the string "link-aggregate"):
A search bar and pane is oriented on the right side of the window.
There is a single means of searching - cmd+f  enter string into search bar.  Both methods are identical and yield an indexed search.
Preview begins searching in real-time as the search string is entered.  There is no indication (e.g., status bar) of how long the search will take but there is a spinning wheel icon.
My search took 38 seconds and yielded hits for 9 Pages.
Preview doesn't say how many actual hits there are for the string in question but lists how many pages contain the string.

Adobe Acrobat Reader:
There is no search bar pane by default.  There are (2) types of search.
1) Cmd+f launches a small blue pop-down search bar.  Search values entered here are displayed one-at-a-time.  Move forward using cmd+g and backwards using cmd+G
This is not that useful when searching a huge document because it offers no context of where the result is nested within the entire doc.
2) Cmd+F launches a separate advanced search window for an indexed search.  The actual search is only run after the search string has been entered and "search" is clicked.
What's nice about this search is that a pop-up window will appear listing a progress status bar.

The search took about 19 seconds and the results, IMO, are far more useful than those yielded by Preview.  Here we have 35 hits for the string link-aggregate.  

Skim has the search bar located in the left pane over the Table of Contents.  Like Adobe Acrobat Reader, there are (2) ways of executing search - cmd+f launches a pop-up window for line-by-line results whereas entering the search string in the search bar will yield a search index.

The search commences in real-time as the user types and displays a progress bar (on lower righthand corner).  The search yielded 30 hits.

Preview is my least favorite.  Search results, as displayed, are not as useful as Skim and Acrobat.  Preview supports bookmarks.
Adobe is fast and accurate - a great research tool.  Adobe does not support bookmarks although the Comment/Sticky Note features can be used as a pseudo-bookmark system within the local document.
Skim is accurate and with a fantastic interface.  Skim supports bookmarks.

Currently I am using Adobe Acrobat Reader since I like having a large, separate window for searches.  However, Skim is an excellent app that was my default PDF viewer before enhancements to Acrobat made it more competitive.

Friday, January 7, 2011

Creating Smart Folders in Thunderbird v.3.1.x

I completely forgot that I had a blog.

I receive a tremendous volume of email and, for work, am tired of maintaining IMAP folders and filter rulesets except for the most voluminous of mailing lists. Especially when situations change and I'm forced to move mail around.

I like to keep everything in one big pot - the Inbox - but still want organization. has "Smart Mailboxes" which is intuitive and easy to use (albeit sometimes slow). Essentially it is a saved search that is displayed as a folder and can comprise any number of search criteria. Thunderbird at different times referred to them interchangeably as Favorite, Smart, Virtual, and Saved Search folders.

As good as is, however, I returned to Thunderbird for several reasons (tabbed messages being one) and sought to recreate this environment. But Thunderbird has been inconsistent across releases on how to create Smart folders, the method is not intuitive, and documentation on the subject is pitiful.

Here's how it's done using Thunderbird v.3.1.7 on OS X:

1) Initiate a search using either cmd+shift+f or from the menu bar: Edit->Find->Search Messages...
Note that searches must be performed this way and not via either the Search or Filter bars.
2) Enter your search criteria and then select "Save as Search Folder"
3) Create the Smart folder Name, specify where on the tree it will be a subfolder, and double-check the search criteria.
4) Hit OK and you're done.

Thursday, January 7, 2010

CDMA Nexus One on Sprint PCS

The new Google Nexus One smartphone running Android 2.1 is hands down the most attractive new phone on the market. The specs speak for themselves, especially the processor speed and screen resolution. Unfortunately, Nexus One is a GSM phone utilizing SIM cards while Verizon and Sprint use CDMA technology. A CDMA Nexus One is slated to arrive in Spring for Verizon but not for Sprint.

I currently have an HTC Hero Android phone (when the hell is Android 2+ going to come out with HTC Sense already?!) but crave the Nexus One. I'm not alone here.

For those of us who are Sprint PCS subscribers because of their excellent pricing plans and top-notch data service and want the Nexus One I created a petition for folks to sign. Begging for it on message boards is one thing but a focused visible petition is another.

I urge potential or current Sprint PCS subscribers to sign it:

Thursday, September 10, 2009

Building Hadoop and Hypertable on Debian Lenny

Hadoop and Hypertable on Debian


Systems: (2) SUN Virtualbox 3.0.6 Virtual Machines, both Debian Lenny x86
Hypervisor system: Debian Lenny AMD64
Networking: Bridged adapter (not NAT) to eth0
g++: 4.3.2

*Note: The fact that this test deployment is on virtual machines is irrelevant to the configuration of Hadoop or Hypertable. Since the (2) virtual machines are on a shared disk, high-performance IO is not expected. However, should the testbed prove promising, dedicated systems will be deployed and high-performance tuning of both Hadoop and Hypertable will be explored.


At NSLS-II we are toying with the idea of back-ending the next generation of the Channel Archiver with a distributed database atop a distributed filesystem. This for a myriad of reasons other than it being a "cool project". For example, combining Hypertable (a high-performance distributed data storage system) with the MapReduce functionality of Hadoop promises to offer performance, redundancy, reliability, and scalability.


Build a (2) node Hadoop and Hypertable cluster. The (2) nodes are "systemA" (master) and "systemB" (slave).

(1) Prerequisites:

*Note: Building Hadoop and Hypertable in this document requires adding the unstable and (optionally) testing repositories to /etc/apt/sources.list.

(1) Make sure to specify that Lenny/stable is the default distribution:
Edit /etc/apt/apt.conf:
# apt.conf
APT::Default-Release "stable"; # Only "stable", "testing", "unstable" are acceptable release names; i.e., "lenny" is not accepted.

$ apt-get update
apt-get -y install g++ cmake libboost-dev liblog4cpp5-dev git-core cronolog libgoogle-perftools-dev libevent-dev zlib1g-dev libexpat1-dev libdb4.6++-dev libncurses-dev libreadline5-dev

(2) Install Hyperic-Sigar

$ wget$ tar -xzvf hyperic-sigar-1.6.2.tar.gz$ cp ~src/hyperic-sigar-1.6.2/sigar-bin/include/*.h /usr/local/include$ cp ~src/hyperic-sigar-1.6.2/sigar-bin/lib/ /usr/local/lib/$ ldconfig

(3) Install Thrift:

$ apt-get -y install sun-java6-jdk ant autoconf automake libtool bison flex pkg-config php5 php5-cli ruby-dev libhttp-access2-ruby libbit-vector-perl liblog4j1.2-java erlang ruby libevent-1.4-2

$ update-java-alternatives --set java-6-sun
$ ln -f -s /bin/bash /bin/sh
$ wget

* Note: Since I am behind a proxy I needed to set this variable:

$ export ANT_OPTS="-Dhttp.proxyHost= -Dhttp.proxyPort=3128"
$ ./
$ ./configure
$ make

(4) Build Hadoop:

(I) Get Hadoop:

Download the latest version of Hadoop. I untarred mine in /opt and made a symlink from hadoop-0.20.1 to hadoop:

The latest version can be found at this mirror:

At the time of writing this document, the latest Hadoop was version 0.20.1

$ wget

You can also retrieve the latest version via git, but the Hadoop directory tree is different than is what is referred to in this document.

$ git clone git://
$ git clone git://
$ git clone git://

$ ln -s /opt/hadoop-0.20.1 /opt/hadoop

Some source needs patching in order for Hypertable to cooperate with MapReduce. This is the list of files requiring patching:

Here are the patches:

--- (revision 765057)
+++ (working copy)
@@ -18,7 +18,8 @@
#include "hadoop/SerialUtils.hh"
#include "hadoop/StringUtils.hh"


--- (revision 765057)
+++ (working copy)
@@ -18,10 +18,11 @@
#include "hadoop/StringUtils.hh"
#include "hadoop/SerialUtils.hh"


using std::string;
@@ -31,7 +32,7 @@

string toString(int32_t x) {
char str[100];
- sprintf(str, "%d", x);
+ snprintf(str, 100, "%d", x);
return str;

@@ -96,7 +97,7 @@
const char* deliminators) {

string result(str);
- for(int i=result.length() -1; i >= 0; --i) {
+ for(int i = result.length() - 1; i >= 0; --i) {
char ch = result[i];
if (!isprint(ch) ||
ch == '\\' ||
@@ -116,7 +117,7 @@
char buff[4];
- sprintf(buff, "\\%02x", static_cast(result[i]));
+ snprintf(buff, 4, "\\%02x", static_cast(result[i]));
result.replace(i, 1, buff);

--- (revision 765057)
+++ (working copy)
@@ -26,9 +26,9 @@

(II) Compile Hadoop:

$ cd /opt/hadoop/src/c++/pipes
$ sh configure
$ make && make install

$ cd /opt/hadoop/src/c++/utils
$ sh configure
$ make && make install

This will place Hadoop headers in:

And Hadoop libraries in:

Add the location of the Hadoop libraries to ld:
$ echo "/opt/hadoop/src/c++/install/lib" > /etc/
$ ldconfig

Next we're ready to compile Hadoop.
$ cd /opt/hadoop

*Note: Again, because I am behind a proxy, I needed to set this environmental variable:
$ export ANT_OPTS="-Dhttp.proxyHost= -Dhttp.proxyPort=3128"
$ ant compile && ant jar

(III) Configure Hadoop:

(A) My Hadoop configuration files are kept in /opt/hadoop/conf
Relevant configuration files in this test are: # Environmental variables
core-site.xml # Default Hadoop filesystem
hdfs-site.xml # HDFS defaults for replication, name, and datanode services
mapred-site.xml # MapReduce defaults for trackers

*Note: All configuration files except for "slaves" and "master" will be the same on both cluster nodes.


By default Hadoop prefers ipv6. Since I am not using ipv6 this required a change:

other variables set in
export JAVA_HOME=/usr/lib/jvm/java-6-sun
export HADOOP_HOME=/opt/hadoop
export HADOOP_LOG_DIR=/var/log/hadoop
export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves
export HADOOP_PID_DIR=/var/hadoop/pids

This will require the creation of directories:
$ mkdir /var/log/hadoop
$ mkdir /var/hadoop/pids

(ii) core-site.xml (where "systemA" is the master node in the cluster):

<description>The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri's authority is used to
determine the host, port, etc. for a filesystem.</description>


(iii) hdfs-site.xml:
*Note: dfs.replication specifies how many nodes are in the Hadoop cluster, in our case, (2).
*Note: This configuration requires the creation of a few directories, namely, "/dfsname", "/hadoop/data". If your test system has more than (1) system disk, it is advisable to separate the DataNode and NameNode directories so as to avoid contentious IO.

<description>Default block replication.The actual number of replications can
be specified when the file is created. The default is used if replication is not
specified in create time.</description>

<description>Path on local filesystem where the NameNode stores the
namespace and transactions logs persistently.</description>

<description>Comma separated list of paths on the local filesystem of
a DataNode where it should store its blocks</description>


(iv) mapred-site.xml:
*Note: This configuration requires the creation of the directory "/hadoop/mapred"

<description>The host and port that the MapReduce job tracker runs at.
If "local", then jobs are run in-process as a single map and reduce task.*lt;/description>
<description>Path on the HDFS where the MapReduce framework stores
system files e.g., /hadoop/mapred/system

(v) slaves:

*Note: both systems are slaves, so both are listed
*Note: this file is only configured on the master, "systemA"

(vi) Master (just the name of the master):

Enable public-key authentication on all nodes within the Hadoop cluster. This will allow the Hadoop service to log-on to other nodes and start/stop services.

I am running Hadoop as the user "hadoop".

On both machines, create the Hadoop user:
$ groupadd hadoop
$ useradd -g hadoop -c "Hadoop User" -d /opt/hadoop -s /bin/bash hadoop

On the master server, systemA:
$ su - hadoop
$ ssh-keygen -t rsa -P ""
$ cp ~hadoop/.ssh/authorized_keys

Copy over the slave system and save it in the same place: ~hadoop/.ssh/authorized_keys

(C) Local name lookup
In /etc/hosts on both machines:
Enter all Hadoop node IP addresses and remove " localhost". Have "localhost" point to an assigned IP address

(IV) Initialize Hadoop:

*Note: Initialization is only necessary on the Master node.
*Note: For convenience-sake I've added Hadoop to my path:

$ export PATH=$PATH:/opt/hadoop/bin

$ hadoop namenode -format

Check to see what's running on both the master and slave:
systemA $ jps
4388 Jps
28444 NameNode
28795 JobTracker
2181 main
28696 SecondaryNameNode
28575 DataNode
28903 TaskTracker

systemB $ jps
13033 DataNode
13142 TaskTracker
1675 Jps
539 main

systemA $ netstat -ptlen

(V) Create a directory in the Hadoop namespace:

$ hadoop dfs -mkdir /hypertable
$ hadoop/bin/hadoop dfs -chmod 777 /hypertable
$ hadoop/bin/hadoop dfs -ls /
$ hadoop dfsadmin -report

(5) Hypertable installation:

(I) Get Hypertable:

$ apt-get -y install git sparsehash libbz2-dev doxygen graphviz
$ git config --global "First Lastname"
$ git config --global ""
$ git clone git://

(II) Pre-build:

(A) Fixes:

The Hypertable source requires some patching so that the build cooperates with Debian's g++. The files that need to be patched are:

--- a/
+++ b/
@@ -24,7 +24,7 @@ TableReader::TableReader(HadoopPipes::MapContext& context)
HadoopUtils::deserializeString(start_row, stream);
HadoopUtils::deserializeString(end_row, stream);

- scan_spec_builder.add_row_interval(start_row, true, end_row, true);
+ scan_spec_builder.add_row_interval(start_row.c_str(), true, end_row.c_str(), true);

if (allColumns == false) {
std::vector columns;
@@ -32,7 +32,7 @@ TableReader::TableReader(HadoopPipes::MapContext& context)

split(columns, job->get("hypertable.table.columns"), is_any_of(", "));
BOOST_FOREACH(const std::string &c, columns) {
- scan_spec_builder.add_column(c);
+ scan_spec_builder.add_column(c.c_str());
m_scanner = m_table->create_scanner(scan_spec_builder.get());

--- a/
+++ b/
@@ -28,7 +28,7 @@ namespace Mapreduce

startrow = tmprow;

- meta_scan_builder.add_row_interval(startrow, true, startrow + "\xff\xff", true);
+ meta_scan_builder.add_row_interval(startrow.c_str(), true, (startrow + "\xff\xff").c_str(), true);

/* select columns */

(B) Hypertable config

Edit ~src/hypertable/conf/hypertable.cfg and enter the information about the Hadoop Master:
# HDFS Broker

(III) Build Hypertable:

Assuming that Hypertable has been unpacked in ~src then
$ mkdir ~src/build/hypertable
$ cd ~src/build/hypertable

*Note some important variables that need to be set in order for a successful compile on the Debian platform:
HADOOP_INCLUDE_PATH = /opt/hadoop/src/c++/install/include
HADOOP_LIB_PATH = /opt/hadoop/src/c++/install/lib
JAVA_INCLUDE_PATH = /usr/lib/jvm/java-6-sun/include
JAVA_INCLUDE_PATH2 = /usr/lib/jvm/java-6-openjdk/include

$ cmake -DBUILD_SHARED_LIBS=ON -DHADOOP_INCLUDE_PATH=/opt/hadoop/src/c++/install/include -DHADOOP_LIB_PATH=/opt/hadoop/src/c++/install/lib -DJAVA_INCLUDE_PATH=/usr/lib/jvm/java-6-sun/include -DJAVA_INCLUDE_PATH2=/usr/lib/jvm/java-6-openjdk/include ../../hypertable

$ make -j <number of cores>
$ make install
$ make doc

(IV) Had issues starting up Hypertable:
: error while loading shared libraries: cannot open shared object file: No such file or directory

Temporary fix:
*Note: this is a stop-gap since this library is linked to files in the source. Don't delete the source. Need to fix this.
$ cp ~src/build/hypertable/src/cc/ThriftBroker/ /opt/hypertable/

(IV) Resolve issue with Hypertable connecting to Hadoop:
Replace the Hypertable-Hadoop jar with Hadoop's jar:
$ cp /opt/hypertable/ /opt/hypertable/
$ cp /opt/hadoop-0.20.1/hadoop-0.20.1-core.jar /opt/hypertable/

(V) Initialize Hypertable:
*Note: Added Hypertable to path:

$ export PATH=$PATH:/opt/hypertable/
$ hadoop

(V) Hypertable Scripts (reference): (local|hadoop|kfs) [<server options>] [<server options>] [<server options>] [<server options>] hadoop

And a wrapper script to start all services: (local|hadoop|kfs) [<server options>]


Friday, August 21, 2009

Debian dbus - ldap error messages on boot

I recently updated a system using LDAP for authorization/authentication and happily discovered it wouldn't reboot. The boot screen was scrolling ad infinitum with these helpful messages:
udevd: nss_ldap failed to bind to LDAP server .....

Quick search displayed many bug reports, obviously not fixed since I'm endured this on a freshly upgraded Debian Lenny system. Some comments I read were silly, like folks having the system boot with a nsswitch.conf using only "files" for passwd, group, and shadow, then switching to "ldap files" or "compat" after boot.

Fix for me was (2) steps:
1. Leave /etc/nsswitch.conf the way it was -- "compat ldap" only and add the problematic user to the LDAP server. And the problematic user is [drumroll] messagebus. All that did was prevent the infinite udevd messages on boot -- still errors though.

2. Edit /etc/libnss_ldap and add:
bind_policy soft

Server boots, fixed, no error messages, done, back to work...