Friday, January 17, 2014

Configuring Debian 6 and 7 for IBM Mid Range Storage Systems (DCS3700 and DS3500) 
Part 1

Overview:
I like IBM storage and, while I don't use their storage exclusively, am rather fond of the DS3500 (traditional low-midrange integrated controller storage array) and the DCS3700 (midrange high-density storage arrays) solutions.  Both units are easily expandable with SAS expansion shelves and the 3700 series in particular receives frequent line refreshes that boost performance and scalability.
 
However, a minor hassle is that the IBM DS Storage Manager and Remote Management software are not made available to Linux platforms other than Red Hat and SUSE Enterprise.  I run a Debian-only shop (for historical rather than religious reasons) and was not prepared to make exceptions for our storage servers for the sake of convenience.

Here's a quick how-to for installing and configuring a Debian system for use with the aforementioned IBM storage arrays.  These steps apply to both 64-bit Debian 6 and 7 (32-bit should be the same).  It is most probable that these instructions will fare well for other IBM storage systems (both SAS and FC) that work with IBM's DS Storage Manager software.  

Set-up:
IBM x3650 M4 server with 2x QLogic 8Gb dual-port FC HBA for IBM System X (QLE2562)
IBM DCS3700 dual FC controllers
1x connection from each HBA to each disk controller
In-band management and out-band SNMP and mail alerts.

Process:
1) Download the latest IBM DS Storage Manager Software and, optionally, Remote Manager Software from IBM.  At the time of writing this can be found by doing a product search at ibm.com -> support & downloads -> downloads -> fixed, updates & drivers
The current file is SM10.86_Linux_64bit_x86-64_SMIA-10.86.x5.43.tar

2) Some packages to install before the journey:
apt-get install multipath-tools sysfsutils qlogic-tools alien

3) Ensure that the installed HBA is properly detected by the OS. Some command variations:
$ dmesg | grep qla
[    7.712817] qla2xxx [0000:20:00.1]-00fa:10: QLogic Fibre Channed HBA Driver: 8.03.07.12-k.
...

$ lspci | grep -i qlogic
11:00.0 Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02)
...

$ lsmod | grep qla
qla2xxx               304826  5
scsi_transport_fc      39180  1 qla2xxx

$ systool -c fc_host -v
$ systool -m qla2xxx -v

4) Untar the IBM DS Storage Manager software and begin installation (follow the prompts for the default install):
$ cd Linux_x64_10p86/Linux_x86-64
$ chmod +x SMIA-LINUXX64-10.86.0A05.0043.bin
$ ./SMIA-LINUXX64-10.86.0A05.0043.bin

5) The installation will install dump RPMs and rest of the software in /opt/IBM_DS.  Go there are convert those RPMs into DEBs:
$ alien --scripts SMagent.rpm
$ alien --scripts SMutil.rpm
$ alien --scripts SMesm.rpm
$ alien --scripts SMruntime.rpm
$ alien --scripts SMclient.rpm

Then dpkg -i each of the resulting .deb packages

SMagent and SMmonitor daemons will have init scripts created and should start automatically after installation.

6) Configure multipathing.  
Create a file named /etc/multipath.conf.  Since this is an IBM array we're going to use the RDAC driver for multipath I/O.

But first we need to get the WWN:
/lib/udev/scsi_id -g -u /dev/sdc (or whatever sd device the storage mapped to)
My output was: 360080e5000324c18000000a852d67c5e

Copy the WWN for use in /etc/multipath.conf:
#####################################################
defaults {
        udev_dir                /dev
        path_checker            rdac
        user_friendly_names     yes
}
blacklist {
        # devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
        devnode "^(ram|raw|loop|fd|md|sr|scd|st)[0-9]*"
        devnode "^hd[a-z][[0-9]*]"
        devnode "^sd[a-b]"
}
multipaths {
       multipath {
       wwid    360080e5000324c18000000a852d67c5e
                alias                   3700-array-1
        }
}

devices {
        device {
                vendor                  "IBM"
                product                 "1818"
                path_grouping_policy    group_by_prio
                getuid_callout          "/lib/udev/scsi_id -g -u /dev/%n"
                path_checker            rdac
                prio                    rdac
                hardware_handler        "1 rdac"
                path_selector           "round-robin 0"
                failback                immediate
                features                "1 queue_if_no_path"
                rr_weight               uniform
                rr_min_io               100
                no_path_retry           queue
        }
}

#####################################################


7) Restart the multipath-tools daemon:
$ /etc/init.d/multipath-tools restart

8) List the multipath devices:
$ multipath -ll

9) Launch the SMclient utility to configure your disk array:
$ /opt/IBM_DS/client/SMclient &























Monday, June 6, 2011

Fereyel and Debbo Engal the Witch

Just recently I remembered the African folktale "Fereyel and Debbo Engel the Witch" which was part of a old Caedmon recording named Spirits and Spooks for Hallowe'en Summoned up by William Conrad.  I recalled the story as best I could to my daughter but was disappointed to learn there was no copy of the text on the web.
Here it is, faithfully transcribed from the record.

+++ Fereyel and Debbo Engal the Witch +++
     A long time ago there lived a witch called Debbo Engal.  She had ten daughters who were beautiful girls whom all men sought after.  And from time to time youths would make the long journey to the house where they lived hidden away in the bush.  But none of these young men ever returned to their villages again.  Although nobody knew the reason why.          
     Debbo Engal knew, however.  When young men called to see her lovely daughters, she would pretend to be delighted to meet them, giving them palm wine to drink and serving them choice food until night fell.  
     And then she would say, "It is too late and the night is too dark for you to walk back to your homes through the bush.  Why not stay the night here and then go home at daybreak in safety?"
     The young men would gladly agree.  And Debbo Engal would tell them to lie down around the fire she kept burning in the biggest hut in the compound.  And soon all would be asleep.  The wicked witch would then sharpen her large knife, creep up to the lads, and kill them silently, one-by-one, with the skill of long practice.  Then in the morning she would -- eat them.  Debbo Engal did not feed on rice or corn or yams -- only human flesh satisfied her cruel appetite.
     
     Now in a village some miles away lived a woman who had ten sons.  And they heard of the beauty of Debbo Engal's daughters and wanted to visit them.  Their mother entreated the boys not to go.  
     "It is an evil compound; keep away my sons!" she begged.  "So many young men have gone never to return and I don't want to lose all my sons at once!"
     But the lads laughed at her fears and assured her that they could look after each other and that ten men would be a match for any woman.  Besides, the daughters were said to be so very beautiful that none of the young men could rest until they had seen the maidens.
     
     Early the next morning the ten brothers set off in high spirits, singing and laughing as they walked along the narrow paths which led to the bush to Debbo Engal's compound.  
No sooner had they left their mother then she gave birth to an eleventh son.  But what a strange looking child he was, being scarcely the size of his mother's little finger.  Then he stood upright straight away and spoke to her.  
     "Good mother," he said, his bright little black eyes gazing fixedly at her face.  "Where are my brothers?"
     "They have gone to Debbo Engal's compound," she replied in amazement, wondering how it was that he knew he had any brothers.
     At this the little boy gave a shout exclaiming, "Then I must go after them and save them!"  And he ran swiftly down the path which his brothers had taken.  
     Very soon he saw the ten lads in the distance and called after them, "Hey!  Hey, wait for me!"
     The brothers stopped and turned to see who was calling.  And when the tiny boy ran up to them they stared open-mouthed.  
     Presently, one of them managed to say, "Who are you and what do you want?"
     "My name is Fereyel and I'm your youngest brother," he replied.
     "Indeed you are not for there are only ten of us," they replied.  "Now go away and leave us in peace!"
     "I want to come with you to save you from harm," said Fereyel.
     At this the brothers were angry and began to beat him, saying, "Don't be so silly; how can you be our brother?  Now go away and leave us in peace!"
     They beat him so hard that he lay senseless on the ground.  And then the unkind brothers went on their way toward Debbo Engal's home. 
     
     Some time later, one of the brothers found a piece of beautiful cloth lying across the path.  
     "Look what I found!" he exclaimed.  "Some careless person has dropped this fine cloth!  This really is a lucky journey, isn't it?"
     He picked up the cloth, slung it over his shoulder, and continued on his way.  But somehow the cloth seemed to get heavier and heavier.  
     And presently, he said to the second brother, "Will you carry this for me?  It's so very heavy on my shoulder."
     The second brother laughed at him for a weakling.  But very soon he, too, found the cloth too heavy and passed it on to the third.  And so it went on until it reached the eldest of the ten brothers.  
     And when he complained about the weight, a shrill voice from inside the cloth called out, "I'm inside!  That's why you find the cloth so heavy.  It's Fereyel, your youngest brother!"
     The young men were furious.  And shaking Fereyel out of the cloth, they beat him again and again until once more they left him lying senseless beside the path.
     "Well that's the end of him," they said.  "Lying little scoundrel!"
     So they went on their way; for it was a long journey.  And they began to hurry since they had wasted some time in beating Fereyel.  
     Suddenly one of the brothers kicked his toe against a piece of metal.  And as he bent to pick it up, he saw that it was a silver ring.  
     "Heh-heh, what luck!" he exclaimed.  "Somebody has dropped a ring and now it's mine!"  And placing it on his finger, he swaggered happily along.  But after a few minutes, his hand hung heavily at his side.  It was all that he could do to walk, so weighty had the ring become.  And the same thing then happened with the ring as with the cloth -- each brother taking turns to wear it, but passing it on when it got too heavy until, at last, it reached the eldest.
     "There's something odd about this ring," he said.
     And was just taking it off his finger when Fereyel's voice piped up saying, "I'm inside!  Th-that's why it's so heavy!"  And he jumped out of the ring onto the ground.  
     Well, the brothers were about to beat him again when the eldest said, "He seems determined to follow us, and he's certainly been very cunning about it.  Leave him alone and let him follow us to Debbo Engal's place after all."

     So on they went until at last they reached the compound they were seeking.  And Debbo Engal came out to greet them.
     "Welcome!" she cried.  "Welcome to our home!  Come and meet my daughters."
     The ten girls were very lovely and the brothers could scarcely take their eyes away from them.  They were led away to the largest hut, and Debbo Engal brought them delicious food and drink.  
     At first, she did not see Fereyel for he was hidden behind the eldest brother's foot.  
     But suddenly she caught sight of him, picked him up, and exclaimed, "What a charming little fellow you are!  Come with me to my hut and I will see that you are properly looked after.  Never have I seen anyone so tiny - you must stay with me and be mine!"
     The brothers were surprised when Fereyel allowed himself to be led away without protest.  But they soon forgot all about him as they feasted and drank and danced with the ten beautiful girls.  

     Night came.  And the brothers talked about going home.  But Debbo Engal persuaded them to stay where they were.  
     "There is no moon," she said, "and you might lose your way.  There are many snakes and wild animals about at this season, too.  So stay with us and return to home by daylight tomorrow."
     The lads needed little persuasion and soon began another dance, while Debbo Engal brought more palm wine to refresh them.  At last, however, the ten boys and girls had to admit that they were too tired to stay awake any longer.  And Debbo Engal lent the brothers some mats and pillows on which to rest in the large hut, where the girls were already almost asleep.  The wicked witch went back to her hut and gave Fereyel a comfortable mat to sleep on and a specially soft pillow for his head.  
     "There you are!" she said.  "Go to sleep now and do not wake until the morning.  I shall sleep on the mat beside you, my little man, so you'll be quite safe."
     So saying, she lay down and closed her eyes.  And soon the compound was wrapped in silence.  
     Presently, Debbo Engal sat up and bent over Fereyel to see if he was asleep.  He closed his eyes and kept perfectly still.  She stoop up, went to the corner where she kept her big knife, but just as she was taking hold of it, Fereyel called out, "What are you doing?!"
     Hastily replacing the knife, Debbo Engal said sweetly, "Aren't you asleep yet, little man?  Let me smooth your pillow for you."  
     And she tidied his bed and shook up the pillow and begged him to sleep in peace.  
     
     Once again she lay down beside him and once again Fereyel pretended to sleep.  So that after an hour, the wicked witch got up for the second time and took out her knife, ready to sharpen it.
     "What are you doing?!" called Fereyel again.  
     So making some excuse, Debbo Engal came back to her bed and told him to go to sleep again.  For a long time after that, all was quiet.  But Fereyel did not sleep.  He waited until the steady breathing of the woman on the mat beside him told him she was asleep.  And then silently, he crept out of the hut, and made his way to where his brothers and the ten beautiful maidens were.  Gently and silently, he changed all their clothes, putting the white gowns the boys wore over the girls.  And covering his brothers in the blue robes of the women.  Then he returned to Debbo Engal's hut, lay down again and waited.  
     Sure enough, Debbo Engal soon woke with a start.  And for the third time, she crept to the corner of her hut, seized her knife, and began to sharpen it.  Fereyel did not interrupt her this time, and she slipped out of the door, holding the gleaming blade in her hand.  
     Stealthily, she entered the young people's hut, bent over the ten sleeping forms wrapped in white clothes, and cut their throats with practiced skill.  
     "Heh-heh!  They'll make me a splendid meal tomorrow!" she muttered to herself as she lay down contentedly and fell asleep again.
     As soon as he was sure Debbo Engal would not wake, Fereyel hurried into the big hut and shook each of his brothers by their shoulders.
     "Get up!  Get up!" he whispered.  "Debbo Engal meant to kill you all and had I not changed over your clothes she would have done so.  Look...!"
     And he pointed to the ten girls who lay with their throats cut.
     "The old witch thinks it is you she has killed!"
     The brothers needed no second bidding, but tumbled hastily out of the door, and began their journey home through the bush, anxious to get as far away from Debbo Engal as soon as possible before she woke up again.  But it was no use.
     As soon as the witch woke and discovered that Fereyel was no longer by her side, she rushed into her daughter's hut and saw that she had killed them by mistake in the darkness.  Uttering a fearful cry she called up the wind, mounted on its back, and flew towards the brothers who were as of yet scarcely half-way home.
Fereyel saw her coming.  "Look out!" he shouted to his brothers.  "Here come the old witch!"
     The brothers were panic stricken but Fereyel knew what to do.  Seizing a hen's egg from under a bush, he dashed it on the ground between them and Debbo Engal.  The egg immediately turned into a wide, deep river.  And the young men were able to continue on their way.  

     Debbo Engal was furious and turned about at once, and made for home.  But the brothers had not got rid of her so easily.  For she came back with her magic calabash and began to empty out all the water from the swiftly flowing river.  Soon there was not a drop left and she was able to continue her journey once more.  
     Fereyel saw her coming and shouted, "Look out!  Here comes the old witch again!" while he seized a large stone, flung it in her path.  Immediately it changed into a high mountain, and the brothers continued on their journey, certain that Debbo Engal could not get to them now.  
     But the witch was not defeated yet.  She went back to her home on another puff of wind and fetched her magic axe.  And then she hacked and chopped and chopped and hacked until at last the whole mountain disappeared, and she was able to continue on her way.  
     But she was too late.  Just then, Fereyel saw her coming again and gave his brothers a warning shout.  
     "Look out!" he cried as they saw their village ahead.  
     And with one final effort, they reached their house.  Debbo Engal knew that she could not touch them there, and went away defeated, muttering fearful curses under her breath.  
     But Debbo Engal did not let the matter rest there.  She was determined to get hold of the young men and kill them.  Even though she had mistakenly killed her own daughters.  So she lay in hiding and waited her chance. 

     Early next morning, the village headman told the brothers to go into the bush and collect logs.  Somewhat fearfully, they went, keeping close together, and glancing over their shoulders from time-to-time in case the witch turned up again.  They did not see her, however, for the very good reason that she had heard the headman's instructions and had immediately turned herself into a log of wood.  As the lads collected the logs, they stacked them beside the path.  
     "Come on!" one of the called to Fereyel.  "Don't be so lazy!  Why are you standing still while we do all the work?"
     "Because Debbo Engal has turned herself into a log and I do not want to be the one who picks her up," he explained.
     On hearing this, the brothers threw down the logs they were carrying and raced for home.  Debbo Engal, who was furious that she had not yet been picked up, changed herself back into a witch, and hid in the bush still longing for revenge.  

     A few days later, the brothers went off into the bush to collect wild plums.  At first, they only found trees with somewhat withered fruit.  But suddenly they came upon a bush with bright, green leaves, and luscious, juicy plums hanging from its branches.  
     "Look at this; what luck!" exclaimed the eldest brother, reaching out his hand to pluck the fruit.
     "Stop!" commanded Fereyel.  "Don't you realize it's a magic tree and Debbo Engal is inside it?  If you fill your calabashes with fruit, she'll soon have you under her spell." 
     The brothers dropped their calabashes and ran home with haste.  And once again, Debbo Engal's plans were frustrated.  

     The next morning when the brothers came out of their compound, they saw a grey donkey grazing on the communal grass at the edge of the village.  It seemed to belong to no one and the brothers thought it must have strayed from a nearby village.  
     "What luck!" said the eldest.  "Let's all have a donkey ride!"
     And one-by-one, they climbed onto the donkey's back, until all ten of them were perched up there precariously.  
     And then they turned to Fereyel standing beside them and called, "Room for one more -- jump on!"
     "There's no room at all," replied Fereyel.  "Even I, as small as I am, could not get on that donkey's back now."
     Immediately, the strangest thing happened.  The donkey began to grow longer and there was plenty of room for Fereyel.  
     "Hah hah!" he shouted.  "You won't catch me climbing on the back of such an elongated donkey!"
     Then much to everyone's surprise, the donkey shrank back to its normal size.  
     Fereyel laughed.  "You have all been tricked again!" he said.  "Donkeys don't usually understand what human beings are saying, but this one does, so it must be Debbo Engal again.  Get off if you value your lives!"
     The brothers tumbled off the donkey's back and the animal went braying back to the bush where it changed into Debbo Engal.  

     And now the witch was desperate.  She had tried all her magic tricks, save one.  And she was determined to make this a success.  
     "If I can only catch Fereyel, I shall be sure of the others," she said to herself, and sat in deep contemplation planning another wicked scheme.

     The next morning, a beautiful maiden walked into the village.  The villagers crowded round her and asked, why she had come?
     "I want to see Fereyel," she replied in a clear, bell-like voice.  "Will you lead me to his house?"
     Fereyel was amazed to see such an attractive girl and asked her to come into the visitor's hut.  And then he went out and killed a young goat and told his mother to cook the meat for his beautiful guest.  All day long her entertained the maiden, giving her delicious food to eat and talking to her all the while.  The villagers, who had never seen such beauty before, came peeping into the hut from time-to-time, and went away exclaiming loudly at the wonderful sight.  

     When evening came, the maiden said she must go back to her home.   
     "Will you lead me through the bush, Fereyel?" she asked.  "It is too dark for me to go alone."
     Fereyel willingly agreed and the whole village turned out to bid them goodbye.  
     It was very dark.  And Fereyel led the way along the little winding path that the maiden had told him led to her home.  And then suddenly, she disappeared behind a thick tree trunk and was completely hidden.  Fereyel stood still, alert and waiting, straining his eyes in the dark.  And then, out slithered a horrible fat python which made straight for Fereyel and would have coiled itself round him and crushed him to death had he not been waiting for this moment.  
     "Hah-hah!  Debbo Engal!" he laughed, and changed himself into a roaring fire.  
     The python had no time to turn around.  It could not stop its huge rippling body from dashing straight into the fire where it immediately perished.  

     Great was the joy in Fereyel's village when he went home and told his brothers the tale.  
     And great was the feasting and dancing they had that night to celebrate the death of the wicked witch, Debbo Engal.





Wednesday, January 26, 2011

OSX PDF Readers - Search Functionality - Acrobat vs. Skim vs. Preview

Since I'm a heavy reader of technical PDF documents, here is a quick survey of the search functionality in Preview (v.5.0.3), Skim (v.1.3.10), and Adobe Acrobat Reader (v.10.0.0).

For starters, if you use only default Apple apps and expect the most seamless integration, then Preview is your best bet as it integrates nicely with Safari.  However, since I use Chrome, I really don't care about that.

For this comparison, I won't be discussing the full feature-set offered by these PDF viewers but rather only those features that are of interest to me - specifically search accuracy and functionality.

My test doc will be "Brocade FastIron_07200_ConfigGuide.pdf", a technical document of 1,854 pages.

Test 1 (search for the string "link-aggregate"):
Preview:
A search bar and pane is oriented on the right side of the window.
There is a single means of searching - cmd+f  enter string into search bar.  Both methods are identical and yield an indexed search.
Preview begins searching in real-time as the search string is entered.  There is no indication (e.g., status bar) of how long the search will take but there is a spinning wheel icon.
My search took 38 seconds and yielded hits for 9 Pages.
Preview doesn't say how many actual hits there are for the string in question but lists how many pages contain the string.





Adobe Acrobat Reader:
There is no search bar pane by default.  There are (2) types of search.
1) Cmd+f launches a small blue pop-down search bar.  Search values entered here are displayed one-at-a-time.  Move forward using cmd+g and backwards using cmd+G
This is not that useful when searching a huge document because it offers no context of where the result is nested within the entire doc.
2) Cmd+F launches a separate advanced search window for an indexed search.  The actual search is only run after the search string has been entered and "search" is clicked.
What's nice about this search is that a pop-up window will appear listing a progress status bar.

The search took about 19 seconds and the results, IMO, are far more useful than those yielded by Preview.  Here we have 35 hits for the string link-aggregate.  

Skim:
Skim has the search bar located in the left pane over the Table of Contents.  Like Adobe Acrobat Reader, there are (2) ways of executing search - cmd+f launches a pop-up window for line-by-line results whereas entering the search string in the search bar will yield a search index.

The search commences in real-time as the user types and displays a progress bar (on lower righthand corner).  The search yielded 30 hits.

Thoughts:
Preview is my least favorite.  Search results, as displayed, are not as useful as Skim and Acrobat.  Preview supports bookmarks.
Adobe is fast and accurate - a great research tool.  Adobe does not support bookmarks although the Comment/Sticky Note features can be used as a pseudo-bookmark system within the local document.
Skim is accurate and with a fantastic interface.  Skim supports bookmarks.

Currently I am using Adobe Acrobat Reader since I like having a large, separate window for searches.  However, Skim is an excellent app that was my default PDF viewer before enhancements to Acrobat made it more competitive.

Friday, January 7, 2011

Creating Smart Folders in Thunderbird v.3.1.x


I completely forgot that I had a blog.

I receive a tremendous volume of email and, for work, am tired of maintaining IMAP folders and filter rulesets except for the most voluminous of mailing lists. Especially when situations change and I'm forced to move mail around.

I like to keep everything in one big pot - the Inbox - but still want organization. Mail.app has "Smart Mailboxes" which is intuitive and easy to use (albeit sometimes slow). Essentially it is a saved search that is displayed as a folder and can comprise any number of search criteria. Thunderbird at different times referred to them interchangeably as Favorite, Smart, Virtual, and Saved Search folders.

As good as Mail.app is, however, I returned to Thunderbird for several reasons (tabbed messages being one) and sought to recreate this environment. But Thunderbird has been inconsistent across releases on how to create Smart folders, the method is not intuitive, and documentation on the subject is pitiful.

Here's how it's done using Thunderbird v.3.1.7 on OS X:

1) Initiate a search using either cmd+shift+f or from the menu bar: Edit->Find->Search Messages...
Note that searches must be performed this way and not via either the Search or Filter bars.
2) Enter your search criteria and then select "Save as Search Folder"
3) Create the Smart folder Name, specify where on the tree it will be a subfolder, and double-check the search criteria.
4) Hit OK and you're done.

Thursday, January 7, 2010

CDMA Nexus One on Sprint PCS

The new Google Nexus One smartphone running Android 2.1 is hands down the most attractive new phone on the market. The specs speak for themselves, especially the processor speed and screen resolution. Unfortunately, Nexus One is a GSM phone utilizing SIM cards while Verizon and Sprint use CDMA technology. A CDMA Nexus One is slated to arrive in Spring for Verizon but not for Sprint.

I currently have an HTC Hero Android phone (when the hell is Android 2+ going to come out with HTC Sense already?!) but crave the Nexus One. I'm not alone here.

For those of us who are Sprint PCS subscribers because of their excellent pricing plans and top-notch data service and want the Nexus One I created a petition for folks to sign. Begging for it on message boards is one thing but a focused visible petition is another.

I urge potential or current Sprint PCS subscribers to sign it:
http://www.petitiononline.com/GN1oSPCS/petition.html

Thursday, September 10, 2009

Building Hadoop and Hypertable on Debian Lenny

Hadoop and Hypertable on Debian

Environment:

Systems: (2) SUN Virtualbox 3.0.6 Virtual Machines, both Debian Lenny x86
Hypervisor system: Debian Lenny AMD64
Networking: Bridged adapter (not NAT) to eth0
g++: 4.3.2

*Note: The fact that this test deployment is on virtual machines is irrelevant to the configuration of Hadoop or Hypertable. Since the (2) virtual machines are on a shared disk, high-performance IO is not expected. However, should the testbed prove promising, dedicated systems will be deployed and high-performance tuning of both Hadoop and Hypertable will be explored.

Background:

At NSLS-II we are toying with the idea of back-ending the next generation of the Channel Archiver with a distributed database atop a distributed filesystem. This for a myriad of reasons other than it being a "cool project". For example, combining Hypertable (a high-performance distributed data storage system) with the MapReduce functionality of Hadoop promises to offer performance, redundancy, reliability, and scalability.

Goals:

Build a (2) node Hadoop and Hypertable cluster. The (2) nodes are "systemA" (master) and "systemB" (slave).

(1) Prerequisites:

*Note: Building Hadoop and Hypertable in this document requires adding the unstable and (optionally) testing repositories to /etc/apt/sources.list.

(1) Make sure to specify that Lenny/stable is the default distribution:
Edit /etc/apt/apt.conf:
# apt.conf
APT::Default-Release "stable"; # Only "stable", "testing", "unstable" are acceptable release names; i.e., "lenny" is not accepted.

$ apt-get update
$
apt-get -y install g++ cmake libboost-dev liblog4cpp5-dev git-core cronolog libgoogle-perftools-dev libevent-dev zlib1g-dev libexpat1-dev libdb4.6++-dev libncurses-dev libreadline5-dev

(2) Install Hyperic-Sigar

$ wget http://internap.dl.sourceforge.net/sourceforge/sigar/hyperic-sigar-1.6.2.tar.gz$ tar -xzvf hyperic-sigar-1.6.2.tar.gz$ cp ~src/hyperic-sigar-1.6.2/sigar-bin/include/*.h /usr/local/include$ cp ~src/hyperic-sigar-1.6.2/sigar-bin/lib/libsigar-x86-linux.so /usr/local/lib/$ ldconfig

(3) Install Thrift:

$ apt-get -y install sun-java6-jdk ant autoconf automake libtool bison flex pkg-config php5 php5-cli ruby-dev libhttp-access2-ruby libbit-vector-perl liblog4j1.2-java erlang ruby libevent-1.4-2

$ update-java-alternatives --set java-6-sun
$ ln -f -s /bin/bash /bin/sh
$ wget www.hypertable.org/pub/thrift-r796538.tgz


* Note: Since I am behind a proxy I needed to set this variable:

$ export ANT_OPTS="-Dhttp.proxyHost=192.168.1.130 -Dhttp.proxyPort=3128"
$ ./bootstrap.sh
$ ./configure
$ make


(4) Build Hadoop:

(I) Get Hadoop:

Download the latest version of Hadoop. I untarred mine in /opt and made a symlink from hadoop-0.20.1 to hadoop:

The latest version can be found at this mirror:
http://ftp.wayne.edu/apache/hadoop/core/

At the time of writing this document, the latest Hadoop was version 0.20.1

$ wget http://ftp.wayne.edu/apache/hadoop/core/hadoop-0.20.1/hadoop-0.20.1.tar.gz

You can also retrieve the latest version via git, but the Hadoop directory tree is different than is what is referred to in this document.

$ git clone git://git.apache.org/hadoop-common.git
$ git clone git://git.apache.org/hadoop-hdfs.git
$ git clone git://git.apache.org/hadoop-mapreduce.git


$ ln -s /opt/hadoop-0.20.1 /opt/hadoop

Some source needs patching in order for Hypertable to cooperate with MapReduce. This is the list of files requiring patching:
/opt/hadoop/src/c++/utils/impl/StringUtils.cc
/opt/hadoop/src/c++/utils/impl/SerialUtils.cc
/opt/hadoop/src/c++/pipes/impl/HadoopPipes.cc

Here are the patches:

(i) SerialUtils.cc:
--- SerialUtils.cc (revision 765057)
+++ SerialUtils.cc (working copy)
@@ -18,7 +18,8 @@
#include "hadoop/SerialUtils.hh"
#include "hadoop/StringUtils.hh"

-#include
+#include
+#include
#include
#include
#include

(ii) StringUtils.cc:
--- StringUtils.cc (revision 765057)
+++ StringUtils.cc (working copy)
@@ -18,10 +18,11 @@
#include "hadoop/StringUtils.hh"
#include "hadoop/SerialUtils.hh"

-#include
+#include
#include
-#include
-#include
+#include
+#include
+#include
#include

using std::string;
@@ -31,7 +32,7 @@

string toString(int32_t x) {
char str[100];
- sprintf(str, "%d", x);
+ snprintf(str, 100, "%d", x);
return str;
}

@@ -96,7 +97,7 @@
const char* deliminators) {

string result(str);
- for(int i=result.length() -1; i >= 0; --i) {
+ for(int i = result.length() - 1; i >= 0; --i) {
char ch = result[i];
if (!isprint(ch) ||
ch == '\\' ||
@@ -116,7 +117,7 @@
break;
default:
char buff[4];
- sprintf(buff, "\\%02x", static_cast(result[i]));
+ snprintf(buff, 4, "\\%02x", static_cast(result[i]));
result.replace(i, 1, buff);
}
}

(iii) HadoopPipes.cc:
--- HadoopPipes.cc (revision 765057)
+++ HadoopPipes.cc (working copy)
@@ -26,9 +26,9 @@
#include
#include
#include
-#include
-#include
-#include
+#include
+#include
+#include
#include
#include

(II) Compile Hadoop:

$ cd /opt/hadoop/src/c++/pipes
$ sh configure
$ make && make install

$ cd /opt/hadoop/src/c++/utils
$ sh configure
$ make && make install


This will place Hadoop headers in:
/opt/hadoop/src/c++/install/include/hadoop

And Hadoop libraries in:
/opt/hadoop/src/c++/install/lib

Add the location of the Hadoop libraries to ld:
$ echo "/opt/hadoop/src/c++/install/lib" > /etc/ld.so.conf.d/hadoop.conf
$ ldconfig


Next we're ready to compile Hadoop.
$ cd /opt/hadoop

*Note: Again, because I am behind a proxy, I needed to set this environmental variable:
$ export ANT_OPTS="-Dhttp.proxyHost=192.168.1.130 -Dhttp.proxyPort=3128"
$ ant compile && ant jar


(III) Configure Hadoop:

(A) My Hadoop configuration files are kept in /opt/hadoop/conf
Relevant configuration files in this test are:
hadoop-env.sh # Environmental variables
core-site.xml # Default Hadoop filesystem
hdfs-site.xml # HDFS defaults for replication, name, and datanode services
mapred-site.xml # MapReduce defaults for trackers
slaves
master

*Note: All configuration files except for "slaves" and "master" will be the same on both cluster nodes.

(i) hadoop-env.sh:

By default Hadoop prefers ipv6. Since I am not using ipv6 this required a change:
HADOOP_OPTS=-Djava.net.preferIPv4Stack=true

other variables set in hadoop-env.sh:
export JAVA_HOME=/usr/lib/jvm/java-6-sun
export HADOOP_HOME=/opt/hadoop
export HADOOP_LOG_DIR=/var/log/hadoop
export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves
export HADOOP_PID_DIR=/var/hadoop/pids

This will require the creation of directories:
$ mkdir /var/log/hadoop
$ mkdir /var/hadoop/pids


(ii) core-site.xml (where "systemA" is the master node in the cluster):
<property>
<name>fs.default.name</name>
<value>hdfs://systemA:9000</value>

<description>The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri's authority is used to
determine the host, port, etc. for a filesystem.</description>

</property>

(iii) hdfs-site.xml:
*Note: dfs.replication specifies how many nodes are in the Hadoop cluster, in our case, (2).
*Note: This configuration requires the creation of a few directories, namely, "/dfsname", "/hadoop/data". If your test system has more than (1) system disk, it is advisable to separate the DataNode and NameNode directories so as to avoid contentious IO.

<property>
<name>dfs.replication</name>
<value>2</value>
<description>Default block replication.The actual number of replications can
be specified when the file is created. The default is used if replication is not
specified in create time.</description>

</property>
<property>
<name>dfs.name.dir</name>
<value>/dfsname</value>
<description>Path on local filesystem where the NameNode stores the
namespace and transactions logs persistently.</description>

</property>
<property>
<name>dfs.data.dir</name>
<value>
/hadoop/data</value>
<description>Comma separated list of paths on the local filesystem of
a DataNode where it should store its blocks</description>

</property>

(iv) mapred-site.xml:
*Note: This configuration requires the creation of the directory "/hadoop/mapred"
<property>
<name>mapred.job.tracker</name>

<value>systemA:9001</value>
<description>The host and port that the MapReduce job tracker runs at.
If "local", then jobs are run in-process as a single map and reduce task.*lt;/description>
</property>
<property>
<name>mapred.system.dir</name>
<value>/hadoop/mapred</name>
<description>Path on the HDFS where the MapReduce framework stores
system files e.g., /hadoop/mapred/system
</description>
</property>

(v) slaves:

*Note: both systems are slaves, so both are listed
*Note: this file is only configured on the master, "systemA"
systemA
systemB

(vi) Master (just the name of the master):
systemA

(B) SSH
Enable public-key authentication on all nodes within the Hadoop cluster. This will allow the Hadoop service to log-on to other nodes and start/stop services.

I am running Hadoop as the user "hadoop".

On both machines, create the Hadoop user:
$ groupadd hadoop
$ useradd -g hadoop -c "Hadoop User" -d /opt/hadoop -s /bin/bash hadoop


On the master server, systemA:
$ su - hadoop
$ ssh-keygen -t rsa -P ""
$ cp id_rsa.pub ~hadoop/.ssh/authorized_keys


Copy id_rsa.pub over the slave system and save it in the same place: ~hadoop/.ssh/authorized_keys

(C) Local name lookup
In /etc/hosts on both machines:
Enter all Hadoop node IP addresses and remove "127.0.0.1 localhost". Have "localhost" point to an assigned IP address

(IV) Initialize Hadoop:

*Note: Initialization is only necessary on the Master node.
*Note: For convenience-sake I've added Hadoop to my path:

$ export PATH=$PATH:/opt/hadoop/bin


$ hadoop namenode -format
$ start-dfs.sh
$ start-mapred.sh


Check to see what's running on both the master and slave:
systemA $ jps
4388 Jps
28444 NameNode
28795 JobTracker
2181 main
28696 SecondaryNameNode
28575 DataNode
28903 TaskTracker

systemB $ jps
13033 DataNode
13142 TaskTracker
1675 Jps
539 main

systemA $ netstat -ptlen

(V) Create a directory in the Hadoop namespace:

$ hadoop dfs -mkdir /hypertable
$ hadoop/bin/hadoop dfs -chmod 777 /hypertable
$ hadoop/bin/hadoop dfs -ls /
$ hadoop dfsadmin -report


(5) Hypertable installation:

(I) Get Hypertable:

$ apt-get -y install git sparsehash libbz2-dev doxygen graphviz
$ git config --global user.name "First Lastname"
$ git config --global user.email "something@something.com"
$ git clone git://scm.hypertable.org/pub/repos/hypertable.git


(II) Pre-build:

(A) Fixes:

The Hypertable source requires some patching so that the build cooperates with Debian's g++. The files that need to be patched are:
~src/hypertable/contrib/cc/MapReduce/TableReader.cc
~src/hypertable/contrib/cc/MapReduce/TableRangeMap.cc

(i) TableReader.cc:
--- a/TableReader.cc
+++ b/TableReader.cc
@@ -24,7 +24,7 @@ TableReader::TableReader(HadoopPipes::MapContext& context)
HadoopUtils::deserializeString(start_row, stream);
HadoopUtils::deserializeString(end_row, stream);

- scan_spec_builder.add_row_interval(start_row, true, end_row, true);
+ scan_spec_builder.add_row_interval(start_row.c_str(), true, end_row.c_str(), true);

if (allColumns == false) {
std::vector columns;
@@ -32,7 +32,7 @@ TableReader::TableReader(HadoopPipes::MapContext& context)

split(columns, job->get("hypertable.table.columns"), is_any_of(", "));
BOOST_FOREACH(const std::string &c, columns) {
- scan_spec_builder.add_column(c);
+ scan_spec_builder.add_column(c.c_str());
}
}
m_scanner = m_table->create_scanner(scan_spec_builder.get());

(ii) TableRangeMap.cc:
--- a/TableRangeMap.cc
+++ b/TableRangeMap.cc
@@ -28,7 +28,7 @@ namespace Mapreduce

startrow = tmprow;

- meta_scan_builder.add_row_interval(startrow, true, startrow + "\xff\xff", true);
+ meta_scan_builder.add_row_interval(startrow.c_str(), true, (startrow + "\xff\xff").c_str(), true);

/* select columns */
meta_scan_builder.add_column("StartRow");

(B) Hypertable config

Edit ~src/hypertable/conf/hypertable.cfg and enter the information about the Hadoop Master:
# HDFS Broker
HdfsBroker.Port=38030
HdfsBroker.fs.default.name=hdfs://systemA:9000
HdfsBroker.Workers=20

(III) Build Hypertable:

Assuming that Hypertable has been unpacked in ~src then
$ mkdir ~src/build/hypertable
$ cd ~src/build/hypertable


*Note some important variables that need to be set in order for a successful compile on the Debian platform:
HADOOP_INCLUDE_PATH = /opt/hadoop/src/c++/install/include
HADOOP_LIB_PATH = /opt/hadoop/src/c++/install/lib
JAVA_INCLUDE_PATH = /usr/lib/jvm/java-6-sun/include
JAVA_INCLUDE_PATH2 = /usr/lib/jvm/java-6-openjdk/include

$ cmake -DBUILD_SHARED_LIBS=ON -DHADOOP_INCLUDE_PATH=/opt/hadoop/src/c++/install/include -DHADOOP_LIB_PATH=/opt/hadoop/src/c++/install/lib -DJAVA_INCLUDE_PATH=/usr/lib/jvm/java-6-sun/include -DJAVA_INCLUDE_PATH2=/usr/lib/jvm/java-6-openjdk/include ../../hypertable

$ make -j <number of cores>
$ make install
$ make doc


(IV) Had issues starting up Hypertable:
: error while loading shared libraries: libHyperThriftConfig.so: cannot open shared object file: No such file or directory

Temporary fix:
*Note: this is a stop-gap since this library is linked to files in the source. Don't delete the source. Need to fix this.
$ cp ~src/build/hypertable/src/cc/ThriftBroker/libHyperThriftConfig.so /opt/hypertable/0.9.2.6/lib/

(IV) Resolve issue with Hypertable connecting to Hadoop:
Replace the Hypertable-Hadoop jar with Hadoop's jar:
$ cp /opt/hypertable/0.9.2.6/lib/java/hadoop-0.20.0-core.jar /opt/hypertable/0.9.2.6/lib/java/hadoop-0.20.0-core.jar.hypertable
$ cp /opt/hadoop-0.20.1/hadoop-0.20.1-core.jar /opt/hypertable/0.9.2.6/lib/java/hadoop-0.20.0-core.jar


(V) Initialize Hypertable:
*Note: Added Hypertable to path:

$ export PATH=$PATH:/opt/hypertable/0.9.2.6/bin
$ start-all-servers.sh hadoop


(V) Hypertable Scripts (reference):

start-dfsbroker.sh (local|hadoop|kfs) [<server options>]
start-hyperspace.sh [<server options>]
start-master.sh [<server options>]
start-rangeserver.sh [<server options>]
start-dfsbroker.sh hadoop
clean-database.sh

And a wrapper script to start all services:
start-all-servers.sh (local|hadoop|kfs) [<server options>]

Enjoy!

Friday, August 21, 2009

Debian dbus - ldap error messages on boot

I recently updated a system using LDAP for authorization/authentication and happily discovered it wouldn't reboot. The boot screen was scrolling ad infinitum with these helpful messages:
udevd: nss_ldap failed to bind to LDAP server .....

Quick search displayed many bug reports, obviously not fixed since I'm endured this on a freshly upgraded Debian Lenny system. Some comments I read were silly, like folks having the system boot with a nsswitch.conf using only "files" for passwd, group, and shadow, then switching to "ldap files" or "compat" after boot.

Fix for me was (2) steps:
1. Leave /etc/nsswitch.conf the way it was -- "compat ldap" only and add the problematic user to the LDAP server. And the problematic user is [drumroll] messagebus. All that did was prevent the infinite udevd messages on boot -- still errors though.

2. Edit /etc/libnss_ldap and add:
bind_policy soft

Server boots, fixed, no error messages, done, back to work...

Thursday, August 20, 2009

OpenNMS and Apache2 on Debian Lenny

OpenNMS and Apache2 on Debian Lenny 5.0

What's the motivation? Jetty as built via OpenNMS does not have a configurable cipher suite or at least an obvious and/or intuitive method that wouldn't necessitate web crawling. Weak ciphers create noise on a Nessus scan listing it as a medium-level vulnerability. I found (2) how-tos on the OpenNMS wiki and both of them entailed using Jetty with AJP support. I wanted something simpler.

1. Edit /etc/opennms/opennms.properties and uncomment this line:
opennms.web.base-url = https://%x%c/

Restart OpenNMS

2. Install mod_proxy for Apache2 and add module:
$ a2enmod proxy

Beware that the default Debian proxy configuration disallows all proxy access. My installation required me to loosen it up a bit.
Edit "/etc/apache2/mods-available/proxy.conf", get rid of "Deny from all" and add:
Allow from 127.0.0.1/8 192.168.90.0/24

3. Add the virtual host entry for OpenNMS on Apache. Edit "/etc/apache2/sites-available/default-ssl" and add:

ProxyPass http://127.0.0.1:8980/opennms
ProxyPassReverse http://127.0.0.1:8980/opennms


Done. Everything works fine. Nessus is happy. Deployment secure.

Tuesday, January 27, 2009

Cray pas haunted house

I really like drawing with Cray pas. I don't feel any pressure to achieve something great when working with them. It is not technically demanding like, say, painting. I've tried my hand at painting and found it to be frustrating since my ideas never map themselves to the brush.

My daughter likes haunted houses...

Friday, January 23, 2009

Cray pas -- first go

Last night my older daughter was real demanding. In fact, the evening was so emotionally draining that after everyone went to bed I decided to grab a big piece of paper, sit on the floor with a box of cray pas, and draw a picture. It was actually cathartic. Here it is...

Thursday, January 22, 2009

Mind mapping software: XMind vs. FreeMind vs. MindManager vs. Inspiration

I've always had a disorganized mind brimming with ideas and while it has served me well over the years I felt it was time to 1) optimize the way I think, 2) delineate and partition work, and 3) preserve state. Why do I need to preserve state? When I get distracted or pulled off on some wild tangent I need to focus quickly again. Likewise, when dragging myself into the office on Monday mornings I want to pick up precisely where I left off on Friday. Finally, and maybe it's because I'm getting older, I spend more time thinking about what I've been thinking about. It's a sort of vicious cycle wherein the moment I'm situated to perform a task, right when I've knifed fresh paints on the palette, I have to go to the restroom or get a coffee or eat lunch and all is lost.

Enter mind mapping. I've spent a considerable amount of time this past week evaluating various mind mapping software on OS X. The candidates were MindJet MindManager 7, Inspiration, FreeMind, and XMind

Here's a summary of my impressions:

Inspiration:
  • Old MacOS9 look and feel
  • Costs $$$$ -- but why should I pay that for a something that feels so outdated?
  • Crashed twice and felt somewhat unwieldy for very large corporate or engineering projects
  • Very very nice outline mode handy for cut and paste right into a doc or email.
  • Might consider the Kidspiration for the youngins
MindManager7 for Mac:
  • $$$ but the newer MM8 is only available on Windows
  • Frequently crashed (due to evaluation? Doubt it)
  • Overview mode was annoying with no automatic way of expanding every node in the tree
  • Nice interface and keyboard shortcuts
  • Good documentation
  • Supports floating nodes
  • Worked quite well -- felt productive from the start.
FreeMind:

  • OpenSource. This is the product I wanted to like the most being a fan of the opensource community.
  • Full export
  • Cloud functionality mimics XMind boundaries.
  • Text mode is somewhat clumsy -- how does one delete the text icon when there is no longer any text without deleting the node itself?? This is precisely the sort of information I don't want to waste time digging around for.
  • Documented key-mappings for Mac didn't always match reality
  • Overview mode?
  • Annoying options menu -- especially choosing default colors
  • No floating nodes and difficult to place nodes where you want them (they're always snapping back to the way FreeMind wants it)
  • Somewhat primitive look and feel
  • I took some time before I felt truly productive with this product



XMind:


  • Some components OpenSource
  • Extremely polished, friendly interface
  • Good documentation
  • Floating nodes
  • Intuitive -- felt productive almost at once.
  • Easy to both add and delete notes using function+F4
  • Compact legend of key-bindings instantly accessible via ↑⌘L
  • Optional tri-pane window featuring outline view and properties
  • Cool Boundary and Summary functions
  • Nice auto-styling like multi-branch coloring and line tapering
  • Flexible node styling such as rounded, rectangle, callout, fishhead
  • Useful templates
  • Can't export to PDF in the free version


Ranking: 1) XMind, 2) FreeMind, 3) MindManager, 4) Inspiration.

XMind just wins hands-down in $$$, ease of use, form, and function. I would even consider purchasing the yearly subscription for some advanced features (Gantt charting) @ $49 a bargain.

Freemind is nice, too, but not as polished, full-featured, or easy to use as XMind. As mentioned before, I like opensource community projects, but in the end I need to get work done.

Shortly after XMind was made available at no cost, one of the FreeMind developers initiated a thread on the FreeMind mailing list asking whether or not it was worthwhile to continue that project (see here). Obviously the answer should be yes -- why crumple in the face of competition? Reading through the thread, however, I noticed a lot of commentors stated that while XMind is good, it's also slow whereas Freemind is lean and fast and, as such, Freemind is more suitable for quick off-the-cuff mapping. While it is true that Freemind is a bit spryer I feel this is negligable -- at least on a modern desktop.

Wednesday, December 17, 2008

Keeping my iTunes Library on an external hard drive across systems

Real simple: I have both an Apple desktop and laptop with limited disk space and want all my iTunes media to reside on an external HD that I can switch between systems.

This is fairly straight-forward.  In "Advanced Preferences" I changed the "iTunes Music folder location" to my external drive (in my case:  WD Passport/iTunes:).  This worked well for me until today.  Now to backtrack a little, when I first set-up this schema I even symlinked the ~/User/Music/iTunes directory to my external disk with the notion that I'd even want " iTunes Music Library.xml" (the dictionary file that maps every file in iTunes) on the external drive.  Ultimately this was stupid because if the external drive was not connected I couldn't fire up iTunes to listen to, say, the radio, without the application wanting to create an new library (equivalent to launching iTunes while pressing alt).

But that was then and this is now.  I'm not sure why my iTunes directory got screwed today but it did.

What happened was I accidentally "reset" the "iTunes Music folder location" which defaulted to my local hard drive.  When I tried to add the external hard drive location, the iTunes directory was greyed out (grayed out). Greyed out meaning I couldn't add it.  Then I attempted "Add to Library" (command+o) and the external iTunes directory was also greyed out.  WTF??

At first I thought there was some defaults com.apple.iTunes RootDirectory entry I could fool with like I did when I wanted to default my iPhoto Library to the external hdd some time ago.  But no go (another case of an OS X inconsistency).

It wasn't until I thought about doing something inane with permissions that I noticed that the iTunes directory on the external disk was an iPhoto-esque package content directory.  "Get Info" on iTunes even listed "open with iPhoto".  Hmmm. I stupidly changed it to "open with iTunes" but that was low-brow desperation.

Opening a shell and doing a "ls -lad" on iTunes showed:

drwxr-xr-x@ 20 rpetkus staff 680 Dec 17 19:40 iTunes/

What is "@"?  "man ls" informs me that @ = extended attributes which I can list with "-@"

$ ls -lad@ iTunes/

drwxr-xr-x@ 20 rpetkus staff 680 Dec 17 19:40 iTunes/
  com.apple.FinderInfo 32

What is com.apple.FinderInfo? Turns out this is a Uniform Type Identifier (UTI) which ascribes a lot more meaningful type data to a file than a mere file extension (.jpeg) or MIME type.  I found this older link which was a informative read about UTIs on OS X.  

After my reading assignment I surmised that I want to get rid of this extended attribute using the "xattr" command which doesn't have a man page but "xattr -help" is pretty self-explanatory.

Make iTunes a normal folder again:

$ xattr -d com.apple.FinderInfo iTunes/

Viola! iTunes on my external hard drive is no longer greyed-out, I can set it as my Music Folder location and "Add to Library".  Resolution.

As a side note, if you're doing the same thing with your external hdd, make sure under "Advanced Preferences" that you select "Copy files to iTunes Music folder when adding to library".  This way, if you add more media to iTunes on one system with the external drive, it indeed gets copied there for consumption on the second system.

Monday, December 1, 2008

Solaris 10, Kerberos, and OpenLDAP

Solaris 10 - Solaris in general - annoys me and Linux is favored for all instances except those where I need to reap the benefits of ZFS. Today as a precursor to deploying SUDO on a number of Solaris systems, I sought to eliminate ssh-key access and allow individual accounts to logon. Piece of cake in Linux but Solaris?

I want to use kerberos for authentication and ldap for authorization. I'm using OpenLDAP and not SUN LDAP so this already creates a bit of chafe, but I'm not going to install SUN LDAP or install a 3rd party PAM module - I want everything to be default so the next person to inherit this mess doesn't go mad.

Now I don't want to use pam_ldap for account authorization since this implies 1) I'm using SUN LDAP and 2) I'm storing account information like PASSWD in LDAP. We don't.

I *do* want to use vanilla pam_unix_account and rely on nsswitch.conf to state my name service preferences.

Now one would assume, given my requirements, that a reasonable SSH PAM stack would look like this:
other auth requisite pam_authtok_get.so.1
other auth required pam_dhkeys.so.1
other auth required pam_unix_cred.so.1
other auth sufficient pam_krb5.so.1
other auth required pam_unix_auth.so.1
other account requisite pam_roles.so.1
other account required pam_projects.so.1
other account sufficient pam_unix_account.so.1

Then why doesn't it work???? Why, in debug mode, does it always complain that account so and so is not found?

After some investigation it turns out that if using OpenLDAP for authorization one needs to add "objectClass=shadowAccount", and that's it, as an attribute to every single uid=xxx,ou=People,dc=place,dc=org. At least for those users that want to be authorized for login.

What a pain. And how much precious time did I waste on this? 2, maybe 3 hours? Unbelievable.

Monday, November 17, 2008

The Great Farce Part I

About 2 years ago there was a major change at my lovely place of employment -- out went the Old Director (OD) and in came the new. The new fellow was much vaunted, real tech-savvy, vocal and strong. New Director (ND) had actually helped develop a piece of software that we consumed heavily and this implied that ND was relevant and valuable. OD was swell but perhaps ill-suited for the prominent Director position as he was burdened with a modicum of introversion. The daily trade was plied in the confines of his office and rarely did he speak, if at all, to the common folk that produced many of the real, tangible items that were consumed by the facility. Nevertheless, OD was both well-respected and well-liked. He allowed only a close coterie of esteemed folks - the upper echelon - to influence and provide council. But at the time, occupying a lower caste, I had to be content lapping at the meager drip of information which fell in my pan.
Then, one day, without warning, word spread that OD was to retire from his storied post. Since OD rarely communicated anything to the larger group, the advertisement for the ND was discovered in a vocationally relevant circular. Change was in the wind and would arrive in the Spring.
About a month or so before the arrival, there were a series of secret meetings amongst the mid and upper-level managers. It was crucial to strategize earlier rather than later. Would the pecking order be preserved, could new territory be up for grab? Old allegiences were strengthened and enemies were kept close. There would be alignment with the new power at all costs. Every ounce of data obtained about ND was parsed for clues and suggestions from the most heralded dissertation to the meagerest plea for help on a mailing list. And yours truly? I was cynical yet optimistic. I was hoping for my chance; a meritocracy.
Enter Spring. It was buzzing that a wholly new organizational chart had been devised. There was an endless queue of covert meetings between ND and the mid/upper level managers. Decisions were being made and, as far as I knew, the common folk that produced real things, such as myself, were never consulted. It was leaked that, at the behest of ND, a quasi-famous Industrial Psychologist (IP) had been hired to analyze the staff, make assessments, and present reportage and council to ND. It was at this point that I become uneasy and warn't too shy about it, neither.
IP had authored some books and wrote vignettes for public radio. IP was disarming enough, as I suspected would be the case. Each staff member was allotted one hour of couch time. Some folks strode in there thinking this was their big opportunity to be an instrument of change, some were paranoid and promised to present to IP an inpenetrable shell, and one particular clown read up on industrial psychology ahead of time, picked up a bit of argot, and sought to turn the tables. I wasn't interested in any of that. I resolved to be forthright and helpful but I wasn't in the mind to put my cards on the table, so to speak. I was more interested in listening to the type of language employed by IP and keen to identify the triggers that implored me to lower my guard and spill forth -- knowledge of those triggers and techniques could be useful.
Upon completion of all the interviews, IP anonymized and categorized the data into a series of talking points and presented it to the staff during a lively 2 hour group meeting. Common issues and concerns were bulletted and reviewed. I list some specimens that will become relevant later on here:
1. People's roles and responsibilties are either ill-defined and nebulous so as to mean nothing or they are too specific and thus constrictive.
2. People want a more transparent organization, one where decisions and logic are communicated and where input is encouraged and well-regarded.
3. People want the opportunity to become more involved, redefine their roles, and grow professionally.

To be continued...

Friday, November 14, 2008

The Mens Room

Here are things that piss me off in the mens room:
  • Talking on cell phones, especially while in the stall
  • People afraid to urinate at the urinal and use the toilet instead - worse when they leave the stall door open
  • People that, given an entire row of empty urinals, chose the one next to yours
  • People that can't urinate unless they keep flushing the urinal
  • People that make a show of excessive shaking, tugging, and groaning upon completion
  • People who drop pants and even worse, underwear, to the floor at the urinal
  • People that stand and have a lengthy conversation after the business is finished
  • People who leave the stall and don't wash their hands
  • People who leave the door knob wet when they leave

CHEP 2009 - Prague

I've always wanted to go to Prague and it finally looks as if the planets are coming into alignment. I just submitted (3) abstracts for the CHEP conference which will be held 3/21-27, 2009. CHEP stands for Computing in High Energy and Nuclear Physics and it's an opportunity for folks such as myself to present papers and give lectures on both the mundane and extraordinary components of our own little computing context. This particular conference is held every (2) years, I think, and carries with it a bit more prestige than the bi-annual HEPiX meetings.
I've always had a romance for Prague -- I imagine it to be a labrynthine, stoney grey web of spires and mystery in the spirit of Kafka. In the 1991 Soderberg movie, Kafka, Prague was an intensely dark and murderous place. The movie rehashed many of Kafka's own themes: isolation, transformation, paranoia, and institutional oppression. I especially liked when the movie moves from black and white to full color once Kafka (Jeremy Irons) enters the Castle -- with all the appropriate elbows and winks to Dorothy in Oz.
Then I recall reading something about Einstein, Kafka, and Freud hanging out in Prague but I don't know if that's real or just a myth...

Anyway, here are the abstracts -- it took a minimal of effort to write them up -- if they get accepted then I'll craft something good.

Title: dCache Storage Cluster at BNL

Abstract content
Over the last (2) years, the USATLAS Computing Facility at BNL has managed a highly performant, reliable, and cost effective dCache storage cluster using SunFire x4500/4540 (Thumper/Thor) storage servers. The design of a discreet storage cluster signaled a departure from a model where storage resides locally on a disk-heavy compute farm. The consequent alteration of data flow mandated a dramatic re-construction of the network fabric.
This work will cover all components of our dCache storage cluster (from door to pool) including OS/ZFS file-system configuration, 10GE network tuning, monitoring, and environmentals. Performance metrics will be surveyed within the context of our Solaris 10 production system as well as those rendered during evaluations of OpenSolaris and Linux. Failure modes, bottlenecks, and deficiencies will be examined.
Lastly, we discuss competing architectures under evaluation, scaling limits in our current model, and future technologies that warrant close surveillance.

Presentation type (oral | poster)
Oral

Primary Authors:
PETKUS, Robert (Brookhaven National Laboratory)

Co-authors:
KARASAWA, Mizuki (Brookhaven National Laboratory)
MCCARTHY, John (Brookhaven National Laboratory)
SMITH, Jason (Brookhaven National Laboratory)

Abstract presenters:
PETKUS, Robert

Track classification:
Hardware and Computing Fabrics



Title: Building a Storage Cluster with Gluster

Abstract content
Gluster, a free cluster file-system scalable to several peta-bytes, is under evaluation at the RHIC/USATLAS Computing Facility. Several production SunFire x4500 (Thumper) NFS servers were dual-purposed as storage bricks and aggregated into a single parallel file-system using TCP/IP as an interconnect. Armed with a paucity of new hardware, the objective was to simultaneously allow traditional NFS client access to discreet systems as well as access to the GlusterFS global namespace without impacting production.
Gluster is elegantly designed and carries an advanced feature set including, but not limited to, automated replication across servers, server striping, fast db backend, and I/O scheduling. GlusterFS exists as a layer above existing file-systems, does not have a single-point-of-failure, supports RDMA, distributes metadata, and is entirely implemented in user space via FUSE.
We will provide a background of Gluster along with its architectural underpinnings, followed by a description of our test-bed, environmentals, and performance characteristics.

Presentation type (oral | poster)
Oral

Primary Authors:
PETKUS, Robert (Brookhaven National Laboratory)

Co-authors:
SMITH, Jason (Brookhaven National Laboratory)

Abstract presenters:
PETKUS, Robert

Track classification:
Hardware and Computing Fabrics


Title: Log Mining with Splunk

Abstract content
Robust, centralized system and application logging services are vital to all computing organizations, regardless of size. For the past year, the RHIC/USATLAS Computing Facility (RACF) has dramatically augmented the utility of logging services with Splunk. Splunk is a powerful application that functions as a log search engine, providing fast, real-time access to data from servers, applications, and network devices. Splunk at the RACF is configured to parse system and application log files, script output, snmp traps, alerts, and has been integrated into our Nagios monitoring infrastructure.
This work will detail our central log infrastructure vis-`a-vis Splunk, examine lightweight agents and example configurations, consider security, and demonstrate functionality. Distributed Splunk deployments or clusters between institutions will be discussed.

Presentation type (oral | poster)
Oral

Primary Authors:
PETKUS, Robert (Brookhaven National Laboratory)

Co-authors:
SMITH, Jason (Brookhaven National Laboratory)
RIND, Ofer (Brookhaven National Laboratory)

Abstract presenters:
PETKUS, Robert

Track classification:
Software Components, Tools and Databases

Thursday, November 13, 2008

Social Bookmarking

There is simply too much information and interesting content on the web and not enough hours in the day to waste precious time in front of a monitor. I lead a productive work life and only allow myself small punctuations throughout the day for surfing. Now I've always been behind the trend so I'm beginning to explore vehicles that real technophiles had already spun and crashed years ago: social bookmarking sites.
It's a natural progression for yours truly. At first, many years ago, I'd set my home page to a web portal like Excite! or Yahoo, which I eventually customized to include the news and stock tallies, etc., and revisit throughout the day. Later on I made a ritual out of visiting a selection of bookmarked sites that offered the perspectives I sought during the day: NYTimes, Register, Onion, Slashdot. Then RSS and ATOM made this process simpler and more attractive since I could have information pushed rather than pulled.
But the problem was always the same -- unless I was willing to spend an inordinate amount of time spelunking the web, I would never be exposed to new and innovative sources of interest. This began to change when I began to utilize Del.icio.us and Google Reader. The great thing about Del.icio.us was that once I began to populate my account with all the random bookmarks I deemed valuable I was immediately exposed to others who were interested in the same thing. I could then browse the other, similarly categorized bookmarks these far-flung siblings had chosen to cherish. In effect, I was able to benefit from the research, googling and web-mining of others. I discovered virtual soul-mates who were prolifically bookmarking previously unknown and valuable web pages. I would parasitically attach myself to them via anonymous RSS subscription. For all I know they were attaching themselves to greater source. Perhaps every user of Delicious is merely a decomposer fed from a single autotroph.
Google Reader, a web-based aggregator, is a different animal. A user essentially populates their reader, an empty canvas, with a selection RSS and/or ATOM feeds (the selection is Google-vast), and reads articles through the interface. The application tracks and tallies what you have and haven't read, what you've labeled worthwhile, and begins to suggest new sources of information. The more you use the system, the more intelligent the suggestions. Again, this has had an impact on my daily consumption of information and I appreciate the exposure I receive to different sources of data.
There is something sinister about these tools. In effect, I am allowing myself to be observed and studied. I'm confident dissertations will be penned exploring the reduction of human psychology via web trend analysis. I imagine a superior advertising product will be engineered based on the information obtained when folks like me use a system such as this on a regular basis. But this is another matter.
Today, my attention has been drawn to social bookmarking sites such as Digg and StumbleUpon. I've created accounts with both services and intend on giving them a full evaluation. Will these tools expose me to the information I seek with minimal work? Will I be able to reclaim more hours of the day and spend them away from a monitor? Will I be fed information that I don't necessarily want to read but am meant to read? Only time will tell.

The Farmingdale Library web site isn't entirely lame but alisweb.org is

Yesterday I mentioned how, at first blush, the Northport Library website looked superior to my local digs, the Farmingdale Library. You'll recall how I was smitten by the ability to manage your own account online. Then the final verdict that the NPPL site was really just a heap of bolus draped in semi-professional CSS glaze. Let's cover new ground and make some corrections.
The Farmingdale Library does allow patron account access. I can view material currently checked out, renew online, request inter-library loan (with email notification), and even opt into a system wherein all my activity is tracked for personal analysis (I opted). They use a system utilized by all libraries in Nassau County called the Automated Library Information System Web Catalog, or Alisweb.


This is what it looks like -- as you can see I'm overdue on a number of items.




Now I'm a severe library buff and so are my kids. There was a time many years ago when I would spend many hours and dollars in Borders-n-Barnes&Nobles. Strand was a mecca and old George Lenz in Huntington had all the literary gems (Faulkner, Proust, Joyce, Nabokov) in the ever-so-hip 8x5.2 sizing for $8-10. As a kid my room was stuffed wall to wall with bookcases, books piled 2-3 levels deep, falling off the shelves, and atop every piece of furniture.
Then everything changed when I had kids. Even before progeny one breathed her own air I began building a mini child library. Until I realized that the majority of my selections were flops: Daddy, I don't like this book. I simply cannot predict what my kids are going to like. The library changed this -- I've donated a significant portion of my collection and simply see no reason why I would ever need to purchase a book again. I'm really referring to recreational reading books here, not reference or professional volumes.
The public library evokes all the sentiment it held when I was in elementary school. Now I recall with embarrassment how I had all those $20 Stephen King hardcovers during junior and high school -- did I ever ever read one of those books more than once?

But I'm supposed to be speaking about alisweb. Cool idea -- horrible implementation. In order to access your account one needs to enter 1) username, 2) 14-digit library card barcode, and 3) 4-digit pin. The site doesn't support encryption which is real annoying but I figure it's not like I'm entering my bank account password. At worst, someone can request a bunch of books on my behalf and cause some annoyance for me. Otherwise, I'm not taking out how-to books on nail pipe-bomb or pvc silencer manufacturing so I go ahead, enter the information, close my eyes and inject my clear-text library genome straight into cyberspace.
Since it's not SSL, the Safari browser can't store the password and Firefox only stores the 4-digit pin -- I can remember the 4-digit pin, it's the 14-digit number that's the problem! Now get this: Every time you navigate away from "personal settings" and back again, you have to re-enter everything again -- including the 14-digit number! What a piece of garbage. I want to contact Innovative Interfaces, Inc. and tell them how uninnovative their interface is.

Here is my open letter to Innovatice Interfaces, Inc.:
Please consider SSL and credential management.

Wednesday, November 12, 2008

Northport Library - Luminescent Orchestrii - old people

A friend told me that "Luminescent Orchestrii" would be playing at the Northport Public Library at 7PM.  Since my 3-year old is learning to play the violin and the band includes (2) female violinists in the front perhaps I would be interested?  Indeed I was.  On their MySpace page, the group is self-described as "Gypsy Tango Klezmer Punk".  I wasn't exactly enamored by the clips I heard but figured a group such as this is best appreciated live.  I also hoped the exposure would broaden my daughter's appreciation of the violin, allowing her view it from a different lens. 
Alas, the performance was cancelled at the last minute.  I was maybe 10 minutes from the library, having rushed dinner before fumbling around the house for last minute essentials in a storm of panic and profanity.  Which library is it at, Northport or East Northport?  The libraries share the same website and pdf bulletin.  There was no distinction made regarding the location of the performance.  
/* Side note: at first blush I thought to myself, Wow, this library has a nice website -- much better than my local Farmingdale library.  Look Honey, I hollered from the kitchen table excitedly, members can log in and track the books they've taken out!  But that was a superficial assessment.  The web site sucks.  Basic information like, Calendar of Events, is obfuscated.  Useful information is obtained by downloading a pdf of the most recent bulletin, a decoupage of clip art and goofy fonts.  */
Another great tragedy is that I had my wife last-minute-cancel dinner with the mother-in-law in order to make the gig.  The wife was appropriately disgusted about the cancellation suggesting this was yet another in a long series of atrocities committed by yours truly.  The most fabled of all was when my mother-in-law took it upon herself to invite her son and his (2) rowdy boys to participate in a simple birthday ceremony I organized for my daughter that was intended for nuclear family consumption only.  These are the sort of kids that were bound to put a damper on things -- a duo of sullen, surly boys that are completely unresponsive to authority.  Anyway, I had to call up both mother and brother-in-law and do the uninviting that sad day.  And from that moment on I ceased to be a team-player and was labeled capricious and tacky.
Back to tonight and Luminescent, my wife made it clear that since she was unhappy about the cancellation she would refuse to have fun and resolve herself to a disposition that was grey and mealy.  After dinner and last minute panic, I call the library whilst driving and ask if Luminescent Orchestrii was playing that night.  That was intended as a rhetorical question designed only to give context to my real question: Are they playing at the Northport or East Northport Library?  But even this was a challenge.  I'm at a red light, I'm running 15-minutes late, I have (2) addresses.  I need to finger one of those addresses into the GPS before the light turns green and I have to drive on a dark highway.  What, the Illuminati? is the response.  No, Luminescent Orchestrii.  Let me see, she says.  Long long pause -- the other traffic light turns yellow -- my phone's low battery indicator beeps -- Hello, is anyone there?  No, sir, I don't have any such event scheduled for tonight.  Goddammit, I say, I know for a fact the band is playing but I need to know where -- put me on the phone with someone who knows things.
But like I said before, the band cancelled.  I decided to go to the library anyway -- maybe go to the children's section and let the kids traipse about.  Upon entering, I play dumb and ask the front desk attendant where Luminescent is performing and hear of the cancellation; a band member's mother had past away.   However, another act has filled the void -- go downstairs and see.  A gnarled finger, white as whale bone, points to a downward flight of stairs, the Ghost of Christmas-yet-to-come.  I hear faint music and what sounds like a cow bell.  We all follow the gaze of the specter and enter this room filled with old people.  There may have been young people there but only the old ones stood out.  They were that breed of old people I expect attend every free event offered at the library -- concerts, seminars, movie afternoons.  Every fold out metal chair is occupied.  The faces are disconsolate and drawn.  They look at me with plaintive eyes because they are trapped in the midst of an incomprehensibly lame performance and for whatever reason felt obliged to sit and suffer.  For the sake nobility?  Who can say.  I don't know the fellow's name, and even if I did probably wouldn't write it here to avoid offense, but he's a jolly Bert one-man-band sitting down with some home-made instrument resembling a pedal steel but not worth a tuppence in tone.  He has a small drum set, too -- snare, bass, tom and ride cymbal.  Unseen from my vantage, an incessant triangle or bell.  
A woman came to my side and sympathetically whispered in my ear that this wasn't Luminescent Orchestrii and I didn't have to stay.  Gee, thanks for the permission.  When I left the room, I heard (2) library workers bemoaning the fact that everyone inside looked like prisoners, that they weren't obligated to stay, and were trying to work a strategy where they could make a non-offensive announcement giving people the freedom the leave.  I laughed to myself.  All those poor old folks.  Under what circumstance would they deem it safe to leave?