Setting up RAID for a Mac Pro

Over the last few months collected big volumes of data for processing. For instance ~1.2TB BLOG survey data or data from Twitter. Even the best performing single drive I've got, cannot achieve transfer rate good enough for a reasonable in time data decompression and the followed database solution. Not to mention a drive failure would be devastating. Had to look for another option.

As it turns out, network attached storage (NAS) may be good for archiving, but NAS is not for a real time database solution. Direct attached storage (DAS) options were needed. Unfortunately, there is not that many for Mac Pro. The Apple RAID card only uses the 4 internal drive bays. There is no other connectivity options with it. Not to mention it is too expensive, so it was ruled out almost immediately.

While digging, I came across MaxUpgrades web site. While the site is terrible, the company has interesting products. It is worth checking the the site if you happen to be a Mac Pro owner. One of the products that deserve attention is SAS/SATA back plane attachment. The attachment allows connecting internal bay drives to a third party RAID controller instead of the main board. There was hope.

After some more time online I came across HighPoint Technologies. They offer RAID controllers with external, internal or both SAS/SATA connectivity options. The cards have support for Windows, Mac OS, Linux. They also seem to perform better than the Apple RAID card according to http://www.barefeats.com/hard120.html. The price is reasonable: end up getting a 4322 card for ~$430 from newegg.com:



The next thing to do was deciding on a hard drives for the controller. It would not have been easy if it was not for this high-end drives comparison chart. I was looking for a hard drive with capacity around 1TB. While not the best option for a RAID, I just did not want the additional expense and storage space for a second hard drive enclosure. After considering both price and user feedback, I end up getting 4x SAMSUNG HD103SJ drives from different online stores in attempt to minimize the likelihood of two drives in the array doing down simultaneously.

The last (I thought) piece of equipment was the drive enclosure. This one was though. A lot of time wasted. All the enclosures I found were way overpriced for what they were. After all, the enclosure it is just a box with fan (or two) a power supply unit. Despite this fact, the price tag is from $150 going up to $300-$400. Thinking in retrospect, I might have done far better getting a small computer box instead of a drive enclosure. Anyway, after hours of browsing I ordered the only good looking not so expensive one: SATAVAULT Ultra Fast Quad Bay Open Bay 5.25Inch Aluminum Case!


Since I orderred the last item in the store, the store offered a $50 discount for scratches on the box outside. All that prior to having the order shipped (the store has me; shopping again at USBGear). When the box arrived, I went to the local Microcenter looking for 5 1/4' to 3 1/2' brackets. Found some ugly plastic ones for total of $20. Not so bad, but again somewhat overpriced, considering it was plastic and it takes seconds to manufacture one. With the drives in the enclosure, the RAID controller connected to the drives I turned the box switch on...

The box was noisy. It sounded like my 9 years old DELL Pentium III PC. It was a huge disappointment after the absolute Mac Pro silence. But I did not do anything for a week. I was happy. The driver installation and the array configuration was extremely easy. The performance seemed good, though there were surprises after booting under x64 Windows 2008 Server. Figuring out how to best use the RAID with a SQL Server running in a virtual machine was not straightforward, but after a lot of reboots, different file system partition formats it was all going fine.

In a week time, I decided I could not stand the noise anymore. After some digging, found endpcnoise.com. Ordered a 80mm PAPST 8412NGLLE fan (measured at 9dB when working) and some acoustic noise absorbing foam blocks for the box inside. It worked fine, but still there was something troubling: the two 40mm power supply fans. There was a follow up endpcnoise order. This time was for two 40mm Scythe SY124010L fans (14dB) . Withe the fans, after disassembling the box, losing the warranty on the power supply, and assembling the box there it was: the perfect hard drive enclosure! It is a little bit noisier than the Mac Pro, but still goes unnoticeable when powered on.

---
While replacing the original drive enclosure fans, my concern was airflow. Quiet fans generally have lower RPM and lower airflow rate as result. The acoustic foam installation may further disrupt the airflow for some or all drives. So, when keeping an eye on the drive temperature I wasted few minutes looking for a healthy hard drive temperature range. An interesting paper with somewhat unexpected results was published by Google and can be found at
http://labs.google.com/papers/disk_failures.pdf. Hope you enjoy reading it.

---
In conclusion, I wonder how much time would have been saved if the manufacturer had the obligation to report the overall noise level of each mechanical unit shipped. Seems beneficial. Can it be enforced somehow?!

---
All the links in this post have been recorded/extracted with my local PeerBelt ranking and retrieval browser module.

The end of nature

In the last few days my wife is reading Bill McKibben's The End of Nature:

Couple of days ago my 3 year old daughter saw the book cover and corrected my wife: "You are not reading it right. Hold it this way: "

What I find striking about the story is with our age, education, habits we gradually lose the ability to recognize simple, elegant solutions. Seems like few people praise these solutions anyway. The majority just follows eventually. Unfortunately, by the time this happens, it may be too late for the solution to be implemented. Something to keep in mind... A lesson learned from a three year old.

On a related note, my daughter's comment force me to read again the Antoine de Saint-Exupéry's Little Prince. He did a great job looking at the world from a different point than the every day reference point we take out of habit.

In conclusion, we need to evolve or we may lose it all.

Another week + Halloween

Another week with just steady progress. Though moving toward a complete Peer Belt solution, nothing exceptional has happened...

Unless one counts the Halloween maze prepared yesterday at the Harisson plaza Taekwondo school. The maze with the strobe lights, coffin, fog, live zombies makes the school a kids' favorite destination October 31st each and every year thanks to Master Myre and Instructor Cabral. It was great spending couple of hours tying down maze "walls". There cannot be anything better than seeing all the kids running around the labyrinth laughing, screaming and ultimately having fun. It was great! What does the office environment do to us, so we forget enjoying these small things?!

Open source

I was surprised by the quality of the software available. Consider for instance SQLite. Just one glance at the SQL supported comes to tell you what the goal of the authors is. They want to have something short, simple, effective, fast. And the target was hit. Though one may miss stored procedures initially, the triggers are to the rescue and simple procedures may reside in the trigger body. But think about it: for an engine running in the same process, should one refrain from making series of database calls?! I guess not. Still maintaing the SQL script as a resource outside of the C/C++ code is a good practice and triggers are the enabler.

The simplicity is in no way limiting. Implementing custom functions or even replacing "out of the box" functions cannot be easier. Without prior knowledge, just looking at the source code and the web site documentation I managed to create a simple math and blob manipulation library in just few hours. Intend to post the source code ( or contribute it back to SQLite ) in the next few days.

Just wanted to say big thank you to the SQLite development team. With the excellent design, implementation and just right amount documentation you saved me a lot of development and debugging time. Cheers!

SQLite can be found here: http://www.sqlite.org/

Few more days

Few more days left at the bank... As the author of Reward Systems Steve Kerr put it in the book, ineligibility is nothing less than intentional unavailability. It is interesting how, in the big organizations, by creating, one penalizes himself. No complains apart from not having my late twenties and early thirties back. But it was a personal choice and regardless I might have learned stuff..

There is new things coming and I am sure they will keep my thought away. PeerBelt product is coming along very well. I am truly excited about it. Some of the friends over the years are too. It is a good sign and it is very rewarding.

An interesting article on Techcrunch

Foreigners Attending US Grad Schools Way Down: Wake Up, Xenophobeshttp://tcrn.ch/8SGo

IE8 "crash protection"

Working around IE8 multiple processes "crash protection" end up eating a day of my life. Nothing in the SDK to alleviate the pain. Microsoft used to be far better in the past.

Another GREAT Investor Session at the Institute

It has been few weeks since the last blog update. Thanks to Adeo, I joined the second investor session at the Institute. It was a great experience. For second time on a row, meeting extremely sharp people. If the company is not under pressure of finding money in the next week or two, one can find a great investor. An investor capable of doing a lot for the company he invested money in. Raising money, though important, should not be in any way the only consideration. The economics of the deal is important, but with smart and understanding person on the other side, negotiation has a good chance of not being too painful.

Adeo, if you are reading this, thank you for the email Monday at 6pm.

Browser Helper Objects

Getting back in shape. Let's hope it does not take long: I do not seem to have much time at my disposal these days. Developing COM components with ATL has its specifics. Especially true when dealing with the multiple IDispatch implementations, connection points, Disp events, etc. For instance, today I had hard time figuring why document events were not sinking. During the advise call it was all fine: S_OK every single time. It took both ATL source digging and debugging just to find out IDispEventImpl requires the MSHTML type library major/minor version exactly as in the registry. If this does not hold, no event ever makes it into the callback. Other people were having the same problem, but none of the top Google results contained the solution. Posted it as a reply here: http://tinyurl.com/l4krht. Hope it helps others avoid unneeded debugging effort. Cheers!

Revenue

Another great session at the Founder Institute took place yesterday. It is far different experience being in the room than watching the presenters via the video feed. The video is real-time in most cases, but still it is limiting.

The Investor pitching session was invaluable in terms of what an investor thinks like. What shocked me was, an investor was actually helping us all by giving answers to whatever question we came up with. Seems like a good investor is one that values long term relationship above all. When/If time comes, I am to look for an investor like that. The bad news is, there is just one chance at getting a particular investor interested. There is a whole lot to learn in a meanwhile...

It was interesting how a B2B software vendor sees the big corporations. It intersects unbelievably well with my own experience, though he is on the other side. It comes to tell, one must know the customer base extremely well. Otherwise a valuable time will be spent working on insignificant issues and no real progress will be made toward getting customers or signing a contract.

Yesterday's Roadmap Session

It is interesting how it all goes through a high and low cycle. The last week Intellectual Property session was definitely a low, partially due to the fact uncovering the valuable information that may affect the startup is buried in pages of legal text. In addition, the requirements on how to properly maintain an inventor book, getting witness signatures, etc. it is so much hassle that ends up eating valuable time. Regardless, sticking to the rules and filing an application may keep one away from a courtroom for some time. And going to a courtroom will definitely put an end to a startup.

The high of the Roadmap session was in the fact, there is no roadmap. Laying down a roadmap is needed for the idea to crystallize. This is it. At is all about protecting and growing the product in a direction that makes sense from users perspective.

The Funded Founder Institute Rocks!

It has been three weeks into the Funded Founder Institute (FFI) '09 Inaugural Semester. It is amazing how the Institute managed to attract exceptional Mentors for relatively short period of time. Between the story leak in TechCrunch and the actual semester start, there were no more than 8 weeks. During that period the FFI had to finalize the curriculum, define an application process and execute it in 10 days, setup the web site we all use on daily basis, setup a live video feed for the remote participants, have the legal aspects covered, define the compensation model for Mentors, FFI, and participants, etc.  All that in addition to selecting and inviting truly exceptional mentors. 

It did take just couple of weeks to realize what was wrong with my previous venture and why it end up unsuccessful. Being able to differentiate the company from the crowd, perform a basic research, and pick a name people will remember the first time it is said are just the first steps. Basic rules. So basic people generally do not pay enough attention to. Eager for more. It is a privilege being there. 

All I wish for at the moment is a success to the FFI. What FFI is about is the free will. The undertaking is noble and with potentially great impact on the entire society. At the end it depends on us, the founders participating in the semester. The Institute is doing the best. I hope we are up to the challenge.