Technology Movement

I read somewhere(?) that the Technology Movement always follow the Sun (from East to West).

• More than 2000 years ago, the Chinese invented paper, compass, gunpowder and printing.
• 2000 years ago the Egyptians built the pyramids.
• 600 years ago, Leonardo da Vinci in Italy did his stuffs... too many to write down.
• 400 years ago, an apple hit Newton in England.
• In the last 200 years, various people in US studied electricity, invented plane, phone, transistor and eventually the Internet.

China -> Egypt -> Italy -> England -> US. The Technology Movement always follow the Sun (East to West). There are more countries in between but I think I made my point. The question now is, not 'if' but 'when' will the Technology Movement jump the Pacific Ocean to Asia? Are we on the verge of the jump because the time span of each Technology Movement is getting shorter and its already been in the US for 200 years.

Is the economic boom in China the prelude to the Technology Movement of Asia? How are countries like Australia, India or my home country Bangladesh going to fare? With the shorter time span trend, I think the Technology Movement will be in Asia for less then 100 years. So if we want to ride it, we need to get ready and get ready right NOW.

Followup: How does the dark age affect the Technology Movement? That is another post.

Seth's Sheepwalking

Seth Godin wrote an interesting article about Sheepwalking. Personally I call it Grinding. Sheepwalking is when you only do what you are told to. Examples are:

The TSA 'screener' who forces a mom to drink from a bottle of breast milk because any other action is not in the manual. A 'customer service' rep who will happily reread a company policy six or seven times but never stop to actually consider what the policy means. A marketing executive who buys millions of dollars of TV time even though she knows it's not working--she does it because her boss told her to.

I call them Grinders. People who punch in at 9, punch out at 5 and wait for the paycheck. They grumble when you ask them to do something out of their job description. Basically, just grinding it out.

At Uni, whenever I discussed technology with my fellow engineering classmates, it saddened me to see them as Grinders. I had spent many a night in the lab to get something working while my team mates went home at 6pm.

Google was a company that defied it all. But lately, all the good people are leaving after cashing out their stock options and they are being replaced by Grinders.

Overhaul the TCP/IP Stack

Today, a friend of mine stated one way to make lots of money is to build a smaller TCP/IP stack. The current stack size is 16K. A smaller stack will enable faster transfer due to more data per packet, faster packet composition/processing and less bandwidth. Companies such as Microsoft, Sun, Cisco would definitely pay a lot of money for the stack.

A way to do that is to move the stack to hardware. To quote the movie Antitrust:

The answer's in the box.

However, I take a different view. The Internet was built in 70's as part of the DARPA project. Technology has advanced so much since then that we need a complete overhaul of the stack. A new, better beginning. You cannot build space-ships out of automobile technology.

Fishing

Last night I went fishing in Shorncliff with my friends. None of us knew how to cast the net, so we improvised. We caught a fish by surprise. Surprise because we thought the net didn't spread when we cast it. So pulled on the net and up came a little fish (3 inch). No idea what fish it was.

The day ended when we accidentally let go of the net completely when we cast it.

Oh well. Headed home after throwing the fish back in the water.

Tech Bust Due Soon

The news article Rampant M&A activity may signal peak of global business cycle puts forward a nice justification that the next bust is near. Current M&A activity ($3.6 trillion) has already exceeded the activity ($3.4 trillion) of the previous tech bust.

I, for one cannot understand YouTube's valuation of $1.65 billion or Skype's valuation of over $3 billion. Just numbers of customers cannot justify those valuation. Right now all of them are supported by ads. What will happen to them at the next bust when advertising money dries up. However, its all part of evolution. We need the next bubble bust to kill off the weak, so that the strong can rise from the ashes.

It takes a long time for a big company to die: so long that it's non-obvious that most of them are in fact dying, or at best treading water. We're a hit-driven industry. A few big successes can make it seem like everyone's doing well. But most of them have only had one hit. Go visit most tech companies, and all you'll find is a fussy henhouse parading around an aging goose that laid one or two golden eggs. All their innovation happened in the first act, and now they're focused on "managing for success." But that kind of managing is just staving off insolvency until a real innovator takes their business away. Any tech company overly focused on (or dependent on) its management is probably a good candidate for short-selling.

Graduation

Finaly had my graduation ceremony today. What can I say…

What I remember most are the:
1. professors (especially the most bizarre ones)
2. staying up late working on assignments
3. craming for exams the night before
4. writing a year long research paper in three weeks

Now I am officially an engineer. =)

Snippet://java/internet connectivity

The following code checks if internet can be accessed. Note, this is not a ping. The Java Socket class isn't capable of such low level function. However, since JDK5, Java java.net.InetAddress.isReachable(int) can be used to check if a server is reachable or not.

isReachable() will use ICMP ECHO REQUESTs if the privilege can be obtained, otherwise it will try to establish a TCP connection on port 7 (Echo) of the destination host. But most Internet sites have disabled the service or blocked the requests (except some university such as web.mit.edu).

public static boolean checkinternet(String url) {
try {
InetAddress address = InetAddress.getByName(url);
System.out.println("Name: " + address.getHostName());
System.out.println("Addr: " + address.getHostAddress());
System.out.println("Reach: " + address.isReachable(1000));
} catch (UnknownHostException e) {
System.err.println("Unable to lookup " + url);
} catch (IOException e) {
System.err.println("Unable to reach " + url);
}
}

Snippet://Java/InputStream

Java code snippet to read InputStream using buffered reader.

NOTE: StringBuffer won't insert extra \n, so the returned string will be exactly as the InputStream. Also, the unusual for statement in the snippet below is 10 times faster than the traditional while statement.

private static String slurp(InputStream in) throws IOException {
StringBuffer out = new StringBuffer();
byte[] b = new byte[4096];
for (int n; (n = in.read(b)) != -1;) {
out.append(new String(b, 0, n));
}
return out.toString();
}

java.lang.NoClassDefFoundError: org/apache/commons/

I'm working on a project where I needed to make HTTP requests. Instead of reinventing the wheel, I decided to use Apache-Commons-Httpclient library. Upon compiling, the code blew up in my face. The debugger says:

java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory

Hmm... so Apache-Commons-Httpclient is dependent upon Apache-Commons-Logging library. After I download and add the required library, I compile the code to have it blow up in my face again. This time the debugger says:

java.lang.NoClassDefFoundError: org/apache/commons/codec/DecoderException

Good god, another dependency. This time Apache-Commons-Codec. That's what I hate about external libraries. None of them is self contained. I need the code small enough to fit in embedded devices like a cell phone. Good thing Apache-Commons library is opensource, so the source code is available. I need to strip it down. So much for reinventing the wheel.

High Avalibility - 5 Nines - 99.999%

Recently machinehead asked me how to build high availibility. For the less technically inclined, high availibility is a measure of the reliability of a system and sometimes indicated by "Five nines" or 99.999% reliability. Basically what it means is the system has a total downtime of no longer than five minutes per year.

Having done a course on distributed systems, I decided to take a crack at it. There are three main issues that needs to be addressed:

• Hardware


• Software


• Data

Hardware
Everyone thinks high availibility means only hardware. Its not. Its only the beginning. Firstly, you need at least two of everything located geographically apart. For example, most people believe RAID is insurance against data loss. But RAID only protects against one or two drive failure. What happens if the PSU shorts and pump 240V instead of 5V. That will fry every disk in the array. Plus, rebuilding disk takes time which isn't high availibility. It should also be geographically located apart to protect against fire, earthquake, tsunami, plane etc.

Secondly, you need smooth, automatic switchover incase of crash. For example, using heartbeat to monitor servers and changing the IP at DNS server to the failover server. This makes it smooth and you don't need to make any other server aware of the server failure.

Software
Software should be built from ground up with high availibility in mind. Meaning they should be scalable and clusterable. The best way to do this is to make them stateless. For example, when you click on "2" to goto the second result page on Google, the second page doesn't necessarily have to be processed by the same server that did the initial first page. It can be done by any server. This is the power of stateless.

Data
Data is the hardest problem to solve. If you fragment and replicate the data for performace and scalability, you need to address sync issues. How would you lock and commit multiple partitions? How would you detect deadlocks? 2 phrase lock and 2 phrase commit is not an easy answer. eBay takes down the site on Monday 12-4AM every week to archive sold items so as to keep the fragments small (smaller fragments means faster searching by the database). Yes, this 4 hours is "planned" downtime and no, planned downtime does count towards high availibility.

Sun Just Killed Java

Yup, its official, Sun open-sourced Java by changing its license to GPLv2. The announcement is at http://www.sun.com/software/opensource/java/.

This is a good news for the open-source community. Especially GNU Linux which can now include all Java software with the platform like JBoss, IBM WebSphere, BEA Weblogic because all of them are nor GPL. Hehe.

Bad news is for all commercial firms who's propreity Java applications are now GPL. The GPL requires that any code combined with GPL code must be distributed under the same license. Developers must provide their contributions back to the community. This provision provides a mechanism to ensure that Java continues as a unifying platform for innovation. Most probably this will give rise to .NET technology as all commercial firms move to .NET.

P.S. This does not affect Java applications developed in-house for use in-house ONLY. It mostly effects commercial development of Java.

Format External SATA Drive

I just bought a Seagate SATA 250GB HD and an external SATA enclosure on eBay. Windows 2003 recognized the enclosure in Device Manager but the HD did not show up in My Computer. Reason being the drive was uninitialized and unformatted (since its new).

So how do you format a drive that does not show up on My Computer. Normally, a new HD connected directly to IDE shows up as unallocated drive in My Computer which you can then partition and format. However, what do you do when its connected to USB?

After half hour of playing around, the answer was simple enough:

1. Start -> Administrative Tools -> Computer Management -> Disk Management.
2. The HD will show up as Disk1 (unallocated)
3. Right click on Disk1 and select Initialize.
4. A Wizard will walk you through initializing, partitioning and formatting.

I partitioned it into 4 primary partition (maximum allowed by NTFS).

Debian Hard Disk Spindown

This article Debian hdd spindown - Kurobox Central has an interesting note on how to spin down the hard drive in Debian. When will Linux be as easy to use as Windows?

Install hdparam: apt-get install hdparm

Hard drive info: hdparm -i /dev/hda

Hard drive performance: hdparm -tT /dev/hda

Here's my 7 years old Toshiba Satellite2755 laptop Debian Server 6GB hard drive performance:

Timing cached reads: 62 MB in 2.01 seconds = 30.86 MB/sec
Timing buffered disk reads: 40 MB in 3.07 seconds = 13.01 MB/sec

Set spin down to 35 seconds: hdparm -S7 /dev/hda

To turn on spin down at 35 seconds by default, edit /etc/hdparm.conf:

/dev/hda {
mult_sect_io = 16
write_cache = off
dma = on
spindown_time = 120
}

Run to add hdparm setting to run control 2 (with shutdown running at reboot, shutdown, and single-user mode):

update-rc.d -f hdparm remove
update-rc.d hdparm start 19 2 . stop 19 0 1 6 .

Add to /etc/init.d/sysklogd so SYSKLOGD doesn't log MARK to /var/log/messages every 20 minutes and force the hard drive to turn on

SYSLOGD="-m 0"

Installing .rpm in Debian

Installing a RedHat .rpm in Debian means converting to .deb and then installing it. And here's how to do it:

Convert .rpm to .deb: alien -k package.rpm

NOTE: The -k option preserves the minor version

Install .deb: dpkg -i packagename.deb

Uninstall .deb: dpkg -r packagename

How to Install WordPress On Debian Etch

Complete the following the steps:

1. # apt-get install mysql
2. # apt-get install apache2
3. # apt-get install php5
4. # apt-get install wordpress
5. # mysql -u root
6. mysql> CREATE DATABASE wordpress
7. mysql> GRANT ALL PRIVILEGES ON *.* TO wordpress@localhost
   IDENTIFIED BY ‘wordpresspassword’ WITH GRANT OPTION;
8. mysql> FLUSH PRIVILEGES;
9. #mv wp-config-sample.php wp-config.php
10. Edit wp-config.php to enter the correct setting:

    // ** MySQL settings ** //
    define('DB_NAME', 'wordpress');
    define('DB_USER', 'wordpress');
    define('DB_PASSWORD', 'wordpresspassword');
    define('DB_HOST', 'localhost');

11. Add to /etc/apache2/sites-available/default:

    Alias /blog "/usr/share/wordpress/"
    <Directory "/usr/share/wordpress/">
    Options FollowSymLinks
    AllowOverride Limit Options FileInfo
    DirectoryIndex index.php
    </Directory>

12. #/etc/init.d/apache2 restart
13. Goto: http://localhost/blog/wp-admin/install.php
14. Follow the instruction to create a login
15. Use generated login to log in and start blogging.

Sun Blackbox

Sun just released Sun Blackbox, a new readymade data-center in a shipping container complete with 25 racks, 1.5PB storage, networking, cooling and shock absorbers. The idea is they ship you the whole container and you just power it up. A la data-center in a box.

How cool is that? Do you see whats happening here. Sun Microsystem is changing their whole business model. I remember a few years ago, Cemex, a cement manufacturer based in Mexico changed their business model from '$ per weight' to 'just-in-time' model. Cement manufacturing is a highly standardized business and the margins are too thin. So Cemex came up with a new idea. See, the problem is cement starts setting from the moment it leaves the factory, so you can't have it lying around. Also, sometimes the site is not ready or labours lying around too long waiting for the cement to arrive. Cemex identified these issues and told their customers that from now on, they will only pay for the right amount of cement at right time i.e., time is the business model. Customers phones the call-center to say when and how much cement they want and their ERP system optimizes and routes trucks in the field (fitted with GPS monitoring and comminicator) to the customer. Benefit for customer is they get right amount of cement at right time (usually within plus minus an hour). Benefit for Cemex is they bacame a multibillion dollar company and the 3rd largest cement manufacturer in the world. All by changing their business model to 'just-in-time'.

And this is what Sun Microsystem is doing. Shortening the time and effort for customers to set up a data-center. Sure you won't get the Blackbox within an hour of ordering. But even then, it will be there in a few days all set up and ready to go. Compare this with time taken to plan and build a data-center, from evaluating vendors to purchasing, building and testing infrastructure, etc.

Scaling MySQL

Have been experimenting with MySQL 5. It's not optimized right out of the box which is expected. But I can't find a lot of literature on the web to solve them. Seems MySQL is mostly used by kiddies to run their website who does no performance tunning. Of course, it could also mean I haven't looked well enough.

Anyway, my concern is with scaling. Cluster or replication... that is the question. Replication works because its mostly read based. You can have a central master with slaves replicating from it. Thankfully, I don't need real-time replication. The data size is now in giga bytes which will eventually approach tera bytes. More data means more time for replicating. The slaves will be down during replication but need them to be up 24/7. One way to solve this is to have two sets of slaves. Replicate one set while the other one is up and then switch over. However, this is a waste of resource as one set will always be unsed.

Basically, the problem with replication is it doesn't scale the writes. Expensive raid can speed it up but I can't afford that. As soon as I hit 100% of write transaction operation, I've hit the scaling wall with replication.

I need to investigate vertical and horizontal partitioning and clustering. I'm using my text book, Distributed Systems by Coulouris, Dollimore and Kindberg to base my theory. The book is supposed to be a bible on Distributed Systems. But its only general theory. I need something more geared towards MySQL. Anyone has any lead?

Google uses BigTable which is IO based instead of SQL based. Too bad I can't use that. Think at some point I'll have to develop my own custom thingy. Of course, the other bootleneck with replication is the network. Gigabit ethernet isn't going to cut it anymore. I have seen some cheap optical network card on eBay for $30, but optical cables cost about $100 per 10 meters. I can't afford that either. Oh well.

Thesis Demo

The demo went quite well. The examiner grilled me for 45 minutes, but I had an answer for everything. You will too if you wrote an 85 page thesis (excluding appendix). I think I demoed to about 20 public visitor. All of them said it was quite good (but I'm sure they said that to everyone). But a couple of the visitors really appreciated my product.

Basically what my thesis did was implement a metric monitoring and reporting platform. You can use it to records metrics from different points and then the application will allow you to do whatever you want with the data. The project is actually implemented in the largest bank in the state where the data is hooked into Cognos for balanced scorecarding and other business intelligence thingies. At the bank, what used to take 2 days to perform now takes less than 5 minutes with my product.

The people who appreciated my product were mostly managers who frquently has to collect and analyze data for monitroing and reporting. When they saw what my product, they started grilling me on how it was implemented and the nuts and gritty. One guy even asked if I wanted to commercialize the product. I wasn't since the bank had sponsered me on the project and they were clear from the beginning that they would own the code. Well, the actual reason I don't want to commercialize my product is because there is nothing innovative about it. It basically leveraged existing product to create a new product. Anyone can do it in 6 months (which is how long it took me). Now, after learning from my mistakes, I can make a better one. But I'm still not keen on commercializing it. If I do and it is successful, they I know compititors will spring up on less than 6 months, since they can work on it full time (I worked on it part-time while I had a full load at uni and also working). Hell, with Cognos's resources, they can probably do it in 2 weeks.

So, yup, no commercial product from me. I wish I had asked the bank to pay me for it. I can't believe I did it for free just so I can write a good thesis. Another of life's lesson.

No more free lunch.

Finished My Thesis

I've finally finished my thesis this week. Submitted the electronic version today followed by printed/binded version in a week. That was quite a journey. I realized a lot of things about myself that I never knew. Tomorrow I demo the project at the university fair from 1-6PM which is open to the public. Anyone in the area is welcome to drop by.

I recommend using LaTeX to anyone planning to write a thesis. Basically, LaTeX is a typesetting application that takes care of formatting for you so you can concentrate on the content. It takes care of all formatting and creates table of content, figures, chapter, bibligraphies etc. LaTeX might seem an overkill, but when you approach more than 50 pages, you'll be thankful you chose LaTeX. Don't go with Word and EndNote like I did in the beginning.

Debian Etch

Just installed the latest Debian Etch (Debian 4.0) on my server using netinst cd. The netinst cd contains the base install and is about 150MB in size (better than downloading the full CD). Wow, is it me or is Etch much faster than Sarge.

Since netinst only has the base packages, I need to manually install everything I need, which is better because I get to keep the server lean. I need ssh, svn, apache, mysql, and perl (basically the whole LAMP thingy). Java, Tomcat, JUnit, ANT and Continuous Integration server will come later on. But I ran into a bit of a trouble. Since I installed from CD, I don't have the web mirrors set up. The fix is simple:

1.Add to /etc/apt/sources.list:
deb http://mirror.optus.net/debian/ etch main
2. apt-get update

Which threw up the following error:

# apt-get update
......
Fetched 5562B in 13s (421B/s)
Reading package lists... Done
W: There are no public key available for the following key IDs: A70DAF536070D3A1
W: You may want to run apt-get update to correct these problems

Bit of googling showed the fix is simple

3. apt-get install debian-archive-keyring
4. apt-get update

NOTE: You need to be root when you do all these.