SANOG (South Asian Network Operators Group) is a conference where various stakeholders from the Internet infrastructure ecosystem can come together, share operational experiences and learn from each other. SANOG is targeted towards the SAARC Countries (India, Pakistan, Nepal, Sri Lanka, Bangladesh, Bhutan and Maldives). SANOG is loosely modeled on APRICOT conference with 5 days of Workshops, 2 days of tutorials and 2 days of conference.

The 16th edition of SANOG was held in Paro, Bhutan. This was the second time that Bhutan was hosting SANOG. SANOG was held in the the Paro Engineering College beside the Paro river in very picturesque settings. The earlier edition of SANOG in Bhutan was held in the capital, Thimphu. This was the 2nd time I was attending SANOG, having attended an earlier edition of SANOG in Mumbai in 2006.

I attended the workshop on Network Security by Gaurab Raj Upadhyay and Johhny Martin from PCH. It covered the basics of security specifically for ISPs and large network providers. There were some good discussions on how to manage the different security audit process as well as an incident management program in case of network security breaches. The hands-on part of the workshop concentrated heavily on securing backbone routers and exchanging routes information securely. Some aspects of filtering and verifying network traffic were also covered. The last day had demos of several tools such as nessus and nmap. The slides can be downloaded from the SANOG Program page. Also in between the workshop breaks and during one of the days of the workshop, Devdas and I wrote a improved whois server that is hopefully in production now at the NIC website.

In the tutorials part of SANOG, I was giving a half-day tutorial on application-level performance measurement [Slides,PDF]. There was an small but interested crowd in the tutorials. I ended up covering a lot more of the web-facing and measurement tools as many of the participants were application developers who had written quite a bit of PHP code. It was the first time I was giving a tutorial on this topic and it helped that it was interactive. In addition to the material on the slides, I talked a bit about front-end performance and tools such as Yslow (Yahoo), Pagespeed (Google) and Webpage Test (AOL). There was a lot of whiteboarding and veered a little away from the slides. I also spoke about the network measurement work being done in the IPPM, BMWG and PMOL working groups in the IETF. The feedback was pretty good and I plan to give a longer version tutorial at later versions of SANOG/APRICOT. I skipped the second day of the tutorials and went to Chele La pass.

The conference had several talks that I was looking forward to and I was not disappointed. The standout talk were on long distance wireless network deployment by Matt Peterson and F-root update by Pete Losher. Both has interesting networking insights and interesting traffic data. I also gave a talk on my IETF fellowship experience [Slides,PDF]. Some of the slides were liberally lifted from “The Tao of the IETF” written by Paul Hoffman. As (good) luck might have have it, I ran into Paul Hoffman at the IETF 78 and told him about it . There were a few questions about the fellowship after the conference so I hope it would inspire more people to apply to the IETF fellowship.

I had bought a Asus EeePC (model 1005HA) sometime back. I use it to take notes and as portable browsing and storage device whenever I travel. It came pre-installed with Windows and a crippled version of Office. So when Ubuntu 10.04 – Lucid Lynx came out last week, I decided to install Ubuntu – netbook edition on it as I had heard good reviews of it. This would also help me to do any casual programming on the go. Ubuntu 10.04 is also a long term support (LTS) release. I have written a small howto as installing linux on a netbook is slightly more involved than popping a CD and clicking next (as most Netbooks do not have a CDROM drive).

Step-by-Step guide to install Lucid Lynx Ubuntu 10.04 on a Asus EeePC 1005HA netbook.
1. Go to the Lucid Lynx release page and download the netbook ISO. I suggest you get the torrent and download using a bittorrent client. It is faster and you save some bandwidth (for the mirrors) for the less tech-savvy.

2a. If on Windows, download the USB disk creator, choose the downloaded ISO image and the amount of read-write space you want and follow instructions to create the disk.

2b. If you are on Linux, you can burn the ISO image to a disk and boot from it (by selecting the CDROM drive as the primary drive). Once you are logged in go to System => Administration and select Startup Disk creator. Select the ISO image and the drive to use and click next. You are done in a few minutes.

3. Next reboot the Asus EeePC and press F2 to go to the boot setting screen. Select Boot => Boot Device Priority. Set Removable Device as the 1st boot Device and disable “Boot Booster”. “Boot Booster” feature does some caching to enable fast booting. Since we are changing the boot sequence we need to disable this.

4. Once the BIOS setting are done, plug in the USB drive with the image and boot from it. It will give you an option to install to disk. From there on it is a normal Ubuntu Install.

5. Wireless did not work out of the box. Run the following commands to
get it working.
$ sudo apt-cache search linux-backports

See the output of the above command and install the wireless x86 packages.

But before that, check if the wireless is switched on in the BIOS. I wasted a lot of time realising that it had been switched off in BIOS and the keyboard function keys wouldn’t work.

Posting this on an ASUS EeePC on Lucid Lynx with wireless working
For more info look here.

The corridor discussions with a bunch of Hadoop committers were insightful. I also found out more about Mahout. Mahout is a Apache project to build scalable machine learning libraries. It is not restricted to Hadoop implementations, but much of the current activity seems to be around Hadoop.

Notes and embedded slides from the sessions I attended follow:

Hadoop summit Keynote

Data Management on Grid

Notes:

• Y! uses a HDFS replication factor of 3 (the hadoop default) in most cases. Exceptions are big clusters with large number of applications running simultaneously.
• Y! does not use Avro yet due to large amount of legacy data. Twitter uses Avro.
• Data ingestion layer uses MapReduce for heavy lifting and format conversion for storage.
• LZO is used for compression. gzip (not ideal due to non-block-level indexing) and bzip2 is also used. There are problems with slowness of bzip2 decompression but bzip2 delivers better compression ratios.
• Data ingestion layer also oversees policy for data retention and purging.
• Underlying filesystems is rarely a bottleneck for Hadoop. Mostly the synchronization semantics of HDFS is a bottleneck. A file operation is not successful until all the replicas are in sync.

Machine Learning using Hadoop

Notes:

• There are clear differences between data mining and machine learning.
• ML is harder to implement efficiently on Hadoop. Improving efficiency is still a research problem.
• Hadoop creates one map job / block creating too many empty files and also many reducers.

Optimizing and Benchmarking Hadoop

Notes:

• As a thumb rule, adding as much memory as money can buy is a a good idea for Hadoop
• Consider Network connections as shuffle stage does heavy network I/O
• Solid state disks might make sense at certain price/performance ratios. They are also more power efficient.

Tuning Hadoop To Deliver Performance To Your Application

Notes:

• Several parameters to tune Hadoop but must be used in conjunction with each other.
• Set number of map jobs slightly more than number of cores to ensure better utilization. Makes sure that data is processed in waves. Also better network utilization (as shuffle phase happens parallely with Map phase) along with CPU scheduling
• Choosing a good HDFS block size is important. Number of HDFS blocks is directly proportional to number of Map tasks generated

Links to all presentations

• Hadoop Summit Keynote – USage inside Yahoo and Statistics
• Marketing presentation – Yahoo’s committment to Hadoop and Opensource
• Data Management on Grid
• Machine Learning using Hadoop
• Multiple Sequence Alignment Using Hadoop
• Optimizing and Benchmarking Hadoop
• Challenges And Uniqueness Of Quality Engg And Release Engg Processes In Hadoop
• Tuning Hadoop To Deliver Performance To Your Application

The ACM India launch was addressed by 3 Turing Award Winners – Barbara Liskov, C.A.R Hoare (Tony Hoare) and Raj Reddy. The ACM Turing award is “The Nobel Prize for Computing” and it is rare to see three Turing Award winners address the audience at any event. Barbara Liskov is the most recent awardee of the Turing award (the 2nd woman to win it) and she spoke on the power of abstraction. She spoke about the problems early programmers faced when writing large and complex programs. She explained how she tried to solve it using abstractions similar to (what is now called) Object-oriented programming. She talked at length on how her insights and experiences with these programming problem led to design of the CLU language. CLU was the first language to implement iterators and generators (as well as exception handling). It was a good lesson in computer history listening to her. I learned later that she was the first woman to get her PhD from a Computer Science Department. (Her doctoral advisor was the legendary John McCarthy). Her presentation and the mentioned references in it make for good reading.

Dr Raj Reddy is the only Indian who has won the Turing award for his contributions to field of Artificial Intelligence. Incidentally, his PhD advisor was also John McCarthy – AI Pioneer and Turing Award winner. Dr Raj Reddy spoke about the growth of computing over the years and the challenges of reaching the “bottom of the Pyramid”. He explained why there was need to move from the WIMP-paradigm in user interfaces to the SILK (Speech, Image, Language and Knowledge) to increase the reach of computing. His Turing award lecture (“To dream the possible dream”) makes for interesting read as well.

C.A.R Hoare (Tony Hoare) was the next speaker. He is a living legend in computer science. I was looking forward to hearing him speak as I had studied the Quicksort algorithm (which he invented) and Communicating Sequential Processes paper in college. He was remarkably witty and his enthusiasm for computer science shone through in his talk. In particular he spoke about the Verified Software initiative which he contended was similar in scope and impact (for Computer Science) to the Hubble Telescope and the Human genome project.

The following 2 days, we had the ACM Compute 2010 conference and there were several hands-on Tutorials on Cloud Computing, Rich Internet Applications and Web 2.0 apps, Widgets and Mobile Applications. The RIA tutorial was conducted by Mrinal Wadhwa (slides embedded below) and the Facebook connect tutorial by Prateek Dayal (of Muziboo).

(Disclosure:I am the secretary of the Bangalore Chapter and am on the program committee for ACM Compute 2010.)

Once in a while, you read a book that is filled with ‘aha’ moments. If you have written complex software for a while or want to become a good programmer then ‘Coders at work’ is a must read. This fantastic book interviews 15 master programmers. Some of the people interviewed in the book are well-known names such as Don Knuth, Ken Thompson, Jamie Zawinski and Peter Norvig.

Some comments on the content of the book:
Programming languages
Many of programmers interviewed started with BASIC and considered it an okay language. What is probably more surprising is the universal hatred of C++ in this group. In fact several people such as Peter Norvig and Ken Thompson (who goes on a tirade against C++) consider it a downright ugly and cumbersome language to work with.

Jamie Zawinski – C++ is just an abomination
Brad Fitzpatrick – The syntax is terrible and totally inconsistent and the error messages, at least from GCC, are ridiculous.
Ken Thompson – - By and large I think it’s a bad language. It does a lot of things half well and it’s just a garbage heap of ideas that are mutually exclusive. Everybody I know, whether it’s personal or corporate, selects a subset and these subsets are different. So it’s not a good language to transport an algorithm—to say, “I wrote it; here, take it.” It’s way too big, way too complex. And it’s obviously built by a committee.

On Programming and Curiosity
Almost everyone interviewed still programs (some do occasionally) and enjoys hacking and taking things apart. Many were misfits and took unusual career paths to get to where they are today. There is a rebel and hacker streak in all of the them. Most of them stumbled into programming and discovered that they were good at that at some point. Everyone emphasized the practice of writing good code readable code. Everyone laments that you cannot understand a system from the bottom upwards as systems have become more and more complex and layers of abstraction have multiplied manifold.

On categorizing programming and building software
The opinion is pretty much evenly split on whether programming is a science, art, craftsmanship or engineering with a slight bias towards craftsmanship.

On Recommended Books
Among the books recommended, “The Art of computer programming” by Don Knuth topped the list for obvious reasons. Another books which was recommended by several people was the “Psychology of computer programming” by Gerald Weinberg.

On the state of computer science
The mood on the state of developments in computer science was fairly pessimistic and most people pointed to the fact that many of the breakthrough ideas for computer science were conceived in the ’70s (with the notable exception of the internet and web programming)

The only downside here is the interview of Fran Allen. It should not have made the book. I got the distinct feeling that much of the work that she claimed credit for is implemented by others and she was the manager of those projects (probably a good one but that is hardly the same as being a good programmer).

I have added some notes (for further reading) and quotes from the book on the wiki

Alexis Ohanian ( kn0thing on twitter) – the co-founder of reddit (and the creator of the beloved Reddit Alien) was in Mysore for the TED conference. He took a break from the TED conference to meetup with a bunch of redditers. For those who don’t know he is also the publisher of XKCD books and all the proceeds from the book go to building a school in Laos. It was interesting talking to him about startups, startup school, Paul Graham, ycombinator, traveling in India, the startup scene in India, Social media [link to TED Presentation] and of course reddit.

We gave him a sampling of Indian food (Coconut Groove) and sweets (K C Das). Thanks to @dhempe and @pswam for organising this tweetup.

It all started with a faint memory in the increasingly transient nature of the Internet. I vaguely remembered reading about a datacenter which was based somewhere near Stockholm and was built inside an (decommissioned) nuclear bunker. A little bit of googling later I realised that I had read about it on the Pingdom site. I thought to myself that since this is a conference where networking geeks converge, there is a good chance that some attendee knows the people in the Bahnhof ISP. So I sent this mail. Initially no one replied, but then the number of people interested in seeing the datacenter just ballooned. Two people stepped forward to arrange the visit during the lunch break at the conference and the CEO of Bahnhof, Jon Karlung personally gave guided us through this fantastic datacenter.

The entrance to the Datacenter

There is actually a house on top of that hill and there is pathway from the datacenter that opens up onto the top of the hill. The datacenter is actually dug out of hard rock (granite) as can be clearly seen from the pictures of the server floor at the end of the post. It was originally a military bunker and nuclear shelter during the Cold War era. The Code name from the military days Pionen White Mountains can be seen in the photos of the entrance to the datacenter.

The Backup Generators

2 Maybach MTU diesel engines that produce 1.5 Megawatt of AC power provide backup power. The engines were originally designed for German submarines. There is a warning horn from a German submarine that add to the effect

The Conference Room

The Conference room and the pathway leading to it is made completely out of metal and glass and it hangs above the server flow adding tothe futuristic space staion look of the datacenter (as can be seen from the picture at the top of the post). There is also a Tintin theme rocket [ See Destination Moon for the Tintin Reference ]

The Fountains

There are lots of plants around the datacenter to reduce the claustrophobic feel of the bunker and make it like a more natural working environment. The Fountains at the entry also make up the decor of the place but are generally switched off as they make a lot of noise.

The Netops and the Leisure room

This is Bahnhof’s biggest facility in Sweden and this Network operations room is used for running the ISP. The leisure room has a huge fishtank to add to natural feel of the place.

The Server Floor

Some of the walls of the server floor are unadroned and are made of bare rock giving away the initial use of the facility as a Cold war Era nuclear bunker.

Bahnhof uses the unique nature of the datacenter for marketing purposes. IT is actually possibe to co-locate your servers here. The Pionen Datacenter gives a whole new meaning to disaster recovery backup.

ACM Compute 2010 – 22nd & 23rd Jan, 2010.

ACM Bangalore chapter is organising ACM Compute 2010 which is into it’s third year now. This year the broad theme is Cloud computing and Information retrieval, management and analytics. The aim of this conference is to bring together researchers, practitioners, technology market movers, and thought leaders, with a view to advance the state of the art, and the state of the practice in applied research. This year we are planning to do something special – details soon The Call of papers (CFP) is out for sometime now and the last date for submissions is Oct 1 2009. You can also submit a proposal for a half-day or day long tutorial. Last year we had a bunch of good tutorials and also the symposium on Cloud Computing co-located with ACM Compute 2009 which was a great success.

Disclaimer: I am on the program committee of ACM Bangalore and am the secretary of the ACM Bangalore chapter.

Pycon India 2009 – 26th & 27th Sep 2009

Also this weekend (26th and 27th September 2009) India’s first Pycon India 2009 is being held in Bangalore. There is an interesting list of talks lined up. So do register if you are interested in attending.

Foss.in – December 1-5, 2009.

Foss.in shifts to a new venue this year NIMHANS convention centre. This year promises to be interesting as the venue is available for longer durations. Also there are going to be hacker evening/nights where tinkerers can meetup and talk about a whole range of stuff not restricted to just FOSS. My educated guess is something on the lines of CCC in Germany. Definitely something to look forward to. Plus I think there will be atleast one evening where we will have music . So join the mailing list if you are interested in presenting/attending as more details should emerge soon.

Photo Credits Lincoln McNey (ISOC – Internet Society)

Ronald Nsubuga (Uganda)	Hamid Mukhtar (Pakistan)	Eduardo Ascenço Reis (Brazil)	Dorcas Gachari (Kenya)	Md. Aminul Haque Chowdhury (Bangladesh)
Afaf El Maayati (Morocco)	Alberto Castro (Uruguay)	Alejandro Acosta (Venezuela)	Vinayak Hegde (India)	Muhammad Haris Shamsi (Pakistan)

Later we had a dinner at the Clarion Sign Hotel in Stockholm with our mentors and ISOC Staff. It was addressed by Vint Cerf.

Vint Cerf’s Business card

In Feb 2009, I won an ISOC fellowship to attend the IETF conference. The ISOC fellowship pays for the airfare, stay in the hotel and the conference fees of the IETF participants. The process to apply for the ISOC fellowship is competitive and only participants from third world countries can apply. In Jan 2009 there were 145 applicants who wrote proposals for the work they plan to do at the IETF out of which 4 people were selected. My proposal was regarding Internet Measurement and Analysis – the area of work I have been working at Akamai for the last 5 years. More specifically I was working on drafts which outlined the metrics and the ways for measuring them for websites/webapps and streaming quality. Network application measurements has been one area which has been weakly represented and not much work was happening in these area. There was some related work happening in IP Performance metrics groups and PMOL working groups but none in the above mentioned areas. My work was to fill in those gaps by writing drafts and then soliciting comments and finally working through the IETF process to get it standardized. I was also vouched for by two people who knew about my work which helped me win this fellowship.

The IETF and conference details

The IETF conference is the topmost conference where network engineers meet to discuss RFCs, best practices and share operational knowledge of running various that make up the Internet. I attended the 75th edition which was held in Stockholm from July 26 – July 31st 2009. The IETF is different from most other conferences in the sense that it is a collection of working group meetings. Each of the working groups decides it’s own agenda and the drafts that they want to discuss in the meeting but bulk of the work happens on the mailing list beforehand. Also anyone can join the mailing list and participate in the discussions. So the IETF meeting really is a loose collection of several miniconferences (for the lack of the better word). Please read the Tao of the IETF if you are interested in how the IETF works. The following quote (from the Tao of the IETF) quite accurately describes how the IETF works.

“In many ways, the IETF runs on the beliefs of its participants. One of the “founding beliefs” is embodied in an early quote about the IETF from David Clark: “We reject kings, presidents and voting. We believe in rough consensus and running code”. Another early quote that has become a commonly-held belief in the IETF comes from Jon Postel: “Be conservative in what you send and liberal in what you accept”.

In that sense there are strong parallels (low barrier to entry, open access, easy to contribute, dominated by hands-on engineers) with the Free/OpenSource world as Linus Torvalds said

“Talk is cheap. Show me the code.”

If you participate in the IETF, be prepared to have your draft (and data as well) questioned, scrutinized, critiqued and commented upon. It requires lots of patience (and a thick skin , convincing power with proper data to prove what you have proposed is worth standardizing and being accepted as an RFC.

My Experience

The IETF conference attracts the best and brightest in the field of networking. It was a pleasure to meet so many great engineers. It was easy to talk to most of other engineers as they were very down to earth. There was no ivory-tower syndrome.

On Sunday, I attended the first-timers meeting. It was useful as it gave an overview of what to expect at the IETF meeting and how to make the best use of it. On Monday, I attended the Audio-Video technology working group meeting. An engineer from Huawei was presenting a draft and participants asked him several questions but he could not answer. When asked about the use-cases when scaling to hundreds of thousands of users, he said that they had tested for only 3-5 users. On hearing this one of the participants said sarcastically that we design protocols that scale for millions of users and not just a few users. This was first of many such blunt and sarcastic comments that I had heard during the conference. It was clear from then on that, if you have not carefully considered all the cases, the draft would not go for further review and will be shot down mercilessly. The next meeting was the Benchmarking Methodology WG. IT was a small crowd as compared to the AVT WG and the proceedings were more cordial. There was some remote participation in this meeting. Personally I found this meeting useful as it was related to my area of network application metrics.

On Wednesday, There was a PMOL (Perfomance Metrics for Other Layers) BoF Session. I talked about perfomance metrics for Webapps and streaming at the PMOL BoF and got some good comments from Al Morton (my mentor) and Henk Uijterwaal (WG Chair for IPPM WG). Al later helped me iwth the xml2rfc tools that are needed to format the RFC draft. Overall the response was positive and I got some good pointers to some earlier work in other standardization bodies such as the ITU. On the subsequent days, I attended the Operations & Administration and Technical plenaries and a talk on “Securing the DNS”.

On Friday, I attended the IPPM (IP Performance Metrics) working group meeting. I was really well-prepared this time having attended several WG group meetings. Also since I had read the drafts and the background materials on the drafts that were going to be presented, I was able to make meaningful comments on some of the drafts that were being presented.

Among the other highlights of the meeting were having a photosession and one-on-one chat with the Vint Cerf – one of the inventors of TCP/IP and the Internet and the welcome reception at the City hall in Stockholm where they have the Nobel Prize banquet. It was addressed by the mayor of Stockholm among others and it felt great to occupy the same space as some of the brightest minds of this century (of course, separated by time .