Digg on the LAMP Stack

This is an interesting article on how Digg leverages the LAMP stack (Linux, Apache, MySQL and PHP). The article describes how Digg started with one server, added a load balancer and other tricks to scale up. Right now, they work with well over 100 servers and scalability always remains a challenge because they are growing so fast.

I would have liked a longer article with more technical details, but something is better than nothing I guess.

The Shield of Achilles

by W. H. Auden

I have always liked W.H. Auden‘s poetry. It gets under your skin, and disturbs you, even though it is funny on the surface. All of us would remember The Unknown Citizen, a verse that describes what modern human civilization has been reduced to, where men are mere numbers, record names in the state file cabinet on whom the state maintains statistics, but no face, no disposition, no countenance.

Digression enough. Coming back to The Shield of Achilles, which was published in 1955 and which Auden won the National Book Award. The poem is a take on the passage in Book 18 of Homer‘s Iliad, where the smith god Hephaestos makes a shield and armor for Achilles at the request of Achilles’ mother, the goddess Thetis. (Achilles’ armor was lost when his friend Patroclus wore it into battle and was defeated by Hector). In the original poem, Homer described cornucopia inscribed on the shield, the planets, gigantic cities, fields full of produce, cattle (heifer) and so on.

Auden takes a very different view. Thetis looks for symbols of wealth and power, but she finds desperation on the shield. Instead of crops and full fields, she finds ‘A plain without feature, bare and brown’, signs of desperation all around, troops lined up for battle without a twinkle in their eye, ‘an unintelligible multitude’, men who respond not to invocation, but to ‘a voice without a face’. She wants to see sacrifice and worshipping, but instead finds ‘three pale figures were led forth and bound/ To Three posts driven upright in the ground.’ This reminds us of Christ, ‘That carries weight’, and for whom there was no help. But unlike Christ, ‘they lost their pride/ And died as men before their bodies died’. She wants to see beautiful men and women sashaying across, athletes competing for the grand prize, but all she finds is a street urchin killing birds, chaos and anarchy, rapes and fights, a world where word is meant to be forgotten, and where it was unimaginable that ‘one could weep because another wept’. The great warrior Achilles, the man slayer, could not live long. Not when the world around him was such.

The poem has a lot of reflection of modern times, times of poverty and anarchy following the Second World War. The contrast that the poet draws between the idyllic world of bravery and self-sacrifice that Homer had described and the modern world with its disregard for rules, for honour and for pride, where men are but war machines without a thought for why they move to battle, and prisoners are executed without any thought for their weakness, is exceptional. The poet has matched word for word, phrase for phrase, every word of Homer that alluded to honour has been decimated to a symbol of our troubled times.

The verses below have been copied from here. This article was very helpful for understanding, while a good account of Auden’s life in short is given here.

    She looked over his shoulder
   	   For vines and olive trees,
     Marble well-governed cities
   	   And ships upon untamed seas,
     But there on the shining metal
   	   His hands had put instead
     An artificial wilderness
   	   And a sky like lead.

A plain without a feature, bare and brown,
   No blade of grass, no sign of neighborhood,
Nothing to eat and nowhere to sit down, 
   Yet, congregated on its blankness, stood
   An unintelligible multitude,
A million eyes, a million boots in line, 
Without expression, waiting for a sign.

Out of the air a voice without a face
   Proved by statistics that some cause was just
In tones as dry and level as the place:
   No one was cheered and nothing was discussed;
   Column by column in a cloud of dust
They marched away enduring a belief
Whose logic brought them, somewhere else, to grief.

     She looked over his shoulder
   	   For ritual pieties,
     White flower-garlanded heifers,
   	   Libation and sacrifice,
     But there on the shining metal
   	   Where the altar should have been,
     She saw by his flickering forge-light
   	   Quite another scene.

Barbed wire enclosed an arbitrary spot
   Where bored officials lounged (one cracked a joke)
And sentries sweated for the day was hot:
   A crowd of ordinary decent folk
   Watched from without and neither moved nor spoke
As three pale figures were led forth and bound
To three posts driven upright in the ground.

The mass and majesty of this world, all
   That carries weight and always weighs the same
Lay in the hands of others; they were small
   And could not hope for help and no help came:
   What their foes like to do was done, their shame
Was all the worst could wish; they lost their pride
And died as men before their bodies died.

     She looked over his shoulder
   	   For athletes at their games,
     Men and women in a dance
   	   Moving their sweet limbs
     Quick, quick, to music,
   	   But there on the shining shield
     His hands had set no dancing-floor
   	   But a weed-choked field.

A ragged urchin, aimless and alone, 
   Loitered about that vacancy; a bird
Flew up to safety from his well-aimed stone:
   That girls are raped, that two boys knife a third,
   Were axioms to him, who'd never heard
Of any world where promises were kept,
Or one could weep because another wept.

     The thin-lipped armorer,
   	   Hephaestos, hobbled away,
     Thetis of the shining breasts
   	   Cried out in dismay
     At what the god had wrought
   	   To please her son, the strong
     Iron-hearted man-slaying Achilles
   	   Who would not live long.

Voices In the Head

Another one from the Little Mag stable: Voices in the Head by Altaf Tyrewala. Some excerpts:

A woman who had been with her told me that Ma ran the fastest that morning. She pushed at the burgeoning crowds the hardest. She cursed the Devil the loudest. Like some hysterical lioness whose cub was being snatched away. But in that crowd, there were people far more desperate than Ma. People whose sons were worse than abortionists. They, too, wanted to attack and vanquish the Devil with all their might. These very people, these desperate, god-fearing fathers and mothers of sinners, were the ones who ran over Ma and pounded her body into the Holy Ground.

I see him often. He too works in Colaba. Has been a salesman in a shoe-shop for fifteen years. Sometimes, we take the same train to work. If we spot each other on the platform, we wordlessly board the same compartment and wade toward each other through the working-class crowd. Our actions would seem comic to an acquaintance. A father and son going through all this trouble to be near each other in a packed Harbour Line train, only to not exchange a single word. Like sulky kids.

I’d need a careless fuck-you attitude towards everything around me, including my own body, to throw away everything I have. Or else a greedy, gigantic embrace of everything life has to offer and spend my time doing, getting, doing, getting, getting, getting, getting…

No, people like me get the rawest of deals. We do our jobs. Live moderate, upright lives. Don’t drink. Don’t stone ourselves on bits of paper dipped in hallucinogenic acids. Have no burning ambitions.


What would you like to Read today?

Minekey was a wonderous experience. It all started off about a year and a half back, when Delip came to IIT Kharagpur with a vision to solve the world’s information overload problem. The aim was simple — let content consumers get access to the information they are interested in.

If we look at the world today, most content producers and aggregators produce content for the general audience. That means that the news will be of all flavors, all topics and categories. However, it also means that if you wish to track news on a particular topic from a plethora of sources (and there is no dearth of them!), it is like finding a needle in a haystack, or a key in an information mine – which is incidentally where Minekey got its name from (if you couldn’t guess it already ;-).

As it goes, a handful of super charged students from IITKGP got together with Delip and Prof. Sudeshna Sarkar, and started working on this new project with a new model for incubation. The company was based in Santa Clara, and we were doing the R&D and the initial bootstrapping from Kharagpur.

Ideas flew — personalization, networking, recommendation, search, social connections, groups, communities, feedback, ranking, clustering, collaborative clustering, geo-personalization — you name it. Lots of ideas, debates, deliberations, re-iterations, progress spreadsheets, throwaway code, reusing old code, search, open source services, days and months later, we had a strategy in place. We created a news portal for the world at large (which was still an Alpha), we had strong customer leads, a model seemed to be emerging.

A lot of water has since flowed in the Ganges. There is a definite strategy, the company is well funded, there are people who have left plum jobs to work at Minekey, the business model has been refined and Minekey is staring at an immense opportunity. Kudos to the current team to take a prototype and build a real product.

Minekey has now launched recommendations for blogs! You get a sweet looking widget on your sidebar in a matter of minutes, and your friends would be able to get recommendations. Minekey monitors their clicks and as the users click from the widget, the personalization kicks in, to recommend more and more stories according to the users taste. (Sadly, WordPress doesn’t support JavaScript, otherwise you would have seen one right here. I need to work on a workaround).

Go get it now!

NY to London in 63 easy steps

This takes the cake. You want to get from NY to London, and want the fastest, easiest, cheapest way to get there. Right?

Try Google, or specifically Google Maps. Go to ‘Get Directions’ and put in ‘New York’ in the source and ‘London’ in the destination fields respectively. Viola! You get 63 easy steps to get from NY to London.

Check step 23. Can you beat that? It’s the cheapest way without any doubt.

[For those lazy to follow the very simple directions above, please click here ;-)]


A very good piece of writing by Amit Chaudhuri – Surpanakha.

Andrew Tomkins on Web Search and Online Communities

Went to another of the talks in the Big Thinkers Series by Yahoo! Bangalore. Andrew Tomkins talked about Web Search and Online communities. Andy is the Director of Search Research at Yahoo! Research.

I was very disappointed by the talk. I had expected a lot more from the person who possibly determines the future of Web Search at one of the leading Search Engines in the world. The talk started off with Andy giving a slide show of images from Flickr which rate high on interestingness. That was the good part – the pictures were cool. But thereafter he went into why Flickr was a better social network than other networks — he gave some quantifiable metrics such as the size maximal strongly connected components in the relationship graph and the number of nodes in the graph with a degree of more than a number k, which was parameterized on the X-axis. For Flickr, it seems there are a number of people with 450 friends or more while for another social networking site (a la LinkedIn/Orkut) the number is an order of magnitude smaller. I did not buy his argument that this indicates that Flickr was a more successful social network. Being able to maintain 450 friends is very difficult (ask me! I don’t even interact with many of my 800 odd friends on Orkut). Besides the nature of the two social networks was very different. He also touched upon how social networks are interacting (Upcoming and Flickr).

He then went into how the Internet is growing and the amount of data being generated. Some back of the envelopes (6B people typing away on computers for 4 hours everyday) would generate data with an upper bound of about 150 PetaBytes. However, that data is more and more decentralized. The amount of data which passes through Yahoo! network is only about 11% of the web’s data right now and that is falling fast. Nobody else even comes close (according to Andy). At the same time, one can consume only the data one wants — thanks of course to del.icio.us and RSS feeds and better personalization algorithms. That indicates that both content sources and content consumption is becoming more and more decentralized and democratized. At current rates, the storage of the amount of data being created will cost about $ 25M which should fall. Smaller players can crawl and store the content present on the web. This is great for entrepreneurers because this means that they can match the Big 3 (GOOG, MSFT, YHOO) at least in terms of storage!

Some of the latest developments in search have been special treatment of specialized domains. For instance, if you search for weather, or movies, or flight information, current generation search engines are able to figure out the domain of the search query and provide custom UI for the results based on the domain. For instance, they might give movie timings at theaters near your ZIP code. This is going to become more ubiquitous with special treatment for a lot more domains being added. However, I am not sure how many domains can be supported by a rule-based treatment for each of them. Integration of search results of different types and genres of media (images, video, text) and ranking algorithms for the combined result set remain a challenge. We are going to see the addition of more and more social features as days go by. Crawling and collecting data in the light of new programming models (like Ajax) are going to be a challenge. Andy was not aware of any good solutions to this problem.

The last part of the talk was a real disappointment. He started talking about some of his recent research on estimating properties about a hidden corpus on the basis to the answers to a number of queries. While there is no doubting that his research is worthwhile, it was perhaps a wrong forum to get into mathematical equations and that too suddenly after having talked about general technology. I got the feeling that he wanted to talk about it just to show that he still does some technical work :-).

Overall, the talk didn’t meet my expectations. The last one by Raghu Ramakrishnan was by far better.

[If I have missed out something, please point out in the comments. Thanks!]

School Chale Hum

This is a video by Bharatbala productions on the Sarva Siksha Abhiyaan (Wikipedia). The music is by Shankar, Ehsaan, Loy. The video shows students from all over the country running to school in the morning. The blurb reads:

192 million children between 6-14 years of age across 1.1 million places in India are not going to school. This film for Sarva Shiksha Abhiyan (Universilisation of Elementary Education) addresses the needs of these children.

The film catches the moment when children all across India from Kashmir to Kerala wake up in the morning and run to go to school.

Directed by: Kanika and Bala, Bharatbala Productions (BBP) for the Ministry of Human Resource Development, India.

It is good to see the government releasing such inspiring videos for its campaign. We need a lot of infrastructure, organization, resource mobilization (both material and human), and determination if such a bold target has to be achieved. There is not doubting that universal education, which only remains a dream at this point in India, will unlock the potential of millions of children and help them build a respectable and rewarding life for themselves.

It is sad that even now, after 60 years of independence, we have been unable to provide the right of education to its children. Unless we are united in our campaign to bring the benefits of education to the population at large, we will find unemployment spiralling up as these young kids grow up. Unemployment breeds most other social evils since ‘an idle mind is the devil’s workshop’. As the blurb above says 192 million children are unable to receive the benefits of education. As these children grow up and we are unable to provide for their ‘fish’ everyday since we have been unsuccessful in ‘teaching them how to fish’, we will find that more and more of them will turn to unacceptable and often anti-social means of earning their livelihood.

As the discontent grows into unrest, we will find our country being divided into a ‘developed India’ and a ‘under-developed India’. This is foreboding since unlike developing and developed nations, the two Indias will be geographically co-located and very finely interspersed. In a shirt, every thread is important. Even if one thread is unable to hold together, a gaping hole appears which only grows with time. No amount of stitching can help (which most of our upliftment programs try to do). It is important to ensure all threads are strong and sturdy right from the beginning. Disharmony and unemployment has the potential to cast a shadow on the shining India we take so much pride in.

A divided India will bring the progressive India down like a pack of cards.

If those of us who have been lucky to receive the benefits of education, and a world of opportunities opening to us feel that we can progress and enrich our lives while these kids remain illiterate, we are only fooling ourselves. We need to worry about the India we see outside our tinted car windows, the children who should be going to school but go to work, kids who if guided well will be India’s strength, but if left misguided will pull it back.

It is also important to focus on the education of the girl child. While girls have equalled and even surpassed men in many spheres, vast hinterlands of the country still don’t understand the need of the equality, which is sad considering that in a lot of cases women are actually the breadwinners of the family. Even in the cities, we have so many maids working in our houses. I don’t have any statistics but I am confident that in a lot of cases they just dont supplement the income of the man of the house but in fact perhaps are the sole breadwinners in the family. I am certain that instances of alcoholism and other social evils are much less amongst women than men in the lower strata of the society, and it certainly makes more sense that the money stays with them which would be used for caring for the family, rather than spent on a drinking binge. Not long back I went to a temple where we paid our guide (who took us around) handsomely since we were happy with the services rendered. The next morning I found him stinking of cheap alcohol. I just felt sorry for his family who could not even get the benefits.

It is important that we realize that with the family cushion gradually degenerating from around us, it is essential that girls can not only read and write, but also be able to pursue opportunities in at least one vocation so that they can support their family in case of any adversity. They could actually do a better job at it. And it’s not the elite who need to understand this, it is really the multitude of Indians who despise modern society who need to get this hammered into theri heads!

%d bloggers like this: