Popularity: 2% [?]

How Google Ranks Tweets

To deliver useful search returns from the so-called real-time Web–such as seconds-old Twitter “tweets” reporting traffic jams–Google has adapted its page-ranking technology and developed new algorithmic tricks and filters to keep returns relevant, according to a leading Google engineer.

Google rolled out real-time search technology last month, to offer searchers access to brand-new blog posts and news items far faster than the five to 15 minutes it previously took Google’s Web crawlers to discover newly created items.

Bing, Cuil, and other search engines also provide various kinds of real-time results. Both Google and Bing have also forged major deals with Twitter to get real-time access to tweets, those 140-character microblog posts sent out by Twitter members. But Google claims to offer the most comprehensive real-time results by scanning news headlines, blogs, and feeds from Facebook, MySpace, Twitter, and other sources.

The tweets are a mainstay of Google’s real-time results, but Google has not previously discussed how it ranks them. A fundamental Google strategy for identifying tweet relevance is analogous to that used by Google’s PageRank technology, which helps find relevant Web pages with traditional Web search. Under PageRank, Google judges the importance of pages containing a given search keyword in part by looking at the pages’ link structure. The more pages that link to a page–and the more pages linking to the linkers–the more relevant the original page.

In the case of tweets, the key is to identify “reputed followers,” says Amit Singhal, a Google Fellow, who led development of real-time search. (Twitterers “follow” the comments of other Twitterers they’ve selected, and are themselves “followed.”)

“You earn reputation, and then you give reputation. If lots of people follow you, and then you follow someone–then even though this [new person] does not have lots of followers,” his tweet is deemed valuable because his followers are themselves followed widely, Singhal says. It is “definitely, definitely” more than a popularity contest, he adds.

“One user following another in social media is analogous to one page linking to another on the Web. Both are a form of recommendation,” Singhal says. “As high-quality pages link to another page on the Web, the quality of the linked-to page goes up. Likewise, in social media, as established users follow another user, the quality of the followed user goes up as well.”

But Google’s social-ranking tricks are hardly the only method the search giant uses to extract relevance from tweets. Google also developed new ways to choose which (if any) tweets to surface for common terms like “Obama”–and to avoid spam or low-quality tweets–all within seconds.

Popularity: 1% [?]

We saw innovations in real-time search, streaming video, and an operating system for the cloud.

This year will be remembered for cloud computing, real-time search, and the appearance of Google’s Web-based operating system.

Credit: Microsoft

Real-Time Search

The Web’s dominant search company, Google, got some serious competition in 2009. Thanks to a long-awaited technology-sharing deal with Yahoo and a focus on product-related searches, Microsoft’s revamped search engine, Bing, began nipping at Google’s heels (“What’s Microsoft’s Bing Strategy?”).

The physicist Stephen Wolfram also shook up the search-engine scene by developing a “computational knowledge engine” designed to provide all sorts of useful information via a search-like interface (“Search Me,” “Alpha and Google Face Off,” and “Wolfram Alpha Braces for Overload“). The arrival of Wolfram Alpha also forced Google to explore a more sophisticated approach to presenting meaningful information from online databases (“Google Unveils Google Squared“).

In the battle to gain an edge, both Google and Microsoft turned their attention to the “real-time Web” and this year’s hottest Web company, Twitter. Keen to tap into the freshest online information, these search engines began delivering seconds-old snippets from Twitter, as well as other sites in their results (“Google Takes Search Real-Time“). Several startups also hope to tap into real-time online activity (“In Search of What Everyone’s Clicking“). But collecting real-time information and presenting it in meaningful ways remains a tricky challenge, as Google CEO Eric Schmidt explained during recent a visit to Cambridge, MA (“Searching for Real-Time Search“).

Faster search is just one sign of a general increase in the speed of information flow across the Web. This year Internet experts revealed new protocols for collecting bits of related information no matter where they appear online (“Who’s Talking About Me?“). Others focused on analyzing the huge quantities of data being produced at any given moment (“Startups Mine the Real-Time Web“).

In Search of Context

With so much interest in mining Twitter posts, the trustworthiness of online information was another hot issue. Several research projects and startups have revealed ways to automatically determine who can be trusted to provide accurate information on a particular topic (“A Smarter Way to Dig Up Experts” and “Computers Can’t Answer Everything“). Researchers have also developed ways to add context to information posted online. In this area, Wikipedia has come under particular scrutiny (“Who’s Messing With Wikipedia?” and “Adding Trust to Wikipedia, and Beyond“).

A Web OS

Even as Microsoft encroached on Google’s turf, the search giant took a bold step into Microsoft’s backyard. Google’s browser, Chrome, evolved into a complete operating system, as the search giant continued to press into ever-broader areas of computing technology (“An Operating System for the Cloud“). The Chrome OS isn’t yet available to the public, but there have been some early indications of the radical approach that Google has taken (“Hints of How Google’s OS Will Work” and “Google Gives a First Look at the Chrome OS“).

Google’s vision relies on the power of Web applications and sophisticated browser technology. New browser features will make it easier to work both online and off, to handle heavier processing tasks, and to incorporate multimedia into the browser without plug-ins (“An Upgrade for the Web“).

Cloud Computing

Businesses everywhere were still fascinated by the promised convenience and cost savings of cloud computing (“Technology Overview: Conjuring Clouds“). But as companies began adopting the technology this year, kinks that still need to be worked out revealed themselves (“Industry Challenges: The Standards Question“). Companies have found it difficult to move data between different cloud services, even though some startups are now offering solutions (“Virtual Apps Drift Into the Cloud” and “Moving Data around the Clouds“). Security has been another worry (“Vulnerability Seen in Amazon’s Cloud Computing“). And some companies turned to different kinds of cloud computing that offer greater security and control (“A More Secure, Trustworthy Cloud” and “Big Blue Sees Clouds on the Horizon“).

Content Transformed

Another of the year’s big trends was the growth of online streaming media. This is one reason that downloading copyrighted material over file-sharing networks seems to be on the way out (“Peer-to-Peer File Sharing Usurped by Streaming Video“). Much excitement was also generated over a streaming media project called OnLive, which promises to bring high-end video games to users whose devices don’t have the processing power to handle the advanced graphics (“Moving Video Games to the Clouds“).

Companies in a variety of media industries have also looked for ways to use piracy to make money off content (“Embracing Piracy“). And the music industry sought entirely new ways to sell music online (“Can Video Games Be the New MTV?” and “What Will Happen to Lala’s Music Plans?”).

Source: http://www.technologyreview.com/web/24230/page2/

Popularity: 6% [?]

Researchers are working on ways to make encrypted data easier to find.

Recent advances in cryptography could mean that future cloud computing services will not only be able to encrypt documents to keep them safe in the cloud–but also make it possible to search and retrieve this information without first decrypting it, researchers say.

Credit: Technology Review

“This will be a challenging endeavor,” says Dawn Song, a computer scientist at the University of California, Berkeley, who has made fundamental research contributions to using encrypted search strings to find encrypted documents. “However, some of these recent advances are very powerful and, if cleverly engineered and deployed, could lead to significant advances,” in adding security and privacy to cloud computing over the next few years.

At the ACM Cloud Computing Security Workshop in Chicago tomorrow, Microsoft Research will propose a theoretical architecture that would stitch together several cryptographic technologies in various stages of development to make the encrypted cloud more searchable. The basic idea is that cloud users could download software that would encrypt their data before it’s sent into the cloud. In addition, the software would issue encrypted strings, called tokens, which can be used to check that documents are intact and–crucially–to search their contents without first having to decrypt them.

While the underlying technologies weren’t developed by Microsoft, “we want to show how existing and emerging cryptographic techniques can be combined to make data in the cloud more secure,” says Kristin Lauter, head of the Cryptography Group at Microsoft Research, who will describe the proposal tomorrow.

While cloud computing has exploded in popularity in recent years thanks to the potential efficiency and cost savings of outsourcing the management of data and applications, a few high-profile glitches and hacks have left many potential users worried, and prompted experts to suggest that new technologies may be needed.

For example, early this year, a hacker who guessed the correct answer to a Twitter employee’s security question was able to extract all of the documents stored in Twitter’s “Google Apps” account. And, in March this year, a software bug led to a foul-up in the sharing privileges of Google Docs. As a result, for a small number of users (a fraction of 1 percent), choosing to share a single document instantly gave that contact access to all other shared documents, too.

Popularity: 27% [?]

A real-time search engine bases its results on users’ browsing habits.

Later this week, a new “real-time” and “social” search engine called Wowd will open a beta version of its service to the public. The company says that its search results include only pages that have actually been visited by its users, and that its ranking algorithms offer information based on its freshness and popularity.

Real-time search: Wowd indexes pages visited by its users and ranks them based on either their popularity or their freshness. Credit: Wowd

According to CEO Mark Drummond, Wowd is trying to strike a balance between the up-to-date but chaotic results produced by a site like Twitter and the slower-to-change results that come from traditional search engines such as Google. He expects Wowd to be particularly valuable to users who want to know what content is currently popular, and who see search as an additional feature.

“If what you’re interested in is reference search, Google is great, and we don’t imagine that we or anyone else is going to displace that,” Drummond says. “If, instead, you want to find out what the planet is talking about, that’s where we think Wowd can win.”

Anyone will be able to visit Wowd’s website and perform basic searches, but the company hopes that many users will agree to install Wowd’s software on their machines and have their browsing tracked anonymously. This is how the company hopes to build its search index. Unlike traditional search engines, which use crawlers to build up a database of pages on the Web and store that information in vast data centers, Wowd’s users will be both the crawler and the storage.

Drummond says users have an incentive to install the code because it also unlocks additional features, including a recommendation engine and advanced search features such as the option to search through one’s own Web history.

Users who agree to install the software will feed back to Wowd a list of the sites they visit. Drummond says the company views a visit to a Web page as evidence that the page contains some interesting content. The idea is similar to that used by recommendation sites such as Digg or StumbleUpon, only Wowd offers users a more passive way to participate. “Voting for something [on these sites] requires an order of magnitude more emotional commitment than just looking at something,” Drummond says.

Wowd’s system will also store bits of its search index on each user’s computer. The company uses an algorithm called Kademlia, which is more commonly employed in peer-to-peer file-sharing systems, to keep its index available even when some users shut their computers down. This is done by storing each bit of information on several machines and by using statistical analysis to determine how to share it in order to keep it available.

Drummond stresses that the system is designed to safeguard privacy. For example, when a user visits a page–nominating it for Wowd’s index in the process–that page is actually accessed and indexed by a different user’s computer. This process protects against accidentally collecting information that is protected by passwords or cookies. The site also does not index any pages protected by encrypted Internet protocols.

Wowd doesn’t simply report on what its users are doing in real time, as some Twitter search engines do, for example. Instead it uses its own algorithms that balance a page’s freshness against its apparent popularity. When searching in Wowd, a user can choose to see results ranked by popularity or by freshness. In either case, the page will automatically update and adjust its results over time if the user leaves it up.

Wowd contains several ways for users to discover new content, including a “hot list” on its front page that simply shows the top sites being visited by users. Later, Drummond says, the site will add more ways for people to discover what’s popular in particular interest areas.

Other search companies, for example the German company Faroo, have also tried to use a distributed architecture to obtain more real-time search results. Faroo’s CEO, Wolf Garbe, says that the approach significantly reduces the costs for a search startup. He also believes that any real-time search engine will have to depend on users in some way, since even a company with resources as large as Google’s can’t crawl the Web anew every five minutes.

Petar Maymounkov, a researcher at MIT who helped design the Kademlia algorithm, says that Kademlia is suited to Wowd’s architecture. However, he notes that Kademlia isn’t well-protected against malicious users who may want to game the system. Whether any distributed architecture can be secure when there’s no central control over who participates remains an open question, he says.

Maymounkov adds that it will be interesting to see how well Kademlia works as the foundation for a search engine. In the 10 years that the algorithm has been used, he explains, it’s mainly powered file sharing. And since file sharing is most often used to obtain free movies and music, he says that users of such software tend to be willing to donate a lot of computing power and to put up with some hiccups in availability to retrieve that information. User expectations for a general search engine may be different, he says.

Wowd “is a nice idea, but they’ve got some work to do,” says Daniel Tunkelang, cofounder of a search company called Endeca, based in Cambridge, MA. The preview version of the site, in his opinion, doesn’t provide enough high-quality search results to make it stand out from other attempts at real-time search. As more users sign up after the site’s launch, this may provide better-quality data and improve the results, Tunkelang says, adding that he’s reserving judgment until then.

Source: http://www.technologyreview.com/web/23756/page2/

Popularity: 100% [?]

Next Page »