A recent posting on the Official Google Blog by a senior search engineer reveals some more information about how the Google search engine ranks sites and the ongoing work that goes into improving ranking results. As the article says, the many ranking criteria that drive the search algorithms remain a trade secret to protect themselves from competitors and abuse of the system, but more insight is provided here.

It outlines all the different factors that can make an automatic assessment of a web page a difficult task and the need to match a short search query with the most relevant results within milliseconds. This can only have become harder with the introduction of 'universal' search results over the past year. The original PageRank algorithm remains a core part of Google's ranking criteria but this is combined with other issues such as different models to cope with language, query usage, recency, personalisation and regional results.

A team of engineers works on the evaluation of search results quality and many changes or enhancements can be made during the year - for example, in 2007, over 450 new improvements were introduced, ranging from simple improvements to more complicated changes. The article reveals that significant changes were made to the PageRank algorithm in January which could have dramatically affected the rankings for some websites.

Other teams work on new features and new user interfaces, with the latter group assisted by a team of usability experts who conduct user studies and evaluate new features with Google users around the world. Then, of course, there is the team of engineers who focus on fighting webspam and other types of search engine abuse, such as hidden text to off-topic pages stuffed with irrelevant keywords and other attempts to fix ranking positions. The team may spot new spam trends (or have them reported to them) and then works to counter those trends within the ranking algorithms.

Overall it's a well-written, clearly explained introduction to what goes on 'behind the scenes' at Google and indicates a new willingness to share some of this information with the wider public. It's also a sign of better PR being undertaken with possibly an attempt to avert some of the more negative press and comment that a company of Google's size and prominence starts to attract.

The Yahoo! Search Marketing blog comments on some of the recent results that emerged from research conducted by JupiterResearch for iProspect in the US (which we reported on last month). The 2 tables shown in the blog entry illustrate how search behaviour has changed over the past 6 years with users apparently becoming more impatient.

For example, in answer to the question of how many results users tend to look at before clicking on a link, 16% in 2002 said 'just a few' whereas in 2008 that figure had increased to 27%. The same figures for those scanning the whole of the first page has also changed from 32% six years ago to 41% this year. These figures also imply that 68% of searchers will tend to click on one or more results within the first page in 2008, compared to 48% in 2002.

A second question, which asked searchers at which point they revised their search query or tried another search engines if they didn't find what they were looking for, shows that in 2002 14% would do this after reviewing just a few results, compared to 23% this year. Those who revised their search after reviewing the first page of results rose from 14% in 2002 to 26% in 2008.

These results do show an interesting trend which may indicate the impatience of searchers, or an improved level of searching skill using more key terms to find a specific result. It can also indicate the improved relevancy of results (and probably more people now using Google) as well as the higher number of optimised sites appearing in the search results and, through highlighted content, appearing to give users the results they are looking for.

Of course the Yahoo! article encourages website owners to target their paid search advertising to create relevant rankings for users and to get first page positions to attract the majority of the search traffic. It also recommends focusing optimisation efforts on all aspects of the 'blended' search results that are now being served up - such as news stories, video, images and local business listings - to ensure that all opportunities are being taken for the website to put itself infront of relevant searchers.

In a new move widely reported in the press (including by the Washington Post), Microsoft has announced a new incentive to try to encourage more users to conduct their online shopping through the Live Search engine. Cash incentives are being offered to shoppers through the new 'Live Search Cashback' scheme, so that listed products could offer users discounts of up to 10% of the sale price.

This is a bold - or desperate - move by Microsoft, depending on how it works. The clear intention is to provide a monetary incentive to get users to change their search habits, since the quality of the search results offered by Live Search don't seem to be making any headway against the dominance of Google. It may certainly lead to more traffic from online shoppers looking for a bargain, but it may not significantly change their online search habits.

Microsoft says that its new cashback program covers more than 10 million products from over 700 merchants. These participating merchants will pay Microsoft a fee each time a customer completes a sale through Live Search Cashback and this fee will be a percentage of the retail price. When the purchase is completed, Microsoft will then return the fee to the consumer in the form of a cash rebate, so in reality Microsoft is simply acting as a middleman through their search tool, in the form of an affiliate programme.

The Washington Post article reports that this new scheme is part of Microsoft's plan to "innovate and disrupt" in the search industry, and certainly it is thought that if elements of the cashback plan are adopted more widely by other search services, then it is likely to change the Internet advertising business in fundamental ways, including adapting the ways that search engines provide an 'independent' service, to one where conversions and profits could drive search results.

Not surprisingly, the Microsoft - Yahoo! story is unlikely to go away for some time and there are still possible discussions and initiatives to be developed by either party to get to their ultimate objective. As widely reported, including by The Sydney Morning Herald, Microsoft are apparently talking to Yahoo! again about some form of joint partnership, partly to keep the door open to an eventual takeover but also to block any moves that Google may be trying to fill the space of the abandoned acquisition attempt.

Yahoo! is also having to keep an eye on a potential proxy fight from shareholders, led my billionaire investor Carl Icahn who has criticised the Yahoo! board for their handling of the Microsoft offer. The rumours are that Microsoft are discussion some form of joint advertising arrangement with Yahoo!, based on search advertising or targeted display advertising, to try to ward off a stronger partnership being formed between Yahoo! and Google.

There's almost an inevitability that the 3 main players within the search market will merge in some format eventually -and most likely a Microsoft-Yahoo! deal - which will not be good news for the overall market and advertisers, but probably a natural progression for the runners-up to try to tackle Google's strengthening monopoly.

Google has said that the company is witnessing a "significant inflow" of advertising spend from companies who are moving their budgets from mainstream 'above the line' media into more targeted and measurable forms of marketing like PPC advertising.

As reported by The Sydney Morning Herald, there are mixed reports about how the threat of a recession is affecting companies, but this trend does seem to indicate concerns from companies who want to make the most of their advertising spend when budgets are being tightened, either in response or anticipation of a market downturn. In light of this trend, the reported 30% growth in search advertising in Australia for the March quarter could be significantly underestimated.

At the same time as this continued growth in Internet advertising continues, another article in the SMH claims that, dollar for dollar, the Internet accounts for more carbon emissions than any other form of advertising. A study by consultancy P3 estimated the environmental cost of different forms of advertising and claimed that Internet advertising is the equivalent of junk mail, with high levels of water generated by large numbers of page impressions.

Perhaps this is good publicity for the green auditing division of the company publishing the research but it also raises many questions about the methodology of the research.

The New York Times reports that Google executives have expressed a desire to develop the advertising relationship with Yahoo! that was tested for several weeks recently with AdWords appearing alongside Yahoo!'s search results. This is partly a move to ward off any possible future attempts by Microsoft to make a bid for Yahoo! but could also be a lucrative development for both Google and Yahoo!

Any arrangement, should it go ahead, would face antitrust investigation in the US due to the dominance of the search advertising market by the two companies. It would also have serious implications for Yahoo!'s own PPC advertising system, which is still struggling to compete against Google, despite a facelift and relaunch in recent years. Such a move would also place even more power within Google's control and advertisers would have even fewer options to place search advertising so that bid rates would continue to rise.

An interesting article from The Boston Globe explains how more companies are starting to embrace the online community and include negative feedback from customers on their websites, as well as positive ones. The main example given is for the Bank of America who have now decided to be open to criticism on their products and services, however harsh it may be.

This trend reflects the growing use of consumer opinion sites, or comments made on discussion forums or blogs. The new attitude of these companies is that it's better to have some awareness and control of these negative comments as well as to balance off all the positive ones that it receives (although of course it's human nature to make complaints rather than pass praise).

Such a move provides a challenge to companies who want to protect their brand and implement reputation management strategies online. It does give them a chance to respond and may well attract more criticism directly to the corporate site rather than through other channels. Of course the same issues remain as always, with the opportunities for competitors to pitch in or for the company to provide their own positive reviews!

The latest quarterly figures for online advertising spend in Australia have been released by the Interactive Advertising Bureau, using data collected by Pricewaterhouse-Coopers. These show that online display advertising has lost further market share in this sector to classified and search advertising, although the sector as a whole continues to show good growth.

The figures just published for the quarter to March show that the total online advertising market grew by over 30% on the same period last year, up $90.5m to $384.5m. Display advertising's share of this spend fell from 25.5% to 24.6% year-on-year, whereas classifieds grew from 27% to 27.7% and search rose from 47.4% to 47.7% - although this latter sector is still dominated by Google who refuse to reveal actual figures, so this data is based on PwC's estimate.

However, the change in sector share between the March quarter and the previous quarter to the end of December shows that display advertising lost 3 share points while classifieds rose 1.8 points and search by 1.2 share points. This may be indicating the start of a slowdown in online advertising due to possible concerns in an economic slowdown, although overall the market is still growing at a much healthier rate than any other advertising sector.

A recent article by The New York Times reports on a research paper presented at the International World Wide Web Conference in Beijing by 2 Google scientists, which described the development of 'VisualRank' by the search engine. This is the prototype algorithm that combines image-recognition software techniques with the methods for weighting and ranking images that look most similar. Described as the equivalent of Google's main PageRank system adapted for digital images, it's being seen as potentially a big step forward for the quality of image searching on the web.

Image search has long been one of the most popular 'vertical' search options on the main search engines and more recently Google, Ask and the others have been 'blending' image results into the main search results list, when relevant, thereby giving image search results far more exposure to mainstream searchers. Up until now, image results are usually generated from information in the file name and surrounding page content, but Google - and others - are working on ways to analyse image form and shape similarities.

The article says that Google has been working on a sample of around 2,000 of the most popular product queries from Google’s product search - so items such as iPod and Xbox - and then sorting the top 10 images both from its main ranking system and the standard Google image search results. Using a team of 150 Google employees, the researchers then created a scoring system for image “relevance”, which is being referred to as VisualRank. The outcome of this development, according to the researchers, was to retrieve new image results that were 83% less irrelevant.

The ongoing saga of Microsoft's attempts to buy Yahoo! came to an abrupt halt on Saturday when a top level meetings between the executives of both companies failed to reach an agreement. Despite Microsoft upping their bid price to nearly US$50bn, Yahoo! still held out their position that the bid was undervaluing the business and so Microsoft finally withdrew their offer.

Shares in Yahoo! fell as news of this development reached the markets, and the question now remains - what will Yahoo! do now? Will they continue to pursue the other business alignments that were being investigated during the takeover offer period, such as with AOL, or try to build on the recent advertising trial with Google?

It's no doubt good news for the web search market that Microsoft hasn't pursued this acquisition to a successful conclusion, but it now leaves Yahoo! in a weaker position from which they need to develop their business and services to help them maintain their position in the market, and to protect themselves from other predatory takeover attempts. Microsoft are also likely to be reviewing the outcome and deciding where to spend their cash now and it may eventually come about that another purchase attempt will be made if Yahoo!'s value takes another hit in the future.

The new May issue of the monthly Web Marketing newsletter has been published, covering some of the recent stories on web search and online marketing trends.

This month's edition looks at several pieces of research that have just been published. The first analyses the trend for online shopping in Australia and how growth seems to have halted due to the limitations being set by retailers. The second research report looks at how web searchers are using 'universal' search results on Google and the other main search engines and what this means for search engine marketing campaigns. Finally the newsletter takes a technical look at the issues involved when moving a website to a new hosting location or domain name and what impact this can have on search engine indexing and rankings.

If you want to sign up for future issues of this newsletter, please do so by using the form at the bottom of this page. To view back issues of this newsletter you can see the archive by date or by subject.