The Google Webmaster blog has announced a 'secret project' that the company has been working on - the next-generation architecture for Google's web search engine. And now they are opening the test siteup to users, requesting feedback on the performance of the new technology.

Google says their aim is to improve the size, indexing speed, accuracy and comprehensiveness of their search experience, but there's little coincidence that this announcement comes soon as the launch of Microsoft's new Bing search engine and the attention that has generated. Of course, this Google announcement has generated lots of press and online activity and comment, such as this favourable review from Information Week.

Dubbed 'Google Caffeine' the new search engine infrastructure can be viewed and tested at http://www2.sandbox.google.com/.

The new search engine seems to be faster, which probably reflects lower usage and less integration of 'universal' search results at this stage. Some searches show little difference in results to the existing search engine, whereas others do show a notable change in ranking positions, so there could be implications for some companies relying on search engine optimisation performance for their site visits. This new version will continue to be reviewed and assessed over the next few months to see what wider impact it may have on the search and online business market.

An article in the New York Times reviews the issue of the 'Deep Web' - sometimes known as the 'Invisible Web' - and the difficulties for search engines to find this information. Despite Google claiming to now index over one trillion web pages, this is still believed to represents just a fraction of the entire web, since there is much more content that lies beyond the reach of search engines - such as database information, content controlled by login access, financial information, shopping catalogues, medical research, transport timetables and more.

The report focuses on a number of new search and index technologies that are trying to improve this coverage of the web's hidden content, such as Kosmix and DeepPeep. The former service, for example, has developed software that matches searches with the databases most likely to yield relevant information, then returns an overview of the topic drawn from multiple sources. If tools such as this do manage to delve deeper into the web's content, the quality and application of search results will be greatly expanded and, as the article claims, could ultimately reshape the way many companies do business online.

Adding another post in their series about the 'inner workings' of Google, the Official Google Blog published an article earlier this week about the process of search evaluation - how they measure the quality of search results being produced and how they meet users' expectations.

As the post illustrates, there are many different factors that need to be taken into account and different approaches considered, but ultimately Google (and any other serious search engine) need to constantly review and revise the ways that search results are generated automatically and can meet the needs of the user as closely as possible, since that is the way that users will be retained as regular users.

There are 4 main stages of evaluating search results - understanding what a user's intent is when they search, comparing the quality of search results from different sources, assessing what is 'good' or most relevant, and finally considering different geographic locals and search options (particularly now that Google's universal search results combine answers from other sources, such as news, local search or video).

Google uses a team of human evaluators around the world to conduct search tests, as well as live traffic experiments to assess and review search quality. This can include statistical evaluation and a review of user actions, or a manual rating of the appropriateness, usefulness, and relevance of each individual related search suggestion. Clearly this can be a never-ending process that will never be perfect but at least tries to continually enhance the search experience for users as the knowledge and technology of search develops.

The Google Webmaster blog has outlined ways that they are now developing better access to Adobe Flash files, which have traditionally been very difficult for search engine spiders to index, either for content or to follow links. Adobe has been working with Google and with Yahoo! to improve the performance of these files, which is certainly progress but still doesn't make Flash as accessible as standard HTML content.

Google reports that they are now able to better index text content within all types of SWF files now, as well as identifying URL links and following these to index additional content or page files that may be linked from the Flash file. However, images and video files cannot be indexed in this way. There are also limitations with Flash files loaded through Javascript, as these can't currently be identified and indexed, nor will content loaded from an external source within the Flash file.

Further developments are apparently still underway and this area may improve further in the future, so that Flash developers need to be aware of these requirements if the web content needs to be indexed for search.