Starting with version 3.5 Apache Solr comes with a stemmer that uses Hunspell. The Hunspell stemmer works with dictionaries and rules to do its job. The great thing is that the Hunspell stemmer can use the dictionaries and rules provided by the Apache OpenOffice project.

The dictionaries and rules are provided as OpenOffice extensions. Download the dictionary extension you need and simply unzip it - the extensions are simple zip files. In the unpacked extension you'll find a .aff file and a .dic file with the same name, these are the files needed to get your Hunspell stemmer working.

Example for slovenian Hunspell stemmer:

<filter class=”org.apache.solr.analysis.HunspellStemFilterFactory” dictionary=”sl_SL.dic” affix=”sl_SL.aff”>

Using these OpenOffice dictionaries you get stemming support for a lot of languages.

After having a quick look at the specification it seemed an easy enough task for the night. At the end of the night we actually got most of the needed things implemented in a way so that it is very easy to set up in just two steps.

• First add the new OpenSearch TypoScript template
• Then adjust the name and the description of your site's search

The TypoScript template configure's your page object and add a new page type providing the OpenSearch description file for the browsers. After these two simple steps your browser will detect the site's search and you can start using it from the browser's search field.

We hope you enjoy this little new feature when it will ship in the next version.

This is mainly a maintenance release with quite a few bug fixes and a hand full of small new features. An update is encouraged.

New in this release

• Failed items in Index Queue no longer stop the whole indexer
• Improved logging of failures during page indexing 
• Removed the result count from the suggest drop down - caused too much confusion
• Change of index document ID - Requires re-indexing of your content!
• Status Report checks for allow_url_fopen = On and a PHP bug in filter_var()
• Example Index Queue configuration for EXT:news
• Support for content fallback
• Support for FILE: path in LLL viewhelper
• (EAP) File Detector for files linked in Templavoila FCE 

Notable bug fixes

• Fixed handling of multivalue fields in template output
• Fixed issues with Mount Pages indexing an Index Queue initialization
• Fixed handling of filters configured through TypoScript
• (EAP) Garbage collection of files in indexes other than the default language
• Fixed some issues with Workspaces 

Updating

Simply update the extension, no need to update anything with the Solr server. You must re-index all your content though due to a change in the index document ID format: Go to the search module, empty index, and initialize the Index Queue. This will cause the Index Queue worker task to re-index your site.

Contributors

(patches, comments, bug reports, ... no particular order)

• Jigal van Hemert
• Dmitry Dulepov
• Jochem de Groot
• Pierrick Caillon
• Michael Voehringer
• Stefan Galinski
• Georg Ringer
• Markus Friedrich
• Dorit Rottner
• Hendrik Nadler
• Michael Knoll
• Thorsten Kahler
• Lars Tode
• Ivan Kartolo
• Constantin Lebrecht

Thanks to everyone who helped with this release!

The issue was that in our TYPO3 specific Solr schema.xml we have a field called the siteHash. The site hash helps to filter results so that you can index multiple sites into one index while still only getting results of the site you are on.

To allow the documents indexed using Nutch to be found they also need to have a site hash. Of course does not know about the site hash and its implementation; to overcome this we now provide a small REST style API to request a site hash to be used when indexing external resources.

A GET request to
www.typo3-solr.com/index.php &api=siteHash&domain=www.dkd.de
would then for example provide the site hash needed to index a page from www.dkd.de.

To prevent abuse we also added a protection for the API by having to provide the correct API key. By that you are now able to index external resources into a Solr index that is searched by TYPO3.

This and lots more will be part of the soon to be released version 2.0 of Apache Solr for TYPO3.

With the Index Queue  we provide a way to define an easy mapping of database record fields to Solr index fields. TYPO3 is awesome in how flexible it is in regards to page rendering; in this case this kind of turned out to be a disadvantage at first since you can't (read: want to) "re-implement" all the logic. So to index pages we will still rely on frontend indexing, but it's still way better than the classic way. Even better, the Index Queue will find any access restrictions and translations and take care of indexing those variants automatically, too.

The Index Queue comes with a Record Monitor to track any changes and can then re-index just those changed records or pages. In the same way the Garbage Collector will clean up after your editors when they delete records.

So the Index Queue is an easy and fast way to get your site's pages and extension records indexed. However, raw database data often is not what you want in your index right away, usually you want to do some cleaning or transformation before handing the data to Solr. Because of that the Index Queue also supports mapping TYPO3 content objects (cObj) - or more specifically their output - into index fields. To make it even easier to index custom extension records we came up with custom content objects provided by the Apache Solr for TYPO3 extension.

Currently there are three content objects:

• SOLR_CONTENT - cleans database fields like rich text editor fields from HTML code and formatting
• SOLR_MULTIVALUE - allows to index data into - you guessed it - Solr multivalue fields, acting like the PHP explode() functions
• SOLR_RELATION - resolving relations between records, indexing data into multivalue or singlevalue fields. Useful for news and news categories f.e.

We hope you are looking forward to EXT:solr v2.0 as we do!

EXT:solr Templating explained

The issue that appeared was that you cannot use the Solr schema provided by our TYPO3 extension because it requires several non-default fields to be set.

The most important field is sitehash. It usually contains a MD5 hash of the site's domain, the encryption key and the extension name. The implementation of the site hash is located in tx_solr_Site (classes/class.tx_solr_site.php):

/** 
* Generates the site's unique Site Hash.
*
* The Site Hash is build from the site's main domain, the system encryption
* key, and the extension "tx_solr". These components are concatenated and
* md5-hashed.
*
* @return	string	Site Hash.
*/
public function getSiteHash() {
	$siteHash = md5(
		$this->getDomain() .
		$GLOBALS['TYPO3_CONF_VARS']['SYS']['encryptionKey'] .
		'tx_solr'
	); 	return $siteHash;
}

Because Nutch does not provide such a logic out of the box, we need to add it via a custom Nutch plugin.

The remaining fields can be set using Nutch's Solr mappings that are located in conf/solrindex-mapping.xml.

Get in touch with us if you're interested in indexing external websites with your Solr installation.

Apache Solr is a state-of-the-art Enterprise search technology. It has proven many times that it does its job pretty well. But what if you are in need for a feature that is not supported by Solr?
Besides indexing our own TYPO3 website we also want to index external websites. Unfortunately, Apache Solr does not support this itself.
Anyway, Apache Solr integrates smoothly in the whole Apache ecosystem.

Apache Nutch is a highly scalable web crawler that has a Solr integration. In this article I will show you how to set up, configure and use Nutch with Solr.

We will use Nutch 1.3 which supports Apache Solr 3.X.

Installation

Let's get our hands dirty. We change to the directory /opt

cd /opt

and download Apache Nutch 1.3:

wget apache.mirror.clusters.cc//nutch/apache-nutch-1.3-bin.tar.gz

When the download is finished we unpack the archive:

tar xvfz apache-nutch-1.3-bin.tar.gz

Configuration

We move on to the directory nutch-1.3/runtime/local

cd nutch-1.3/runtime/local

and change the permissions of the command nutch:

chmod +x bin/nutch

Please ensure that you have set the variable JAVA_HOME. Nutch needs to know where your Java installation is located:

export JAVA_HOME=/usr/lib/jvm/java-6-openjdk

In the next step we configure the crawler. We open the file conf/nutch-site.xml and save the following configuration: 

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>http.agent.name</name>
    <value>Solr Nutch spider</value>
  </property>
</configuration>

We create the folder urls. Nutch reads each file inside urls in order to retrieve websites to visit.
Therefore we create the file urls/seeds having the following content:

www.dkd.de

Nutch provides an own Solr schema located in conf/schema.xml. We copy the schema to our Solr installation after making a small fix.
We need to change the line

<field name="content" type="text" stored="false" indexed="true"/>

to

<field name="content" type="text" stored="true" indexed="true"/>

By setting the option stored to true we enable saving the crawled website's content in the Solr index.

Usage

After finishing the configuration we are ready to enjoy the power of Nutch.

For crawling the configured website we are using the command crawl. We call it with our running Solr server and the depth of links to follow:

bin/nutch crawl urls -solr 127.0.0.1/solr/ -depth 2

The command supports the following options:

• The option solr defines the used Solr server.
• depths defines the depth of links to follow.
• You can set a maximum of websites to crawl by using the option topN.

You can view the indexed websites using the Solr admin interface. If you do not want to search external websites via your TYPO3 website we can provide alternative solutions like Tempo.

Conclusion

Apache Nutch provides an easy to use solution for crawling and indexing external websites. It integrates Apache Solr perfectly.

This use case proves the great flexiblity of the Apache eco system once again. If you want to search external websites using your Solr installation or are interested in an alternative solution for displaying search results get in contact with us!

Resources

• Apache Solr: http://lucene.apache.org/solr/
• Apache Nutch: http://nutch.apache.org/
• Tempo with Solr: http://tempojs.com/examples/solr