In the first step, we'll add an indexing configuration for the table tx_cal_event. Due to the fact that the configuration is written in TypoScript we can create ISO 8601 conform dates using stdWrap functions:

plugin.tx_solr {
	index.queue {
		events = 1
		events {
			table = tx_cal_event
			fields {
				start_date_dateS = TEXT
				start_date_dateS.cObject = COA
				start_date_dateS.cObject {
					10 = TEXT
					10.field = start_date
					10.substring = 0,4
					20 = TEXT
					20.value = -
					30 = TEXT
					30.field = start_date
					30.substring = 4,2
					40 = TEXT
					40.value = -
					50 = TEXT
					50.field = start_date
					50.substring = 6,2
					60 = TEXT
					60.value = T00:00:00Z
				}
				end_date_dateS = TEXT
				end_date_dateS.cObject = COA
				end_date_dateS.cObject {
					10 = TEXT
					10.field = end_date
					10.substring = 0,4
					20 = TEXT
					20.value = -
					30 = TEXT
					30.field = end_date
					30.substring = 4,2
					40 = TEXT
					40.value = -
					50 = TEXT
					50.field = end_date
					50.substring = 6,2
					60 = TEXT
					60.value = T00:00:00Z
				}
				title = title
				content = SOLR_CONTENT
				content.field = description
			}
		}
	}
}

After initializing the indexing queue for events we can index the calendar entries by executing the scheduler task Index Queue Worker Task. Now, the events are available as search results.

"Starts in..." facet

In the next step we'll create a facet that breaks down calendar entries based on their start date. We want to divide them by the following options:

• starts in less than 1 week
• starts in 1 week to 1 month
• starts in 1 to 6 months
• starts in 6 months to 1 year
• starts in 1 year or later

The following TypoScript code configures the facet:

plugin.tx_solr {
	search.faceting = 1
	search.faceting.facets {
		age {
			label = Date
			field = start_date_dateS
			selectingSelectedFacetOptionRemovesFilter = 1
			renderingInstruction = CASE
			renderingInstruction {
				key.field = optionValue
				default = TEXT
				default.field = optionValue
				week = TEXT
				week.value = Less than one week
				month = TEXT
				month.value = 1 week to 1 month
				halfYear = TEXT
				halfYear.value = 1 to 6 months
				year = TEXT
				year.value = 6 months to 1 year
				old = TEXT
				old.value = over a year
			}
			type = queryGroup
			queryGroup {
				week {
					query = [NOW/DAY TO NOW/DAY+7DAYS]
				}
				month {
					query = [NOW/DAY+7DAYS TO NOW/DAY+1MONTH]
				}
				halfYear {
					query = [NOW/DAY+1MONTH TO NOW/DAY+6MONTH]
				}
				year {
					query = [NOW/DAY+6MONTHS TO NOW/DAY+1YEAR]
				}
				old {
					query = [NOW/DAY+1YEAR TO *]
				}
			}
			requirements {
				typeIsCal {
					facet = type
					values = tx_cal_event
				}
			}
		}
	}
} 

By using the requirements option we can specify that the facet is shown only if every result has the type tx_cal_event.

Quarter facet

Another approach to breaking down dates is grouping them by the quarter of the current year they belong to. This can be achieved by adding the following TypoScript configuration:

plugin.tx_solr {
	search.faceting = 1
	search.faceting.facets {
		quarters {
			label = Quarters
			field = start_date_dateS
			selectingSelectedFacetOptionRemovesFilter = 1
			renderingInstruction = CASE
			renderingInstruction {
				key.field = optionValue
				default = TEXT
				default.field = optionValue
				first_quarter = TEXT
				first_quarter.value = 1st quarter
				second_quarter = TEXT
				second_quarter.value = 2nd quarter
				third_quarter = TEXT
				third_quarter.value = 3rd quarter
				fourth_quarter = TEXT
				fourth_quarter.value = 4th quarter
			}
			type = queryGroup
			queryGroup {
				first_quarter {
					query = [NOW/YEAR TO NOW/YEAR+3MONTHS]
				}
				second_quarter {
					query = [NOW/YEAR+3MONTHS TO NOW/YEAR+6MONTHS]
				}
				third_quarter {
					query = [NOW/YEAR+6MONTHS TO NOW/YEAR+9MONTHS]
				}
				fourth_quarter {
					query = [NOW/YEAR+9MONTHS TO NOW/YEAR+1YEAR]
				}
			}
			requirements {
				typeIsCal {
					facet = type
					values = tx_cal_event
				}
			}
		}
	}
} 

The facet gets rendered as following:

Summing up

These were only two examples for facets that work on a date field. Apache Solr offers a lot more possibilities for working with date fields.

This release marks a new milestone in the development of Apache Solr for TYPO3. Until now there have been two versions of the extension, one publicly available in the TYPO3 Extension Repository (TER), and a "private" one available from dkd with additional features over the public one.

Starting with version 2.8.0, there will be only one version of the extension, publicly available from TER. Many additional features previously exclusive to the EAP version have been moved to the public version now. Other features are now available as add-on extensions from dkd.

This allows to reduce the amount of work involved with maintenance of the extension. On the other hand it also allows to choose add-on functionality as needed and to distribute them to customers as they require them.

New in this Release

Apache Solr 3.6

With this release we're moving to Apache Solr 3.6. This brings bug fixes on the Solr server side of course, but also allows us to now provide support for start and stop time in TYPO3 since we are using functionality introduced with Solr 3.6.
The index schema and Solr configuration files have been adopted and optimized for Solr 3.6. This includes enabling the Solr index browser (browse request handler), JSON update request handler, and analysis request handler.

General Changes

• We added support for workspaces so that changes will get indexed when they are published to LIVE, but not before that. Until now we of course did not index unpublished changes either, however the extension did not recognize the workspace publish and swap events.
• Support for start and stop time based publishing has been added. Pages and records will be indexed once their start time has been reached. When the stop time passes, content will be removed from the index again.
• Selected features previously exclusive to the EAP version of the extension are now available free of charge and ship with the extension released in TER. New features include hierarchical facets, frequent searches, last searches, and a result document score analyzer.

Other Changes and Notable Bug Fixes

• New hooks and interfaces to customize functionality
• General bug fixes and stability improvements
• Improvements to the backend module for index maintenance
• PHP 5.4 compatibility
• Support for indexing of records that exist only in specific languages, but not in the default language

Updating

This release requires Apache Solr 3.6 to work! Start and stop time support depends on functionality only available from Apache Solr 3.6 and up. Also the Solr configuration and schema files have been updated for Solr 3.6. The easiest way to install Solr is to use the install script that ships with the extension. This script will install the latest version, Apache Solr 3.6.1.

Although we try to change the schema only when necessary and thus making updating the extension painless we had to adopt schema and configuration files with this release. After updating Solr you will need to re-index your content due to the schema changes. Apologizes for any inconvenience caused by this.

Updating the TYPO3 extension should be straight forward.

Other changes can be found in the update guide.

Contributors

(patches, comments, bug reports, ... no particular order)

• Steffen Gebert
• Stephan Schuler
• Jonas Götze
• Heiko Kromm
• Stefan Sprenger
• Michel Tremblay
• Peter Kraume
• Constantin Lebrecht
• Laurent Cherpit
• Andreas Beutel
• Rolf Hofmann
• Florian Megyesi
• Pierrick Caillon
• Dominik Steinborn
• Dmitry Dulepov
• Albert van der Veen
• Witali Rott
• Alexander Wende
• Stefan Galinski
• Jigal van Hemert

Thanks to everyone who helped in creating this release!

General Changes 

• The HTTP GET parameter for the user's query is now freely configurable. Instead of using the default tx_solr[q], you can now also just a more simple q for example. This change makes it easier to analyze the user's search behavior using tools like Piwik.
• Solr connections can now be initialized per site through a new entry in the page tree's click menu. This should be a relieve for large installations with many sites where the Solr connection initialization could run into a PHP execution time limit.
• Solr connections can now also be initialized through TYPO3's command line interface (CLI) which should be very much appreciated for Continuous Integration and Deployment scenarios.
• The checks in TYPO3's system status report have once been improved making it even easier to detect issues in the extension's configuration.

Index Queue

• Priorities for indexing configuration. It is now possible to set priorities for different indexing configurations, allowing to have products indexed before news articles before pages, or any other order.
• Index Queue indexing configurations can now be initialized selectively. Before you could just initialize the whole Index Queue, now you can just re-initialize f.e. news or products.
• A sorting field can now be configured for the SOLR_RELATION indexing configuration helper. This makes it easier to f.e. index just the main category of a news record instead of all categories.

Search and Faceting

• Hierarchical facets now support the configuration options selectingSelectedFacetOptionRemovesFilter and singleOptionMode. When building menus from hierarchical facets, the ACT state is supported, too. Hierarchical facets are a feature exclusive to the extension's early access program (EAP).
• Sites to be searched can now be easily configured, which allows to index multiple sites into one index and search different or multiple sites depending from where a user starts its search request.
• Support for OpenSearch has been added, allowing a user to start a search on your site from his browser's search input field.
• Conditional Facets: It's now possible to define requirements for a facet that must be fulfilled before the facet is being rendered. For example a category facet may only be rendered if the user has selected news from the type facet.
• Query Facets: Instead of creating a facet's options from an index field's values it is now possible to create a facet's options from the results of several queries.
• The spell checker (did you mean ...) can now use the suggestions to show the actual results instead of making the user search for the suggestions, creating a better experience for the user by saving time and additional page requests.

Minor Changes and Notable Bug Fixes

• RealURL did not create URLs when indexing records
• Garbage Collector does not wait for Solr's commit to complete anymore. Previously, when an editor caused the Garbage Collector to clean up documents from the index, by for example deleting or hiding a record, that action could take a while before the editor could continue working in the TYPO3 backend. The Garbage Collector now returns immediately so that the editor does not need to wait anymore.
• Performance of the template engine has been improved. Depending on the complexity of a template this can speed up result page rendering by a factor of four to five times.
• New template markers have been added related to faceting

Other Changes

• The public version on forge has been migrated to the git and Gerrit infrastructure, making it even easier to contribute to the project.
• The default filter on the index' language field has been removed as it is recommended to separate languages into separate indexes.

Updating

Updating should be strait forward. Simply update the extension, no need to update anything with the Solr server, and also no need to re-index your content.

Contributors

(patches, comments, bug reports, ... no particular order)

• Stefan Sprenger
• Stefan Galinksi
• Cornel Boppart
• Steffen Ritter
• Dimitri Ebert
• Andre Westbunk
• Jochem de Groot
• Michel Tremblay
• Rik Willems
• Dirk Wenzel
• Jigal van Hemert
• Steffen Gebert
• Peter Niederlag
• Oliver Hader
• Olivier Dobberkau
• Peter Kraume
• Axel Böswetter
• Constantin Lebrecht
• Frédéric Jaeger
• Ingo Pfennigstorf
• Christoph Moeller
• Jari-Hermann Ernst
• Marco Bresch
• Marc Bastian Heinrichs
• Tomas Norre Mikkelsen
• Thorsten Kahler

Contributors for this release were 26, up from 15 for 2.1. Thanks to everyone who helped with this release!

Starting with version 3.5 Apache Solr comes with a stemmer that uses Hunspell. The Hunspell stemmer works with dictionaries and rules to do its job. The great thing is that the Hunspell stemmer can use the dictionaries and rules provided by the Apache OpenOffice project.

The dictionaries and rules are provided as OpenOffice extensions. Download the dictionary extension you need and simply unzip it - the extensions are simple zip files. In the unpacked extension you'll find a .aff file and a .dic file with the same name, these are the files needed to get your Hunspell stemmer working.

Example for slovenian Hunspell stemmer:

<filter class=”org.apache.solr.analysis.HunspellStemFilterFactory” dictionary=”sl_SL.dic” affix=”sl_SL.aff”>

Using these OpenOffice dictionaries you get stemming support for a lot of languages.

After having a quick look at the specification it seemed an easy enough task for the night. At the end of the night we actually got most of the needed things implemented in a way so that it is very easy to set up in just two steps.

• First add the new OpenSearch TypoScript template
• Then adjust the name and the description of your site's search

The TypoScript template configure's your page object and add a new page type providing the OpenSearch description file for the browsers. After these two simple steps your browser will detect the site's search and you can start using it from the browser's search field.

We hope you enjoy this little new feature when it will ship in the next version.

This is mainly a maintenance release with quite a few bug fixes and a hand full of small new features. An update is encouraged.

New in this release

• Failed items in Index Queue no longer stop the whole indexer
• Improved logging of failures during page indexing 
• Removed the result count from the suggest drop down - caused too much confusion
• Change of index document ID - Requires re-indexing of your content!
• Status Report checks for allow_url_fopen = On and a PHP bug in filter_var()
• Example Index Queue configuration for EXT:news
• Support for content fallback
• Support for FILE: path in LLL viewhelper
• (EAP) File Detector for files linked in Templavoila FCE 

Notable bug fixes

• Fixed handling of multivalue fields in template output
• Fixed issues with Mount Pages indexing an Index Queue initialization
• Fixed handling of filters configured through TypoScript
• (EAP) Garbage collection of files in indexes other than the default language
• Fixed some issues with Workspaces 

Updating

Simply update the extension, no need to update anything with the Solr server. You must re-index all your content though due to a change in the index document ID format: Go to the search module, empty index, and initialize the Index Queue. This will cause the Index Queue worker task to re-index your site.

Contributors

(patches, comments, bug reports, ... no particular order)

• Jigal van Hemert
• Dmitry Dulepov
• Jochem de Groot
• Pierrick Caillon
• Michael Voehringer
• Stefan Galinski
• Georg Ringer
• Markus Friedrich
• Dorit Rottner
• Hendrik Nadler
• Michael Knoll
• Thorsten Kahler
• Lars Tode
• Ivan Kartolo
• Constantin Lebrecht

Thanks to everyone who helped with this release!

The issue was that in our TYPO3 specific Solr schema.xml we have a field called the siteHash. The site hash helps to filter results so that you can index multiple sites into one index while still only getting results of the site you are on.

To allow the documents indexed using Nutch to be found they also need to have a site hash. Of course does not know about the site hash and its implementation; to overcome this we now provide a small REST style API to request a site hash to be used when indexing external resources.

A GET request to
www.typo3-solr.com/index.php &api=siteHash&domain=www.dkd.de
would then for example provide the site hash needed to index a page from www.dkd.de.

To prevent abuse we also added a protection for the API by having to provide the correct API key. By that you are now able to index external resources into a Solr index that is searched by TYPO3.

This and lots more will be part of the soon to be released version 2.0 of Apache Solr for TYPO3.

With the Index Queue  we provide a way to define an easy mapping of database record fields to Solr index fields. TYPO3 is awesome in how flexible it is in regards to page rendering; in this case this kind of turned out to be a disadvantage at first since you can't (read: want to) "re-implement" all the logic. So to index pages we will still rely on frontend indexing, but it's still way better than the classic way. Even better, the Index Queue will find any access restrictions and translations and take care of indexing those variants automatically, too.

The Index Queue comes with a Record Monitor to track any changes and can then re-index just those changed records or pages. In the same way the Garbage Collector will clean up after your editors when they delete records.

So the Index Queue is an easy and fast way to get your site's pages and extension records indexed. However, raw database data often is not what you want in your index right away, usually you want to do some cleaning or transformation before handing the data to Solr. Because of that the Index Queue also supports mapping TYPO3 content objects (cObj) - or more specifically their output - into index fields. To make it even easier to index custom extension records we came up with custom content objects provided by the Apache Solr for TYPO3 extension.

Currently there are three content objects:

• SOLR_CONTENT - cleans database fields like rich text editor fields from HTML code and formatting
• SOLR_MULTIVALUE - allows to index data into - you guessed it - Solr multivalue fields, acting like the PHP explode() functions
• SOLR_RELATION - resolving relations between records, indexing data into multivalue or singlevalue fields. Useful for news and news categories f.e.

We hope you are looking forward to EXT:solr v2.0 as we do!