concept file in category solr

This is an excerpt from Manning's book Solr in Action.
SimplePostTool version 1.5 Posting files to base url http://localhost:8983/solr/update using content- type application/xml.. POSTing file gb18030-example.xml POSTing file hd.xml POSTing file ipod_other.xml POSTing file ipod_video.xml POSTing file manufacturers.xml POSTing file mem.xml POSTing file money.xml POSTing file monitor.xml POSTing file monitor2.xml POSTing file mp500.xml POSTing file sd500.xml POSTing file solr.xml POSTing file utf8-example.xml POSTing file vidcard.xml 14 files indexed. COMMITting Solr index changes to http://localhost:8983/solr/update..
Equally important to the speed of your queries, however, can be whether your Solr index can fit completely into the available memory not allocated to your JVM. If the Solr index is larger than the amount of extra RAM available on your server, this means hard disk seeks will be necessary throughout the course of your search requests in order to load files into memory, which can significantly affect both query speed and query throughput.
As discussed in section 12.3.1, having enough RAM on your server to fit your entire Solr index (plus the memory you allocate to your JVM) can tremendously help Solr’s performance, as it allows the OS’s filesystem cache to keep loaded files from the hard drive in memory after the first time the files are loaded. Unfortunately, it still requires all of the files to be loaded initially, which means the first time you start up Solr after a system restart (or after a long delay in which other files are loaded and kick the Solr index out of memory), Solr will run much more slowly than usual—every time it hits a file it hasn’t used before—until it has eventually been able to pull all files back into memory.
There is a clever trick to bypass the slowness associated with files in the Solr index lazy loading on the first access: preload all of the files into memory before Solr even starts using the following command.
find $SOLR_INSTALL/example/solr/*/data/ -type f -exec cat {} \; > /dev/nullAssuming that you use the default directory structure for storing your Solr indexes for each Solr core ($SOLR_INSTALL/example/solr/$CORE_NAME/data/), this command will load all of the files one by one into memory from disk for each of your Solr cores without trying to display them anywhere. If you store your data directories elsewhere, you need to modify the path in the command accordingly. For large indexes (tens to hundreds of gigabytes), the process of preloading all files into the OS filesystem cache can take several minutes to finish executing, but searches will run much faster if you run the command before starting Solr, especially if you have just restarted your server.
Due to the nature of Lucene’s incremental indexing process, an index can be composed of hundreds of files, all of which may need to be opened at once. If you have many indexes in Solr, this means that Solr can have thousands of files that it must keep open at any given time to enable searching. Unfortunately, having this many files open at once can exceed the upper limit for available file handles set as the default on some Unix-based OSes. Hitting this upper limit can result in Solr crashing due to being unable to open additional index files during the normal course of index fragmentation. The number of file descriptors Solr is using relative to the limit (the maximum number of files that can be opened at once) is tracked on the main dashboard on the Solr Admin page, as shown in figure 12.1.
Figure 12.1. The file descriptor usage and limit displayed with other system resource information on the Solr Admin page.
![]()
In figure 12.1, you can see that the number of files Solr has open (mostly from indexes belonging to multiple Solr cores) has reached the system file descriptor limit, which means Solr is likely about to crash. This is fortunately an easy problem to fix through the use of the ulimit -n command. Issuing the ulimit -n command will tell you the user limit for file descriptors available. In order to increase this limit, which is important for safely running Solr, you need to specify a new limit.
ulimit –n 100000Many systems have a default file descriptor limit of 1024, but since each Solr index can consist of hundreds of files (or even thousands depending upon your MergePolicy settings), it may be necessary to increase this limit to 100,000 (from our example) or something even higher, especially if you expect to have many Solr cores on your server. You will probably want to set this limit permanently (running the command only applies to the current bash session) by setting it in a system-wide configuration such as /etc/security/limits.conf. You should ensure that your new file descriptor limit is sufficiently large that you will never run the chance of hitting it.
So far, you have seen one way to post documents to Solr: through sending a document over HTTP to the Solr /update handler. We have utilized an included post.jar file as a convenience library for posting files containing Solr documents primarily in XML format, but under the covers it posts the contents of a file to Solr’s /update handler for you. It’s also possible to have Solr ingest documents in other ways, either through pushing documents to Solr in other formats or through having Solr import documents itself from any number of external data sources.