concept this task in category lucene

appears as: this task
Lucene in Action, Second Edition

This is an excerpt from Manning's book Lucene in Action, Second Edition.

  • Test the speed of creating the documents and tokenizing them by using the ReadTokens task in your algorithm. This task steps through each field of the document and tokenizes it using the specified analyzer. The document isn’t indexed. This is an excellent way to measure the document construction and tokenization cost alone. Run this algorithm to tokenize the first 200K docs from Wikipedia using StandardAnalyzer:
    analyzer=org.apache.lucene.analysis.standard.StandardAnalyzer

    content.source=org.apache.lucene.benchmark.byTask.feeds.LineDocSource

    docs.file=/x/lucene/enwiki-20090306-lines.txt

    { "Rounds"
    ResetSystemErase
    { ReadTokens > : 200000
    NewRound
    } : 3

    RepSumByPrefRound ReadTokens which produces output like this:
    Operation round run recsPerRun rec/s elapsedSec avgUsedMem avgTotalMem
    ReadTokens_N 0 1 161783312 1,239,927.9 130.48 2,774,040 2,924,544
    ReadTokens_N 1 - 1 161783312 1,259,857.2 - 128.41 2,774,112 - 2,924,544
    ReadTokens_N 2 1 161783312 1,253,862.0 129.03 2,774,184 2,924,544
  • If the commands available for use in the algorithm don’t meet your needs, you can add commands by adding a new task under the org.apache.lucene.benchmark.byTask.tasks package. You should extend the PerfTask abstract class. Make sure that your new task class name is suffixed by Task. For example, once you compile the class SliceBreadTask.java and ensure it’s on the classpath that you specify to Ant, then you can invoke this task by using SliceBread in your algorithm.