concept this task in category lucene
appears as: this task

This is an excerpt from Manning's book Lucene in Action, Second Edition.
Test the speed of creating the documents and tokenizing them by using the ReadTokens task in your algorithm. This task steps through each field of the document and tokenizes it using the specified analyzer. The document isn’t indexed. This is an excellent way to measure the document construction and tokenization cost alone. Run this algorithm to tokenize the first 200K docs from Wikipedia using StandardAnalyzer:
analyzer=org.apache.lucene.analysis.standard.StandardAnalyzer
content.source=org.apache.lucene.benchmark.byTask.feeds.LineDocSource
docs.file=/x/lucene/enwiki-20090306-lines.txt
{ "Rounds"
ResetSystemErase
{ ReadTokens > : 200000
NewRound
} : 3
RepSumByPrefRound ReadTokens which produces output like this:
Operation round run recsPerRun rec/s elapsedSec avgUsedMem avgTotalMem
ReadTokens_N 0 1 161783312 1,239,927.9 130.48 2,774,040 2,924,544
ReadTokens_N 1 - 1 161783312 1,259,857.2 - 128.41 2,774,112 - 2,924,544
ReadTokens_N 2 1 161783312 1,253,862.0 129.03 2,774,184 2,924,544
If the commands available for use in the algorithm don’t meet your needs, you can add commands by adding a new task under the org.apache.lucene.benchmark.byTask.tasks package. You should extend the PerfTask abstract class. Make sure that your new task class name is suffixed by Task. For example, once you compile the class SliceBreadTask.java and ensure it’s on the classpath that you specify to Ant, then you can invoke this task by using SliceBread in your algorithm.