I am in need of an engine to index my whole site and about
2GB of docs(PDF/HTML/DOC/XLS...etc). Have tried
htdig/asearch which both take the machine down and just can't scale to the size of file-store to be indexed.
Would appreciate some pointers. Thanks in advance.
Htdig should be fine, dunno why it crashes for you. Tried out swish-e?