Well, I haven't tried to run crawler with large depth setting yet. I discovered, that it have strange problems with threading and almost unable to process in parallel. Then I tried to investigate why. Will make a separate post on it, when will get some valuable information.
