Skip to content
  • unarist's avatar
    Improve error handling on LinkCrawlWorker (#5250) · 5c8ca024
    unarist authored
    * Improve error handling on LinkCrawlWorker
    
    * Ignore TimeoutError and InvalidURIError too
    * Record errors to debug log
    * Enable dead job queue on LinkCrawlWorker
    
    Since most of acceptable errors were already ignored, only our side issue should go to dead job queue.
    
    * Ignore all http gem errors
    5c8ca024