2026-01-08 22:30:04 [scrapy.utils.log] INFO: Scrapy 2.12.0 started (bot: magicpin) 2026-01-08 22:30:04 [scrapy.utils.log] INFO: Versions: lxml 5.3.0.0, libxml2 2.12.9, cssselect 1.2.0, parsel 1.9.1, w3lib 2.2.1, Twisted 24.11.0, Python 3.10.12 (main, Nov 4 2025, 08:48:33) [GCC 11.4.0], pyOpenSSL 24.3.0 (OpenSSL 3.4.0 22 Oct 2024), cryptography 44.0.0, Platform Linux-6.8.0-1044-aws-aarch64-with-glibc2.35 2026-01-08 22:30:04 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): cron.restaverse.com:443 2026-01-08 22:30:04 [urllib3.connectionpool] DEBUG: https://cron.restaverse.com:443 "POST /api/db_services/fetch-query HTTP/11" 200 5782 2026-01-08 22:30:04 [scrapy.addons] INFO: Enabled addons: [] 2026-01-08 22:30:04 [py.warnings] WARNING: /home/ubuntu/restaverse_spiders/venv/lib/python3.10/site-packages/scrapy/utils/request.py:120: ScrapyDeprecationWarning: 'REQUEST_FINGERPRINTER_IMPLEMENTATION' is a deprecated setting. It will be removed in a future version of Scrapy. return cls(crawler) 2026-01-08 22:30:04 [asyncio] DEBUG: Using selector: EpollSelector 2026-01-08 22:30:04 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor 2026-01-08 22:30:04 [scrapy.utils.log] DEBUG: Using asyncio event loop: asyncio.unix_events._UnixSelectorEventLoop 2026-01-08 22:30:04 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor 2026-01-08 22:30:04 [scrapy.utils.log] DEBUG: Using asyncio event loop: asyncio.unix_events._UnixSelectorEventLoop 2026-01-08 22:30:04 [scrapy.extensions.telnet] INFO: Telnet Password: 0d237826bb56dc4d 2026-01-08 22:30:04 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.memusage.MemoryUsage', 'scrapy.extensions.logstats.LogStats', 'scrapeops_scrapy.extension.ScrapeOpsMonitor'] 2026-01-08 22:30:04 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'magicpin', 'CONCURRENT_REQUESTS': 8, 'DEPTH_LIMIT': None, 'DOWNLOAD_DELAY': 2, 'FEED_EXPORT_ENCODING': 'utf-8', 'LOG_FILE': '/home/ubuntu/restaverse_spiders/logs/magicpin/magic/magicpin ' 'reviews_2026-01-08T22_30_02.log', 'NEWSPIDER_MODULE': 'magicpin.spiders', 'REQUEST_FINGERPRINTER_IMPLEMENTATION': '2.7', 'SPIDER_MODULES': ['magicpin.spiders'], 'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor'} 2026-01-08 22:30:04 [scrapy.middleware] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware', 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapeops_scrapy.middleware.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats'] 2026-01-08 22:30:04 [scrapy.middleware] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2026-01-08 22:30:04 [scrapy.middleware] INFO: Enabled item pipelines: ['magicpin.pipelines.MagicpinPipeline'] 2026-01-08 22:30:04 [scrapy.core.engine] INFO: Spider opened 2026-01-08 22:30:04 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2026-01-08 22:30:04 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): api.scrapeops.io:443 2026-01-08 22:30:05 [urllib3.connectionpool] DEBUG: https://api.scrapeops.io:443 "POST /api/v1/setup/ HTTP/11" 200 None 2026-01-08 22:30:05 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6024 2026-01-08 22:30:05 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): api.scrapeops.io:443 2026-01-08 22:30:05 [urllib3.connectionpool] DEBUG: https://api.scrapeops.io:443 "POST /api/v1/normalizer/domain/?domain=magicpin.in HTTP/11" 200 242 2026-01-08 22:30:19 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): api.scrapeops.io:443 2026-01-08 22:30:19 [urllib3.connectionpool] DEBUG: https://api.scrapeops.io:443 "POST /api/v1/stats/ HTTP/11" 200 109 2026-01-08 22:30:34 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): api.scrapeops.io:443 2026-01-08 22:30:34 [urllib3.connectionpool] DEBUG: https://api.scrapeops.io:443 "POST /api/v1/stats/ HTTP/11" 200 109 2026-01-08 22:30:49 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): api.scrapeops.io:443 2026-01-08 22:30:49 [urllib3.connectionpool] DEBUG: https://api.scrapeops.io:443 "POST /api/v1/stats/ HTTP/11" 200 109 2026-01-08 22:31:04 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): api.scrapeops.io:443 2026-01-08 22:31:04 [urllib3.connectionpool] DEBUG: https://api.scrapeops.io:443 "POST /api/v1/stats/ HTTP/11" 200 109 2026-01-08 22:31:04 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2026-01-08 22:31:05 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:31:07 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:31:09 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:31:12 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:31:14 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:31:16 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:31:18 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:31:19 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): api.scrapeops.io:443 2026-01-08 22:31:19 [urllib3.connectionpool] DEBUG: https://api.scrapeops.io:443 "POST /api/v1/stats/ HTTP/11" 200 109 2026-01-08 22:31:19 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:31:34 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): api.scrapeops.io:443 2026-01-08 22:31:34 [urllib3.connectionpool] DEBUG: https://api.scrapeops.io:443 "POST /api/v1/stats/ HTTP/11" 200 109 2026-01-08 22:31:49 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): api.scrapeops.io:443 2026-01-08 22:31:49 [urllib3.connectionpool] DEBUG: https://api.scrapeops.io:443 "POST /api/v1/stats/ HTTP/11" 200 109 2026-01-08 22:32:04 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): api.scrapeops.io:443 2026-01-08 22:32:04 [urllib3.connectionpool] DEBUG: https://api.scrapeops.io:443 "POST /api/v1/stats/ HTTP/11" 200 109 2026-01-08 22:32:04 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2026-01-08 22:32:05 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:32:07 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:32:10 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:32:13 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:32:14 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:32:17 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:32:20 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:32:21 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:32:22 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) 2026-01-08 22:32:34 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): api.scrapeops.io:443 2026-01-08 22:32:34 [urllib3.connectionpool] DEBUG: https://api.scrapeops.io:443 "POST /api/v1/stats/ HTTP/11" 200 109 2026-01-08 22:33:04 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): api.scrapeops.io:443 2026-01-08 22:33:04 [urllib3.connectionpool] DEBUG: https://api.scrapeops.io:443 "POST /api/v1/stats/ HTTP/11" 200 109 2026-01-08 22:33:04 [scrapy.extensions.logstats] INFO: Crawled 1 pages (at 1 pages/min), scraped 0 items (at 0 items/min) 2026-01-08 22:33:07 [scrapeops_scrapy.middleware.retry] ERROR: Gave up retrying (failed 3 times): 502 Bad Gateway 2026-01-08 22:33:07 [scrapy.core.engine] DEBUG: Crawled (502) (referer: None) 2026-01-08 22:33:10 [scrapeops_scrapy.middleware.retry] ERROR: Gave up retrying (failed 3 times): 502 Bad Gateway 2026-01-08 22:33:10 [scrapy.core.engine] DEBUG: Crawled (502) (referer: None) 2026-01-08 22:33:13 [scrapeops_scrapy.middleware.retry] ERROR: Gave up retrying (failed 3 times): 502 Bad Gateway 2026-01-08 22:33:13 [scrapy.core.engine] DEBUG: Crawled (502) (referer: None) 2026-01-08 22:33:15 [scrapeops_scrapy.middleware.retry] ERROR: Gave up retrying (failed 3 times): 502 Bad Gateway 2026-01-08 22:33:15 [scrapy.core.engine] DEBUG: Crawled (502) (referer: None) 2026-01-08 22:33:17 [scrapeops_scrapy.middleware.retry] ERROR: Gave up retrying (failed 3 times): 502 Bad Gateway 2026-01-08 22:33:17 [scrapy.core.engine] DEBUG: Crawled (502) (referer: None) 2026-01-08 22:33:20 [scrapeops_scrapy.middleware.retry] ERROR: Gave up retrying (failed 3 times): 502 Bad Gateway 2026-01-08 22:33:20 [scrapy.core.engine] DEBUG: Crawled (502) (referer: None) 2026-01-08 22:33:22 [scrapeops_scrapy.middleware.retry] ERROR: Gave up retrying (failed 3 times): 502 Bad Gateway 2026-01-08 22:33:22 [scrapy.core.engine] DEBUG: Crawled (502) (referer: None) 2026-01-08 22:33:25 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:33:34 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): api.scrapeops.io:443 2026-01-08 22:33:34 [urllib3.connectionpool] DEBUG: https://api.scrapeops.io:443 "POST /api/v1/stats/ HTTP/11" 200 109 2026-01-08 22:33:48 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) 2026-01-08 22:34:04 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): api.scrapeops.io:443 2026-01-08 22:34:04 [urllib3.connectionpool] DEBUG: https://api.scrapeops.io:443 "POST /api/v1/stats/ HTTP/11" 200 109 2026-01-08 22:34:04 [scrapy.extensions.logstats] INFO: Crawled 9 pages (at 8 pages/min), scraped 0 items (at 0 items/min) 2026-01-08 22:34:07 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:34:10 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:34:13 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:34:15 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:34:18 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:34:21 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:34:26 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:34:34 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): api.scrapeops.io:443 2026-01-08 22:34:34 [urllib3.connectionpool] DEBUG: https://api.scrapeops.io:443 "POST /api/v1/stats/ HTTP/11" 200 109 2026-01-08 22:34:48 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:35:04 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): api.scrapeops.io:443 2026-01-08 22:35:04 [urllib3.connectionpool] DEBUG: https://api.scrapeops.io:443 "POST /api/v1/stats/ HTTP/11" 200 109 2026-01-08 22:35:04 [scrapy.extensions.logstats] INFO: Crawled 9 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2026-01-08 22:35:07 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:35:10 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:35:13 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:35:15 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:35:18 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:35:21 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:35:26 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:35:34 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): api.scrapeops.io:443 2026-01-08 22:35:34 [urllib3.connectionpool] DEBUG: https://api.scrapeops.io:443 "POST /api/v1/stats/ HTTP/11" 200 109 2026-01-08 22:35:48 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://magicpin.in/Mumbai/Andheri-East/Restaurant/Persian-Darbar/store/254a8/reviews) 2026-01-08 22:35:48 [scrapy.dupefilters] DEBUG: Filtered duplicate request: - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates) 2026-01-08 22:35:48 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:36:04 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): api.scrapeops.io:443 2026-01-08 22:36:04 [urllib3.connectionpool] DEBUG: https://api.scrapeops.io:443 "POST /api/v1/stats/ HTTP/11" 200 109 2026-01-08 22:36:04 [scrapy.extensions.logstats] INFO: Crawled 10 pages (at 1 pages/min), scraped 0 items (at 0 items/min) 2026-01-08 22:36:07 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:36:10 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:36:13 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:36:19 [scrapeops_scrapy.middleware.retry] ERROR: Gave up retrying (failed 3 times): 502 Bad Gateway 2026-01-08 22:36:19 [scrapy.core.engine] DEBUG: Crawled (502) (referer: https://magicpin.in/Mumbai/Andheri-East/Restaurant/Persian-Darbar/store/254a8/reviews) 2026-01-08 22:36:21 [scrapeops_scrapy.middleware.retry] ERROR: Gave up retrying (failed 3 times): 502 Bad Gateway 2026-01-08 22:36:21 [scrapy.core.engine] DEBUG: Crawled (502) (referer: https://magicpin.in/Mumbai/Andheri-East/Restaurant/Persian-Darbar/store/254a8/reviews) 2026-01-08 22:36:26 [scrapeops_scrapy.middleware.retry] ERROR: Gave up retrying (failed 3 times): 502 Bad Gateway 2026-01-08 22:36:26 [scrapy.core.engine] DEBUG: Crawled (502) (referer: https://magicpin.in/Mumbai/Andheri-East/Restaurant/Persian-Darbar/store/254a8/reviews) 2026-01-08 22:36:34 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): api.scrapeops.io:443 2026-01-08 22:36:34 [urllib3.connectionpool] DEBUG: https://api.scrapeops.io:443 "POST /api/v1/stats/ HTTP/11" 200 109 2026-01-08 22:36:48 [scrapeops_scrapy.middleware.retry] ERROR: Gave up retrying (failed 3 times): 502 Bad Gateway 2026-01-08 22:36:48 [scrapy.core.engine] DEBUG: Crawled (502) (referer: https://magicpin.in/Mumbai/Andheri-East/Restaurant/Persian-Darbar/store/254a8/reviews) 2026-01-08 22:36:50 [scrapeops_scrapy.middleware.retry] ERROR: Gave up retrying (failed 3 times): 502 Bad Gateway 2026-01-08 22:36:50 [scrapy.core.engine] DEBUG: Crawled (502) (referer: None) 2026-01-08 22:37:04 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): api.scrapeops.io:443 2026-01-08 22:37:04 [urllib3.connectionpool] DEBUG: https://api.scrapeops.io:443 "POST /api/v1/stats/ HTTP/11" 200 109 2026-01-08 22:37:04 [scrapy.extensions.logstats] INFO: Crawled 15 pages (at 5 pages/min), scraped 0 items (at 0 items/min) 2026-01-08 22:37:07 [scrapeops_scrapy.middleware.retry] ERROR: Gave up retrying (failed 3 times): 502 Bad Gateway 2026-01-08 22:37:07 [scrapy.core.engine] DEBUG: Crawled (502) (referer: https://magicpin.in/Bangalore/Indiranagar/Restaurant/The-Filter-Coffee/store/15a5bb4/reviews) 2026-01-08 22:37:10 [scrapeops_scrapy.middleware.retry] ERROR: Gave up retrying (failed 3 times): 502 Bad Gateway 2026-01-08 22:37:10 [scrapy.core.engine] DEBUG: Crawled (502) (referer: https://magicpin.in/Bangalore/Indiranagar/Restaurant/The-Filter-Coffee/store/15a5bb4/reviews) 2026-01-08 22:37:13 [scrapeops_scrapy.middleware.retry] ERROR: Gave up retrying (failed 3 times): 502 Bad Gateway 2026-01-08 22:37:13 [scrapy.core.engine] DEBUG: Crawled (502) (referer: https://magicpin.in/Mumbai/Andheri-East/Restaurant/Persian-Darbar/store/254a8/reviews) 2026-01-08 22:37:19 [scrapeops_scrapy.middleware.retry] ERROR: Gave up retrying (failed 3 times): 502 Bad Gateway 2026-01-08 22:37:19 [scrapy.core.engine] DEBUG: Crawled (502) (referer: None) 2026-01-08 22:37:22 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:37:26 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:37:34 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): api.scrapeops.io:443 2026-01-08 22:37:34 [urllib3.connectionpool] DEBUG: https://api.scrapeops.io:443 "POST /api/v1/stats/ HTTP/11" 200 109 2026-01-08 22:37:48 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:37:51 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:38:04 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): api.scrapeops.io:443 2026-01-08 22:38:04 [urllib3.connectionpool] DEBUG: https://api.scrapeops.io:443 "POST /api/v1/stats/ HTTP/11" 200 109 2026-01-08 22:38:04 [scrapy.extensions.logstats] INFO: Crawled 19 pages (at 4 pages/min), scraped 0 items (at 0 items/min) 2026-01-08 22:38:07 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:38:10 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:38:13 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:38:19 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:38:22 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:38:26 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:38:48 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:38:51 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:39:03 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) 2026-01-08 22:39:04 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): api.scrapeops.io:443 2026-01-08 22:39:04 [urllib3.connectionpool] DEBUG: https://api.scrapeops.io:443 "POST /api/v1/stats/ HTTP/11" 200 109 2026-01-08 22:39:04 [scrapy.extensions.logstats] INFO: Crawled 20 pages (at 1 pages/min), scraped 0 items (at 0 items/min) 2026-01-08 22:39:07 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:39:10 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:39:13 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:39:19 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:39:22 [scrapeops_scrapy.middleware.retry] ERROR: Gave up retrying (failed 3 times): 502 Bad Gateway 2026-01-08 22:39:22 [scrapy.core.engine] DEBUG: Crawled (502) (referer: None) 2026-01-08 22:39:42 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) 2026-01-08 22:39:48 [scrapeops_scrapy.middleware.retry] ERROR: Gave up retrying (failed 3 times): 502 Bad Gateway 2026-01-08 22:39:48 [scrapy.core.engine] DEBUG: Crawled (502) (referer: None) 2026-01-08 22:39:51 [scrapeops_scrapy.middleware.retry] ERROR: Gave up retrying (failed 3 times): 502 Bad Gateway 2026-01-08 22:39:51 [scrapy.core.engine] DEBUG: Crawled (502) (referer: None) 2026-01-08 22:40:03 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:40:04 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): api.scrapeops.io:443 2026-01-08 22:40:04 [urllib3.connectionpool] DEBUG: https://api.scrapeops.io:443 "POST /api/v1/stats/ HTTP/11" 200 109 2026-01-08 22:40:04 [scrapy.extensions.logstats] INFO: Crawled 24 pages (at 4 pages/min), scraped 0 items (at 0 items/min) 2026-01-08 22:40:10 [scrapeops_scrapy.middleware.retry] ERROR: Gave up retrying (failed 3 times): 502 Bad Gateway 2026-01-08 22:40:10 [scrapy.core.engine] DEBUG: Crawled (502) (referer: None) 2026-01-08 22:40:13 [scrapeops_scrapy.middleware.retry] ERROR: Gave up retrying (failed 3 times): 502 Bad Gateway 2026-01-08 22:40:13 [scrapy.core.engine] DEBUG: Crawled (502) (referer: None) 2026-01-08 22:40:19 [scrapeops_scrapy.middleware.retry] ERROR: Gave up retrying (failed 3 times): 502 Bad Gateway 2026-01-08 22:40:19 [scrapy.core.engine] DEBUG: Crawled (502) (referer: None) 2026-01-08 22:40:22 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:40:42 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:40:48 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:40:51 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:41:03 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:41:04 [scrapy.extensions.logstats] INFO: Crawled 27 pages (at 3 pages/min), scraped 0 items (at 0 items/min) 2026-01-08 22:41:10 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:41:13 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:41:19 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:41:22 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:41:42 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:41:48 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:41:51 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:41:58 [scrapy.core.engine] DEBUG: Crawled (404) (referer: None) 2026-01-08 22:42:00 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) 2026-01-08 22:42:03 [scrapeops_scrapy.middleware.retry] ERROR: Gave up retrying (failed 3 times): 502 Bad Gateway 2026-01-08 22:42:03 [scrapy.core.engine] DEBUG: Crawled (502) (referer: None) 2026-01-08 22:42:04 [scrapy.extensions.logstats] INFO: Crawled 30 pages (at 3 pages/min), scraped 0 items (at 0 items/min) 2026-01-08 22:42:10 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:42:13 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:42:19 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:42:22 [scrapeops_scrapy.middleware.retry] ERROR: Gave up retrying (failed 3 times): 502 Bad Gateway 2026-01-08 22:42:22 [scrapy.core.engine] DEBUG: Crawled (502) (referer: None) 2026-01-08 22:42:42 [scrapeops_scrapy.middleware.retry] ERROR: Gave up retrying (failed 3 times): 502 Bad Gateway 2026-01-08 22:42:42 [scrapy.core.engine] DEBUG: Crawled (502) (referer: None) 2026-01-08 22:42:48 [scrapeops_scrapy.middleware.retry] ERROR: Gave up retrying (failed 3 times): 502 Bad Gateway 2026-01-08 22:42:48 [scrapy.core.engine] DEBUG: Crawled (502) (referer: None) 2026-01-08 22:43:01 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:43:04 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:43:04 [scrapy.extensions.logstats] INFO: Crawled 33 pages (at 3 pages/min), scraped 0 items (at 0 items/min) 2026-01-08 22:43:10 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:43:13 [scrapeops_scrapy.middleware.retry] ERROR: Gave up retrying (failed 3 times): 502 Bad Gateway 2026-01-08 22:43:13 [scrapy.core.engine] DEBUG: Crawled (502) (referer: None) 2026-01-08 22:43:19 [scrapeops_scrapy.middleware.retry] ERROR: Gave up retrying (failed 3 times): 502 Bad Gateway 2026-01-08 22:43:19 [scrapy.core.engine] DEBUG: Crawled (502) (referer: None) 2026-01-08 22:43:22 [scrapeops_scrapy.middleware.retry] ERROR: Gave up retrying (failed 3 times): 502 Bad Gateway 2026-01-08 22:43:22 [scrapy.core.engine] DEBUG: Crawled (502) (referer: None) 2026-01-08 22:43:42 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 1 times): 502 Bad Gateway 2026-01-08 22:44:01 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:44:04 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:44:04 [scrapy.extensions.logstats] INFO: Crawled 36 pages (at 3 pages/min), scraped 0 items (at 0 items/min) 2026-01-08 22:44:10 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:44:42 [scrapeops_scrapy.middleware.retry] DEBUG: Retrying (failed 2 times): 502 Bad Gateway 2026-01-08 22:45:01 [scrapeops_scrapy.middleware.retry] ERROR: Gave up retrying (failed 3 times): 502 Bad Gateway 2026-01-08 22:45:01 [scrapy.core.engine] DEBUG: Crawled (502) (referer: None) 2026-01-08 22:45:04 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): api.scrapeops.io:443 2026-01-08 22:45:04 [urllib3.connectionpool] DEBUG: https://api.scrapeops.io:443 "POST /api/v1/stats/ HTTP/11" 200 109 2026-01-08 22:45:04 [scrapeops_scrapy.middleware.retry] ERROR: Gave up retrying (failed 3 times): 502 Bad Gateway 2026-01-08 22:45:04 [scrapy.core.engine] DEBUG: Crawled (502) (referer: https://magicpin.in/Gurgaon/Sector-7/Restaurant/Om-Sweets-and-Snacks/store/57a5c7/reviews) 2026-01-08 22:45:04 [scrapy.extensions.logstats] INFO: Crawled 38 pages (at 2 pages/min), scraped 0 items (at 0 items/min) 2026-01-08 22:45:10 [scrapeops_scrapy.middleware.retry] ERROR: Gave up retrying (failed 3 times): 502 Bad Gateway 2026-01-08 22:45:10 [scrapy.core.engine] DEBUG: Crawled (502) (referer: https://magicpin.in/Gurgaon/Sector-7/Restaurant/Om-Sweets-and-Snacks/store/57a5c7/reviews) 2026-01-08 22:45:23 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) 2026-01-08 22:45:23 [scrapy.core.engine] INFO: Closing spider (finished) 2026-01-08 22:45:23 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): cron.restaverse.com:443 2026-01-08 22:45:23 [urllib3.connectionpool] DEBUG: https://cron.restaverse.com:443 "POST /insert_data/ HTTP/11" 201 112 2026-01-08 22:45:23 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): api.scrapeops.io:443 2026-01-08 22:45:24 [urllib3.connectionpool] DEBUG: https://api.scrapeops.io:443 "POST /api/v1/stats/ HTTP/11" 200 109 2026-01-08 22:45:24 [scrapy.statscollectors] INFO: Dumping Scrapy stats: {'downloader/request_bytes': 36447, 'downloader/request_count': 116, 'downloader/request_method_count/GET': 116, 'downloader/response_bytes': 327668, 'downloader/response_count': 116, 'downloader/response_status_count/200': 7, 'downloader/response_status_count/404': 1, 'downloader/response_status_count/502': 108, 'dupefilter/filtered': 7, 'elapsed_time_seconds': 919.357718, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2026, 1, 8, 22, 45, 23, 851524, tzinfo=datetime.timezone.utc), 'httpcompression/response_bytes': 2389975, 'httpcompression/response_count': 8, 'items_per_minute': None, 'log_count/DEBUG': 176, 'log_count/ERROR': 32, 'log_count/INFO': 25, 'log_count/WARNING': 1, 'memusage/max': 86372352, 'memusage/startup': 86372352, 'request_depth_max': 2, 'response_received_count': 40, 'responses_per_minute': None, 'retry/count': 76, 'retry/max_reached': 32, 'retry/reason_count/502 Bad Gateway': 76, 'scheduler/dequeued': 116, 'scheduler/dequeued/memory': 116, 'scheduler/enqueued': 116, 'scheduler/enqueued/memory': 116, 'start_time': datetime.datetime(2026, 1, 8, 22, 30, 4, 493806, tzinfo=datetime.timezone.utc)} 2026-01-08 22:45:24 [scrapy.core.engine] INFO: Spider closed (finished)