2025-09-16 00:32:42 [scrapy.utils.log] INFO: Scrapy 2.12.0 started (bot: swiggy_checkin)
2025-09-16 00:32:42 [scrapy.utils.log] INFO: Versions: lxml 5.3.0.0, libxml2 2.12.9, cssselect 1.2.0, parsel 1.9.1, w3lib 2.2.1, Twisted 24.11.0, Python 3.10.12 (main, Aug 15 2025, 14:32:43) [GCC 11.4.0], pyOpenSSL 24.3.0 (OpenSSL 3.4.0 22 Oct 2024), cryptography 44.0.0, Platform Linux-6.8.0-1015-aws-aarch64-with-glibc2.35
2025-09-16 00:32:42 [scrapy.addons] INFO: Enabled addons:
[]
2025-09-16 00:32:42 [asyncio] DEBUG: Using selector: EpollSelector
2025-09-16 00:32:42 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor
2025-09-16 00:32:42 [scrapy.utils.log] DEBUG: Using asyncio event loop: asyncio.unix_events._UnixSelectorEventLoop
2025-09-16 00:32:42 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor
2025-09-16 00:32:42 [scrapy.utils.log] DEBUG: Using asyncio event loop: asyncio.unix_events._UnixSelectorEventLoop
2025-09-16 00:32:42 [scrapy.extensions.telnet] INFO: Telnet Password: d7d9a88b8b418899
2025-09-16 00:32:42 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.logstats.LogStats',
 'scrapeops_scrapy.extension.ScrapeOpsMonitor']
2025-09-16 00:32:42 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'swiggy_checkin',
 'FEED_EXPORT_ENCODING': 'utf-8',
 'LOG_FILE': '/home/ubuntu/restaverse_spiders/logs/swiggy_checkin/swiggy/7506001972_2025-09-16T00_30_01.log',
 'NEWSPIDER_MODULE': 'swiggy_checkin.spiders',
 'SPIDER_MODULES': ['swiggy_checkin.spiders'],
 'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor'}
2025-09-16 00:32:42 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2025-09-16 00:32:42 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2025-09-16 00:32:43 [scrapy.middleware] INFO: Enabled item pipelines:
['swiggy_checkin.pipelines.SwiggyCheckinPipeline']
2025-09-16 00:32:43 [scrapy.core.engine] INFO: Spider opened
2025-09-16 00:32:43 [botocore.credentials] INFO: Found credentials in environment variables.
2025-09-16 00:32:43 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2025-09-16 00:32:43 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6025
2025-09-16 00:32:43 [botocore.hooks] DEBUG: Event choose-service-name: calling handler <function handle_service_name_alias at 0xfb9f309a5a20>
2025-09-16 00:32:43 [botocore.hooks] DEBUG: Event creating-client-class.ssm: calling handler <function add_generate_presigned_url at 0xfb9f30b1a8c0>
2025-09-16 00:32:43 [botocore.configprovider] DEBUG: Looking for endpoint for ssm via: environment_service
2025-09-16 00:32:43 [botocore.configprovider] DEBUG: Looking for endpoint for ssm via: environment_global
2025-09-16 00:32:43 [botocore.configprovider] DEBUG: Looking for endpoint for ssm via: config_service
2025-09-16 00:32:43 [botocore.configprovider] DEBUG: Looking for endpoint for ssm via: config_global
2025-09-16 00:32:43 [botocore.configprovider] DEBUG: No configured endpoint found.
2025-09-16 00:32:43 [botocore.endpoint] DEBUG: Setting ssm timeout as (60, 60)
2025-09-16 00:32:43 [botocore.client] DEBUG: Registering retry handlers for service: ssm
2025-09-16 00:32:43 [botocore.hooks] DEBUG: Event before-parameter-build.ssm.GetParameter: calling handler <function generate_idempotent_uuid at 0xfb9f309a71c0>
2025-09-16 00:32:43 [botocore.regions] DEBUG: Calling endpoint provider with parameters: {'Region': 'ap-south-1', 'UseDualStack': False, 'UseFIPS': False}
2025-09-16 00:32:43 [botocore.regions] DEBUG: Endpoint provider result: https://ssm.ap-south-1.amazonaws.com
2025-09-16 00:32:43 [botocore.hooks] DEBUG: Event before-call.ssm.GetParameter: calling handler <function add_recursion_detection_header at 0xfb9f309a6dd0>
2025-09-16 00:32:43 [botocore.hooks] DEBUG: Event before-call.ssm.GetParameter: calling handler <function add_query_compatibility_header at 0xfb9f309c5870>
2025-09-16 00:32:43 [botocore.hooks] DEBUG: Event before-call.ssm.GetParameter: calling handler <function inject_api_version_header_if_needed at 0xfb9f309c4a60>
2025-09-16 00:32:43 [botocore.endpoint] DEBUG: Making request for OperationModel(name=GetParameter) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'AmazonSSM.GetParameter', 'Content-Type': 'application/x-amz-json-1.1', 'User-Agent': 'Boto3/1.35.79 md/Botocore#1.35.79 ua/2.0 os/linux#6.8.0-1015-aws md/arch#aarch64 lang/python#3.10.12 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.35.79'}, 'body': b'{"Name": "api-secrets", "WithDecryption": true}', 'url': 'https://ssm.ap-south-1.amazonaws.com/', 'context': {'client_region': 'ap-south-1', 'client_config': <botocore.config.Config object at 0xfb9f2ec26ce0>, 'has_streaming_input': False, 'auth_type': None, 'unsigned_payload': None}}
2025-09-16 00:32:43 [botocore.hooks] DEBUG: Event request-created.ssm.GetParameter: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0xfb9f2ec26c50>>
2025-09-16 00:32:43 [botocore.hooks] DEBUG: Event choose-signer.ssm.GetParameter: calling handler <function set_operation_specific_signer at 0xfb9f309a7010>
2025-09-16 00:32:43 [botocore.auth] DEBUG: Calculating signature using v4 auth.
2025-09-16 00:32:43 [botocore.auth] DEBUG: CanonicalRequest:
POST
/

content-type:application/x-amz-json-1.1
host:ssm.ap-south-1.amazonaws.com
x-amz-date:20250916T003243Z
x-amz-target:AmazonSSM.GetParameter

content-type;host;x-amz-date;x-amz-target
9a6e6c3d3792249e4c40138fec610bff203c6fcb1822afa8fe899a5746c68f71
2025-09-16 00:32:43 [botocore.auth] DEBUG: StringToSign:
AWS4-HMAC-SHA256
20250916T003243Z
20250916/ap-south-1/ssm/aws4_request
190a85338f948e7911fd33014ffa1cfa08c506be5eaf13dadf92e57dc07515e0
2025-09-16 00:32:43 [botocore.auth] DEBUG: Signature:
abd7d5e7b81a71cbfdd02add36684bb782d641445fdebf533cffbe06311740ee
2025-09-16 00:32:43 [botocore.hooks] DEBUG: Event request-created.ssm.GetParameter: calling handler <function add_retry_headers at 0xfb9f309c51b0>
2025-09-16 00:32:43 [botocore.endpoint] DEBUG: Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://ssm.ap-south-1.amazonaws.com/, headers={'X-Amz-Target': b'AmazonSSM.GetParameter', 'Content-Type': b'application/x-amz-json-1.1', 'User-Agent': b'Boto3/1.35.79 md/Botocore#1.35.79 ua/2.0 os/linux#6.8.0-1015-aws md/arch#aarch64 lang/python#3.10.12 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.35.79', 'X-Amz-Date': b'20250916T003243Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIA5G4U4VR4LBE4GOWJ/20250916/ap-south-1/ssm/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-target, Signature=abd7d5e7b81a71cbfdd02add36684bb782d641445fdebf533cffbe06311740ee', 'amz-sdk-invocation-id': b'13b9a0e4-66e5-445f-9316-b1469ee56565', 'amz-sdk-request': b'attempt=1', 'Content-Length': '47'}>
2025-09-16 00:32:43 [botocore.httpsession] DEBUG: Certificate path: /home/ubuntu/restaverse_spiders/venv/lib/python3.10/site-packages/certifi/cacert.pem
2025-09-16 00:32:43 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): ssm.ap-south-1.amazonaws.com:443
2025-09-16 00:32:44 [urllib3.connectionpool] DEBUG: https://ssm.ap-south-1.amazonaws.com:443 "POST / HTTP/11" 200 2038
2025-09-16 00:32:44 [botocore.parsers] DEBUG: Response headers: {'Server': 'Server', 'Date': 'Tue, 16 Sep 2025 00:32:44 GMT', 'Content-Type': 'application/x-amz-json-1.1', 'Content-Length': '2038', 'Connection': 'keep-alive', 'x-amzn-RequestId': '907df593-51db-4291-84b0-f08a8d0afeb3'}
2025-09-16 00:32:44 [botocore.parsers] DEBUG: Response body:
b'{"Parameter":{"ARN":"arn:aws:ssm:ap-south-1:908160969848:parameter/api-secrets","DataType":"text","LastModifiedDate":1.757514600441E9,"Name":"api-secrets","Type":"String","Value":"{\\n\\t\\"rds_host\\": \\"prod.covz56o9ujx3.ap-south-1.rds.amazonaws.com\\",\\n\\t\\"rds_user_name\\": \\"restaverse_deployment_user\\",\\n\\t\\"rds_password\\": \\"D]m%))QDshFY-3P\\",\\n\\t\\"rds_db_name\\": \\"restaverse\\",\\n       \\"rds_region_name\\":\\"ap-south-1\\",\\n       \\"website_google_auth_client_id\\":\\"507355443751-pqleh0im90nt53csl5fu8brquvdmujeq.apps.googleusercontent.com\\",\\n\\t\\"google_auth_client_id\\": \\"696642202818-u9jcv3kv3filak7mijs85b9i58fefgo8.apps.googleusercontent.com\\",\\n\\t\\"google_auth_client_secret\\": \\"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\\",\\n        \\"microsoft_auth_client_id\\" : \\"7810b64e-05ba-4355-9c27-941c08b88154\\",\\n        \\"microsoft_auth_client_secret\\" : \\"LQc8Q~TU5mT.vdOBCy.xtBkQuJC8Kno9Beh_wbjs\\",\\n        \\"zomato_google_auth_client_id\\" : \\"628153421598-7mpo9933k6r6nqtg91k6s6nhq4c3ghsl.apps.googleusercontent.com\\",\\n        \\"zomato_google_auth_client_secret\\" : \\"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\\",\\n\\t\\"ACCESS_JWT_SECRET\\": \\"restaverse-access-secret\\",\\n\\t\\"REFRESH_JWT_SECRET\\": \\"restaverse-refresh-secret\\",\\n\\t\\"facebook_app_id\\": \\"1607551373115759\\",\\n\\t\\"facebook_app_secret\\": \\"01d4721055d3a2eeca294267e04d3874\\",\\n        \\"frontend_callback_url\\": \\"https://app.restaverse.com/\\",\\n        \\"frontend_callback_url_stage\\": \\"https://staging.restaverse.com/\\",\\n        \\"backend_url\\": \\"https://prod.restaverse.com\\",\\n        \\"backend_url_stage\\": \\"https://stage.restaverse.com\\",\\n        \\"ec2_backend_url\\" : \\"https://prod.restaverse.com\\",\\n        \\"open_ai_api_key\\": \\"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\\",\\n        \\"META_WEBHOOK_SECRET\\": \\"abbphwrpwp40wfntpfsdsg349cyqt93\\",\\n        \\"SCRAPEOPS_API_KEY\\": \\"dbc08e8a-4f73-4088-9f13-c2db0a774557\\"\\n}","Version":44}}'
2025-09-16 00:32:44 [botocore.hooks] DEBUG: Event needs-retry.ssm.GetParameter: calling handler <botocore.retryhandler.RetryHandler object at 0xfb9f2ec27fa0>
2025-09-16 00:32:44 [botocore.retryhandler] DEBUG: No retry needed.
2025-09-16 00:32:44 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): cron.restaverse.com:443
2025-09-16 00:32:44 [urllib3.connectionpool] DEBUG: https://cron.restaverse.com:443 "POST /api/db_services/fetch-query HTTP/11" 200 198
2025-09-16 00:32:44 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): partner.swiggy.com:443
2025-09-16 00:32:44 [urllib3.connectionpool] DEBUG: https://partner.swiggy.com:443 "POST /authentication/v1/login HTTP/11" 200 74
2025-09-16 00:32:44 [scrapy.core.engine] INFO: Closing spider (finished)
2025-09-16 00:32:44 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): api.scrapeops.io:443
2025-09-16 00:32:44 [urllib3.connectionpool] DEBUG: https://api.scrapeops.io:443 "POST /api/v1/stats/ HTTP/11" 200 108
2025-09-16 00:32:44 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'elapsed_time_seconds': 1.056435,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2025, 9, 16, 0, 32, 44, 437589, tzinfo=datetime.timezone.utc),
 'items_per_minute': None,
 'log_count/DEBUG': 42,
 'log_count/INFO': 11,
 'memusage/max': 95297536,
 'memusage/startup': 95297536,
 'responses_per_minute': None,
 'start_time': datetime.datetime(2025, 9, 16, 0, 32, 43, 381154, tzinfo=datetime.timezone.utc)}
2025-09-16 00:32:44 [scrapy.core.engine] INFO: Spider closed (finished)