Hi. WebParser gives parse error, and RainRegExp says "Unable to connect to web site", for the following and similar URL's:
https://weather.gc.ca/warnings/report_e.html?on61#2121537321901824111201902060503wo1171cwto. It can connect to the base URL https://weather.gc.ca/warnings/report_e.html, but that page is just an error page, because the ?on61#2121537321901824111201902060503wo1171cwto is missing. Is WebParser not able to handle such a URL?
It is currently April 24th, 2024, 10:13 pm
WebParser can't connect to site
-
qwerky
- Posts: 182
- Joined: April 10th, 2014, 12:31 am
- Location: Canada
-
jsmorley
- Developer
- Posts: 22629
- Joined: April 19th, 2009, 11:02 pm
- Location: Fort Hunt, Virginia, USA
Re: WebParser can't connect to site
Yeah, I get a 400 "Bad Request" error when I connect to that URL outside of a browser. It could be they have some protections built-in to reduce being hammered by requests from outside a browser, where they can show you their advertising on the page.
I tried using a UserAgent option to fake the user agent string to be the same as my browser, and that didn't work, so it may be that you just can't get to that URL with WebParser.
I tried using a UserAgent option to fake the user agent string to be the same as my browser, and that didn't work, so it may be that you just can't get to that URL with WebParser.
-
qwerky
- Posts: 182
- Joined: April 10th, 2014, 12:31 am
- Location: Canada
Re: WebParser can't connect to site
Thanks for trying and confirming. I had the same thoughts, and wondered whether UserAgent would help, but you already tried that.jsmorley wrote: ↑February 7th, 2019, 12:23 am Yeah, I get a 400 "Bad Request" error when I connect to that URL outside of a browser. It could be they have some protections built-in to reduce being hammered by requests from outside a browser, where they can show you their advertising on the page.
I tried using a UserAgent option to fake the user agent string to be the same as my browser, and that didn't work, so it may be that you just can't get to that URL with WebParser.
The WebParser options also allow setting custom headers; does anyone know of a way to see all the headers that my browser is sending?-
qwerky
- Posts: 182
- Joined: April 10th, 2014, 12:31 am
- Location: Canada
Re: WebParser can't connect to site
It seems that the #2121537321901824111201902060503wo1171cwto portion is what is preventing the page loading. Using just https://weather.gc.ca/warnings/report_e.html?on61 does work, and I am able to parse it for the text in the warning.
What I suspect is happening is this: the "on" part is the province (Ontario), and the "61" refers to the city (Toronto). The trailing part (beginning with the "#"), I think refers to the particular watch/warning. So without that trailing part, the link goes to the current/latest bulletin for that location. That is completely adequate for my needs.
So now I need to figure out how to strip the trailing portion so it will load, and then put the parsed text into a tooltip.
What I suspect is happening is this: the "on" part is the province (Ontario), and the "61" refers to the city (Toronto). The trailing part (beginning with the "#"), I think refers to the particular watch/warning. So without that trailing part, the link goes to the current/latest bulletin for that location. That is completely adequate for my needs.
So now I need to figure out how to strip the trailing portion so it will load, and then put the parsed text into a tooltip.