Weather Skins Not Working

[Yincognito]

I thought maybe there were ways to bypass cloudflare. I have an idea, but lack the knowledge to implement it. There is a skin PluginWebViewExample. He successfully accesses the Gismeteo website. WebWiev2, which is the basis of this skin, can save the page to a txt file. This txt file can be parsed and information can be retrieved. It works manually. I have not yet seen a captcha request in WebWiev2. If sometimes a request appears, you can answer it.

Yeah, but WebView and WebParser are different things. WebView is basically placing the original webpage into the skin, accessing it is done by the Edge browser, and the webpage area will not behave like a skin but as a browser tab. WebParser has nothing to do with browsing, it just parses the text in the webpage source and looks for the desired regex pattern in it (it also doesn't support cookies or other session-based authentication, so it cannot be used to retrieve information from web sites requiring a login, except through HTTP authentication like https://myname:mypassword@somesite.com - which might explain your cloudflare / captcha problem).

To see if WebParser can retrieve the desired data when accessing an URL address, all you have to do is open that address in your browser and left click to show its Page Source from the right click context menu (at least, that's how you do it in Chrome). In the page source text page that opens, if you find your desired data using CTRL + F (aka Find), then WebParser will find it as well. If not, then you have to use alternatives like WebView and instead "navigate" that page just as you'd do it in a browser, but from your skin.

The main inconvenience with WebView is that you're not parsing any text in the page source, you're simply showing the page in the skin, which means that any action on the browsing area (like clicking, dragging, etc.) will have nothing to do with the skin, only with the browser itself (Edge) and the webpage you're on and its HTML / CSS / JavaScript code.

[Kotofanchik]

The idea is that WebWiev would save a text version of the page. Webparser does not have access to the site due to cloudflare technology. But webwiev2 has no problems with this. But I don’t know whether it can control commands. There is a manual function to save as txt. But I don’t understand whether it’s possible to manage teams. Perhaps not. Another option is to access the site using selenium and undetected_chromedriver. But I don't know Python at all. And so the idea is the same, go to the site. save in text form, process the text document with webparser.

[Yincognito]

The idea is that WebWiev would save a text version of the page. Webparser does not have access to the site due to cloudflare technology. But webwiev2 has no problems with this. But I don’t know whether it can control commands. There is a manual function to save as txt. But I don’t understand whether it’s possible to manage teams. Perhaps not. Another option is to access the site using selenium and undetected_chromedriver. But I don't know Python at all. And so the idea is the same, go to the site. save in text form, process the text document with webparser.

Well, I suppose you might be able save the text version of a webpage "automatically" via the Javascript code, but only if you get that webpage from another local webpage that you write (and whose code you can control in the first place) and open with WebView in the skin. The same thing might probably be possible via a Python script, as you mentioned, without necessarily involving another local page. That being said, these are not exactly beginner level things to do, so I would instead try to use the WebParser options, maybe you can bypass the cloudflare stuff.

Have you tried looking into the Debug, Download, UserAgent, Header or Flag options (especially the UserAgent one) of WebParser? Having a suited UserAgent for the request might "trick" the site into believing the request was made from a browser, so given that you said that the cloudflare thingy doesn't show up when you open the page via the browser or WebView (which is also using a browser), you might be able to get the desired webpage directly in WebParser.

[Kotofanchik]

Well, I suppose you might be able save the text version of a webpage "automatically" via the Javascript code, but only if you get that webpage from another local webpage that you write (and whose code you can control in the first place) and open with WebView in the skin. The same thing might probably be possible via a Python script, as you mentioned, without necessarily involving another local page. That being said, these are not exactly beginner level things to do, so I would instead try to use the WebParser options, maybe you can bypass the cloudflare stuff.

Have you tried looking into the Debug, Download, UserAgent, Header or Flag options (especially the UserAgent one) of WebParser? Having a suited UserAgent for the request might "trick" the site into believing the request was made from a browser, so given that you said that the cloudflare thingy doesn't show up when you open the page via the browser or WebView (which is also using a browser), you might be able to get the desired webpage directly in WebParser.

Yes, I tried changing the user agent. I tried to pass it off as a browser, I also changed different flags. Installed and deleted them
Flags=Resync | NoCookies | ForceReload
UserAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:81.0) Gecko/20100101 Firefox/81.0.
In regular browsers there is always a captcha prompt when you first log in. Then there may or may not be a request. The only browser that did not request a captcha was webwiev probably there is some kind of flag needed there
What flags are worth trying?

[Kotofanchik]

Well, I suppose you might be able save the text version of a webpage "automatically" via the Javascript code, but only if you get that webpage from another local webpage that you write (and whose code you can control in the first place) and open with WebView in the skin. The same thing might probably be possible via a Python script, as you mentioned, without necessarily involving another local page. That being said, these are not exactly beginner level things to do, so I would instead try to use the WebParser options, maybe you can bypass the cloudflare stuff.

Perhaps you didn’t understand me or I don’t understand you. Everything is exactly the opposite

Open the site in webweiv. Save the page as txt. Open the txt page in webparser. I tried. it works. What is needed is saved in txt. webparser handles this successfully.
The site thinks it works with a person and gives information.

[Yincognito]

Perhaps you didn’t understand me or I don’t understand you. Everything is exactly the opposite

Open the site in webweiv. Save the page as txt. Open the txt page in webparser. I tried. it works. What is needed is saved in txt. webparser handles this successfully.
The site thinks it works with a person and gives information.

Yes, I understood what you meant. The problem is that you have to do this manually, which defeats the purpose of having a skin doing it "automatically" for you (i.e. without user intervention). So besides trying with an UserAgent (which didn't change anything, based on your feedback) in WebParser, the only way to possibly attempt to "automate" the saving process (assuming you can get the actual page and not some cloudflare or captcha nonsense) would be through a script of some sort - either included in a local webpage loaded in WebWiev that acts as an "intermediary" between the skin and the actual webpage, or an external one (e.g. PowerShell, Python, etc.) that can be run by the skin.

[Kotofanchik]

So I'm trying to involve scripting specialists. But they don’t want to connect.

[Yincognito]

So I'm trying to involve scripting specialists. But they don’t want to connect.

Yep, now that's a problem indeed...
I'm a little busy now, but later on I might take a look into this. I don't guarantee anything though, and it'll be up to you to do the testing on these sites.

[Kotofanchik]

Yes, I'll wait and I'm ready to test.

[Yincognito]

Yes, I'll wait and I'm ready to test.

Well, it took a little longer because I had some WebView issues I had to solve before getting into this, since communicating between the plugin and the skin via the RainmeterAPI object was key in this case, but here it is...

[SkinFolder]\@Resources\Page.html:

Code: Select all

<!DOCTYPE html>
<html>
  <body style="margin: 0; background-color: red;">
    <script>
      var r = new XMLHttpRequest(), p = "";
      r.onreadystatechange = function() {if (r.readyState == 4 && r.status == 200) {p = r.response; RainmeterAPI.Bang('[!SetOption Data String """' + p + '"""][!UpdateMeasure Data][!UpdateMeter *][!Redraw]');};};
      r.open("GET", `https://api.allorigins.win/get?url=${encodeURIComponent("https://google.com")}`, true); r.responseType = "text"; r.send();
    </script>
  </body>
</html>

[SkinFolder]\Skin.ini:

Code: Select all

[Rainmeter]
Update=1000
AccurateText=1
DynamicWindowSize=1

---Measures---

[Page]
Measure=Plugin
Plugin=WebView
X=0
Y=0
W=5
H=5
URL=file:///#@#Page.html
UpdateDivider=-1

[Data]
Measure=String
String=
UpdateDivider=-1
RegExpSubstitute=1
Substitute='(?siU)(?:^.*lang=(.*)>.*$|^.*$)':"\1",'\\(?=")':"","^\\\d+$":""

---Meters---

[Background]
Meter=Shape
Shape=Rectangle 0,0,600,350 | Fill Color 0,0,0,128 | StrokeWidth 0 | Stroke Color 0,0,0,0

[PageData]
Meter=String
X=(600/2)
Y=(350/2)
W=600
H=350
FontFace=Consolas
FontSize=12
FontColor=255,255,255,255
FontEffectColor=0,0,0,255
StringEffect=Shadow
StringAlign=CenterCenter
AntiAlias=1
ClipString=2
MeasureName=Data
Text=Language: %1

Preview:

WebViewPageData.jpg

This uses the Page.html code to get the page source of the google.com site. The Substitute in the .ini can be adjusted to extract a different part of the response from the page source (here, the value of lang is left in the string, somewhat similar to how RegExp options work in a WebParser measure). Getting the response makes use of this nice and free API to bypass CORS and get the response even if ran from the above local webpage. I don't know if it will work with your CloudFlare / Captcha issue though, but for "normal" cases and sites, it will, as you can see from the screenshot.

P.S. The local page doesn't really need to be shown in the skin, since the response is passed to the [Data] measure automatically by the RainmeterAPI.Bang() part of the Javascript code, but I let it as a red 5 x 5 pixel "dot" in the top left corner of the skin, in case it's needed. If you want to see if getting the entire page source for a site works, just comment out (i.e. precede lines in the .ini with a ; symbol) the RegExpSubstitute and Substiitute lines, save the .ini and refresh the skin (the text will wrap and be clipped to fit the skin area, so it'll be just the part at the start of the desired page source).

P.S.S. In the Javascript code above, I used the "deprecated" XMLHttpRequest() to get the response. The "modern" fetch() can be used instead, if needed.