⭐ Weather.com - Parsing the V3 JSON

[Yincognito]

If you want to see this "unexpected / bad" behavior in the wild, just do this:

Get into Internet Explorer, and log in to your Google account. Open GMail, and send yourself an email.

Close Internet Explorer

Have a skin that has a measure like:

Code: Select all

[MeasureMail]
Measure=WebParser
URL=https://mail.google.com/gmail/feed/atom/
RegExp="(?siU)(.*)$"
DecodeCharacterReference=1
Debug=2

Run that, and you will find the information for your logged in account in WebParserDump.txt.

Just exploring the other side of the issue here, but wouldn't that (i.e. whatever cookie) have been useful if you did something using WebParser that set such a cookie, then the next time doing the same thing everything going seamless, based on the cookie already stored? I can't find right now a good example of this, but hopefully you'll get what I mean. In other words, couldn't be such a cookie useful, if it hypothetically helped the skin having the expected behavior (in case of repeated access of an URL)?

[jsmorley]

Just exploring the other side of the issue here, but wouldn't that (i.e. whatever cookie) have been useful if you did something using WebParser that set such a cookie, then the next time doing the same thing everything going seamless, based on the cookie already stored? I can't find right now a good example of this, but hopefully you'll get what I mean. In other words, couldn't be such a cookie useful, if it hypothetically helped the skin having the expected behavior (in case of repeated access of an URL)?

But you can't "set" a cookie with WebParser. You can't "login" to a site or in anyway react to something on the site that asks any "question". Goodness knows you can't in any way deal with the complexities of OAuth and all that.

All you can do is "accept" a cookie already set, and when you connect to that site with WebParser, it will "send" that cookie back to the site as-is. Any "setting" of the cookie would have to be explicitly and manually done in a conversation between the site and Microsoft Internet Explorer.

In reality, you NEVER "set" a cookie on a site. The site sets the cookie, based on whatever information it wants to remember about you, and it just sends you that cookie. You just give that same cookie back to the site the next time you connect, so it can say "Hey, I remember you!".

All you can react to is "Here is a cookie from me", and in a sense, "Do you have that cookie I sent you?" Anything that sets any values for the cookie are well beyond the capabilities of WebParser.

I see no possible value in this, and only possible confusion or harm.

[Yincognito]

But you can't "set" a cookie with WebParser. You can't "login" to a site or in anyway react to something on the site that asks any "question". Goodness knows you can't in any way deal with the complexities of OAuth and all that.

All you can do is "accept" a cookie already set, and when you connect to that site with WebParser, it will "send" that cookie back to the site as-is. Any "setting" of the cookie would have to be explicitly done in a conversation between the site and Microsoft Internet Explorer.

All you can react to is "Here is a cookie from me", and in a sense, "Do you have that cookie I sent you?" Anything that sets any values for the cookie are well beyond the capabilities of WebParser.

I see no possible value in this, and only possible confusion or harm.

Yeah, I get it, I didn't mean to login on a site, set a value for the cookie with WebParser or anything, just, I don't know, preserving the last settings or whatever from when you visited the site - assuming this was done through a cookie the site sent to the user's computer (and that was, by default, accepted by WebParser).

Bottom line, I agree with you on what you said earlier, that's not the point. I was just trying to find out if there were cases when accepting cookie(s) from a site would have been useful, in terms of expected skin behavior - that's all. After all, nothing is just black or white, isn't it?

[jsmorley]

Yeah, I get it, I didn't mean to login on a site, set a value for the cookie with WebParser or anything, just, I don't know, preserving the last settings or whatever from when you visited the site - assuming this was done through a cookie the site sent to the user's computer (and that was, by default, accepted by WebParser).

Bottom line, I agree with you on what you said earlier, that's not the point. I was just trying to find out if there were cases when accepting cookie(s) from a site would have been useful, in terms of expected skin behavior - that's all. After all, nothing is just black or white, isn't it?

Certainly, but I just can't see much value, given that the only reasonable way to get a site to set and send a cookie destined for Internet Explorer is by using Internet Explorer, and really, who uses that for anything? I'm not even sure it still comes installed by default on all PC's in Europe. I think once Microsoft is happy with Edge, IE is likely to just go away entirely.

[Yincognito]

Certainly, but I just can't see much value, given that the only reasonable way to get a site to set and send a cookie destined for Internet Explorer is by using Internet Explorer, and really, who uses that for anything? I'm not even sure it still comes installed by default on all PC's in Europe.

I see, you're right. Ok then, got my answer - thanks.

[pul53dr1v3r]

probably some of you guys know for this but i just noticed that using UserAgent
UserAgent=Mozilla/5.0 (Linux; Android 10; Pixel 4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Mobile Safari/537.36 uses a little bit less data (1,9MB) than
UserAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0 (2,3MB).

@Morley
Thanks for mentioning the SuperParents!

Apart from the Flags, using SuperParent measure(s) to control Parents really can decrease amount of data downloaded from a site and speed up things quite a bit. Without using it, my Weather skin downloaded 4,6MB per update. Now 2,3 or 1,9 depending on UserAgent configuration.
i guess you used it for a century now but i didn't pay attention at that.

[jsmorley]

probably some of you guys know for this but i just noticed that using UserAgent
UserAgent=Mozilla/5.0 (Linux; Android 10; Pixel 4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Mobile Safari/537.36 uses a little bit less data (1,9MB) than
UserAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0 (2,3MB).

@Morley
Thanks for mentioning the SuperParents!

Apart from the Flags, using SuperParent measure(s) to control Parents really can decrease amount of data downloaded from a site and speed up things quite a bit. Without using it, my Weather skin downloaded 4,6MB per update. Now 2,3 or 1,9 depending on UserAgent configuration.
i guess you used it for a century now but i didn't pay attention at that.

I guess it makes sense that the data to fill up a little 6" screen would been less than my 32" monitor.

[Yincognito]

Apart from the Flags, using SuperParent measure(s) to control Parents really can decrease amount of data downloaded from a site and speed up things quite a bit. Without using it, my Weather skin downloaded 4,6MB per update. Now 2,3 or 1,9 depending on UserAgent configuration.
i guess you used it for a century now but i didn't pay attention at that.

Personally, I can't stand polling the site more than once per "session" / "refresh" or not getting all the needed data in one go (before I break it into pieces according to its structure), so I've always used a "tree-like" structure of WebParser parents / children measures with the help of StringIndex and StringIndex2. Not exactly jsmorley's trademark "SuperParent", but close to it, I believe.

P.S. I can't imagine how you downloaded 4.6 MB per update, or even 2.3 or 1.9 MB. If I set Debug=2 to my own "superparent" I get slightly less than 1 MB. Didn't pay attention to it, but I believe jsmorley's skin gets about the same amount, probably...

P.S.S. The UCS-2 LE BOM encoding in Notepad++ can double the file size, by the way, but Debug=2's WebParserDump.txt is UTF-8 in Notepad++.

[jsmorley]

Personally, I can't stand polling the site more than once per "session" / "refresh" or not getting all the needed data in one go (before I break it into pieces according to its structure), so I've always used a "tree-like" structure of WebParser parents / children measures with the help of StringIndex and StringIndex2. Not exactly jsmorley's trademark "SuperParent", but close to it, I believe.

Very close to it indeed, just that StringIndex2 is used to create a "child" measure. I didn't want that for organizational purposes that pleased me.

I would use StringIndex / StringIndex2 to create "child" measures that are based on parsing a StringIndex of a "parent" measure. StringIndex2 creates a "child" stringindex. I'm not doing that in this case. The "parent" measures are still just fully parents, each with a RegExp that creates their own top-level StringIndex numbers, just based on parsing the "superparent" instead of parsing the "site". The "child" measures then reference the "parent" measures. That way I only need to hit the site once.

So in short, StringIndex2 creates a "child", based on parsing one of perhaps many StringIndex's of a "parent". I preferred having distinct "parents", based on parsing a single StringIndex from a "grandparent" if you will. Then I do child measures the same as always. I like having "parents" that represent distinct areas of the JSON, like "Units", "Location", "Current", "Forecast", etc.

I could have done this with StringIndex / StringIndex2, and actually had a few fewer measures, but I think it would have been harder to read and follow.

[jsmorley]

With the desktop browser UserAgent string, EntireSite.txt is 1.02 MB (1,077,849 bytes).