⭐ Weather.com - Parsing the V3 JSON

[SilverAzide]

Hey guys,
I have a minor enhancement request for both the JSON and HTML template includes...

I occasionally have a situation where my network connection isn't fully up before Rainmeter launches. In this case, the call to weather.com will fail, and the skin will just sit there looking sad and empty.

Would anyone find it of benefit to add a "hook" method to catch failures? Specifically, adding an OnConnectErrorAction to the super-parent measures, @CurrentAll or @EntireSiteSuperParent as the case may be. For example:

Code: Select all

[@CurrentAll]
Measure=WebParser
Group=Weather | WeatherCurrent
Url=#URLcurrent#
RegExp=(?siU)^(.*)$
UpdateRate=#UpdateRate#
OnConnectErrorAction=#DoSomething#

Where #DoSomething# is a variable that is blank by default (in the variables include file), but developer can set it to some series of bangs to show an error or retry, etc.

[jsmorley]

Hey guys,
I have a minor enhancement request for both the JSON and HTML template includes...

I occasionally have a situation where my network connection isn't fully up before Rainmeter launches. In this case, the call to weather.com will fail, and the skin will just sit there looking sad and empty.

Would anyone find it of benefit to add a "hook" method to catch failures? Specifically, adding an OnConnectErrorAction to the super-parent measures, @CurrentAll or @EntireSiteSuperParent as the case may be. For example:

Code: Select all

[@CurrentAll]
Measure=WebParser
Group=Weather | WeatherCurrent
Url=#URLcurrent#
RegExp=(?siU)^(.*)$
UpdateRate=#UpdateRate#
OnConnectErrorAction=#DoSomething#

Where #DoSomething# is a variable that is blank by default (in the variables include file), but developer can set it to some series of bangs to show an error or retry, etc.

I actually don't think this is the right way to approach this. The temptation is going to be to "retry" or "refresh" in some way, which is doomed to get into an endless loop if your internet connection is actually down for some reason.

I think the better approach is doing this in the "skin" itself. Have a String measure that checks to see if for instance @LocationName is an empty string, and if so, hide meters, display an error, set up an action to retry on a click, whatever you want or need. As soon as @LocationName has some value, things can pop to life... I really think this is a "skin design issue". I don't think this should be baked into the @Include(s).

Code: Select all

[MeterTestData]
Meter=String
String=[@LocationName]
DynamicVariables=1
IfMatch=^$
IfMatchAction=[!HideMeterGroup UsualMeters][!ShowMeterGroup ClickToRefresh]
IfNotMatchAction=[!HideMeterGroup ClickToRefresh][!ShowMeterGroup UsualMeters]

Different authors are going to have different ways they want to deal with this, and I don't think a single OnConnectErrorAction is always going to be enough.

[SilverAzide]

Thanks, that does sound like a better approach!

[SilverAzide]

I think there might be a glitch in the JSON regex. For some reason it is returning the wrong lat/long values and I'm not sure why.

While doing some experimenting, I pointed the TWC URL at a spot in the Azores (specifically Cabouco, Azores, Portugal). The data that comes back has two chunks of location data. The first set is correct, with the correct lat/long, but the second set is a bajillion miles away in Portugal itself (Lagoa).

Weirdly, the regex seems to pull the correct displayName value, but the wrong lat/long (it's pulling the second one, in Lagoa)

Code: Select all

{
  "language:en-US:placeid:9236e17c9d6ad352c976ad39e84734fc58f4b32c535d8777b4314a44b659fa4b": {
    "loading": false,
    "loaded": true,
    "data": {
      "location": {
        "latitude": 37.767,
        "longitude": -25.567,
        "city": "Cabouco",
        "locale": {
          "locale1": null,
          "locale2": "Cabouco",
          "locale3": null,
          "locale4": null
        },
        "neighborhood": null,
        "adminDistrict": "Azores",
        "adminDistrictCode": null,
        "postalCode": "9560",
        "postalKey": "9560:PT",
        "country": "Portugal",
        "countryCode": "PT",
        "ianaTimeZone": "Atlantic\u002FAzores",
        "displayName": "Cabouco",
        "dstEnd": "2019-10-27T01:00:00-0100",
        "dstStart": "2020-03-29T00:00:00+0000",
        "dmaCd": null,
        "placeId": "9236e17c9d6ad352c976ad39e84734fc58f4b32c535d8777b4314a44b659fa4b",
        "disputedArea": false,
        "canonicalCityId": "7b7bfe5fb9a5f0f8fb49717156ee225a6073baf0b0c6a15d38b23324a0d8853e",
        "countyId": null,
        "locId": null,
        "locationCategory": null,
        "pollenId": null,
        "pwsId": null,
        "regionalSatellite": "trop",
        "tideId": null,
        "type": "city",
        "zoneId": "PT003"
      }
    },
    "ttl": 86400,
    "date": 1582165741000
  },
  "canonicalCityId:7b7bfe5fb9a5f0f8fb49717156ee225a6073baf0b0c6a15d38b23324a0d8853e:language:en-US": {
    "loading": false,
    "loaded": true,
    "data": {
      "location": {
        "latitude": 37.135,
        "longitude": -8.453,
        "city": "Lagoa",
        "locale": {
          "locale1": null,
          "locale2": "Lagoa",
          "locale3": null,
          "locale4": null
        },
        "neighborhood": null,
        "adminDistrict": "Azores",
        "adminDistrictCode": null,
        "postalCode": "8400",
        "postalKey": "8400:PT",
        "country": "Portugal",
        "countryCode": "PT",
        "ianaTimeZone": "Europe\u002FLisbon",
        "displayName": "Lagoa",
        "dstEnd": "2019-10-27T02:00:00+0000",
        "dstStart": "2020-03-29T01:00:00+0100",
        "dmaCd": null,
        "placeId": "fc61059422cefc8e569780a054f6fec9619a30de0f64bc663e483dd52611d9a4",
        "disputedArea": false,
        "canonicalCityId": "3ae64a81cc0d12d1dfca2158d505721c6b0d90db087d9242b355711090aa0af0",
        "countyId": null,
        "locId": null,
        "locationCategory": null,
        "pollenId": null,
        "pwsId": null,
        "regionalSatellite": "eur",
        "tideId": null,
        "type": "city",
        "zoneId": "PT021"
      }
    },
    "ttl": 86400,
    "date": 1582165742557
  }
}

[Yincognito]

I think there might be a glitch in the JSON regex. For some reason it is returning the wrong lat/long values and I'm not sure why.

While doing some experimenting, I pointed the TWC URL at a spot in the Azores (specifically Cabouco, Azores, Portugal). The data that comes back has two chunks of location data. The first set is correct, with the correct lat/long, but the second set is a bajillion miles away in Portugal itself (Lagoa).

Weirdly, the regex seems to pull the correct displayName value, but the wrong lat/long (it's pulling the second one, in Lagoa)

It's not a glitch, it's intentional. The 2nd subsection (i.e. "chunk") of Location is usually the most complete (i.e. where the 1st subsection field is null, the corresponding 2nd subsection field has valid data to extract). Therefore, I suggested to jsmorley that he could take only the displayName from the 1st subsection, and the rest from the 2nd one. The fact that the latitude and longitude are wrong is a glitch on TWC's side, since they incorrectly entered the geocode for Lagoa: instead of entering the one for Azores' Lagoa (i.e. this one), they entered the one for mainland Portugal's Lagoa, (i.e. this one). So basically, it's TWC's mistake.

That being said, yes, I've noticed too some rare instances when the data from the 1st subsection is more reliable, like when in the 2nd subsection it throws an error (try to look at the data for Sevastopol, Crimean Peninsula) and you'll see what I mean (by the way, by checking the existence of "location" - small caps - or "loaded":true you can avoid the "erroneous" subsection). But then, those instances are rare, as I said. Generally, you'd want data from the 1st subsection when you want things in your native language (like displaying Chinese cities in Chinese, for example), and the data from the 2nd subsection when you want things in English and as few null fields as possible.

Anyway, due to the volatile nature of TWC data, you can't 100% rely on either subsection, as you have these "exception to the rule" situations when the data you thought was reliable ends up either not being there, entered incorrectly, or throwing some error. Some of these issues dissapear on a subsequent parse of the TWC data (on another day, for instance), but some persist.

[SilverAzide]

Ah, I see now... OK!

Well then, I guess what I propose is that the lat/long ALSO be taken from the first set. I've checked repeatedly and in almost every case the lat/long of the first block of location data is more correct, and I have never seen null values there. It is especially true in this particular example in the Azores. It is also true of the case where I live; if I use the "short location code" (the 8 character one), or my Zip code, I get a lat/long closer to me, and the second block of data gives a lat/long in the exact center of the city, which is not where I am. In fact, the bigger the city, the more important the first block of data is. For example, Houston Texas is huge; if you pick a location on the edge of town, the first block of data is more correct, while the second block shows a location that is many miles away in the center of the city.

I suspect that, TWC errors aside, that the first block of location data more closely corresponds to the general area of the observation station, and the second block widens that view out to the city/etc. The first block may be coming out blank where there is insufficient data for that location (remote areas, etc.) whereas the second one is filled out because there is more geographical data available.

[Yincognito]

Well then, I guess what I propose is that the lat/long ALSO be taken from the first set.

Well, I have no problem with your proosal - hopefully jsmorley won't have either. I'm just refraining from making any more proposals on this, since it seems that it will always be the case that in some (hopefully) occasional circumstances there will be these sort of "glitches", where the data from one subsection will be "good" and the other won't. You can't therefore predict which of these two will be suitable to extract and be right in 100% of the cases, unless, as I said, you do some sort of "aggregation" of both subsections to grab the complete and correct data by "filtering" them out.

Personally, I use a different regex to capture things, but this isn't suited for jsmorley's approach, as I first take the "big chunks" of data and only "split" them later using StringIndex2 and appropriate regexes for each section/subsection. Anyway, as far as I remember, the only "trick" needed in jsmorley's regex to grab everything from the 1st subsection is to "move" both the displayName pattern in the regex and its associate WebParser child measure after the countryCode pattern / associated measure, while renumbering the StringIndexes according to the "new order" of the captures. If you only want the LAT & LONG to be taken from the 1st subsection, just "move" them before the displayName pattern/measure (while also renumbering StringIndexes properly), in the regex's current form. It's a 1 minute job, and it's not difficult at all, but if you need assistance on this, I'll be glad to help.

I suspect that, TWC errors aside, that the first block of location data more closely corresponds to the general area of the observation station, and the second block widens that view out to the city/etc. The first block may be coming out blank where there is insufficient data for that location (remote areas, etc.) whereas the second one is filled out because there is more geographical data available.

Maybe, I don't know. All I know was that having city and adminDistrict missing from the 1st subsection at the beginning (curiously, this doesn't seem to happen anymore, or maybe I don't test enough locations to be able to see that) was not tolerable, in my view, which is why I suggested this regex form to jsmorley in the first place. In the end, I'm open to any suggestion, it's jsmorley's choice to weigh in the pros and cons of each.

[jsmorley]

I think the differences in the first section and the second section, and the reliability of either, has much to do with the kind of location code you are using. I think the long "hash" version the online site uses now results in one physical location being primary, and the older "locid" (i.e. USVA0944) version results in another.

I gather that if you use the locid code, the "first" set of location information is for the very specific site you specified, and the "second" set of location information is the nearest "city".

It's all pretty confusing, and I'm not sure how best to proceed, given that any of three types of location codes can be used. "hash", "locid", or "zipcode".

However, in general, I think this is pretty close to right..

I suspect that, TWC errors aside, that the first block of location data more closely corresponds to the general area of the observation station, and the second block widens that view out to the city/etc. The first block may be coming out blank where there is insufficient data for that location (remote areas, etc.) whereas the second one is filled out because there is more geographical data available.

Right now I'm getting all location information from the "second" set, as while perhaps not as accurate as far as latitude/longitude, otherwise seemed more complete and reliable.

displayName is just one thing and is ALWAYS there. After that adminDistrict is the "state or region", and that will presumably always be a logical match with displayName, and will presumably always be there. Same is true for country. I don't get, and don't trust, city in this case. I feel that displayName logically replaces and is better than that. So really it's just latitude / longitude that is the sticking point.

[jsmorley]

In my view, if you have your users use this:

https://forum.rainmeter.net/viewtopic.php?f=118&t=34714

and always just use the locid (i.e. USVA0944) version of the location code, they will get far more accurate and reliable results.

[jsmorley]

I'm fine with switching to getting latitude / longitude from the "first" set, and all the rest from the "second" set, if that seems to work better.

The question is, is there ever a time that latitude / longitude is missing from the "first" set...

I don't know what weather.com's intentions are with this new long "hash" version of location code. I'm not sure why it was needed, as the 1,000,000 or so "locid" codes seem to be fine. My fear is that maybe they are looking to "migrate" to using the hash version, and the older locid version might be at some long-term risk. Presumably they would always and forever "cross-reference" it if needed.

There is only so much benefit that "granularity" can buy you with a weather service. It always has to be based on some "observation station", and there can only be so many of those.