It is currently April 8th, 2020, 3:34 pm

⭐ weather.com - Parsing the JSON

Our most popular Tips and Tricks from the Rainmeter Team and others
User avatar
Yincognito
Posts: 1152
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ weather.com - Parsing the JSON

Post by Yincognito »

jsmorley wrote:
February 20th, 2020, 2:10 pm
It's all pretty confusing, and I'm not sure how best to proceed, given that any of three types of location codes can be used. "hash", "locid", or "zipcode".
You nailed it.
jsmorley wrote:
February 20th, 2020, 2:10 pm
displayName is just one thing and is ALWAYS there. After that adminDistrict is the "state or region", and that will presumably always be a logical match with displayName, and will presumably always be there. Same is true for country. I don't get, and don't trust, city in this case. I feel that displayName logically replaces and is better than that. So really it's just latitude / longitude that is the sticking point.
I might be wrong, but from my observations, displayName is just the one of the localeN fields (where N=1..4) taken from the 1st subsection (or viceversa). I have in front of my eyes the source page of TWC from when the adminDistrict from the 1st subsection was null:

Code: Select all

					"Location": {
						"language:en-GB:locationType:locid:locid:ROXX4613:1:RO": {
							"loading":false,
							"loaded":true,
							"data": {
								"location": {
									"latitude":47.95,
									"longitude":26.4,
									"city":null,
									"locale": {
										"locale1": null, "locale2": "Dorohoi", "locale3": null, "locale4": null
									}
									,
									"neighborhood":null,
									"adminDistrict":null,
									"adminDistrictCode":null,
									"postalCode":"715200",
									"postalKey":"715200:RO",
									"country":"Romania",
									"countryCode":"RO",
									"ianaTimeZone":"Europe\u002FBucharest",
									"displayName":"Dorohoi",
									"dstEnd":"2019-10-27T04:00:00+0200",
									"dstStart":"2020-03-29T03:00:00+0300",
									"dmaCd":null,
									"placeId":"308812117dc37d5579836cb12451d875f7d56d0f6110b0039d6f7dd92ca06004",
									"disputedArea":false,
									"canonicalCityId":"de6a0e21dc1d414329278dfa9961435380b25b76a2cfeb7b7b6156364e9afcbb",
									"countyId":null,
									"locId":"47.950\u002F26.400:29:RO",
									"locationCategory":null,
									"pollenId":null,
									"pwsId":null,
									"regionalSatellite":"eur",
									"tideId":null,
									"type":"locid",
									"zoneId":"RO005"
								}
							}
							,
							"ttl":86400,
							"date":1580559384000
						}
						,
						"canonicalCityId:de6a0e21dc1d414329278dfa9961435380b25b76a2cfeb7b7b6156364e9afcbb:language:en-GB": {
							"loading":false,
							"loaded":true,
							"data": {
								"location": {
									"latitude":47.96,
									"longitude":26.4,
									"city":"Dorohoi",
									"locale": {
										"locale1": null, "locale2": "Dorohoi", "locale3": null, "locale4": null
									}
									,
									"neighborhood":null,
									"adminDistrict":"Botoșani County",
									"adminDistrictCode":null,
									"postalCode":"715200",
									"postalKey":"715200:RO",
									"country":"Romania",
									"countryCode":"RO",
									"ianaTimeZone":"Europe\u002FBucharest",
									"displayName":"Dorohoi",
									"dstEnd":"2019-10-27T04:00:00+0200",
									"dstStart":"2020-03-29T03:00:00+0300",
									"dmaCd":null,
									"placeId":"15bc0d7380e2eb48a62bf7775bf74c3d5e7239934fd863d0827a029a5433f320",
									"disputedArea":false,
									"canonicalCityId":"de6a0e21dc1d414329278dfa9961435380b25b76a2cfeb7b7b6156364e9afcbb",
									"countyId":null,
									"locId":null,
									"locationCategory":null,
									"pollenId":null,
									"pwsId":null,
									"regionalSatellite":"eur",
									"tideId":null,
									"type":"city",
									"zoneId":"RO005"
								}
							}
							,
							"ttl":86400,
							"date":1580559385000
						}
					}
					,
and the location was displayed in the HTML part (as well as the actual main page) as - notice how the district is missing here as well - :

Code: Select all

<h1 class="h4 today_nowcard-location" className="h4 today_nowcard-location">Dorohoi, Romania<span ...
making it clear that the HTML part of the source takes its data from the 1st subsection (but apparently from the localeN fields in the case of displayName).

By the way, when adminDistrict was missing from the 1st subsection, I was using the "old" ROXX4613:1:RO code, while for instances that adminDistrict was present in the 1st subsection, I used the "new" long 15bc0d7380e2eb48a62bf7775bf74c3d5e7239934fd863d0827a029a5433f320 code. I don't know if that's a rule, but worth mentioning anyway.
User avatar
SilverAzide
Posts: 732
Joined: March 23rd, 2015, 5:26 pm

Re: ⭐ weather.com - Parsing the JSON

Post by SilverAzide »

jsmorley wrote:
February 20th, 2020, 2:10 pm
displayName is just one thing and is ALWAYS there. After that adminDistrict is the "state or region", and that will presumably always be a logical match with displayName, and will presumably always be there. Same is true for country. I don't get, and don't trust, city in this case. I feel that displayName logically replaces and is better than that. So really it's just latitude / longitude that is the sticking point.
Yes, the lat/long is the only issue I am concerned about, because of a number of conditions. I have never seen these blank.

First, for me, the lat/long is actually pretty critical, because I use these values to calculate sunrise/set and moonrise/set (I ignore TWC data; wxdata was never correct in the forecast section) and to calculate the exact moon age/phase (not the more generic 8 phases used by TWC). It also is the basis for Dark Sky's API calls, as they use lat/long instead of codes, and I was hoping to put together a drop-in replacement set of includes to allow me (or others) to easily/quickly switch to Dark Sky if TWC ever starts blocking or obfuscating the data.

My second concerned lines up with yours regarding the code in general. I was going save this for a separate topic later (perhaps on the new "Rainmeter hosted codes" thread), but I suggest that folks might want to consider dumping these annoying weather codes altogether. Why? Because the TWC URL can handle lat/longs as easily as codes, and I will say might work even better. Let TWC figure out the closest station, forget trying to figure it out yourself with all these annoying lookups. For example, in the JSON test project skin, make this simple change:

Code: Select all

;LocationCode=USVA0944
LocationCode=38.713279,-77.054059
(Make sure you don't have any spaces.) You'll see that TWC has no problem fetching the correct data, and the FIRST set of location data will have the lat/long of the observation area, the second set will have the lat/long of the city. (Sometimes these are very close, sometimes not.)

There is a free API site that will allow you to search for locations and it will return lat/longs (JelleDekkers uses it in his skins). You can also easily grab lat/long from Google Maps, just drop a pin and it will tell you the decimal coordinates.
DeviantArt Gadgets More...
User avatar
jsmorley
Developer
Posts: 20457
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ weather.com - Parsing the JSON

Post by jsmorley »

Yeah, I think it is correct that when a very specific locid is used, there may not be details of the area (adminDistrict, city etc.) available to them. That is likely why there is always that "second" set that is based on the nearest "known" city or town.

As I said though, I think displayName is just always going to be right and exist. After that, with the exception of latitude / longitude, the information from the "second" set is going to be pretty much just as accurate, and much more reliable.

The only question in my mind is how best to handle latitude / longitude. Again, the question is whether or not it is ALWAYS there in the "first" set of information. If the answer is "yes", that is the more accurate version.
User avatar
SilverAzide
Posts: 732
Joined: March 23rd, 2015, 5:26 pm

Re: ⭐ weather.com - Parsing the JSON

Post by SilverAzide »

OK good... I've been beating on the URL in every way I can think of and so far I have never see blank lat/longs in the first data set except on total failures (like when you give it the coordinates of a spot in the middle of an ocean), or -- for some weird reason -- Bermuda. No idea why TWC doesn't like Bermuda.
DeviantArt Gadgets More...
User avatar
jsmorley
Developer
Posts: 20457
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ weather.com - Parsing the JSON

Post by jsmorley »

SilverAzide wrote:
February 20th, 2020, 2:45 pm
Yes, the lat/long is the only issue I am concerned about, because of a number of conditions. I have never seen these blank.

First, for me, the lat/long is actually pretty critical, because I use these values to calculate sunrise/set and moonrise/set (I ignore TWC data; wxdata was never correct in the forecast section) and to calculate the exact moon age/phase (not the more generic 8 phases used by TWC). It also is the basis for Dark Sky's API calls, as they use lat/long instead of codes, and I was hoping to put together a drop-in replacement set of includes to allow me (or others) to easily/quickly switch to Dark Sky if TWC ever starts blocking or obfuscating the data.

My second concerned lines up with yours regarding the code in general. I was going save this for a separate topic later (perhaps on the new "Rainmeter hosted codes" thread), but I suggest that folks might want to consider dumping these annoying weather codes altogether. Why? Because the TWC URL can handle lat/longs as easily as codes, and I will say might work even better. Let TWC figure out the closest station, forget trying to figure it out yourself with all these annoying lookups. For example, in the JSON test project skin, make this simple change:

Code: Select all

;LocationCode=USVA0944
LocationCode=38.713279,-77.054059
(Make sure you don't have any spaces.) You'll see that TWC has no problem fetching the correct data, and the FIRST set of location data will have the lat/long of the observation area, the second set will have the lat/long of the city. (Sometimes these are very close, sometimes not.)

There is a free API site that will allow you to search for locations and it will return lat/longs (JelleDekkers uses it in his skins). You can also easily grab lat/long from Google Maps, just drop a pin and it will tell you the decimal coordinates.
My fear with having users use LocationCode=38.713279,-77.054059 is that there is no logical "match" between that, and what weather.com has as "locations". I think when you use lat/long in the URL, weather.com will always go find the nearest "city" that is the closest match, and that isn't going to end up as accurate as it might seem on the surface.

USVA0944 has a very specific and "known" meaning to weather.com.
User avatar
jsmorley
Developer
Posts: 20457
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ weather.com - Parsing the JSON

Post by jsmorley »

In other words,

USVA0944 might mean 38.713279,-77.054059

But I have no confidence that 38.713279,-77.054059 means USVA0944

If you see what I mean...
User avatar
jsmorley
Developer
Posts: 20457
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ weather.com - Parsing the JSON

Post by jsmorley »

Later today, I will change to getting Latitude / Longitude from the "first" section.
User avatar
SilverAzide
Posts: 732
Joined: March 23rd, 2015, 5:26 pm

Re: ⭐ weather.com - Parsing the JSON

Post by SilverAzide »

jsmorley wrote:
February 20th, 2020, 2:54 pm
In other words,

USVA0944 might mean 38.713279,-77.054059

But I have no confidence that 38.713279,-77.054059 means USVA0944

If you see what I mean...
Yes, I do see what you mean. I'm still chewing on that example... But a sincere question, what about huge cities like Houston TX or Jacksonville FL? There are dozens of location codes that could apply, and I would have no clue which is which. If I used the lat/long, TWC would tell ME (instead of me telling it) what the nearest observation station was to my selected spot. Wouldn't that be better? :???:
DeviantArt Gadgets More...
User avatar
jsmorley
Developer
Posts: 20457
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ weather.com - Parsing the JSON

Post by jsmorley »

SilverAzide wrote:
February 20th, 2020, 3:02 pm
Yes, I do see what you mean. I still chewing on that example... But a sincere question, what about huge cities like Houston TX or Jacksonville FL? There are dozens of location codes that could apply, and I would have no clue which is which. If I used the lat/long, TWC would tell ME (instead of me telling it) what the nearest observation station was to my selected spot. Wouldn't that be better? :???:
Maybe, maybe not. I have found that if a user knows where they are, even in a big city like Houston, there is some "named area" that is the best match for them. Just like for me, where weather.com will always use "Alexandria" as the city if I use any "hash" code or "zipcode", it will properly display, and use FAR more accurate lat/long (in the first section) if I specify USVA0944, the "locid" for "Fort Hunt". I live in an area called Fort Hunt, I don't live in Alexandria, in spite of what the post office thinks.

So if a user lives in New York, "Greenwich Village" might be a far more accurate choice than "New York City". I don't know if that even exists, just an example...
User avatar
Yincognito
Posts: 1152
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ weather.com - Parsing the JSON

Post by Yincognito »

jsmorley wrote:
February 20th, 2020, 2:47 pm
The only question in my mind is how best to handle latitude / longitude. Again, the question is whether or not it is ALWAYS there in the "first" set of information. If the answer is "yes", that is the more accurate version.
You know, when I wanted to suggest some things a couple of days ago (touching the error occurences as well), I too was just about to write that LAT / LONG are the most reliable and that they always exist (and that they could possibly be used instead of the long location code) ... until SliverAzide's post today. :lol: Now, I'm not sure of anything anymore - even though this was just a misfortunate occurence of bad LAT / LONG from TWC's part... :???:
SilverAzide wrote:
February 20th, 2020, 3:02 pm
Yes, I do see what you mean. I'm still chewing on that example... But a sincere question, what about huge cities like Houston TX or Jacksonville FL? There are dozens of location codes that could apply, and I would have no clue which is which. If I used the lat/long, TWC would tell ME (instead of me telling it) what the nearest observation station was to my selected spot. Wouldn't that be better? :???:
I believe you'd still have to check a bit the localeN for a more precise localization of the "spot" you want to see. That would make the difference from a location like "London, Greater London, UK" and (just guessing here) "Highbury, North London, UK". I don't know if Lat / Long and the displayName would be enough to get a more precise location string. Just saying.