Source for live currency exchange data?

[Novgorod]

Hi!

I went down the rabbit hole of making something as simple as a ticker with live currency exchange rates (e.g. USD to EUR with minutes of time resolution), but it turned out to be a huge pain. Apparently there are zero providers with public APIs (unlike crypto listing sites which have several convenient API providers), so it comes down to web scraping.

Obviously the big sites (xe, Google finance etc.) protect against scrapers by being a nuisance, first and foremost by responding with 0.5-1MB of HTML garbage to each request. In addition, Google requires session-specific consent cookies for European IPs (replacing the old "consent=YES+" cookie), and xe.com returns outdated (non-live) currency rates when the gzip compression header isn't sent. Rainmeter's WebParser can't read compressed http responses and has no conditional logic to catch the consent prompt and create the required session cookies. But even if there was a way around, it's kind of preposterous to generate hundreds of MB of traffic per day just to display a few bytes of data in a ticker. The "proper" way (without paid API access) would be to run an actual headless browser engine to extract the temporary credentials from the js code for the API access used by the website frontend - but that's certainly far beyond the scope of the poor WebParser, though gzip support would be really really useful and simple enough to enable.

Finally I compromised and settled for Finacial Times data (https://markets.ft.com/data/currencies/tearsheet/summary?s=EURUSD), which is not really live but good enough, and wastes "only" ~100kB per request. Am I making it harder than it should be? Is there a suitable data source out there for "hobbyists" which I just haven't found?

[Yincognito]

Hi!

I went down the rabbit hole of making something as simple as a ticker with live currency exchange rates (e.g. USD to EUR with minutes of time resolution), but it turned out to be a huge pain. Apparently there are zero providers with public APIs (unlike crypto listing sites which have several convenient API providers), so it comes down to web scraping.

Obviously the big sites (xe, Google finance etc.) protect against scrapers by being a nuisance, first and foremost by responding with 0.5-1MB of HTML garbage to each request. In addition, Google requires session-specific consent cookies for European IPs (replacing the old "consent=YES+" cookie), and xe.com returns outdated (non-live) currency rates when the gzip compression header isn't sent. Rainmeter's WebParser can't read compressed http responses and has no conditional logic to catch the consent prompt and create the required session cookies. But even if there was a way around, it's kind of preposterous to generate hundreds of MB of traffic per day just to display a few bytes of data in a ticker. The "proper" way (without paid API access) would be to run an actual headless browser engine to extract the temporary credentials from the js code for the API access used by the website frontend - but that's certainly far beyond the scope of the poor WebParser, though gzip support would be really really useful and simple enough to enable.

Finally I compromised and settled for Finacial Times data (https://markets.ft.com/data/currencies/tearsheet/summary?s=EURUSD), which is not really live but good enough, and wastes "only" ~100kB per request. Am I making it harder than it should be? Is there a suitable data source out there for "hobbyists" which I just haven't found?

It's true that free source data for ALL currencies is harder to find, but a free source for EUR to USD data should be easier to find. Possibilities:
1) keep the ft.com source or further look for others
2) use already made skins or the sources used by them:
- see https://forum.rainmeter.net/viewtopic.php?t=43496 for various links and techniques related to Mordasius' skins that can include currency data
3) use the WebView plugin for direct outcome but less control:
- see https://forum.rainmeter.net/viewtopic.php?t=43716 for examples that relate to crypto / bitcoin stuff that include currency data and WebView samples

[Novgorod]

It's true that free source data for ALL currencies is harder to find, but a free source for EUR to USD data should be easier to find. Possibilities:
1) keep the ft.com source or further look for others
2) use already made skins or the sources used by them:
- see https://forum.rainmeter.net/viewtopic.php?t=43496 for various links and techniques related to Mordasius' skins that can include currency data
3) use the WebView plugin for direct outcome but less control:
- see https://forum.rainmeter.net/viewtopic.php?t=43716 for examples that relate to crypto / bitcoin stuff that include currency data and WebView samples

Thanks for the reply! I thought the scripts using CNBC as data source parse the html output, which is >1MB, but someone discovered the unrestricted API, which is great! The data seems to be live and quite complete. There seems to be a typo in the query posted in the thread, which defaults the output to XML instead of JSON. Here's an example with actual JSON output:

https://quote.cnbc.com/quote-html-webservice/quote.htm?noform=1&partnerId=2&fund=1&exthrs=0&requestMethod=quick&output=json&symbols=EUR%3D|EURGBP%3D|EURKRW%3D

WebView seems overkill for this application, so let's see how long the "free" API stays free...

[Yincognito]

Thanks for the reply! I thought the scripts using CNBC as data source parse the html output, which is >1MB, but someone discovered the unrestricted API, which is great! The data seems to be live and quite complete. There seems to be a typo in the query posted in the thread, which defaults the output to XML instead of JSON. Here's an example with actual JSON output:


WebView seems overkill for this application, so let's see how long the "free" API stays free...

Yeah, the CNBC skins by Mordasius weren't parsing the entire HTML but returned an XML/JSON as well, it's just that they used to work by making a request for each individual stock / currency, until we found that GitHub sample demonstrating that one can get multiple stock / currency data as an XML / JSON via a single request to their API. Indeed, there is a typo in one of the replies by emp00 from that thread, though the sample skins posted afterwards are not affected. Personally, I preferred the XML response since its tags made it a bit easier to parse and extract data, compared to the JSON syntax, but you can obviously use which one you prefer.

I only mentioned WebView to cover all possibilities. In some simpler cases when data is hidden behind Javascript functions that WebParser can't handle, WebView is a decent alternative, albeit with some drawbacks when it comes to the skin control.

[balala]

I went down the rabbit hole of making something as simple as a ticker with live currency exchange rates (e.g. USD to EUR with minutes of time resolution),

Not quite a time resolution of minutes, however one of my previous works, does show the EUR / USD exchange rate, with a chart of the evolution of these rates for last five days. The skin is pretty old (or in fact it's extremely old) so, lot of things should be rewritten. If you need it, I can help on rewriting it.

[Novgorod]

Not quite a time resolution of minutes, however one of my previous works, does show the EUR / USD exchange rate, with a chart of the evolution of these rates for last five days. The skin is pretty old (or in fact it's extremely old) so, lot of things should be rewritten. If you need it, I can help on rewriting it.

Thanks, but the ECB data gets only updated once per day. The CNBC API is perfect for as long as it stays public and works fine now. The parsing and the script were very easy to make - I only needed a minimalist ticker to always show the latest rates for some currencies.

[Active Colors]

Does this have currencies?
https://www.alphavantage.co/
Seems promising.

[Yincognito]

Thanks, but the ECB data gets only updated once per day. The CNBC API is perfect for as long as it stays public and works fine now. The parsing and the script were very easy to make - I only needed a minimalist ticker to always show the latest rates for some currencies.

Be aware that some of the CNBC data also has a 15 minutes delay, according to their usage terms. Not sure if that applies to currencies or whether such a time resolution is acceptable to you, but I thought you should know.

[Novgorod]

Does this have currencies?
https://www.alphavantage.co/
Seems promising.

It has currencies, but it's not a free API by any stretch of the imagination. You get 25 requests per day for "free", which isn't even usable for demo or testing purposes.

Be aware that some of the CNBC data also has a 15 minutes delay, according to their usage terms. Not sure if that applies to currencies or whether such a time resolution is acceptable to you, but I thought you should know.

Good to know. If the API is to be trusted (it time-stamps the actual data values in the "last_time" and "reg_last_time" fields as well as the time of the request, "responseTime"), forex values are updated every minute and crypto every second, which is plenty fine for me. Stonks (at least from the US) come with millisecond time-stamps close to real-time and the actual price values change pretty much on every request (sub-second resolution), though that's less relevant for me. Of course it's impossible to know whether there's an artificial delay undisclosed in the time-stamping fields of the API response, or whether these time-stamps just indicate when they updated the data from a (possibly delayed) external source. In any case, it's entirely for private use, there's no big money riding on it and I'm not setting up a quantum entanglement laser link to the NYSE just yet ...

[Yincognito]

Good to know. If the API is to be trusted (it time-stamps the actual data values in the "last_time" and "reg_last_time" fields as well as the time of the request, "responseTime"), forex values are updated every minute and crypto every second, which is plenty fine for me. Stonks (at least from the US) come with millisecond time-stamps close to real-time and the actual price values change pretty much on every request (sub-second resolution), though that's less relevant for me. Of course it's impossible to know whether there's an artificial delay undisclosed in the time-stamping fields of the API response, or whether these time-stamps just indicate when they updated the data from a (possibly delayed) external source. In any case, it's entirely for private use, there's no big money riding on it and I'm not setting up a quantum entanglement laser link to the NYSE just yet ...

Haha, yeah, wait till you'd like more and more features in the skin. All of them start small, but obviously it's always nice to have more.

If you check the Mordasius skins' links where the linked discussion I posted earlier continued, you'll find out that in some cases, there are (at least?) two <last> fields for a stock / currency, so you might plan with that in mind as well (there, I posted a way to get the last <last> via the greediness of the regex pattern):
https://forum.rainmeter.net/viewtopic.php?t=36931&start=210#p221576