Real-time Reverse Beacon Spots Map - Beta

Discussion in 'Amateur Radio Software' started by AF7TI, Oct 16, 2018.

ad: L-HROutlet
ad: l-rl
ad: L-MFJ
ad: Left-3
ad: MessiPaoloni-1
ad: Subscribe
ad: Left-2
ad: l-assoc
  1. AF7TI

    AF7TI XML Subscriber QRZ Page

    -Requires QRZ Subscription for geocoding. Paste API key into password field on top right of page. Get one from
    http://xmldata.qrz.com/xml/current/?username=xx1xxx;password=abcdef

    -Only uses my server for RBN telnet stream -> websocket conversion.

    My goal with this project is to unleash the community to do map development. With visualization code 100% client side anyone can add or change stuff. I'll talk to QRZ about hosting a websocket server so that use API subscribers can better leverage our investments and accelerate development. I imagine a decent map could sell a few API subscriptions . . .

    Map is beta has some bugs, not all lookups are good. Feedback, improvements welcome.

    http://bl.ocks.org/AF7TI/raw/ab566d80629940b6e2087e9fbb6991bc/

    73s, Matt AF7TI
     

    Attached Files:

  2. AA6YQ

    AA6YQ XML Subscriber QRZ Page

    1. I'd be surprised if querying QRZ for location information can keep up with the real-time arrival of spots from the Reverse Beacon Network

    2. If there were significant use of this tool, the load on QRZ could become very large; they could mitigate this somewhat with caching, as each user would be querying QRZ for the same callsigns around the same time; you should give Fred AA7BQ a "heads up"

    3. The location information in QRZ is not always accurate; there are mobile operations, and unreported relocations

    4. All of the above can be addressed by using alternate means of determining station locations, e.g. callsign analysis, gridsquares in spot notes, and databases available from license-granting authorities (like the FCC for US callsigns)

    5. where's your API documented?

    73,

    Dave, AA6YQ
     
  3. AF7TI

    AF7TI XML Subscriber QRZ Page

    Hi Dave,

    1. Load is no problem for real-time use. Browser will melt down well before server handling API.

    2. Agree on potential load; already reached out to QRZ. Ideal situation is QRZ publishes some type of Websocket stream of RBN spots that includes enriched geocoded data. I can do this myself but I don't want to be in the data provider business or divert potential subscription revenue by republishing geolocation data. That said I want to keep cost down for QRZ so I'll be happy to do whatever requires the least resources.

    3 + 4. I know but have to start somewhere. I'd rather stand on existing geolocation efforts than get lost in the weeds doing this myself. The goal is not 100% accuracy but demonstration of tooling.

    5. I don't expose an API as this is just API consumption. All the code is in plain view in html source. https://gist.github.com/AF7TI/ab566d80629940b6e2087e9fbb6991bc .
     
  4. AF7TI

    AF7TI XML Subscriber QRZ Page

    Ok fine - I'll write a server-side version that requires no user key and caches lookups. But I'm not hosting it! Will share code when done.
     
  5. AA6YQ

    AA6YQ XML Subscriber QRZ Page

    Careful, I'm just kicking tires!

    I do plan to replace DXView's built in World Map with one that supports zooming and panning; it's been low priority for years given the available interoperation with "DX Atlas" and "Google Earth". And I need native execution, not web-hosted...
     
    AF7TI likes this.
  6. AF7TI

    AF7TI XML Subscriber QRZ Page

    Take a look at D3 for maps. I use D3 for everything. It's clean and fast. Everything is just a tiny abstraction layer on top of existing web standards like topojson for maps. I've used basemap from matplotlib then cartopy and found D3 to be so much easier to use.

    I appreciate the code tire-kicking for sure. Took at close look at lookups and realized I wasn't caching all previously found lookups. Updated code and so should reduce load on QRZ dramatically.

    If you have an interest in getting real-time RBN data and want to avoid telnet client take a look at how gists connect to my socket server. This page is a simplified version that just gets RBN spots, parses the field and adds to an array http://bl.ocks.org/AF7TI/82ba997de0a11d723daa652486403503 . Server code is all in https://github.com/AF7TI/radio/blob/master/application.py

    73s
    Matt
     

    Attached Files:

  7. AA6YQ

    AA6YQ XML Subscriber QRZ Page

    Many thanks for the D3 recommendation, Matt!

    73,

    Dave, AA6YQ
     
    AF7TI likes this.
  8. AF7TI

    AF7TI XML Subscriber QRZ Page

    Added code to save QRZ lookups to localStorage in browser. This means lookups will persist across application sessions which should further reduce unnecessary load on QRZ. To see the stored data view localNodes in Application section of Web Inspector/Developer Tools. When page loads it logs how many items are loaded from cache.

    Need some cache timeout update logic next; will mull it over. For now you can manually clear the saved nodes in developer tools via 'Clear All' or 'Delete Selected' button.

    Relevant storage code is:

    // Serialize nodes and store to aid lookups
    if (typeof(Storage) !== "undefined") {
    // Code for localStorage
    localStorage['localNodes'] = JSON.stringify(nodesCumulative);

    and retrieval:

    // Retrieve and deserialize
    if (localStorage.getItem("localNodes") != null) {
    nodesCumulative = JSON.parse(localStorage['localNodes']);
    console.log ('loaded ' + nodesCumulative.length + ' nodes from localStorage');
    }
     

    Attached Files:

    AA6YQ likes this.
  9. AF7TI

    AF7TI XML Subscriber QRZ Page

    Added error handling for failed callsign lookups today. Very pleased with QRZ DXCC parameter when using in the second form, dxcc=af7ti. Seems to cover every sign I throw at it.

    Also added session status indication to key input field. Green good. Red bad;

    Use web inspector console log to view lookups and session status details.
     

    Attached Files:

    Last edited: Oct 23, 2018
  10. AF7TI

    AF7TI XML Subscriber QRZ Page

    Added QRZ lookup cache age handling. Default value saves lookups for 7 days. Adjustable with slider. Clean function runs every min and on control change. See screenshot for example of activity logged to console.

    This parameter and all others with gui controls can be set in the browser console through chartProperties global object like chartProperties.data.maxNodeAge = 60000

    Working on lookup accuracy; know I'm not getting remote location at all times and see some obvious improvements.

    Current first-pass callsign strategy is divide callsign at slash take side with most characters. after sending this to QRZ via callsign then dxcc if callsign fails - over 99.99% of the time it returns location info. Perhaps inaccurate info but info. When inaccurate it seems to be a matter of taking the left side except for one operator.

    If anyone has a complete general function to return current location from a callsign string please share!

    // divide callsign at slash take side with most characters
    function getCall(nodeName) {
    var s = nodeName.indexOf('/');
    var l = nodeName.length;
    if (l/2 > s)
    { lookupString = nodeName.split("/").pop(); }
    else
    { lookupString = nodeName.substring(0, s != -1 ? s: nodeName.length);}
    return lookupString;
    }


    73s, Matt AF7TI
     

    Attached Files:

Share This Page