ad: elecraft

FCC Dumps a Load on the Internet (updated)

Discussion in 'Amateur Radio News' started by Guest, Oct 25, 2000.

Thread Status:
Not open for further replies.
ad: L-HROutlet
ad: l-rl
ad: Radclub22-2
ad: abrind-2
ad: L-MFJ
ad: Left-2
ad: Left-3
  1. Guest

    Guest Guest

    Earlier today the FCC released the latest copy of the amateur
    radio database updates. Their system had been down for a few
    days while they installed "upgrades" to their land mobile
    system.



    Unfortunately, the information that they released was totally
    scrambled and as a result some 140,000 callsigns on the QRZ
    database were incorrectly modified.



    Fortunately, we have backups! We restored the database from
    0400 MST and were able to recover from the upset. In the
    meantime, however, we're not loading anything else from the
    FCC until they can explain what happened...



    QRZ will keep you informed as the situation develops...



    -fred AA7BQ



    More comments by Fred on the causes...


    Few people realize how difficult it is to keep the site up to
    date with the FCC, thousands of user updates, and hundreds
    of emails per day. To say that our staff was small would be
    an understatement. Of the people who help keep QRZ running on
    a day-by-day basis, there's myself (the programmer and engineer),
    and our two editors Mark (N7MK) and Steve (N7TX).




    Yesterday, the FCC transmitted some very erroneous files. A usual
    daily update to the amateur database consists of anywhere from
    100 to 1000 updates. Yesterday's file (the now-infamous Monday
    file) contained 98,000 record changes.




    Looking at just the file sizes alone, you can see (below)
    that something was terribly wrong with monday's data. These
    are the files that the FCC was distributing yesterday:

    <pre>
    -r--r--r-- 1 root root 13538 Oct 23 23:13 l_am_fri.zip
    -r--r--r-- 1 root root 16236315 Oct 24 09:51 l_am_mon.zip
    -r--r--r-- 1 root root 26575 Oct 15 07:16 l_am_sat.zip
    -r--r--r-- 1 root root 81427 Oct 24 10:29 l_am_thu.zip
    -r--r--r-- 1 root root 58579 Oct 18 09:09 l_am_tue.zip
    -r--r--r-- 1 root root 28613 Oct 19 09:12 l_am_wed.zip
    </pre>

    Since there's only one of me and because of the fact that I
    have a full time day job with a major computer company, I can't
    be here to monitor the daily updates. To solve this problem,
    I wrote a program which automatically fetches the FCC data each
    day and applies the changes to the QRZ database. It's a great
    system as long as things are running smoothly. On the other
    hand, it does suffer from the GIGO (Garbage In, Garbage Out)
    syndrome on occasion.




    The main problem with our update system is that it blindly
    TRUSTS the FCC's data. The other sites, including the ARRL,
    most likely have people who personally monitor each day's
    updates before they are applied. Had I seen the monday file
    before my system applied it, I would have stopped it cold.
    Perhaps that's what the other online services did, but I'll
    never know. I don't know how they do their updates.




    To compound matters, I learned of the problem about 6 hours
    after it happened as I dialed into QRZ.COM using my laptop
    from a hotel room while on a business trip for my (real) day
    job. It made for an interesting evening, and no dinner to boot.
    In the final analysis, there were 140,000 bad updates to
    our database from the monday file alone! Some callsigns,
    (even my own) were assigned to two or three people after
    the update.




    As I mentioned before, I used the previous day's nightly backup
    to restore the correct data and all is now well. I hope that
    this explanation doesn't sound like whining but it is the
    truth, which is all I can offer.




    I still appreciate hearing from you (our users) and appreciate
    the problem reports because often it's the first that I get
    to hear about a problem.



    73,

    -fred
     
Thread Status:
Not open for further replies.

Share This Page

ad: CQMM-1