10-25-2000, 12:28 AM
Earlier today the FCC released the latest copy of the amateur
radio database updates. Their system had been down for a few
days while they installed "upgrades" to their land mobile
system.
Unfortunately, the information that they released was totally
scrambled and as a result some 140,000 callsigns on the QRZ
database were incorrectly modified.
Fortunately, we have backups! We restored the database from
0400 MST and were able to recover from the upset. In the
meantime, however, we're not loading anything else from the
FCC until they can explain what happened...
QRZ will keep you informed as the situation develops...
-fred AA7BQ
More comments by Fred on the causes... (http://www.qrz.com/article.pl?sid=00/10/24/1732251&mode=thread)
Few people realize how difficult it is to keep the site up to
date with the FCC, thousands of user updates, and hundreds
of emails per day. To say that our staff was small would be
an understatement. Of the people who help keep QRZ running on
a day-by-day basis, there's myself (the programmer and engineer),
and our two editors Mark (N7MK) and Steve (N7TX).
Yesterday, the FCC transmitted some very erroneous files. A usual
daily update to the amateur database consists of anywhere from
100 to 1000 updates. Yesterday's file (the now-infamous Monday
file) contained 98,000 record changes.
Looking at just the file sizes alone, you can see (below)
that something was terribly wrong with monday's data. These
are the files that the FCC was distributing yesterday:
<pre>
-r--r--r-- 1 root root 13538 Oct 23 23:13 l_am_fri.zip
-r--r--r-- 1 root root 16236315 Oct 24 09:51 l_am_mon.zip
-r--r--r-- 1 root root 26575 Oct 15 07:16 l_am_sat.zip
-r--r--r-- 1 root root 81427 Oct 24 10:29 l_am_thu.zip
-r--r--r-- 1 root root 58579 Oct 18 09:09 l_am_tue.zip
-r--r--r-- 1 root root 28613 Oct 19 09:12 l_am_wed.zip
</pre>
Since there's only one of me and because of the fact that I
have a full time day job with a major computer company, I can't
be here to monitor the daily updates. To solve this problem,
I wrote a program which automatically fetches the FCC data each
day and applies the changes to the QRZ database. It's a great
system as long as things are running smoothly. On the other
hand, it does suffer from the GIGO (Garbage In, Garbage Out)
syndrome on occasion.
The main problem with our update system is that it blindly
TRUSTS the FCC's data. The other sites, including the ARRL,
most likely have people who personally monitor each day's
updates before they are applied. Had I seen the monday file
before my system applied it, I would have stopped it cold.
Perhaps that's what the other online services did, but I'll
never know. I don't know how they do their updates.
To compound matters, I learned of the problem about 6 hours
after it happened as I dialed into QRZ.COM using my laptop
from a hotel room while on a business trip for my (real) day
job. It made for an interesting evening, and no dinner to boot.
In the final analysis, there were 140,000 bad updates to
our database from the monday file alone! Some callsigns,
(even my own) were assigned to two or three people after
the update.
As I mentioned before, I used the previous day's nightly backup
to restore the correct data and all is now well. I hope that
this explanation doesn't sound like whining but it is the
truth, which is all I can offer.
I still appreciate hearing from you (our users) and appreciate
the problem reports because often it's the first that I get
to hear about a problem.
73,
-fred
radio database updates. Their system had been down for a few
days while they installed "upgrades" to their land mobile
system.
Unfortunately, the information that they released was totally
scrambled and as a result some 140,000 callsigns on the QRZ
database were incorrectly modified.
Fortunately, we have backups! We restored the database from
0400 MST and were able to recover from the upset. In the
meantime, however, we're not loading anything else from the
FCC until they can explain what happened...
QRZ will keep you informed as the situation develops...
-fred AA7BQ
More comments by Fred on the causes... (http://www.qrz.com/article.pl?sid=00/10/24/1732251&mode=thread)
Few people realize how difficult it is to keep the site up to
date with the FCC, thousands of user updates, and hundreds
of emails per day. To say that our staff was small would be
an understatement. Of the people who help keep QRZ running on
a day-by-day basis, there's myself (the programmer and engineer),
and our two editors Mark (N7MK) and Steve (N7TX).
Yesterday, the FCC transmitted some very erroneous files. A usual
daily update to the amateur database consists of anywhere from
100 to 1000 updates. Yesterday's file (the now-infamous Monday
file) contained 98,000 record changes.
Looking at just the file sizes alone, you can see (below)
that something was terribly wrong with monday's data. These
are the files that the FCC was distributing yesterday:
<pre>
-r--r--r-- 1 root root 13538 Oct 23 23:13 l_am_fri.zip
-r--r--r-- 1 root root 16236315 Oct 24 09:51 l_am_mon.zip
-r--r--r-- 1 root root 26575 Oct 15 07:16 l_am_sat.zip
-r--r--r-- 1 root root 81427 Oct 24 10:29 l_am_thu.zip
-r--r--r-- 1 root root 58579 Oct 18 09:09 l_am_tue.zip
-r--r--r-- 1 root root 28613 Oct 19 09:12 l_am_wed.zip
</pre>
Since there's only one of me and because of the fact that I
have a full time day job with a major computer company, I can't
be here to monitor the daily updates. To solve this problem,
I wrote a program which automatically fetches the FCC data each
day and applies the changes to the QRZ database. It's a great
system as long as things are running smoothly. On the other
hand, it does suffer from the GIGO (Garbage In, Garbage Out)
syndrome on occasion.
The main problem with our update system is that it blindly
TRUSTS the FCC's data. The other sites, including the ARRL,
most likely have people who personally monitor each day's
updates before they are applied. Had I seen the monday file
before my system applied it, I would have stopped it cold.
Perhaps that's what the other online services did, but I'll
never know. I don't know how they do their updates.
To compound matters, I learned of the problem about 6 hours
after it happened as I dialed into QRZ.COM using my laptop
from a hotel room while on a business trip for my (real) day
job. It made for an interesting evening, and no dinner to boot.
In the final analysis, there were 140,000 bad updates to
our database from the monday file alone! Some callsigns,
(even my own) were assigned to two or three people after
the update.
As I mentioned before, I used the previous day's nightly backup
to restore the correct data and all is now well. I hope that
this explanation doesn't sound like whining but it is the
truth, which is all I can offer.
I still appreciate hearing from you (our users) and appreciate
the problem reports because often it's the first that I get
to hear about a problem.
73,
-fred