The past few weeks have been a frustrating time for many users of the ARRL’s Logbook of The World (LoTW) QSO confirmation system. The problems began with a 3-1/2 day outage during the first week of November -- a particularly busy time of the year for log uploads. When the system was brought back on line, a large queue of logs awaiting processing developed and processing times lengthened well past normal expectations.

Another, much more subtle problem cropped up at about the same time: some logs began disappearing from the queue, apparently at random. Users had been advised at the time the log was uploaded that it was “queued for processing,” and so they were understandably upset when the log was not processed, even after the several-day delay that most logs were experiencing.

Because of its random nature, it took the ARRL IT staff a while to figure out what was happening. When LoTW was designed more than a decade ago -- long before the present IT staff was here -- an assumption was made as to how many logs could possibly be in the queue at a given time. The assumption was based on users uploading their most recent QSOs perhaps once a week or once a month. The environment in which LoTW now operates is quite different from that assumption, in that many users now upload logs with small numbers of QSOs in them, almost in real time. This creates a much larger number of separate logs.

When a log is uploaded, it is identified by a file name that is assigned by the user. Because there is no way to avoid duplication of file names that are assigned in this fashion, the LoTW system renames each file. Because of the unusual processing delay -- combined with the dramatic increase in the number of submitted logs -- the system began to run out of unique identifiers for the log files. This resulted in a file sometimes being renamed with an identifier that had already been assigned to a log that was still in the queue, causing the earlier log to be overwritten.

Once the problem was identified, designing a fix was relatively easy. It should be in place by 2359 UTC November 28. Because the number of overwritten logs is relatively small, we have decided to keep the system available for use, even though this may result in a few more logs being lost until the fix is in place.

We apologize for the inconvenience that users have experienced, and especially for being unable to explain what was happening until now. We want to emphasize that no data from processed logs has been lost. That data is secure and backed up. If you have had a log disappear after it was “queued for processing,” the solution is to upload the log file again, preferably after the bug fix is in place. We will announce when that occurs.

73,
David Sumner, K1ZZ
Chief Executive Officer, ARRL
November 28, 2012