From the Richta GPS Home Office in Kansas — March 10, 2024
Earlier this morning, I released version 1.39 of the Android Rallymaster app. This version contains a fix for a bug first reported two years ago. Identifying the root cause of the bug illustrates the care that must be taken in software development to ensure that one fixes the real cause of the reported errors.
During the Summer Alcan 5000 in 2022, I received a report that several cars were mistimed at checkpoints. Were Apple and Android devices affected? Which cars and checkpoints? Could I get debug logs from the affected cars? Entrants were required to drive as many as 500 miles per day, leaving little time to collect necessary descriptions of the errors. Given the nature of the event, initial details were sketchy.
I needed a complete description of the problem. I had yet to learn if it was because of some anomaly in my system at higher latitudes (no) or whether it was a user error because the phones affected had the wrong time zone setting. The Alcan 5000 covers so much territory that it covers three time zones over ten days, even changing time zones during a long day of driving. Lacking sufficient detail, I needed to figure out what to change to correct this problem.
Fast-forward to the Winter Alcan 2024, which ended this past week (Feb 21-March 1st). Reports of the same mistiming at checkpoints started late in the 10-day event. It’s time to collect more data.
It turns out that cars were only mistimed at CZT (car zero time) Restart checkpoints. The first reported occurrence was on the day when entrants transited from Yellowknife, Yukon Territory, to Fort Nelson, British Columbia. Research showed that Yellowknife is in the Mountain time zone while British Columbia is in the Pacific time zone. Wait! Fort Nelson is in BC but elected to stay on Mountain time. With this clue, I postulated that my apps had issues starting in one time zone, and the car moved to a different time zone.
After considerable thought, I devised a way to recreate the problem. I set a CZT on my deck and another 100 feet away on the septic tank cover. I turned off the Use Network-provided Time Zone on my phone and manually changed the time zone after the first CZT and before the second. Failure. The apps worked as desired and expected. So much for Theory #1.
No teams reported problems until the cars got to Fairbanks, Alaska. Each car was timed at the CZT checkpoint an hour earlier than expected and desired. The problem continued as the route wound its way to Anchorage. All cars, Apple and Android(!?!?), experienced the problem.
I called Jim Crittenden, who provided remote scoring support for the event. We eventually devised a series of six test scenarios that would help me get to the bottom of this problem. Finally, on test #6, we replicated the issue.
The problem occurred on Apple and Android Competitor apps when passing through a CZT in the Alaska time zone. The calculated “in time” at the CZT was recorded an hour earlier than expected. Now, the “in time” at a CZT is the restart time from the database for that CP plus the car number in minutes and bears no relation to when the CZT was actually tripped. (There were no reports of mistiming at any checkpoint that was not a CZT Restart).
During a call with Jim, I said, “It’s almost as if they changed the time zone rules for Alaska.” Jim did a Google search and reported that that was indeed the case! In 1983, the main body of Alaska moved an hour earlier, from Alaska-Hawaii Standard Time (GMT-10) to Alaska Standard Time (GMT-9). The light bulb went on.
The Richta database stores the timestamp for CZT Restarts in a unit representing the number of milliseconds since midnight on January 1, 1970, GMT. Interpreting this value requires one to specify the time zone for which that time is being requested. An 8:00 am CZT in Alaska in 1970 was 8:00 am in 1970 but 7:00 am in 2024!
A quick test suggested the eventual solution to the problem. I had been storing CZT Restart times with the 1970 GMT instead of the current year. Adding ten lines of code to Android Rallymaster now stores CZT Restart times with the current year.
Problem solved.
There were several interesting aspects of this problem. First, the Apple version of Rally Master already stored the CZT Restart times in the current year, not 1970, as did the Android Rallymaster app. Second, deep in the bowels of Android and Apple iOS, there is code in the time zone interpretation software that knows about the 1983 change to the time zone used in Alaska. Yet third, and perhaps most importantly, I was reminded that when investigating a complex problem, it is critically important to know when a problem occurs and when it does not, as well as to understand what parts of the system are affected and which are not.
For those interested in a structured approach to solving complex problems or making decisions, I highly recommend a non-computer science book called The Rational Manager by Kempner and Tregoe, first published in 1965. The case studies presented make for entertaining reading.
