PKHunter Postmortem

After a long run, Asheron’s call has finally shut down for good. As such, I’ve also shut down the data processing for the 2 related site’s I’ve run for ~15 years:  TrophyHunter and PKHunter.  Here’s a post about statistics, info, and details about PKHunter.  At some point i might do a similar post about TrophyHunter.  There’s a little less stats involved there, but the final kill map looks a lot more like Dereth than the PK-only kill map

Some stats

First processed update 10/20/2002 10:01:02 pm

First tracked kill: 10/18/2002 12:04 am Shrek-Pk killed Sigma Cubed (a test character I had on DT)

Last processed update 1/31/2017 10:00am

Last tracked kill: Elder Slug killed Plazma Infuzion on Darktide at 9:02 am on 1/31/2017

Total service time: 5218.5 days = 14 years, 3 months, 12 days, 12 hours

Downtime between eras: 565.66 days = ~1 year, 6 months, 16 hours

Active service time: 4652.84 days = ~12 years, 8 months, 12 days, 4 hours

Total submissions by users: 3,121,839

PKHunter had a very active first 3 years. Incoming data almost doubling each year.  Then?  World of Warcraft, and a delayed expansion (which broke 3rd party tools) decimated the player base.

See that big dropoff starting after August of 2004? That’s when a lot of people seemed to get into World of Warcraft betas. WoW officially released in November of 2004.

Legacy PKHunter Era

The original plugin (C++) until the “Throne of Destiny” update in July 2005. Originally, PKHunter was envisioned as a “ladder” system, like in chess. In order to move up the ladder, you have to kill someone above you on the ladder. This didn’t work very well because of how AC tracks kills. One person might get the kill “shot”, but someone else might have done the majority of the damage and has “looting rights” to the corpse.

First tracked kill: 10/18/2002 12:04am Shrek-Pk killed Sigma Cubed (a test character I had on DT)

Last legacy kill (2): Mage of Cruor killed Heltik and Spiritual Frenzy II killed Lamir at 1:04 am on 7/18/2005.

After the ToD update occurred, the plugin system that PKHunter (and all the other plugins used) called Decal, was broken and unavailable for nearly a year. Even after it was available it took me a while to learn C# (which was relatively new at the time) and convert the plugin to the old C++ plugin to the new version of Decal.

Modern PKHunter Era

The 2.0 plugin (C#) released after the ToD update was released on 2/3/2007. The scoring First modern kill: 2/3/2007 ~5:07 pm. In the new system, killshots were tracked separately from the kills according to looting rights on the corpse. A killshot alone was worth 1 point, looting rights worth 5. so you could get 6 total for a kill. The 2.0 plugin also recorded monarchy+patron information, location of kills, and kept track of who witnessed the kill. (everyone who submitted the kill), where the original one only kept the record of “x killed y” but didn’t track who submitted that. The website was updated to be able to show kills on a map. I stopped actively playing AC, and basically just kept the site running, never actually finishing a lot of the “todo” things that still show up on the website.

First tracked kill: Magic-Tcm killed Eggburt of holtberg at 5:07 pm on 2/3/2007.

Last processed kill: Elder Slug killed Plazma Infuzion on Darktide at 9:02 am on 1/31/2017

Character stats

PKHunter has a record of every kill that was witnessed by anyone running the plugin while playing. Because the scoring methods were different in the 2 eras, there is a “modern users” and “legacy users” table with statistics:

	modern	legacy
total characters	243,541	186,224
characters submitting updates	93,435	73,543
% characters submitted at least 1 update	38.4%	39.5%
distinct update sources	14,407	15,203
estimated characters/person	6.5	4.8

64,505 users were seen only in the modern era, and 7188 were only seen in the legacy era, leaving 179,036 characters that were seen over both the legacy and modern eras. That means ~73% of characters who’d played prior to ToD in the legacy era were also seen at some point in the modern era, even though there’s a gap of a year and a half between those eras! AC players were dedicated!

Interestingly, it appears that in both eras, nearly 40% of tracked characters were using the plugin at some time and submitted at least one update. By tracking submitted updates, it that most players were submitting data for 5-6 characters across servers.

The drop in users was a bugfix and cleanup of duplicated users that had slowly occurred over the lifetime of the service, see below.

	modern	legacy
total kills	508,197	3,334,239
total suicides	70,776	65,241
kills by “suicide”	2,372
suicide rate	14.4%	2.0%
total witnesses	669,224
average # witnesses	1.3
most witnesses of a single kill	38
suspect kills	889

Even though there were eventually more players tracked in the modern era, the legacy era players were a lot more active. There were 3.3 million kills tracked in that 3 year period, compared to 500k kills of the last nearly 10 years!

The suicide rate seems to be a lot higher in the modern era. This might be because of the kill/killshot distinction? Not sure. Strangely, it wasn’t until relatively recently that I realized that while I was tracking suicides from killshots, it appears that a player named suicide appears as a killer as well, as the “killer” that got kill credit as well, but wasn’t properly filtered out by server processing. On Darktide, the “suicide” character on that server ended up being the #17 overall rated player, with a lifetime perfect 1758-0 kill/death ratio.

In the modern era, most kills were submitted by more than one person, with an average of 1.3 witnesses for every kill. In many cases, this is probably from both the killer and victim submitting information.

The most witnessed kill was submitted by 38 people, when Rawrsticks killed Biffard R Nasty on 2/8/2007 on Leafcull.

Of all the data submitted, there were only 889 “suspect” kills, generally where submitted data had some kind of issue, usually when a player showed up as being killed twice (or more) in a period of time that shouldn’t have been physically possible given the time it takes in AC while a player is “safe” from being killed. I have an unconfirmed suspicion that some of these might have occurred during things like Daylight Savings Time, where that same hour occurs twice, and maybe kills during that time could get really confusing?

I believe that jump in kills in 2009 was a backlog of a couple weeks of unprocessed data due to hardware failures and rebuilding the database?

Patron / Monarchy stats

Asheron’s call had a very unique patron/vassal and monarchy system, where players pledged fealty to a higher level player (their patron) and fed a percentage of their own experience points up to their patron, based on leadership and loyalty skills. The patron player could pledge to another higher level player, creating another vassal:patron pair. This could continue up until someone who had no patron, and that person was the “Monarch” of a tree of players.

Since the modern era plugin collected vassal and monarchy information, with kills, we can see some statistics of that. This information is probably incomplete and possibly inaccurate. People could change patrons, etc, so this is just from some simple queries against the DB that don’t take any of that into account.

Player with most vassals: Drunk Al-Night had 52 vassals with 437 total kills

Player with most effective vassals: Imor had 2 vassals with 5866 kills!

Monarchy with most players (and kills): Ocom had 580 monarchy members with 12026 kills!

The most dangerous place in Dereth

It appears that the most dangerous place for PK players was near a lifestone up by the Empyrean Ice Propylaeum, in the north east corner of Dereth, which isn’t intuitive to me. There are some cottages and mansions nearby, so I’m not sure if that was a thing I never got into, or if the landblock+coordinates there are actually inside a dungeon, so it just shows there on the map. Some of the other hotspots are near starter cities, near Holtburg and Cragstone, which makes a lot of sense to me. Given that there is an amount of deaths shown in the water in the Inner Sea, I have a feeling a lot of those are actually landblocks that represent the inside of dungeons. If I find time, I really want to generate a higher resolution, more accurate version of this map. (or if you want to, let me know!)

Technical Specs

The original Decal plugin was written in C++. The backend had a website written in PHP, with a mysql database, sitting on top of Apache on Linux, in the early days, making it an old school “LAMP” app for a time. Later on I moved to IIS when I moved from linux to Windows Server, which eventually I migrated all to a VM on Azure instead of running out of my home office or garage. The backend processing was done by an hourly script which invoked processing done in Java. The modern version of the plugin was written in C#, and lived and worked for almost a decade without any updates. Over the years the source was in subversion, then TFS (tfvc). I believe I also tried Perforce, but wasn’t a fan. I’d really wished git would have existed at the early days, but git didn’t show up until 2005, and wasn’t commonly popular until well after that. I moved from subversion to tfvc at some point because I was used to a centralized repo that could be backed up seperately than how I’d been doing subversion and copying repos around. I have many random versions of the code in various forms, but I’m almost positive that I couldn’t build any version of it today if I tried!

At some point (~June 2009) there was a massive hardware failure, which took out the machine hosting the site and the database. I had the code backed up, but had to build a new machine and rebuild the database from a backup. I even sent one of the hard drives off to a disaster recovery company to get what data off of it I could, as it had other things backed up on one that I wanted to save. At another point, an upgrade of PHP (December 2013?) introduced a method into the PHP built in runtime that had the same name as a method I had in code, (which actually did the same thing but subtly differently) that also broke things for several weeks until I figured out what was going on.

Over the 14 years, the service ran on various hardware: a linux box in my home office, then a couple different mini atx or then micro atx machines in my office, then later in the garage once my office became a nursery. For a while, the hardware was in a custom PC I’d built into an old Mac Plus, complete with an LCD screen where the original Mac CRT had been. After overheating / airflow issues, with that, I built a server machine in a standard case, and that lived in the garage until I moved everything to an Azure VM instead. I looked into converting from mysql and php to sql server and asp.net on Azure, but at the time there were limits on sqlazure that would have made it problematic.  That, and i wasn’t actively playing anymore, so even testing any of my changes would have been a problem.

The plugin submitted data regularly while playing, either every 5 minutes (iirc) or when a certain number of kills were seen, or when a user logged out of a server. These submissions were batched up and processed hourly. Unlike the “Trophy Hunter” service kills weren’t processed real time. In the legacy days, since things were a ladder, kills had to be processed in sequence, as killing a very high rated player could dramatically affect your score, so if someone else saw your death, and logged off (submitting that death) before you hit the 5 minute autosubmission of a kill you’d had 4 minutes ago, it could have drastic effects on the ladder. This was one of the reasons I went to a straight up point version in the modern era. Now that there’s no new data being submitted, someone could re-process the data using one of the real ladder algorithms and come up with a final ranking if they really wanted to. Batching things up also generally ensured that any duplicate submissions were processed together.

Random telemetry

In the early days, I had google analytics hooked up for tracking, and hooked up google adwords to try to cover hosting expenses or domain names or something. I believe I got one payout from google when I hit the $100 minimum payment value. I never got anything from google about my account after that payment. I believe at some point they “lost” my payment information and stopped attempting to pay me. At some point I got notification that my account was now “dormant” for too long, so all of the unsent money went to the State of Washington and I have to do work to collect it from them? How convenient, that they couldn’t get a hold of me by any means until that mail. I used google analytics every once in a while just to see usage, but that’s about it.

In late 2013, my group at work got re-orged and retasked with working on Application Insights, Microsoft’s publicly available telemetry solution. Eventually in early 2015 I added Application Insights telemetry to the service and the website using the PHP version of our sdk. Within a few days of doing that, I’d found 2 bugs that had apparently been in the backend processing for probably a decade.

• One bug (2 characters in a php script) meant that there was a small race condition where a user with the same name could be created twice. After that user existed more than once, the bug ensured that forever after that, whenever that user attempted to use the plug-in, they’d get yet another user with their name created. Worst case, there was one character with 5067 duplicate entries! After fixing the bug, identifying the duplicates and deleting 19653 duplicates, the plugin started working for those 13 people who’d had random failures. This is the dip of users in the chart in Feb of 2015.
• The second bug was people failing to authenticate, because their user was showing up as named “LoginNotComplete”, either an issue first logging in, or possibly upon logging out and attempting to send a submission? I’m not sure when this started happening, but it was probably some change in the AC client. Once I found it in failure telemetry, I found a way to work around it in most cases.

In Sept of 2015 telemetry also showed me that something was wrong as all submissions dropped to 0. I worked backwards through all my stuff, only to find out it was a massive AC server migration, so nobody could play, so nobody could submit updates.

The end of telemetry. There’s an interesting little uptick of PK activity the last day or so before the shutdown.