Chapter Seventeen – Another World: Three

Francis Delaney sighed. It was three in the fucking morning.

“Whaaat?” he drawled in annoyance as he answered the phone.

“It's pretty early, so I’m going to forgive that, Mr. Delaney,” came an older, cultured voice that made Francis’s blood run cold.

“Director McDonough! I… I’m s-so sorry!” he blurted, horrified. The woman was his boss’s boss’s boss.

He was incredibly well paid for the long, arduous hours spent killing himself to meet unbelievable deadlines. By contrast, she probably didn’t even buy her own groceries.

And he’d just fucking growled at her like a college kid did at their alarm clock.

“Forgiven. Truth be told, I’m just as annoyed as you right now, but a crisis has occurred. Get to work, immediately. The entire eastern seaboard is unable to login to the game and we are losing money by the minute, while no one in this company seems to be able to figure out why,” she seethed, her tone spelling salvation for himself but a dire warning to whoever was in the room with her.

“I… I’ll be right there! Give me, uhm. Fifteen minutes?” he guessed, already leaping to his feet.

“Fifteen minutes? Punctual. I like that. I will see you soon,” the Director said before ending the call.

Before he even left his apartment, Francis was already trying to figure out the problem and dialing his boss.

“Oh thank god, Frank,” Havery breathed heavily into the phone. Francis hated it when the man shortened his name, but couldn’t bring himself to tell the rotund man after his first day when he’d said Frank was fine.

The man was a complete tool, and how he was the manager of the development process, no one on the team could fathom. He was a decent coder, with horrible comments, horrible people skills, and a penchant for making people uncomfortable on purpose.

Still, he had kept them all on track, and at least made an effort to get to know a little about what each team’s goal was, while creating a coherent if utterly grueling deliverable schedule.

Right now, the man sounded as panicked as he had the day before launch day, and that was not a good sign.

“The world server is down!”

“Really? Why is this a big deal? Shut down the bad node and spin up a new one.”

“No you don’t understand! Not a node, the whole cluster!”

Francis threw on a pair of clean pants and a shirt fit for work before dashing out the door, frantically trying to get his belt on while rushing to his car. He didn’t live very far from his work, only a few blocks, but in the middle of the night, he figured driving would be faster and he was right.

‘The whole server cluster?’ he pondered, momentarily baffled. ‘What the hell could take out the whole cluster?’

A hacking attempt at that scale? For Tread the Sky? Not to toot the company’s horn, but the dynamic scaling nature of the system should require a computer that could scale up faster than their own in order to successfully DDOS a cluster. Shy of hackers compromising some big budget cloud provider and spending millions of dollars on someone else’s dime… Hacking was probably out.

“I’m telling you, the whole thing is down. Every node is red on the monitor, Frank. I am on a bridge with the entire C level executive team just waiting for an update and we’ve already been down for an hour.”

Individual servers had been taken down before, and honestly it was almost a sign of prestige that the game was so successful that it was attracting hackers. Not only that, it was soundly defeating them by shutting down targeted nodes and scaling up replacements quickly. In most cases the playerbase never even noticed when an attack was occuring.

“What if the hacking attempt was exploiting an in-game function? Using the cluster’s own processing power against it. But if that was it then the cluster wouldn’t be down - performance would just be abysmal. Hey, has anyone spoken to the network team yet?”

Havery didn’t hesitate, anticipating the question.

“They are adamant that all the egress links for the region are up and utilization has dropped to near null. BGP hasn’t flapped and we are still receiving the full internet route table and advertising our BGP ASN’s IP space.”

‘So, basically, the lights were on but no one was home. Our end,’ he thought glumly.

“Hnnnn,” he grunted. “I’m almost at the office, Havery. Have someone get me a mug of coffee. I’m still mostly asleep. See you in a few.”

He arrived at GypsEnergy Entertainment’s headquarters and maneuvered his way to the parking garage, taking a moment to marvel at the beautiful garden and fountains that decorated the company’s campus. Even now, after three years and hundreds of hours spent here, he was still awed by the place, and the fact that he worked here.

As soon as he parked, he forgot the majestic fountains in favor of legging it at as mad a dash as a sophisticated office worker could make at three in the morning. He walked briskly down the concrete paths around the cultivated lawn through a well lit early morning atmosphere, taking care to dodge the sprinklers.

He swiped his way past the security guard at the front before continuing on up into the fourth floor where he worked. His job was usually quite leisurely, now that the game had finally launched. Going from development to maintenance was an abrupt shift that sometimes left him floundering and bored, even though the pay was still awesome.

Before he’d taken two steps into the floor he was greeted by his boss, Havery. He thankfully had a mug of ohh so delectable java in hand and after an initial sip, the work began.

“Please… tell me you have some ideas,” Havery began. “If players can even login, they’re finding that all of the dynamic generation elements on the east coast server are inaccessible. The land being generated is shit - random white space and pixelated nonsense, because it doesn’t have an eighth of its data set to draw from! If this keeps up we’re going to have to issue large scale refunds just to appease our player base and it will piss off management something fierce!” he exclaimed.

A part of him wanted to panic, but with practiced patience Francis smiled down at the frantic little man who ran his life. “Whoa… whoa. It isn’t that bad, Havery, okay? It's three A.M. in our main target player base location. It is going to be ok. We will find and fix it. We can’t get anything done if we panic, so help me out here. Any idea about the cause yet?”

The man shook his head and flushed like a kid who didn’t know the answer to a question when called on in class.

“Alright, alright. Diagnostics? Who all has begun working on this already?” he asked.

“Donna-Lou got here first and ran diagnostics. She’s debugging now, trying to find the problem. All she’s been able to tell me so far is that it's on us. It's not a hack. It's a bug. If the Director asks, though, we think it's a hacking attempt. Okay?” he said, wringing his hands furiously.

Poor guy. Francis couldn’t help but be acutely aware of the odd shift in power dynamic between them. His boss’s job depended on his subordinate’s ability to pull off a miracle of programming, from his perspective. That couldn’t feel pleasant.

The man was making mistakes. Rookie mistakes any manager should know better than to make. What he obviously meant was that “We don’t know what’s wrong so we’re assuming it’s a hacking attempt.”

Of course, that shit wouldn’t fly anyway.

“I’m not going to lie. That will only bite us in the ass later at the Incident Review Meeting. You know management is going to be all over this one. The best we can do is do our best with all available urgency,” he said gently, but patted the man on his shoulder with a reassuring smile.

“I got this, sir. Don’t worry.”

The man smiled, but there was visible sweat trailing down his brow. Fortunately, Francis did have this. He lived for this.

He finally got to his desk and with a touch of a mouse all six of his monitors came to life.

“Hey!” he hollered to the floor. “Someone PM me the outage bridge number and start feeding me log data. I need an accurate timestamp for the start of the outage.”

He paused to take another chug of coffee and frowned. “And someone get me another coffee, please?”

A fundamental truth of the IT profession was that there was always a log. There was always some line of code or output showing where someone or something had gone sideways. What was much more difficult was knowing which log to read - there were many.

Francis groused about as he looked over them. The trick was to read them all. It sucked, but them’s the breaks. He noted timestamps and then filtered the results for particular keywords. Armed with the nearest timestamp, a fresh cup of joe, the syslog server and a linux terminal, he felt like a god in the machine, just by his ability to comprehend what most considered gibberish.

./date-time-diff.sh /var/log/*.log "Jul 31 00:15:02" "Jul 31 00:18:30" | grep -E 'write|overfl|read|error'

The output was fairly small, only a few hundred lines. Poring over them until his eyes bled was a defining feature of his job though. Some sixty lines in he found his first lead.

00:16:32 objID 47921846 write error 0x0412798 attribute-list error item-list is full.

“Well. That’s a problem,” he mused.

A little known detail about the game’s inner workings was that mobs had a very limited number of available spaces for their inventory. When the game was first being developed and intended to scale to the planned size, keeping mobs from crashing the game by limiting their inventory space to only a few items was a trade-off made to protect user performance.

In practice, this meant a mob or NPC might look like it had more items, when they actually just had a skin, which was only one item. It was rare for one to have anything beyond their skin, a weapon and armor, depending on whether the NPC was a mob or a villager or a main quest character. There was some space for originality, but the hard limit was six.

Somehow this mob had managed to get more items than they should. Unfortunately this wasn’t something he could just ‘fix’ by adding more space. No more than he could just arbitrarily allow a player's eyes to be more than the initially implemented 256 possible colors. It was hardcoded. Being thorough before he shared his findings, he looked deeper.

./date-time-diff.sh /var/log/*.log "Jul 31 00:15:02" "Jul 31 00:18:30" | grep 47921846 | head -n 3

00:16:32 objID 47921846 write error 0x0412798 attribute-list error item-list is full.

00:16:34 objID 47921846 write error 0x0412798 attribute-list error item-list is full.

00:16:35 objID 47921846 write error 0x0412798 attribute-list error item-list is full.

He breathed a sigh of relief. It was a big bug, but it was fixable. “Ugh. The system is spamming the request. How many times?”

./date-time-diff.sh /var/log/*.log "Jul 31 00:15:02" "Jul 31 04:18:30" | grep -c 47921846

690445762118346

“Shit. We are DOSing our own DB with a spam of write attempts,” he exclaimed, a little annoyed that Donna-Lou hadn’t already figured this out. “No one ever reads the logs…”

“Status, Frank? McDonough wants an update,” Havery called from the entrance. His timing was rather good, as usual.

“I’ve found our root cause. Trying to think up the best way to apply a fix.”

“What is it?”

“An NPC is overflowing the max item limit, the system keeps looping trying to add more. I am trying to think of a way to fix this that doesn’t involve increasing the limit.”

“An NPC… why hasn’t this ever happened before? NPCs need items right?” he asked.

Francis grit his teeth in annoyance, while trying not to show it. Did the man not know his own game? “Well, sir,” he began robotically. “NPCs have a hard limit of–!”

He held up a hand, stalling the explanation with an apologetic expression.

“Got it. Something I should know. Sorry, Frank. Don’t want to waste your time. I’ll appreciate the explanation later. For now, get me a fix. How long do you think you need?”

Francis brightened. He thought to himself for a moment. An NPC picking up too many items was probably a fluke. Coming up with a fix shouldn’t take too long, but better to give himself some wiggle room.

“Check back in with me in 30 minutes,” he said, doubling the actual time he thought it would take, knowing it would make him look like a wizard when he was finished in fifteen.

Havery nodded and wandered off to go update the execs. Meanwhile he pulled up the object viewer to get a closer look at what this little NPC was up to.

ObjID: 47921846

Name: Gell

Type: 0 - NPC

Creator : xd172

Lvl: 13

HP: 150

MP: 380

Spawn.zone: bb1128

Size: tiny

Stats:

0 - str - 5

1 - dex - 30

2 - vit - 15

3 - wis - 15

4 - int - 35

5 - cha - 15

6 - luc - 10

Attribute-list

Weapons:

0 - natural - melee - shocker 10d20 + 2 * int

Drop-table:

0 - 100% - [Killer Class Stat] Crystal (Sovereign)

Armor:

0 - natural - AC 14

Race:

0 - Fae

Subtype:

0 - Jelly

Skill-list

0 - sneak

1 - hide

2 - backstab

3 - shock

4 - fly

Immunities

0 - Flame

Item-list

0 - JellyFae skin

1 - Summoning Horn, lesser unicorn

2 - Glass Tiara

3 - Bugbear Tunic

4 - Leather Glove

5 - Pebble

6 - Health Potion, minor

“What a little klepto! How’d they even... xd172? I don’t recognize that username...” he murmured to himself.

He looked up the username, as well as the behavioral files for the NPC and was astounded. An intern had created this, “Gell” over 3 years ago, before the game had even been released. In typical intern style the character’s behavior files were massively bloated. Terabytes worth of data. Fucking. Terabytes. There was no way in hell the intern coded all this on his own, not to mention there was no way it would’ve made it into the game while taking up that much space initially, which meant it was able to write its own code.

In other words, a massive time bomb just waiting to explode, and he was the one left holding it. Somewhere in all that mess was the code that encouraged this NPC to keep snatching players’ shit and overrun its item limitations.

It wasn’t going to stop any time soon.

Even if he deleted their active inventory that would just be kicking the problem down the road, a few years or a few weeks. No real way of telling without finding the likely four lines of code causing the little thief to pick up items. And of course, the intern didn’t leave comments.

He supposed he could delete the NPC but… it was Terabytes of data. Even if it hadn’t started that way... this thing was someone’s baby. The idea of deleting that much data – or even a fraction of that much data, because he refused to believe that the intern wrote the whole thing – was tantamount to shanking a bitch because they left their laundry in the washer. Not to mention, that was actually an inefficient fix on its own. Deleting the NPC outright would require deleting the asset, doing a zone rebuild, then defragmenting the database, which could take a week because of its size.

'Still… Terabytes. How the hell…?'

Best not to think about that. Patchjob now, figure out what the heck this little NPC was doing with that much behavioral coding later.

Increasing the item limit wasn’t an option. Doubling NPC inventory space globally would be a massive performance hit that would have a real dollar cost executives would not accept, not to mention the thing would likely fill it soon enough anyway.

Deleting the NPC was out. Changing its capacity was out. But… there was already plenty of code already implemented for an overburdened character. He could redesign and improve the NPC template entirely, taking hours and hours of work or… he could flip a 0 to a 1, add an oversized moderator-only backpack and call it a day!

Player characters did not have the NPC item limit of 6, they had an item limit of 254, which exactly zero players had yet managed to reach, since no players were yet of a high enough level to obtain the crafting materials for the bags that would allow them to reach that limit. They even had a built in method to handle exceeding the item limit: the item in question would be mailed to a mailbox and auto deleted after 30 days. Most importantly, this fix would require no code changes and just some changes to the object.

Back in the ObjectViewer he swapped to edit mode and moved his line cursor to the appropriate line and flipped the type:

Type: 1 - PC

Added a bag:

Item-list.add quantity 1 sudobagofholding

And went to save the object. Of course, he received a commit error.

ERROR: Race: Fae is not a legal player type.

“Ugh,” he groaned. Okay.

Race:

0 - Human

ERROR: Type human cannot hold…

The error went into a line of gibberish that one couldn’t understand without hours of study and googlefu, but he got the gist. Basically, the things that made the NPC who it was, wouldn’t work if it lost the Fae Race. Easy enough.

Race:

0 - Human

Subtype:

0 - Fae

1 - Jelly

ERROR: Size: Tiny is not a legal setting for Race: Human.

“Are you freaking kidding me?”

size: medium

This time the object was saved. Thank god. Now to check up on…

“It has been 30 minutes, Frank. Where are we at?” Startled, he turned around to face Havery as a cheer went out. Monitors started showing nodes green. User counts started rising. He wordlessly lifted his arms out to his sides in a victory yawn before cheekily responding.

“All fixed, boss.”

“Damn, Frank, you’re good.”

Little too close. Maybe next time I should tell him 45 minutes? One thing was sure however, he was tracking down that damned intern and telling him about this.

“Yea, and whoever made an item addition error on an NPC a retryable event needs to be smacked. This is what DLQs are for. I mean we are obviously handling the exception, just with an infinite retry? Without backoff? Without an attempt limit? What the fuck? This will be an interesting conversation at the Incident Response Team meeting. I know we have these for player characters, but for some reason the same is not true for NPCs.”

The man was staring at the green monitors and quite literally panting with relief, though his eyes were glossing over as Francis ranted. “We’ll… discuss it. You’re a god-damn life saver, Francis. Please… keep doing you. I’ll need to talk with you later about that item limit, but for now, as long as you’re sure we’re good, go home if you like. If not, feel free to take off early.”

“Will do, Havery,” Francis said. “Just need to send a quick email.”

Havery nodded and went off to share the good news. Francis turned back to his desk, pouring over the lines and lines of behavioral code, and began to compose an email. He rolled his eyes as he actually read the personal email address, but hell, if he’d made a self-updating NPC that had gotten past development checks, perhaps it wasn’t as pretentious as it looked.

To: [email protected]

From: [email protected]

Subject: Your Creation

Good Morning,

This is Francis Powell, lead game designer at GypsEnergy, a company you interned for about three years ago. I’m reaching out to you concerning an object you created for the game Tread the Sky during your stay here named Gell, the Jellyfae…