GeekNights Monday - IT Disasters

MrPeriod

Tonight on GeekNights, in light of Otakon 2014's spectacular pre-registration badge pickup system failure, we discuss how to identify, mitigate, and entirely avoid IT disasters. We've both seen our share of them (from incorrectly configured servers to mystery Ethernet drops, disabled iptables to ENABLED selinux). Learn why your procedures should never involve "copying the /opt directory." In the news, Yahoo is rolling out browser-to-browser email encryption and Google is claiming to uprank search results for encrypted sites.

• President Gilbert Gottfried
• Feynman Diagrams

Download MP3
Source Link

PlugMyDuck

It was a great episode about the problems with Otakon registration. The episode mirrored what was being said in line.

I would only add (basically my excuse to whine)

- Social Media could have been used to effectively provide incident management for Thursday night
- Registration should be trending the processing rates. It'll help cut off lines earlier
- Watch what the official channels say on social media. Commenting about an empty registration line on Friday afternoon after a night of problems only makes the convention look like it is trolling the attendees

Apreche

PlugMyDuck said:

It was a great episode about the problems with Otakon registration. The episode mirrored what was being said in line.

I would only add (basically my excuse to whine)

- Social Media could have been used to effectively provide incident management for Thursday night
- Registration should be trending the processing rates. It'll help cut off lines earlier
- Watch what the official channels say on social media. Commenting about an empty registration line on Friday afternoon after a night of problems only makes the convention look like it is trolling the attendees

I think they're really that oblivious that they don't realize they are trolling the attendees. The staff has a culture of caring about their own interests, and not prioritizing the enjoyment of the attendees. When you volunteer you sacrifice yourself for the pleasure of others. If it takes licking 30,000 envelopes to get the job done, then you lick 30,000 envelopes.

Brand

I was thinking "Why didn't they bring the database with them?" But I am not a tech person so I wasn't sure if that something that would be common to do. I really hope they just mail out badges next year. I know that can have its own issues but really it would be better than waiting in line again.

Other then linecon I do think Otakon made some serious improvements. Traffic overall hand much better flow. They used clickers to count people (though they need to count people in line, not going through the door). There was a simulcast of the AMV contest.

I'm kind of excited for DC at this point though. Baltimore was a cluster fuck last weekend, the con, a marathon, pre-season football, and three baseball games were all going on that weekend. It was kind of crazy to say the least. Also, Baltimore just does not that many food choices. I love burritos but man even I don't want to eat those three days in a row. At least DC has some decent restaurants.

Back on a more related topic, I’m not IT but I am a front end web designer and I can’t tell you how many terribly coded websites I’ve worked with. When I came to work at the place I am at now, it was a shock to the system. Several of the people who had worked her before had been graphic designers (and some of these websites look super nice) but they didn’t really know how to code. So, they either used Front Page or coded in HTML 4.01. And I’m talking about being coded that was several years after XHTML was standard. Tons of the old websites we host have no doctype (and of course adding one messes up the layout of the pages). I also love it when clients update their own website and then we need to fix something and whatever program they used creates like 40 class styles on the page.

Then some of the HTML is so poorly formatted it isn’t even funny. I’ve had to spend a lot of time re-formatting HTML documents just so I can figure out what is wronged with them. There have even been times where my correcting the code caused the page to break. So, the code is just going to be non-standard forever I guess.

Rym

Have you been to DC? That area will have issues. I suspect hotels will be more expensive too...

PlugMyDuck

Rym said:

Have you been to DC? That area will have issues. I suspect hotels will be more expensive too...

Rym, I have been to D.C. a couple of times and the hotels will be noticeably more expensive. My greater concern is what kind of neighborhoods will the affordable hotels be in.

PlugMyDuck

Brand said:

... Back on a more related topic, I’m not IT but I am a front end web designer ...

The reason #LineCon annoyed and fascinated me is that my life in technology is all about Incident and Problem management (ITIL is evil)

I am continually amazed at the way people respond to outages. I have watched as the person responsible for managing a situation focus on what could have been done to prevent the outage instead of resolving the current downtime. Which causes the support teams to become flustered and defensive. I have also witnessed wonderful times where senior management involves themselves joining technical bridges and promptly derails all progress made as they direct teams to follow leads the call has already decided against because it reminds someone of a problem that happened years ago.

I have no idea what happened behind the scenes during the Otakon outage but if it is anything like the other outages I have dealt with I am glad I didn’t need to see the way the teams responded.

MATATAT

We recently had an issue that was purely because our shitty IT guy didn't set up a scheduled backup. Lost about a year of test data. We're looking into solutions that basically eliminate him from the company.

PlugMyDuck

MATATAT said:

We recently had an issue that was purely because our shitty IT guy didn't set up a scheduled backup. Lost about a year of test data. We're looking into solutions that basically eliminate him from the company.

In larger organizations the easiest way to get rid of someone is to promote them

Dragonmaster Lou

PlugMyDuck said:

MATATAT said:

We recently had an issue that was purely because our shitty IT guy didn't set up a scheduled backup. Lost about a year of test data. We're looking into solutions that basically eliminate him from the company.

In larger organizations the easiest way to get rid of someone is to promote them

Ah, the Peter Principle... or in a worse case the Dilbert Principle.

FWIW, flaky con internet (and in the case of Otakon, expensive con internet, at least at the time) is why my panel management system/software used for Anime Boston (though I started working on it at Otakon the one year I ran their panels) is completely offline. It's hacked up, held together with digital duct tape and chewing gum, and really only usable by yours truly, but it mostly works (though it's been on my to-do list forever to rebuild it from the ground up now that I know its warts and inefficiencies)

MATATAT

PlugMyDuck said:

MATATAT said:

We recently had an issue that was purely because our shitty IT guy didn't set up a scheduled backup. Lost about a year of test data. We're looking into solutions that basically eliminate him from the company.

In larger organizations the easiest way to get rid of someone is to promote them

Is it a good or bad thing that I relate life issues to Seinfeld so often?

I guess Dilbert probably did it first but I like Seinfeld more.

Brand

I went to Katsucon several times when it was in DC and I lived in Fairfax for a few years. I did talk to my friend who lives in the area and he said a lot of gentrification was going on in the area where the convention center in DC is. And honestly, I can't imagine it being any more sketchy then Baltimore is already.

The biggest issues I can see is clogging up the Metro and issues with props that look like real weapons.

2bfree

Corporation I worked at once used the ol' Special Projects Manager position. (Manager of one, himself.)

kravenoff42

I noticed you called the feynman diagrams "fain-min" they are are actually pronounced "fine-min" they are named for Richard Feynman. Maybe this was just a slip but I thought I would put it out there.

Rym

Yeah. My physics teacher in high school pronounced his name that way, and it stuck. If I'm not careful, I always say it incorrectly.

netninja

Thanks guys for anther great show. If your looking for show ideas, how about a show about clashes between admins. For example in my experience the DBAs or web developers really don't like the sys admins controlling what they claim is there servers. Perhaps a talk on separation of duties.

As for hotels try to book them through your work. For example I can't be charged more then $88 per night if I book through work. If you don't have a job with discounts or price plans, maybe a person you know does.

Have a nice day.

Jack Draigo

Apreche said:

PlugMyDuck said:

It was a great episode about the problems with Otakon registration. The episode mirrored what was being said in line.

I would only add (basically my excuse to whine)

- Social Media could have been used to effectively provide incident management for Thursday night
- Registration should be trending the processing rates. It'll help cut off lines earlier
- Watch what the official channels say on social media. Commenting about an empty registration line on Friday afternoon after a night of problems only makes the convention look like it is trolling the attendees

I think they're really that oblivious that they don't realize they are trolling the attendees. The staff has a culture of caring about their own interests, and not prioritizing the enjoyment of the attendees. When you volunteer you sacrifice yourself for the pleasure of others. If it takes licking 30,000 envelopes to get the job done, then you lick 30,000 envelopes.

No then you invent an automatic envelope licker and patent that shit.

HMTKSteve

Jack Draigo said:

Apreche said:

PlugMyDuck said:

It was a great episode about the problems with Otakon registration. The episode mirrored what was being said in line.

I would only add (basically my excuse to whine)

- Social Media could have been used to effectively provide incident management for Thursday night
- Registration should be trending the processing rates. It'll help cut off lines earlier
- Watch what the official channels say on social media. Commenting about an empty registration line on Friday afternoon after a night of problems only makes the convention look like it is trolling the attendees

I think they're really that oblivious that they don't realize they are trolling the attendees. The staff has a culture of caring about their own interests, and not prioritizing the enjoyment of the attendees. When you volunteer you sacrifice yourself for the pleasure of others. If it takes licking 30,000 envelopes to get the job done, then you lick 30,000 envelopes.

No then you invent an automatic envelope licker and patent that shit.

Patent a wet sponge?

Jack Draigo

HMTKSteve said:

Jack Draigo said:

Apreche said:

PlugMyDuck said:

It was a great episode about the problems with Otakon registration. The episode mirrored what was being said in line.

I would only add (basically my excuse to whine)

- Social Media could have been used to effectively provide incident management for Thursday night
- Registration should be trending the processing rates. It'll help cut off lines earlier
- Watch what the official channels say on social media. Commenting about an empty registration line on Friday afternoon after a night of problems only makes the convention look like it is trolling the attendees

I think they're really that oblivious that they don't realize they are trolling the attendees. The staff has a culture of caring about their own interests, and not prioritizing the enjoyment of the attendees. When you volunteer you sacrifice yourself for the pleasure of others. If it takes licking 30,000 envelopes to get the job done, then you lick 30,000 envelopes.

No then you invent an automatic envelope licker and patent that shit.

Patent a wet sponge?

A wet sponge on a flywheel with something to keep it damp.

Victor Frost

Accidental double post.

Victor Frost

Jack Draigo said:

HMTKSteve said:

Jack Draigo said:

Apreche said:

PlugMyDuck said:

It was a great episode about the problems with Otakon registration. The episode mirrored what was being said in line.

I would only add (basically my excuse to whine)

- Social Media could have been used to effectively provide incident management for Thursday night
- Registration should be trending the processing rates. It'll help cut off lines earlier
- Watch what the official channels say on social media. Commenting about an empty registration line on Friday afternoon after a night of problems only makes the convention look like it is trolling the attendees

I think they're really that oblivious that they don't realize they are trolling the attendees. The staff has a culture of caring about their own interests, and not prioritizing the enjoyment of the attendees. When you volunteer you sacrifice yourself for the pleasure of others. If it takes licking 30,000 envelopes to get the job done, then you lick 30,000 envelopes.

No then you invent an automatic envelope licker and patent that shit.

Patent a wet sponge?

A wet sponge on a flywheel with something to keep it damp.

Bottom 1/3 of the flywheel is in a trough of water, the top of the trough near the edges of the flywheel has bristles to wipe off the excess, non-absorbed water to keep it from splashing.

Rym

Otakon is slowly almost looking at fixing this. They're doing a poll now on whether they should mail the badges.

https://www.otakon.com/forms/2015_mailbadge.asp

Interestingly, only people who have attended Otakon in the past three years and have a member ID are allowed to answer. So, anyone who stopped going because they don't mail the badges isn't able to answer...

Apreche

Rym said:

Otakon is slowly almost looking at fixing this. They're doing a poll now on whether they should mail the badges.

https://www.otakon.com/forms/2015_mailbadge.asp

Interestingly, only people who have attended Otakon in the past three years and have a member ID are allowed to answer. So, anyone who stopped going because they don't mail the badges isn't able to answer...

I answered using my member ID.

Rym

Apreche said:

Rym said:

Otakon is slowly almost looking at fixing this. They're doing a poll now on whether they should mail the badges.

https://www.otakon.com/forms/2015_mailbadge.asp

Interestingly, only people who have attended Otakon in the past three years and have a member ID are allowed to answer. So, anyone who stopped going because they don't mail the badges isn't able to answer...

I answered using my member ID.

You haven't attended in the past three years: your vote will be discarded.

Apreche

Rym said:

Apreche said:

Rym said:

Otakon is slowly almost looking at fixing this. They're doing a poll now on whether they should mail the badges.

https://www.otakon.com/forms/2015_mailbadge.asp

Interestingly, only people who have attended Otakon in the past three years and have a member ID are allowed to answer. So, anyone who stopped going because they don't mail the badges isn't able to answer...

I answered using my member ID.

You haven't attended in the past three years: your vote will be discarded.

I voted anyway. Fuck 'em.

Also, if that's true it's just another example of Otakons bad culture more concerned with taking care of existing attendees with no regard for adding more. The same thing that killed superhero comics.