So many jobs these days use technology to reach new audiences, sell products and services, and provide entertainment to people all over the world.  But are these considered "tech jobs" because of that? Or is a tech job strictly in the domain of software development? Are we all in tech right now?

We examine this further by starting off with a conversation between our VP of engineering James Turnbull, and one of our software developers Sheridan Kates. TK Dutes talks with author and entrepreneur Mushambi Mutuma about the concept of being "tech adjacent", and we're debuting a new segment titled "Jenngineering", straight from the mind of our director of health and support Jenn Schiffer.

Looking for bonus content? Our app collection for this episode is where it's at, including links to Glitch apps, profiles, articles, and other information mentioned in this episode!

Executive Producers: Maurice Cherry, Keisha "TK" Dutes
Editor: Brittani Brown
Engineer: Keisha "TK" Dutes
Mixing: C.

Are We All In Tech?

Alexa Scordato: This is Shift Shift Forward, a new podcast from Glitch that showcases everything that makes Glitch the best place to create on the web. I'm Alexa, the VP of marketing here at Glitch and in this episode we're tackling a big question. Are we all in tech? Author and entrepreneur Mushambi Mutuma sits down with our producer TK as they discuss being tech adjacent in our modern world.

Alexa Scordato: Later, we're debuting a new segment from Jenn Schiffer, our director of health and support, called Jengineering. Trust me, you won't want to miss that. But first, Maurice from our marketing team sat down with our VP of engineering, James Turnbull, and one of our software engineers, Sheridan Kates, to get their thoughts on tech jobs and industry.

Maurice Cherry: Why don't you start off by telling us how you both got into technology?

Sheridan Kates: My dad worked for IBM, so I had access to computers from a young age. I would say my first school assignment that I did on a computer was actually in the late eighties and I played with Logo. I liked the little turtle that would move around. But then I think a lot changed when the internet became a thing. And I remember the first MP3 I downloaded, which took about four hours on Napster. I'm probably not supposed to say. That really excited me. The kind of potential for communication and it wasn't just a computer, they were people at the end of the other end of those computers. And so I did go, I did engineering but it was general engineering. It was everything from electrical, mechanical, bunch of different things.

Sheridan Kates: And I did a computer science module there but it was done in a very isolated way. And again, wasn't getting that people element. And so initially I went out into consulting and technology consulting, but then I discovered about product management and so really I came into technology through what people might think was a technology adjacent role. And as a product manager, I felt like I could use the technical skills that I had gained but not in a way where I had to be what I thought was a very solitary, sitting by yourself behind a computer profession.

Sheridan Kates: And then as I started out at Google and was there for about seven years and then when I actually was working in a profession, I could see that that's not what a technology job was. The whole point was that you were building things that people could use. And I started to get really sad that I was no longer building that myself. And so that kind of like took me back, steered me back into the world where I wanted to be building things again. And so over the course of a few startups, I got back into software engineering and I'm here at Glitch as a software engineer now.

James Turnbull: I was sort of, I sort of similarly, none of my family is into technology, pretty much my entire family is, my dad was a journalist and my sister's an artist. And there's not a lot of technology in the family. But I liked computers and I think I probably like computer games more than anything else. And then that sort of led to, I worked out you could actually sort of break into computer games and change things and potentially violate copyright licenses. I think the statute of limitations is probably gone on that and the companies are out of business so I'm probably safe. But I was actually working in a library stacking books and the library was run by a mid range computer system, [unintelligible 3:40] 100 and it broke one afternoon and the chap who was supposed to look after it wasn't there. And somebody said, "James knows something about computers, maybe he can fix it."

James Turnbull: And I walked in there and there's a message on the screen and there's a big pile of books. And I looked in the books and found the message and then followed the instructions. And everyone's like, "You're a genius." And then I discovered that people actually get paid money to do this thing. And at the time I think I was making less than minimum wage. Yeah, discovered that that my salary quadrupled or something like that overnight. And yeah, ever since then I've sort of been doing that. And that's probably 20 years ago now.

Maurice Cherry: What would you consider to be a tech job in the classic definition?

James Turnbull: I think it's probably the people who build the technology pieces is probably the classical definition of a technology job.

Sheridan Kates: Yeah. And then classical and then like on top of that, I think anyone who's being creative with a technology that I keep going back to spreadsheets and Anil is going to think I'm just pandering to him or something. But I do think that spreadsheets, what you can do with VBA, those are databases just in visual form. And if you're in finance departments or creating automation, yes of course there are financial tools and if you just use those financial tools then maybe that's not necessarily a tech job, but if you're generating the processes by which things can be improved and maybe that's a tech job too.

James Turnbull: I was just going to say these days I'd probably also put product people and designers and business analysts and data people and who may not necessarily, they are building technology in their own right. It's just not classically considered maybe not, it's writing lines of code or building a software system of some kind.

Sheridan Kates: Do people think that the tech is a cost center or a value generator? And are the engineers treated accordingly?

James Turnbull: That's a good way of thinking about it. I think the days when sort of technology was considered IT and was a sunk cost and whereas now it's actually a business differentiator. If your mortgage app on the phone is more appealing to a particular demographic than your competitors that might make a tangible difference in the number of leads or people who buy mortgages from you.

Maurice Cherry: What does the future of the tech industry look like?

Sheridan Kates: I have to say I was not really into tech for the sake of it. I didn't really relate to the types of people that you'd see spending a ton of time on computers. I saw the stats about during the eighties about how all of these films were like, like Weird Science. These films were very much young white men portrayed as the people who were on computers and then that starts to become the stereotype of who does these tech jobs. That's not the case anymore I hope. I feel like we're all moving towards a world where we understand that those messages were sent to us and it's not just about technology for technology's sake, it's for building stuff. It's there's a range of people excited in this thing and so hopefully people don't see those stereotypes anymore and feel more included.

James Turnbull: Yeah. The biggest blocker I think these days is, is the fact that we need more people who look in industry who don't look like me because we tend to hire like. It's why I keep coming to startups in San Francisco. They're full of Stanford graduates, because that's the only people they know. Engineers look like Mark Zuckerberg in a hoodie and he dropped out of Harvard. And I think that the really interesting problems were solved by people that have a different perspective. People who understand that I could be building a payments app, how do you deal with the fact that a lot of people don't have bank accounts? How do you deal with undocumented people? There's a huge audience of folks out there who are just not serviced by technology because the people who build the technology have no empathy for those people's problems or their experiences.

James Turnbull: I think the more we can do to diversify the industry makes it more powerful for those groups and also, to be honest, it makes the market broader. I have this one thing that really frustrates me is all of these tools that are designed for folks that live within a couple of square miles of Sonoma or Manhattan, when you could actually build tools with a broader audience in mind and make a lot more money because you sell to a lot more people. And to me that, I'm not a huge fan of late stage capitalism but if it's going to exist, we should take advantage of its failings. Yeah, and that's always bewildered me that people will ignore the experiences of others or the needs of others in favor of building something that knocks the rough edges off their affluence versus being a product that has a broader appeal.

Maurice Cherry: Do either of you have any advice for people who want to sort of get into technology and learn how to really make it their own?

Sheridan Kates: The way that has worked for me is having something that I'm personally excited about existing out in the world. One of the things that got me back into this world was I really wanted a way to see bands on a map. If I had nothing to do that evening, I wanted to be able to say, "Okay, who's playing around me?" And there was nothing, I still don't think there's anything that great. That lets you see for a given date, what's on around you, on a map. And then also that you play their music immediately. It's not going to be, oh, this random band, what do they sound like? And so I basically just hooked together a bunch of APIs and created this map for myself.

Sheridan Kates: I basically just hooked together a bunch of APIs and created this map for myself, Gigspace. It does exist out there, but it's not very useful right now because everyone's socially distancing. Being able to do that, I think the times when you're just like, "Oh God, I can't be bothered to learn this part," you're motivated to, because you really want that thing to exist in the world. And then if you want to actually work in a tech job, I find that there's a lot of roles in customer service or that kind of thing. Maybe you can get into a company in a nontechnical role and then get excited about the product, and then move into those types of roles through maybe a different path. That's something else that I think is a possible way to get in technology.

James Turnbull: Yeah. I teach a variety of introduction to tech topics and work adjacent skills, and I always say to people, if you want to... It's really hard to learn, unless you're naturally inclined, I think it's very hard to pick up a book that says Programming 101, or learn a programming language. But if you have a specific problem or thing that annoys you about the world, then solving that problem using some form of technology... I was working with a bunch of people at a coding academy a couple of years ago, and a bunch of them came up with tools to make working with 311, which is New York City's... You dial 311 if your gas is not working or you have a city-related problem. Make it easier to deal with 311. Or identify... People who live in food deserts, identify places that sold various grocery items.

James Turnbull: Otherwise it might be hard to find in a neighborhood. Or things that allow people to cut the rough edges off all the difficulties involved in doing things. And I think those are very motivating because you get an immediate sense of like, I achieved something. And also you are solving a real problem, you're not solving a hypothetical one. I always look at the classic example of, if you could look at a program example, it's always like, here's a to do app. Or here's a blog app. And for the vast majority of people it's like, what am I going to do with this? But if it's an example of like, here is a thing that I can actually practically do something with, it means I'm likely to stick with it and it means I'm likely to iterate on it too.

Sheridan Kates: Yeah. And I think it doesn't even have to be groundbreaking or whatever they call it. Like if you don't have any ideas of stuff that you want, but you can... Something that's yours, like a website that you created from scratch or you found something you liked on the internet and changed it a bit. That is totally fine too. But something that's yours and that you're excited about.

Alexa Scordato: What do you do if you don't come from a classic tech background? Next TK talks with Mushambi Mutuma, the CEO of the South African based app development and design firm Altivex. Mushambi is also the author of Tech Adjacent, the exponential guide to leveraging technology for business success.

Keisha TK Dutes: We've been having this discussion about what is tech, who's considered to be in tech, who can get a tech job, how do you get that? But there's also this other thing that has been swirling in the air for the past few years and it's about being tech adjacent. So I thought that you might be the perfect person to call for this conversation.

Mushambi Mutuma: Much appreciated. I have a little bit of context.

Keisha TK Dutes: So let's go into it. So when someone says to you that they're tech adjacent, what should I be thinking of?

Mushambi Mutuma: Really what tech adjacent means for me is that technology moves really, really fast. At light speed sometimes, and even faster. As we have this conversation, there's going to be new tech that's being developed, being launched, being announced, so it's really difficult for anybody to keep up with technology. Even those who are in it, even developers, even tech entrepreneurs, even those in the valley or anywhere else in the world, it's not possible to keep up with it. It just moves too quick. So we can't expect ourselves to be able to keep up with or expect our staff or anyone we work with to do so. But at the same time, because it's moving so fast, no one has a choice to ignore technology. It's not an option anymore. You can't stay away from it. It's part of our daily life. It's how we're even having this conversation, several thousands and thousands of miles away from one another.

Mushambi Mutuma: So you can't keep up, you can't avoid it. What do you then do? And for me, what tech adjacent means is exactly that, is that you need to be next to, you need to be close, have close proximity to technology. You need to be very close by. And I think fortunately that's something that anyone can do, whether you're a developer, whether you're a techie, whether you're somebody's mother, somebody's grandmother, whether you're quite young, everyone can be very close by technology. And it's really just about having an understanding of the way it works and how it relates to where I am in life, or the industry that I work in. And that's really it. The simpler, the better.

Keisha TK Dutes: In terms of removing the complications around tech, what does that mean for someone that is trying to get into this world? Because there's a lot of opportunity here and people might see themselves as a graphic designer or anything honestly. How does someone remove those barriers to success, in terms of getting into the field?

Mushambi Mutuma: I think I'll use myself as an example to answer. So, I originally was a political science student at Pepperdine University in LA, and I studied politics and I studied art. I had some context to tech growing up because my dad worked at Apple computer. I got to play with a lot of computers. But he never taught me how to code. I didn't learn that side of the business. I was just involved in playing with these things and these gadgets and these Newtons and the old IMAX and everything. And now for the last 10 years, I run a digital innovation lab that builds tech. So we specialize in banking apps and financial services solutions, insurance, all kinds of mobile apps from consumer facing games to deep level payment solutions. But my coding skills are next to not.

Mushambi Mutuma: You do not want me to be the guy that touches any of the code, right? Because it's going to be messy. It might get it done, but it's going to take a long time and it's going to be glitchy. That's the reality. But I've been able to build a very successful business being right next to tech, selling tech, building tech and being the most senior person at the company next to my business partner, but not being able to do any of it actually. And I think it's a perfect example of what being tech adjacent is, is that the careers going forward are really about understanding the ecosystem around technology. So I've got a pretty fair understanding of the different tools that have gone into building an Uber or an Airbnb or any other banking solution. And very rarely do developers this day and age build from scratch.

Mushambi Mutuma: Now what we're doing more than ever is we're integrating. We found this open source solution, we've found this and we've integrated this API and we pull all it together. And so there's some decent knowledge that needs to happen there, but you don't actually have to have all of that raw skill to be able to develop a career. I think, if you think about some of the biggest social media applications that exist, I don't really think that they're focused on hiring the best technical talent. Instagram is really trying to figure out people that understand human behavior more than anything else. It's really not about the next feature. It's really about like, cool, how can we get people to stay on this infinite scroll a little bit longer?

Mushambi Mutuma: So I think careers in tech are no longer technical as much as they used to be.

Keisha TK Dutes: I could see though where some people would be a little bit hesitant to own their place in tech because they don't consider themselves tech. Have you come across that as a block for some people?

Mushambi Mutuma: Definitely. Because the first thing, especially when we talk about... And I speak to a lot of startups and startup entrepreneurs and guys looking to get funding or to get the first series A round or whatever it might be, and for a long time that conversation was, you need a technical co-founder. Or you need to be the technical co-founder or founder. And so it was like you couldn't build a company in tech unless you were a techie, unless you had deep seated roots as an engineer. And I think that conversation still exists on some levels, but I think that perception is now changing.

Mushambi Mutuma:
Technology's job is only here to make your life easier, better, simpler. That's it, that's all tech actually is. It's got nothing to do with the zeroes and ones, actually. The pencils that we use, the pens we use, the whiteboard markers, those are technologies. I think we just have to simplify what our expectation is. It doesn't have to be hyper tech and high tech and these very complex algorithms or applications or solutions, it's really just like, "Does it make my life simpler? Does it make my life better? Does it make it a little bit more efficient?"

Keisha TK Dutes: Awesome. Now I feel more confident myself, owning my space. So, I want to move to a different place. When you go to the store, the cashiers, some of them, are just computers now and there's just one person manning the area. Or Uber drivers or ride share drivers are interfacing with sometimes two, three platforms at a time. Delivery people with different food platforms. So, does it extend down to the everyman that's doing all the, right now, essential work that we need in the world? Are they tech adjacent? Are they in tech?

Mushambi Mutuma: I think so and I think it is from a behavioral standpoint, so they might not be as deep rooted as a cashier or an Uber/Lyft/every other ride share driver, like you mentioned. But if you think about how we integrate with our lives, as soon as we wake up, we're mobile first and that's worldwide. Everything that we consume is right there in the palm of our hand, faster than ever before and in a bigger, wider reach than ever before. So I think we're all interacting with tech every single day, we think of, in the global lockdown that's happening right now we're all communicating on Zoom or FaceTime or What's App calls or Skype for some people or Microsoft Teams or Slack or [inaudible] or whatever it is.

Mushambi Mutuma: So we're all using all of these various tools that are technologies, actually more and more every single day. I always ask people, one of Ray Kurzweil's predictions for the future is, by 2030 or 2040 every human being will have a deep relationship with VR or AR. And I'm like, "Well, cool. How many of you guys have a deep relationship with Siri right now?" A lot of us do, a lot of us might spend some time speaking to Siri or...

Keisha TK Dutes: There's whole movies about people speaking to a Siri like thing and having romantic relationships.

Mushambi Mutuma: Exactly, more now than ever and we do it in a different way. We say, "Hey, what song is playing?" Or, "Hey, how can I get to this next location the fastest route?" So all of those things, I don't think we necessarily acknowledge because it's so much part of our life. Everyone logs into Facebook every morning and checks their memories or their stories for the last couple of years, or into Instagram or into TikTok. I think because all of these things are just normalized, that we forget that we're actually speaking and interacting with robots and AIs all day long.

Keisha TK Dutes: What do you suggest in terms of shifting the thought? If someone now wants to go from where they are at to tech adjacent to tech, what are the mindset shifts that a person should have?

Mushambi Mutuma: I think the very first thing is simplifying it, is that we tend to over complicate what tech is and what it involves. I think it speaks to what I said a little bit earlier, that tech's soul job from the history of man, since the first man existed, the first woman existed on Earth was to make life a little bit simpler, better, and easier. So I think that's the first adjustment that we need to make, is that really understanding that we're already in it.

Mushambi Mutuma: The second thing I would say is that, it's just really about understanding how technology works and moves and there's two things. There's the first thing that I talk about where the technology's really based in convenience and understanding human needs. So if you understand humans, you will always understand technology.

Mushambi Mutuma: The second layer is that technology, it doubles in capacity every, people argue this from 12 to 24 months, but you can pretty much predict the next iPhone or the next Samsung that you get is going to be double in capacity, speed, processing power, et cetera. And that's going to continue, so it's actually very easy to map out where technology is going. A lot of times we think that it's going to be something that dropped here from Mars, that we've never seen before, never heard of. But technology is hereditary, it has a beginning. Whatever it is now, it's going to adapt from.

Mushambi Mutuma: So it's not going to adapt from something you've never seen or heard of, so I think once we get comfortable with that, is simplifying it, understanding that we've always been involved with, that we all know it. And then understanding that it's hereditary and where it's going to jump from now is going to be where it's going to go next. I think it's a lot easier to then predict the future or predict where technology is going or get on board with it because, it doesn't have to be this thing that's so far removed. It's those guys, those guys downstairs in the basement, with headphones on.

Mushambi Mutuma: It's all of us. We can all be next to tech and really become tech adjacent. I think it's in all of us, no matter the job title, no matter the level of experience or education, we can all do this. We're all doing it already and we're going to be part of the future, so it's not going to take us over.

Alexa Scordato: Tech can be one man, every man, or a middle man. Let's break it down in the debut of a new segment on the show called Jenngineering. By one of our most outspoken members of the Glitch Team.

Jenn Schiffer: I'm Jenn Schiffer and I'm the director of community engineering at Glitch. Who we consider to be in tech or what makes a tech company a tech company is a topic I find extremely tedious, very annoying, waste of my time, but also fascinating. The distinctions often used by folks traditionally in tech to further keep the underrepresented in tech underrepresented, with is very gate keep-y. But at the same time it also gives us a nice illustration of what tech means to the people who are building it and who are consuming it. And also, it was used by my students to test me on a weekly basis.

Jenn Schiffer: I used to teach a college course on tech literacy to non computer science majors at a large liberal arts college. A big component of the curriculum was teaching them how to create and use spreadsheets, which are the backbone of any functioning modern society. But I often got pushback from them saying they don't care about spreadsheets or, "I don't understand why I need to know about them." And often they'd just tell me, "It's not like I'm going into tech."

Jenn Schiffer: When I was studying computer science in the early 2000s, in tech was an idea taught to me as building tech in the industry, which at the time was Google and Facebook. Those were the cool places to go work. And also, the industry back then meant Silicon Valley, which is a location. And so, I'm sure there are folks in the industry back then and today who do not consider me to be in tech because I've never left the near metropolitan area.

Jenn Schiffer: And today, the idea has weirdly pervaded into our culture. I say weirdly because a vision of who uses and creates tech has changed quite significantly. Many of the big companies stopped requiring traditional computer science backgrounds. They've also stopped requiring holding college degrees, period. And those businesses that were considered in tech are building in house software and hardware teams to get stuff done.

Jenn Schiffer: I would never think that Casper, the mattress company, is a tech company but they do have a whole in house set of teams, set to set them up for success, which requires technology to allow consumers to complete transactions. And yeah, consumers are using tech to complete most of their transactions today. I want to note, there have been two extremely significant developments in tech between my going to college and me talking about this today, which are the-

Jenn Schiffer: ... Developments in tech between my going to college and me talking about this today, which are the launches of the iPhone and then the iOS app store. Apps. I can talk about apps for days, like how they absolutely vex me but also provide me much entertainment, convenience. And I think a lot of people agree. There's this half joke, half heteronormative misogynist reality where many of the apps that provide any sort of convenience for us were invented by men who were trying to replace their moms. We use them to get rides to the mall, connect with someone to clean our room, get us dinner, so on and so forth. And what we're starting to see is that, especially in quarantine, these apps are becoming many folks' lifelines where there are people or business for better and for worse.

Jenn Schiffer: So here's a cool, fun, free exercise for you. This week, or even just a day this week, I want you to log what apps you use and if it's acting as a middleman between you and a small business. And I want you to see if you can eliminate it or find a way to support those small businesses in other ways. And I also want you to keep an eye out for the impact of letting apps you use, abuse your autonomy, privacy and money. We're seeing the gig economy apps like Uber and Lyft collapsing under the quarantine and they're asking for help from the same government they fought relentlessly to not help their gig workers. And we love to see it. But we also love to see tech be used to remove the middleman.

Jenn Schiffer: I live in Jersey City and they did something really cool, I think. They made a great site cataloging all the local restaurants who were open for takeout during quarantine with links to their menus and how to contact them directly. And this signals to the businesses that the city cares about them. And it signals to consumers a way to directly contribute to that small business and give them their money without letting the 20 sites that Yelp seems to own take a cut of it. So I guess not all tech is bad.

Jenn Schiffer: In fact, here's another cool, free, fun exercise for you. I'm going to look for the folks using tech right now to help people during the chaos and I want you to celebrate them. I look to, and celebrate, tech companies like The Human Utility, which allows folks to pay the water bills of those in Detroit so that they don't lose access to the basic human need of access to water. And I've seen people build apps on Glitch too to share data about the pandemic and even help parents coordinate childcare with each other during quarantine.

Jenn Schiffer: I think the tedious debate of who is in tech or out of tech or whatever that means will continue until the end of time as these debates tend to do. But the reality of who uses tech and creates it will hopefully continue to evolve. I think so. I think that despite our industry's typical gate keeping, I envision code being seen as a normal everyday tool, like a pen and paper, hammer and nail, or whatever. And I think that because of that, folks who find the apps that they use are maybe not serving their needs or harming them or others, I think that they'll use code to make their own tech in response. Whether they are considered in tech or not. And most importantly, everyone in the world will recognize that I am right and spreadsheets are the backbone of a functioning modern society.

Alexa Scordato: That's it for this episode of Shift Shift Forward. Visit us online at glitch.com/ssf, follow us on Twitter, our Twitter handle is @glitch, and let us know what you think about this episode by using the hashtag #ShiftShiftForward. If you really like the show, then subscribe to us on Apple Podcasts, Spotify, Google Podcasts or wherever you find your favorite shows. Leave us a rating and a review.

Alexa Scordato: Shift Shift Forward is produced by Maurice Cherry and Keisha TK Dutes, with editing by Brittani Brown, sound designed by Keisha TK Dutes and mixing by C. Special thanks to the entire team at Glitch and to Kristen from our design team and her friend Mike Sunday for helping with a Jenngineering intro.

Alexa Scordato: Here's what's coming up on Shift Shift Forward. We've all experienced it before. You go to your favorite website, but it's not loading. Or you try clicking on a link to complete a transaction, but your browser times out and you get an error message. It can be frustrating to deal with outages and similar issues from a user perspective, but let's see what it looks like from the other side. What happens when these incidents occur, and what does it take to get everything running smoothly again?

Games can be a great way to learn new concepts, spend time with friends, or just unwind at the end of a long day. And wouldn't you know it — games are actually a big part of Glitch's DNA!

For our first episode, we talk with Glitch user and software engineer Joe Gaffey about his history with game design, hear from our engineering lead Keith Kurson about fun games to play and remix on Glitch, and we have a roundtable conversation with CEO Anil Dash and other Glitch team members where they talk about our history with games, and mention some of their favorite games of all time.

Looking for bonus content? Our app collection for this episode is where it's at, including links to all the Glitch apps and other games mentioned in this episode and much more!

Executive Producers: Maurice Cherry, Keisha "TK" Dutes
Editor: Brittani Brown
Engineer: Keisha "TK" Dutes

Game On!

Hosanna Tolman:
My top number one video game of all time is Link’s Awakening, Legend of Zelda.

Steph Monette:
So, my favorite game right now is backgammon.

Zainab Shah:
So, my favorite game is called Ludo.

Osmose Kelly:
So, my favorite video game is Castlevania: Symphony of the Night.

Tasha Hewett:
My favorite game is this pinball machine that’s called Total Nuclear Annihilation.

Jenn Schiffer:
One of my favorite games is Katamari Damacy. The designer, Keita Takahashi, ended up working on the game Glitch that then became the company Slack…and now we’re Glitch.

Maurice Cherry:
This is Shift Shift Forward, a new podcast from Glitch that showcases everything that makes Glitch the best place to create on the web. I’m Maurice. I work on the marketing team here and as you can probably guess, this episode is all about games: the ones we love to play, the ones that inspire our creativity and the ones that make us want to create our own.

Maurice Cherry:
You’ll meet Joe Gaffey, a software engineer by day and a die-hard game creator in his spare time. Later, our engineering lead, Keith, clues us in on game apps on Glitch that anyone can play or remix. But first, let’s eavesdrop on a conversation about how hard we game at Glitch with some of our fellow team members. Software developers, Tim and Greg, our head of legal, Nkasi, and our CEO, Anil. Our producer, TK, kicks things off. Here we go.

Keisha “TK” Dutes:
I had no idea people at Glitch were such extensive gamers. What are all your favorite games?

Nkasi Okafor Wilken:
Oh my goodness. Okay. This is sort of for nostalgic purposes, but also because it’s one of the franchises that manages to keep bringing it. So, the first time I got really into video games was my sister’s eighth birthday when my dad, which he doesn’t like video games at all, surprised her with a Nintendo. Right? And the game that he bought was the Legend of Zelda in the, if anyone remembers, the gold cartridge. It was such an amazing game. And still Nintendo is putting out the hits with Legend of Zelda, like the last one on the Switch, oh my God, what was the name of it?

Nkasi Okafor Wilken:
Breath of the Wild. So, funny story, I refused to start working at Glitch until I finished that game because it is such a gigantic time suck.

Anil Dash:
The truth comes out now. You were like, “I have some meetings. I can’t start right now.”

Nkasi Okafor Wilken:
Oh, yeah. Legend of Zelda. All of them. I even played it on the Game Cube, and it was Wind Waker on the Game Cube. So yeah, I would say that’s one of my favorite games, if not my favorite game game. Yeah.

Keisha “TK” Dutes:
What about you, Tim?

Tim Kington:
Go’s a strategy game from Japan. It’s, I guess, originally China… It was brought to the US from Japan, but it’s like 4,000 years old. And it’s a one on one game like chess where you have a board, black and white pieces, and it’s a zero sum game. So, somebody win’s, somebody loses. The rules are extremely simple, and it’s really deep

Anil Dash:
And Go’s also how they test a lot of coding and testing AIs and things too. Right? Is it sort of like a common coding challenge as well?

Nkasi Okafor Wilken:
Is that the game in Avatar: The Last Airbender?

Tim Kington:
Oh, Avatar. No, it’s not. It has the same flavor kind of, yes. Something Lotus, I think it is.

Nkasi Okafor Wilken:
Yeah. Yeah. For like the Lotus tile?

Tim Kington:
Yeah. It feels like that kind of, but it’s not the same game.

Tim Kington:
Yeah. Go’s used as an AI challenge, and for a long time it was considered sort of the pinnacle of AI challenge. Chess had been knocked out 20 or 30 years ago, and Go was the next big thing. I thought computers won’t be able to beat professional humans at Go. So, I gave a Friday talk here about Go and AI and why it’s so hard, and talked about how it’s at least 10 years away. And then, I gave another talk about six months later saying a computer just beat a pro.

Nkasi Okafor Wilken:
Greg and Anil, what are your favorite games?

Greg Weil:
Halo 3 because that had this whole custom game system, which was pretty new at the time, where you could to lay out stuff, make your own map, and then they had a separate game type system. And it was just a ton of fun playing like weird modes that people came up with where all of a sudden it’s like, “Oh, everyone moves 10 times faster because there’s no friction and you’re at the bottom of a big slope and there are cars sliding at you really fast that the other team is in, and you just need dodge the cars because if they hit you, they win. But if they miss, they’ll fall off the bottom of the ramp and into the ocean.” Just really weird ideas like that, which were just a ton of fun to play.

Anil Dash:
A couple of quick ones. Back in the ’90s, Lucasfilm, George Lucas’s… like the Star Wars company, they had a company called Lucas Arts, and they would make games. And they were mostly like a star Wars version of whatever games were popular. And then, they did one game called Grim Fandango, that was themed around the Day of the Dead in Mexico, and it was like a a murder mystery, Noir thriller, adventure game with really heavy candy skull art. it was just beautiful and great world building, and I’d never seen anything like it. And I think that really changed my perception of what games were. I think in a similar vein, the first Portal game just blew me away.

Anil Dash:
It was like you felt your brain bending, it was thing teaching you as you went. That was actually in the first notes of when we were prototyping Glitch and stuff. The first thing I thought of was Portal was this idea of what’s possible changes as you use it? And you want something where playing it teaches you what you could do. I just thought that was masterful game design.

Anil Dash:
My all time favorite game is Mario Kart. Obviously, there’s the trash talking part and whatever, but the serious part was like, it was something where there was both immediately accessible to a normal person who could walk in and just button mash and do all right. And then, there was this infinitely deep finesse where the more you want to practice and you wanted to have your preferences… I like this character and this car and whatever. And I just really, really loved the blue shell, which was this idea… Everybody hates the blue shell, and what I loved about it was there was an intention to the game design, which was like, if you’re running away with it, and it’s not fair and it’s not going to be fun for everybody, the game is going to empower somebody to be like, “All right, let’s keep it close. Let’s keep this thing fun.”

Anil Dash:
So, there’s a whatever, a kid that’s playing for the first time, and somebody like me who’s been playing for 30 years or whatever can play together and still have fun. And I see that now all the time where I got a kid, I got an eight year old son, and he’s good. I don’t have to spot him. He’s doing all right. But even before he was, you could play something together and share this thing, and it’s the nostalgia aspect of like, “I grew up playing this and you grow up playing this.” But also, that intentionality, I think about all the time. Somebody made this technology, and you think about all the math behind it to get all these shapes up on the screen, and they’re like, “We’re going to design a system where people can sit together and have a good time together.” And that shows up in the work. And I just thought, “man, those people were doing good with their lives.”

Anil Dash:
Literally from day one, it was always gamers, but it was a lot of… Well, board games was a big part, so it wasn’t always video games. So, the DNA, the gaming DNA, of the company is deep.

Keisha “TK” Dutes:
I heard there was a bridge game. Is that true?

Tim Kington:
It’s true. Yeah. That one got started because our former president, Rich Armstrong, was really into bridge, and I think I mentioned that I played bridge when I was here my first week or something, so he wanted to get a bridge game together. And we drafted a couple of people and made them learn bridge.

Anil Dash:
I came in, and I’d heard about the bridge game. It predated me. And I joined the company, I was like, “This sounds like…” I pictured like a dark room and everybody’s got those accountants visors on, and you’re just about to get… Like they’re going to run the table [inaudible 00:00:08:27]. They’re just going to [inaudible 00:08:28] you because they’re like the bridge experts. There’s Tim’s, who’s like the mathematical mind, and they’re like, “Oh, I’ll just teach you how to play.” Really, you’re just outmatched.

Keisha “TK” Dutes:
I heard from a little birdie that also we had a connection to the folks at Slack or a connection to playing games on Slack?

Anil Dash:
Back in the day, they had started a game called Game Neverending. God, this was probably 20 years ago at this point. And it was beautiful and this great art, and a core base and die hard users, but then everybody else is sort of like, “This is like intense. I don’t know…” You wouldn’t casually go up to Game Neverending and start playing it., And then they built another game, and it was called Glitch, and it was the same as Game Neverending. It was like wild art, massively multiplayer game. Everybody’s in the RPG core base of die hard users, not a whole lot of people playing, and they had built a tool that helped them make the game, and that that tool was called Slack.

Anil Dash:
And then, again, they did the same thing. They were just like, “Nobody wants our games but we’re going to build the tool.” And then, they became Slack, and they’re now multibillion dollar company and I think all of those people have private islands or whatever. But I guarantee you when they’re going to retire, they’re going to be like, “Let’s build a game, and it’s only going to have a couple of users, but it’s going to be really cool looking.”

Keisha “TK” Dutes:
And then, you said we got our name was similar way?

Anil Dash:
Well, what happened was the team had built a prototype of what became Glitch, and we were like, “We need a new name for it,” or whatever. And I think just independently… Tim, you tell me if I’m wrong. My recollection is we came up with the name Glitch independently at one of the early meetings where we’re like, “That’d be a cool name.” But it was like, “Oh, but it’s taken.” And I was like, “Well, I bet they would do a solid.” So, because I knew them, I was just like, “Look, it’s not going to be bad. It’s going to be cool like your game.” And then, they were like, “All right, let’s do a deal and let’s get you the name.” And so, Glitch, for like the first, I don’t know, six months that Glitch was running, our site, our tools of Glitch were running, but one of the top search terms that would come in was Glitch the game because people were still looking for that old game in the [inaudible 00:10:26].

Keisha “TK” Dutes:
Well, we have a programmer that makes games here. Greg, you were telling me over beverages, I don’t want to say what kind of beverages, because you’re here, you’re a web developer at Glitch, but on your off time you make games. Can you talk about that a little bit?

Greg Weil:
I like making games a lot because whenever you’re making one, you end up touching a whole range of different skills, I guess, where you start out where it’s just, “I’m going to code up some movement. I’m going to focus a lot on just making it feel responsive and fun to play.” But then all of a sudden it’s, “I need some sprites for the character moving around, and they need to look good. And also, they need to still fit the gameplay and all the other different kinds of things.” And all of a sudden you need sound effects, and then all of a sudden you need the ability for multiple people to play together. And all of a sudden, it feels like you’re doing like 20 different skill sets but just a little taste of all of, which is a lot of fun to work on.

Keisha “TK” Dutes:
For you guys that make the things that we love so much, what are my game options on Glitch?

Greg Weil:
There are a couple games on Glitch that are kind of surprising and a little tricky to find sometimes. I saw one on the forum just like a week ago that was pretty neat. It looked like an online World War II bomber kind of thing where different players are… It’s massively multiplayer, everyone goes into a single world, and you’re flying around planes in the top down perspective.

Tim Kington:
There’s a Snake game out there that’s a clone of the old Apple II, Snake Byte, the grandfather of the Snake game on your phone.

Tim Kington:
Yeah. And that one has clones of all the levels. So, there were like 20 levels in that game, I think, and it has different walls in different places and gets harder as you go along. I kind of enjoy that one.

Keisha “TK” Dutes:
That’s classic. I’m going to jump over to Nkasi because I don’t think people think that the office’s legal counsel is going to be out here organizing a whole Mario Kart tournament. Can you talk about that?

Nkasi Okafor Wilken:
During my interview, and Neil, I don’t even know how it came up, but he’s talking about Mario Kart and how he was awesome at it. And I was like, “Oh man, I need to confirm all the boasting that was going on in this interview.” So, our onsite came along, and I was like, “Do you know what? This is the perfect opportunity. This is the perfect opportunity.” And so, actually, I got together with Greg and another engineer here, Victoria, and we created the Mario Kart tournament. It was actually really easy. Maybe we accidentally indexed for it, but a lot of people really know how to play Mario Kart, like serious, serious know-how. Like brought their own controllers, like, “No, I want to use this specific controller because then I can actually fly it into things.” So, yeah.

Keisha “TK” Dutes:
Playing games at Glitch sounds like inviting singers to karaoke.

Nkasi Okafor Wilken:
I mean, I guess. That is definitely true. I did come to [inaudible 00:00:14:00], but I didn’t think about it. I was just like, “Oh, maybe there’ll be three or four people that would be interested in that.”

Greg Weil:
Yeah, yeah. All the quiet folks all of a sudden got real loud. It was good.

Keisha “TK” Dutes:
So, with that being said, all of y’all, and especially Nkasi, any plans firstly for another Mario Kart tournament or any plans for any other game nights at Glitch?

Nkasi Okafor Wilken:
What I would really love to do, and whether it’s a game… I think you could consider it a game, but I love Rock Band. Love it. Love it. Love it.

Anil Dash:
Now that I hear that, that’s the most surprising thing I’ve ever learned about you. I’m like, “Yes, that fits.”

Nkasi Okafor Wilken:
I don’t know what you mean by that. We’ll have a discussion later.

Nkasi Okafor Wilken:
Offline. But that’s something I’ve been floating around in my head because, then, you can have people at the same time singing some amazing tunes. And I’ve learned a lot of different rock songs through Rock Band, that I had never heard of, and now are there on my iPhone playing it all day. So, that’s what I’m kicking around in my head for the next onsite.

Maurice Cherry:
Next up, meet Joe Gaffey, a telecoms enterprise software architect by day and a gamer and creative coder by night.

Joe Gaffey:
Yeah, I was into games and coding from quite a young age, actually. I remember seeing arcade machines probably in the early ’80s in local fast food restaurants and such places and thinking these were the best things ever. So colorful and loud and so different from anything else that was around at the time. And these were actually the first computers that I interacted with. Later on, we had an Atari 2600 in the house, and that got lots of use too. Then when I was a little older, I found some coding books in the public library, and I used to read the program listings before I even had access to a real computer. I got my first computer, then, from my parents for Christmas around the age of 10 or so. It was an eight bit era. Commodore 64, one of those ones with the cassette player for loading software. Took about 10 minutes to load 64 [inaudible 00:16:13].

Joe Gaffey:
So, I started coding basic games like Pong and brick breaking games and so forth, and Commodore PET BASIC. Then whenever I was playing commercial games on a C 64, I was always really trying to understand the game mechanics and looking for glitches and insights into how the game worked under the hood. Then once I discovered all of the mechanics of the game, I usually got bored and moved onto the next one or maybe try to recreate something from it in my own code.

Maurice Cherry:
With his love of old school games, some programming knowledge under his belt, and a desire to code, Joe got competitive.

Joe Gaffey:
Yeah. Get Up game off competition turned up with retro gaming team. I taught it. It was really a good opportunity to bring together some of my different interests. I was originally thinking of of Asteroids, a clone of Asteroids, perhaps with a mix of like Lunar Lander mechanics thrown in. But then I thought about those old, colorful, and loud Galaxian and Galaga machines. I thought it would be a great example of a retro gaming game to show my kids and hopefully get them into interested in coding as well. To make it a little more interesting for modern audiences, I added some original mechanics like the [up 00:17:24] red pills, the destruct orb, smart bomb, and the wingman assist, and of course the end game boss battle. If you use one of the destruct orbs just at the right time, you can take out an entire wave in one shot, so there’s a little tip. A [inaudible 00:17:39] in the game off that year, which I was very pleased with.

Maurice Cherry:
Here, Joe weighs in on how he used Glitch to make Galaxeroids.

Joe Gaffey:
Well, I really key benefit for me was portability. I did not much free time when he’s working on Galaxeroids, so I was taking every opportunity to get some coding done using whatever device I had to hand. And having the web based editor made this really easy of course. I actually coded it the whole thing and produced all of the art assets such as they are in the browser as an experiment to see if it all could be done that way. Gethub integration was also a great help, so I could easily create snapshots and forks as needed. Example, for releasing into different app stores. I actually published Galaxeroids into the Microsoft store and the [inaudible 00:18:22] game store as a progressive web app. T. Then making Galaxeroids a PWA on Glitch, I was able to test on any mobile device I can put my hand on, and I was able to send beta links to friends, to family to test some their devices and then make changes on the fly so I could test the fixes immediately.

Maurice Cherry:
It’s no secret. Joe loves retro gaming, but he’s really got his eye on the future.

Joe Gaffey:
I like to play simple games with my kids like platformers, like Mario Bros. and Tetris, but I’ll give most genres a go. So, I’ve done some other experiments with games. In another experiment on Glitch, I built a Guitar Hero/Rock Band clone called Rockstar, which uses the web MIDI and web Gamepad APIs. [inaudible 00:19:07] playing along with MIDI tunes using the actual Rock Band instruments, guitars and drums. It actually worked pretty well in the end. Needs a little bit of UX tidying now, of course. More recently, I’ve gotten into driving simulators in VR, and that’s a whole different experience. And my latest splits experiment, which I called WD-40 for some reason, comes from that interest, and so my first real attempt at 3D programming on the web.

Joe Gaffey:
It’s a physics-based driving simulator based on three.js with ammo.js physics. And it’s pretty basic right now, just extends one of the ammo.js code examples. But my idea is to use it to experiment with upcoming web standards like the upcoming USB web hitch ID, human interface device API, so I want to see if it’s possible to build a full driving simulator where USB-based steering wheel and pedal controls, including full force feedback, just using web standards and technologies running in the browser. I eventually, I’d like to experiment with WebXR, as well, and mixed reality. Maybe driving an RC car remotely on the web over WebRTC, that sort of stuff. Like somebody [inaudible 00:20:24] experiments, I try to push the boundaries of what’s considered feasible to do with web standards technologies in the browser and keeping the source code as open and accessible as possible at the same time, and Glitch is really perfect for that.

Joe Gaffey:
I’m quite passionate about the open web and the idea that anybody can look under the hood of a website and see how it works just by clicking on the view source option. Unfortunately, that’s not so easy to do these days with modern web frameworks, but you want to be able to take advantage of JavaScript and create this rich experience that offers, especially for games and demos, but ideally to do it in a way that supports open and accessible coding. I think in this way, platforms like Glitch shows there’s a real opportunity to realize the vision of the social rewrite web as originally described by Tim Berners-Lee in the early days of the web.

Maurice Cherry:
Now, let’s talk to a member of the team to get the lowdown on some glitch apps that anyone can code, use, or remix for themselves.

Multiple Speakers:
Welcome to the App Review.

Keith Kurson:
Hi, I’m Keith Kurson. I’m an engineering lead at Glitch. I run what is now called the Dotcom team. So, we make everything on Glitch that isn’t part of the editor, so the front page, collections, user profiles, all of that.

Keith Kurson:
One of my favorite is called Figma Pacman, and I use this tool called Figma which is what our designers [inaudible 00:22:01] use and a lot of other designers to build layouts. It’s sort of the design tool at the moment. But we’ve seen some really interesting uses of Figma and other tools that aren’t traditionally used for this on Glitch. And so, Figma Pacman is an example of an app where you, in Figma, draw a little maze like what Pacman runs through, and then Glitch users can plug in that URL and then play Pacman on that maze. And so, it’s a really cool combination of two different tools. And people who aren’t that technical… So, it was made by a designer who uses Figma all day and doesn’t get a lot of time to do programming, and so anyone can jump in and make their own maze without really needing to know how to program. And I think that’s really cool and shows off what Glitch games can look like.

Keith Kurson:
And then, the second one that I really, really love, that I’ve played a bunch of times, is called PowerSlides, and you’ve probably seen some version of this in the city that you live in performed live. But basically, it takes a bunch of random slideshows from all over the internet, so government slideshows, weird marketing slides, things like that, and combines them into one huge database. And when you start the game, it chooses 10 of them randomly and you have five minutes to give a South by Southwest style presentation with these totally random slides that could just have a picture of a dolphin and say, “Think of the dolphins,” or a text line that just says, “iPhones for dogs,” and you have to give a presentation like you’re trying to sell this thing. And that is really fun to watch people do.

Maurice Cherry:
Want more gamer goodies? We made a collection filled with bonus content where you can play the Glitch apps mentioned in this episode, see a list of the team’s favorite games, and check out a game related articles and more on Glimmer. You’ll find a link to the collection in the show notes. That’s it for the first episode of Shift Shift Forward, a show about everything that makes Glitch the best place to create on the web. Our website is glitch.com/SSF, and our Twitter handle is @Glitch. Let us know what you think about this episode on social media by using the hashtag #shiftshiftforward. And if you really liked the show, then subscribe to us on Apple Podcasts, Spotify, Google Podcasts, or wherever you find your favorite shows.

Maurice Cherry:
Here’s what’s coming up on Shift Shift Forward. A lot of us are sheltered in place and adjusting to the reality of living and working from home for the foreseeable future, and technology is a big part of that. So, who is considered to be in tech these days? 

Maurice Cherry:
Shift Shift Forward is produced by Maurice Cherry and Keisha TK Dutes, with editing by Brittany Brown and sound design and mixing by Keisha TK Dutes. Special thanks to the entire team at Glitch.

Anil Dash is CEO of Glitch, and the host of Function, a podcast exploring how technology shapes culture and society. 

He also serves on the boards of Stack Overflow and the Data & Society Research Institute, and was formerly a contributing editor of WIRED. He’s been blogging since 1999 and won’t stop telling people about that one time he was on the Desus and Mero late-night show.

Maurice Cherry is Glitch's creative strategist.

Incidents

The Workers Who Keep the Web Safe in an Unsafe World

Animal Crossing Helped Me Get Back to Real Life

New Rules of the Classroom: Remote Teaching in the Age of Covid-19

Better Tech: Why We Need Calm Technology Now More Than Ever

Function with Anil Dash

Open World is a fiction anthology podcast about how science and technology serve as a backdrop for imagining a better future, helping us envision the positive influence that science and tech can play in a real future. 

Open World is produced by Keisha "TK" Dutes and hosted by critically-acclaimed podcaster Rose Eveleth, for Glitch. By speaking to the pressing concerns of our times through love, friendship, community, compassion, and sustainability, the host will help listeners imagine new ways forward into a future that is less scary and more positive, and something we can start working on constructively.  

Open World launches weekly episodes in Spring 2020. Want to contact the show? Email youropenworld@glitch.com.

Open World

Shift Shift Forward is a podcast that showcases everything that makes Glitch the best place to create on the web. We’ll talk to some of our top creators, explore cool apps, and even hear from the people who work at Glitch!

Shift Shift Forward

Celebrating the makers, creators and coders who shine.

Welcome to Glimmer ✨

— Glimmer

We’ve all experienced it before -- you go to your favorite website, but it’s not loading. Or you try clicking on a link to complete a transaction, but your browser times out and you get an error message. It can be frustrating to deal with outages and similar issues from a user perspective, but let’s see what it looks like from the other side. What happens when these incidents occur, and what does it take to get everything running smoothly again?

On this episode, we look at incidents from three different angles: infrastructure, support, and leadership.  Mads Hartmann, Cori Schlegel, Antoinette Smith, and Emmett Walsh take us through how their team fixes outages and other issues, while senior support engineer Tasha Hewett shows us how she gracefully handles support issues, and we end things off with a few words from our CEO, Anil Dash.

Executive Producers: Maurice Cherry, Keisha "TK" Dutes

Jacky Alciné:
This is Shift Shift Forward, a new podcast for Glitch that showcases everything that makes Glitch the best place to create on the web. I'm Jacky Alciné and I work in a platform team. In this episode, we're taking a look at what happens when websites go down, including ours. We talked to Tasha Hewett, our senior support engineer, and learn what it's like communicating information to users during an outage or incident. Later, we hear from our CEO Anil Dash, who gives us thoughts on these issues from a leadership perspective. But first, TK sat down with some members of infrastructure team. They're the ones who not only get notified first when site issues happen, but also help identify and fix those issues to get things back on track.

Mads Hartmann:
My name is Mads Hartmann and I'm a site reliability engineer at Glitch.

Emmett Walsh:
My name is Emmett Walsh and I'm a site reliability engineer at Glitch.

Cori Schlegel:
My name is Cori Schlagel and I'm a site reliability engineer at Glitch.

Antoinette Smith:
My name is Antoinette Smith and I'm the engineering manager of the infrastructure team at Glitch.

Keisha "TK" Dutes:
What is the process of response for when an incident happens?

Antoinette Smith:
It really depends. When an incident occurs, typically, if it's something that we have an alert for, then whoever the person who is assigned to feed the pager holder is the first person that it gets alerted and that handles it.

Keisha "TK" Dutes:
About the pager, is there a literal pager that goes off or an email notification?

Antoinette Smith:
There is no literal pager. There's a site called PagerDuty and everyone has an account on PagerDuty. And then, you can download the PagerDuty app or it'll go to your email. It'll also go to our Slack channels, but no physical pagers.

Emmett Walsh:
When the page first goes off, I think maybe the most uncomfortable experience is if you're getting woken up three in the morning, I would say, my first response is just pure confusion. I don't know what's happening. It's happened before. You get called by PagerDuty. You have to answer the call to acknowledge it. It's happened before. I actually can't remember what a phone is or how to answer it for the first 10 seconds. Normally, it's just complete debilitation. And then shortly after that, it is normally cases like, "Oh, God, is this a bad one or is it a good one?" Because you don't really know until you get logged onto your computer, so it can be a minute or two of mounting trepidation where you're like, "Oh, is it going to be a sticky one or not?" Until eventually, you get stuck into it. And then, you very quickly try and figure out what the scope of the impact is. Is it something that's really urgent? Is it something that you can snooze both figuratively and literally, or is it something where you need to call in the cavalry and get more support?

Keisha "TK" Dutes:
What's the difference between a good one and a bad one? What's sticky? What's snoozable?

Emmett Walsh:
The kind of page I see and I'll feel relief is one that is narrow in scope, I understand it really well, and I've seen it before. It's a bit annoying that I'm being woken up to deal with something that we know is a problem, but we haven't had a chance to do a more automated fix. But, at least I know exactly what to do and I know I'll be able to get back to sleep very soon. A scary one is one where everything's down, so all users are affected and I don't know why. It's novel, it's unfamiliar, the signals I'm getting from it are confusing. We have different ways of measuring how things are doing. I think that's the case where it's a bit of a tricky one.

Cori Schlegel:
Once you get to the point where you know what the problem is and at least a possible solution, a candidate is something that we can try to resolve things. In some cases, it's still going to be hours and hours before we can recover. That's a little dread-inducing, right? Even if you're confident that you can fix it, but you're looking forward to the other eight hours of monitoring things and restarting services or restarting the hosts, restarting servers, which is one of the things that takes a long time at Glitch. We have a lot of servers. If something affects all of the servers that are serving user projects, that's a thousand servers that we have to recycle. We can't do it quickly. We can only do, I don't know what the rate is, but a few hundred an hour, so that's a 10-hour, an eight or 10-hour outage if it takes us that long to get to basically restart or recreate every single server that's serving a user project. Those make for very, very long days.

Keisha "TK" Dutes:
There's several different time zones across the team, so how do you all manage that? I'll go to you, Mads. When you're receiving the page, are you eating a sandwich or sleeping or what?

Mads Hartmann:
Sometimes, I'm definitely eating a sandwich when I get a page. The way that we have set it up at Glitch is that we have working hours coverage. That means I will take the page during my normal working hours, which is nice because it means that the Americans can sleep a little longer and hopefully don't get interrupted. But, we also have another schedule, which is primary. For that, I am on call at night or in the morning and for all the hours in a day for a week. That might wake me up. Everybody who's on the rotation has a primary like that where they will have to cover the night-time shift as well.

Keisha "TK" Dutes:
How does each person know their role in fixing the incident?

Mads Hartmann:
The way that we have it set up is that there is an initial person who will get the page and it's that person's responsibility to escalate if necessary. That person might initially have a look at it and see if they can fix the problem themselves and go back to sleep or back to work. If they can't, then they can escalate to the secondary on-call. If that person, when that person joins, they'll try to fix it together. If that doesn't happen, you start calling in the cavalry and just pinging everybody and trying to get as many people in who might be able to help.

Cori Schlegel:
There are some cases where even the working hours on-call and the primary on-call and the secondary on-call might all -- some of whom may be the same person occasionally -- will get on together at the same time and they'll be able to isolate where the problem is, but they're not a person who is equipped to fix it. And so, that's when we'll interrupt somebody or get somebody else on a different team, often, out of bed or out of whatever zone they happen to be in to join in and try to figure out what the best way to fix the problem is.

Emmett Walsh:
When we decide who to call, if you're the person who picks up the page first, things we have intention, we don't want to interrupt people who don't necessarily need to be interrupted because we want to keep shipping new features and bug fixes against the very massive concern. If it's the outage that users are being impacted by, then that's a pretty high priority. Normally, we will prioritize getting people off their normal work to address those things at least when we're trying to understand what's going on and maybe all the way through to resolution.

Keisha "TK" Dutes:
In this moment, you're fixing it and it's taking all day. How are you feeling?

Cori Schlegel:
How you feel really depends on what kind of incident and what phase of the incident. When you're in the nobody-can-access-Glitch state and nobody's projects are running because those are two different things. There's at least three different things. Nobody can actually create an account or create a new project. Nobody can edit their existing projects. Then the third type is the projects that everyone has already created. None of those are running. It's kind of three different categories, at least three different categories. When you don't know what's going on and everybody's affected in any one of those categories, that is fear-inducing and stress-inducing. Panic-inducing maybe is the right phrase, at least for me.

Cori Schlegel:
Once you've sort of figured it out or at least you have a little bit of a direction to go, then it can be a little...You might know that it's going to take you awhile to figure it out. But at least you have a pathway and you can feel a little more confident. There's still a lot of pressure. We need to try to fix this as fast as possible, but it's not panicky. It's just "Let's just see how we can do this quickly" rather than "I have no idea what to do", which is that those are the ones that it's...especially at 3:00 AM. I have no idea. I don't even know where I am because it's 3:00 AM. I also have no idea what's going on.

Mads Hartmann:
To me, at least, it is like you're feeling stressed throughout the whole thing, but it can also be quite fun. What you're doing during this incident response is you're trying to figure out what's wrong with the system. That means you're coming up with hypotheses of what might be wrong. Then you're trying to prove or disprove them together with the team. That is extremely challenging and extremely fun for the most of it, I think.

Mads Hartmann:
It's also where you learn a ton of stuff about your systems. You learn more about your systems when they're broke than when they're working just fine. I think that part is really fun and can be very rewarding. Finally, once you've gone through the whole incident and you figure out what's actually wrong with the systems and you fix it, then you're done. That can be extremely fulfilling. I think, at that moment, I'm feeling at least always extremely proud of the team that worked on the incident and proud of myself for dealing with it without breaking down. I think that is what makes it in the end worthwhile. Being on the on-call rotation is that it's not just threat and stress. It can also be very fulfilling and challenging and fun and a great way to bond with your teammates who are in it with you if you're trying to fix the systems.

Keisha "TK" Dutes:
How does the user know that the incident has been resolved?

Antoinette Smith:
Well, what's really helpful is more recently we have a support team. I mean, we've always had a support team, but we haven't had a support team that was actively involved in our incidents. That's been a process change over the past few months. We have a support team and when an incident occurs, they'll update status page at IO so that people will know that Glitch is down or some part of Glitch is down. They also handle communicating with people on social media, with people in our forums. They are part of handling the incident and their major task is making sure that people outside of our team and people that are the users that are interacting with Glitch know what's going on. Once things are resolved, once our team is like, "Oh cool, this is done. Things are looking good," then they say, "Okay, this is resolved." They let everybody know.

Emmett Walsh:
Maybe I could talk more generally about the process things.

Emmett Walsh:
When we have an outage or some dropping function out at Glitch, it's not over when we get things recovered again. The job isn't done. Afterwards, we have a follow-up process. We go through an instant retrospective. We first try as best we can to understand what happened. Sometimes we'll have a really good idea in the middle of an incident. Sometimes we'll never quite get to the bottom of it. Either way, we try to figure out the best we can. Then we get together everyone involved or people who are interested, who might have domain expertise, even if they weren't involved. We'll discuss what happened. We'll try and pick out things that went well in how we dealt with the incident. Be that from how we noticed it to how we dealt with it in the moment to how we communicated about it.

Emmett Walsh:
Likewise, we'll think about things that didn't go so well. Then we also think about where we just got lucky, which depending how you think about it might be a "didn't go so well", depending how cynical you're feeling. Then we'll try and come up with some actions. We try and not be hit by the same thing twice. We'll say, "Hey, in this case, the problem is the way we configure these servers." We change the configuration this way. We make sure this kind of problem won't happen again. Then we try and prioritize when we can get that work done alongside our normal work in shipping new features or trying to do maintenance reliability work.

Keisha "TK" Dutes:
Given all that each of you do on a daily basis to make sure Glitch isn't experiencing issues, running smoothly, what would you like the users to know?

Mads Hartmann:
Alright. One thing I would love the users to know is that sometimes incidents happened because we make mistakes. We ship bad code, we don't follow a process correctly, and it's totally our fault. We try to fix that as quickly as we can. Unfortunately, in other cases, people out to get us. They are trying to use Glitch in ways that is malicious, both for other sites of the incident, but also for Glitch. They might try to use Glitch to attack us or take down another site. In cases like that, it will look the same to you. Your Glitch will be down. The project will not work. But for us, it's very different. We're doing our best to also take care of those cases. It's not always easy.

Emmett Walsh:
When there's an incident and Glitch is down, I would love users to know that we really care and we really hate it when Glitch is down. We work very hard and very quickly to get it back again.

Cori Schlegel:
I guess, as far as what I'd like Glitch users to know, I'd second definitely what Emmett says. We care. We're Glitch users. When Glitch is down, it affects us as well. We do dogfood our own product. Most of us use Glitch for things that are not necessarily in our day-to-day jobs. Support has really engendered this across the board -- Glitch is pretty stable. We don't have a lot of really big incidents. I'd like for users to remember that it's unusual for Glitch, especially Glitch projects, to be down and not responsive.

Antoinette Smith:
I guess what I would like users to know during times when the site is down or some key functionality is down is that Glitch is a really small company still. Our team that are handling these incidents are very small. In addition to shipping features to make Glitch better, on top of that, the team is also handling these incidents. Everyone is working, I feel, as hard as they possibly can. We want Glitch to be stable. That's the thing that we want the most. We're getting there, but it will take us time.

Shift Shift Forward

Subscribe on

Latest Episode

Episode 3

Episode 2

Episode 1

You Might Also Like