We published our look back at our year at Glitch, and I’m incredibly proud of what our team and community have accomplished! But we also encountered our share of challenges — some serious — over the course of 2019. The end of the year is a time of reflection, so I want to talk about those bugs we found, both to acknowledge the bugs we’ve found and share what we’re learning as we work on solving them.

All of this is in keeping with the spirit that underlies our company. When we picked the name “Glitch”, we wanted to emphasize the very human reality that things/people/processes are often imperfect - they have bugs or mistakes. But if we commit to working on them together, we can make great things.

The most visible challenge we had in serving our community this year is keeping up with scaling the technical infrastructure that makes all these millions of Glitch apps possible. Glitch uses an unusual system design (every Glitch app is its own, unique Docker container with a full Linux environment in it) that may not be obvious when simply browsing the site, but it does take some real heavy lifting to make it go.

A few times this year, that system didn’t run as smoothly as everyone would expect. Most notably, we had a fairly lengthy outage on November 4th that we documented here. During each of the times we faced technical issues like this over the course of this year, we’ve been really heartened to see the community be so supportive (so much #hugops!) and understanding about the challenges of scaling.

But if we are serious about Glitch becoming a part of every developer’s toolkit (see Connecting to Coders, below), we have to be a platform everyone can trust. We have to meet and exceed our community's expectations for stability and reliability. We've made a big investment on that front, while also focusing more on user-facing features that makes the creation experience on Glitch smoother and even more enjoyable.

2019 was also the year that many of our talented people chose to come work at Glitch—most of our team has been with the company less than 12 months! It’s been incredible to see our team grow by leaps and bounds and to find out everyone’s superpowers as they work together.

But as we rapidly grew our team, we failed to commensurately grow the processes and infrastructure necessary to support everyone properly. The result has been a lot of needless stress and tension and frustration. On its own, this is a significant problem, but when we’ve talked a lot about wanting to build a company that does these things better, a failure here is twice as painful for the people on our team who are affected. As the person who most often talks publicly about the positive ambitions we have here at Glitch, I’m also the person ultimately responsible for the times we haven’t delivered on those promises.

Underinvesting in HR and people infrastructure while a startup goes through a big period of growth is one of the most common, and egregious, mistakes. And no surprise, this has had the biggest negative impact on the areas where we were trying hardest to do something different — salary transparency, benefits, and supporting remote work. We are still deeply committed to getting these things right, and I’ve been working with our whole leadership team to make sure we get this fixed as quickly and completely as possible. Our next steps are bringing in experienced help immediately, reviewing where we’ve fallen short of our documented goals, and talking to every member of our team to gather their explicit feedback about expectations going forward. We’re also making hiring a head of people a top priority and are actively seeking out qualified candidates.

The third big challenge we’ve faced this year is making it clear and easy for every creator, from professional programmers to brand new coders, to understand how Glitch will fit into their workflow. We have a complicated product, that can be used in many different ways for many different things, and while people love our community and its creations, it’s not always obvious for a new user where to start and how to use Glitch. On top of this, there are so many stresses and distractions in the tech world right now that many people don’t know how a happy web community fits into the problems they’re trying to solve.

We’ll be addressing this by simplifying and sharpening our focus in 2020. None of our capabilities will go away, but we will make it far more obvious how a working coder can take advantage of Glitch’s power and simplicity. We’ll also strengthen Glitch as a platform and as a business by introducing paid services for the features you’ve told us you need most and find most valuable.

You can get an idea of how Glitch will fit into your workflow already with examples like the starter kits for Magic Leap, MapBox, Datasette, and integration into Visual Studio Code. Expect to see more of this in the new year, and if you run an open source project or offer an API to developers, get in touch and we’ll help you make it easier for any coder to get started.

As we’ve reflected back on both the biggest successes and the biggest challenges we faced in 2019, there’s something interesting that connects them both together. Everyone on our team at Glitch, and everyone in our community, is unified in believing in the vision of a creative community. Everyone is inspired by seeing what regular people all over the world are creating, and seeing a flourishing of a more human internet filled with things that surprise or delight us, or simply solve a meaningful problem. Even at our most stressed moments, we still believe deeply in the community we’re building on Glitch, and in the importance of enabling people to create a better internet for themselves.

There’s something really powerful about a mission that a community, and a team, can believe in even when the going gets tough. One of the underlying themes for all of Glitch is that we can make really amazing things together, especially if we’re able to fix our bugs in collaboration with others. We’re excited to do exactly that with all of you in 2020.

The Year in Glitches

For context, that’s more than Apple’s App Store! We mentioned crossing the 2.5 million mark earlier this year, and it’s clear the community on Glitch is creating the cutting edge of the web. We’re proud to enable creativity with code like no tool has done before.

You asked, and we delivered — Glitch for Visual Studio Code

You can now develop Glitch apps entirely using Visual Studio Code, the most widely used developer environment. Download the extension today on the Visual Studio Marketplace!

You helped us partner with MythBusters Jr.

Glitch teamed up with Adam Savage for the premiere of MythBusters Jr., a show where six of the nation’s most talented kids tackle a variety of scientific myths. We created companion Glitch apps for each episode, which take the experiments to a whole new level.

You helped us develop three new starter kits

Since our starter kits for React and website building were such a big hit, we made three new starter kits this year! Learn more about WebVR using A-Frame, build your first Discord bot, and then have some fun with our Glitch Party starter kit!

Anyone can build the web with Glitch, so we put that to the test at some of the Web’s biggest events this year! We brought Appy Hour to XOXO, got our remix on at Capitol Royale, and took questions with Mindy Kaling at this year’s Twilio Signal conference!

Function, our tech podcast hosted by Glitch CEO Anil Dash, returned for season 2 this year with a big question: can we trust the Internet? Anil talked with authors, advocates, and developers to uncover the answer.

You helped us make history with Revision Path

Revision Path became the first podcast to be included in the permanent collection of the Smithsonian’s National Museum of African American History and Culture -- a tremendously significant milestone!

We've been bootstrapped for almost 20 years, so the decision to raise funding wasn't one we took lightly. With this investment, we’re able to turn our words into policies and practices that are unprecedented for a company at our stage.

You kept us honest as we expanded our board member search to the community

We think we have a chance to make a fundamentally more innovative strategy and a dramatically more inclusive company culture by making our call for board members as open as everything else we do at our organization.

We've more than doubled in size and with that growth, we've learned a lot. There's more to say about what these lessons are, but we're going to be putting those into practice in 2020 with a renewed focus on making Glitch a great place for both users and employees alike.

Glitch Year in Review

In early 2020, Glitch is launching a new publication, Glimmer. You're seeing the logo for the first time here, and we can’t wait to show you the site when it launches. In the meantime, we're commissioning reported features, interviews, analysis, and illustrated works about how technology impacts our culture, work, and world.

To view our rates and submit a pitch, complete this form.  If you prefer to use email, send one to pitchglimmer@glitch.com. For more details about our editorial mission and goals, keep reading!

Celebrating the makers, creators, and coders who shine

The seeds of change in tech are planted in the developer notes, social media platform policies, and the software and devices that we use at home and for work. There is a growing public interest to understand the beneficial and harmful aspects of the web.

Glitch is a community and platform built for people who make the web. We’re biased, but we believe our users are at the heart of making the web better. And in 2019, Glitch became a signatory of the Contract for the Web, a consortium of “companies, governments, and citizens” who support sustainable and ethical tech. Our users and audiences consist of the people shaping the future of expressive, ethical, and even eccentric technology.

We're looking for stories on the frontlines of change. Glitch is proud to launch Glimmer, a new magazine highlighting unique stories about how technology informs the work of creators, from developers to DJs to designers. Glimmer will feature perspectives from the experts, but we will also highlight stories about the systemic limitations that prevent people from being heard by industry leaders.

Many of those speaking up are not being heard: the indie artist whose work is appropriated for a hateful meme. The social media user who is being harassed and told the terms of service can’t stop it. A video channel that is shut down without the option of appeal. The app user who doesn’t know how their data is being used and shared.

Send pitches using this form. This form is a remixable Glitch app that automatically posts submissions to our internal Trello board. Pretty cool, eh? For questions about a pitch, email pitchglimmer@glitch.com. We're also looking for like-minded partners to work with us to share stories about building a more exciting, sustainable tech industry. If you've got a great idea, we'd love to talk.

Margarita Noriega
Editor, Glimmer
margarita@glitch.com

Function Podcast, Live from Texas Tribune Fest: Reclaiming Community on the Web

SimCity and “Possibility Spaces”: Reshaping How We Build the Worlds Around Us

How This Accessibility Expert Uses Glitch for Work

Stuck in a Loop: Remembering the 20th Anniversary of Burn All GIFs

Glimmer: Call for Submissions

Often times you might get bitten by the creativity bug and have the urge to make something, but you aren’t sure of where to start. We’ve put together a list of fairly straightforward apps for you to remix. Change around a few colors or add completely new features: this is your playground. Before moving on to your next great achievement, save projects to your My Stuff for easy access later.

The easiest blog in the world, stop worrying about hosting. Just give "Geoglitchies" a remix and maybe change some colors to make it yours. Made by Monica Dinculescu and Jessica Lord.

The holidays are coming up and instead of just inviting everyone to your holiday party on Facebook, you could send them a cool remix of our “appy-hour” app that we made for our Appy Hour party at this year’s XOXO. The site has a lot more personality than your typical event page, all you need to do is remix it and give it your own original flavor.

If you’re looking to make a free link tree for your Instagram bio or just want a place for all your random links, then “link-tree-11ty” is the perfect practical app that you can remix for immediate use. Made by Cassey Lottman.

Given that the 2020 is just around the corner, it’s a great opportunity to spend your Hour of Code remixing something to wrap up 2019! With “music-2019” from Tiff Tseng, you can easily show off all the new albums you picked up this year. It’s as easy as changing the JSON file.

In a similar vein, “year-end-list” from Tiff was made to remember all of the books you read this year, but it could also easily be remixed to be about whatever you want — favorite movies, favorite concerts, favorite places to get croissants, etc.

Play around with some of these cute sparkles with a quick remix — try changing the color, speed and size of the particles. You can also copy them over to another project to spice it up! From Edwin Morris.

Similar to the sparkles, “snow-animation” can give any website a nice subtle holiday feel.

“Trivia-quiz-starter”, from Anjali Patel, gives you the structure of a quiz app, but makes it so easy to fill in your own questions and answers by editing the JSON file.

We’ve also got “odd-care” from Anjali which is your local Mad Lib starter for all your Mad Lib needs! Just give it a remix and change the text and JSON files to make your own story. Mad Libbing has never been easier, and what a fun thing to send a friend or coworker on a bad day.

Generators are a great way to spend an hour if you want something simple to play with. Take this La Croix generator, for example, remix and add onto what Steph Monette started.

Trick your parents into thinking you got a book deal over the holidays with “book-deal-generator” from Margarita Noriega. Add some more book titles or maybe change it altogether with a movie or television deal.

Facebook released a chatbot, called Liam, over Thanksgiving weekend to equip their employees with PR style responses to their families questions. Adrienne Porter Felt made a parody of it to help you with some other family questions. Add your own questions to the bot to help you practice for your next family gathering.

Whatever you do with your creative juices, there are always apps ready for remix. To keep these apps for later, click on this collection and add to your My Stuff or personal collections.

We're building a more powerful Glitch, learn more here.

12 Web Apps that Anyone (and Everyone) Can Remix

We’re obsessed with thinking about the future of the web. Just what will the web look like in ten years? And how will we experience it? Magic Leap feels like a sneak preview of that future.

The Magic Leap 1 is a lightweight, wearable, spatial computing platform, or as we at Glitch like to say: future goggles. It unlocks incredible new experiences and gives digital objects a contextual awareness of your environment. Users can also fire up Helio, the device’s built-in Chromium stack-based web browser, and experience their favorite websites—like Glitch—in a radical new and immersive way.

We’re excited to announce that the spatial computing pros at Magic Leap are bringing a suite of Magic Leap developer tools and resources to Glitch. Now you can create, share, remix, and collaborate on Magic Leap creations on Glitch, either from your computer or from your Magic Leap 1 device.

No download, no installation, no setup required

You may be wondering how this works. The Magic Leap Lumin Web Platform lets you create interactive spatial computing experiences using HTML and JavaScript. With simple HTML tags, some inline attributes, and CSS styling, you can render 3D content that pops off the web page. It allows you to grab content outside of a website and place it into your physical environment. The Lumin Web Platform lets creators use the Lumin APIs for web development. These tools include Prismatic, a JavaScript library that lets you add 3D assets. Once published to the web, folks can view and interact with them using Helio.

The Magic Leap Team page, includes two new Glitch apps: “Hello Prismatic” and “Hello WebXR”. These starter projects give you just what you need to get started with two of Helio’s most powerful and easy to use solutions. In “Hello Prismatic” you’ll learn how to set up a basic Prismatic project, install the library, and display and animate a Robot model. This project is designed to be remixed so you can plug-in and animate your own model. And with “Hello WebXR” you’ll take an even deeper dive into creating AR experiences on Magic Leap. Here you’ll be using WebXR with Three.js to create a giant animated Jellyfish that takes over your entire space. (Watch out!)

This is just the first of many resources to be made available by the team at Magic Leap. In the weeks and months ahead they’ll release more new samples to showcase the new features of the Helio Spatial Web Platform. You can also look forward to seeing detached CSS, page rotation, and more advanced Prismatic and WebXR samples on Glitch, too.

We’re delighted to support the launch of these tools, and can’t wait to see how you make the web a more immersive and more magic place.

Build with Magic Leap on Glitch

In October 2019, the U.S. Supreme Court declined to hear an appeal from Dominos in which they argued that they and other online stores shouldn’t be held to the Americans with Disabilities Act. Because of a 9th Circuit Court of Appeals ruling, online businesses have a legal obligation to make their websites accommodating for people with disabilities. That includes ensuring that users are able to operate your website with assistive technology like text enlargement software or screen readers.

Many Glitch employees are passionate about accessibility but probably none more than Community Health Engineer Cassey Lottman. “In general, we’ve done a lot of work on the community site [to make it more accessible],” Cassey said. “I know there’s been some work done on the editor for accessibility but that’s really our next frontier in terms of making Glitch fully accessible.”

Cassey believes that there’s a lot about Glitch that makes it great for working on accessibility projects. “It’s easy to get a project up and running and have a link you can share with others and test across devices,” she explained. “Also, the ‘view source’ powers are really helpful - especially for apps where you’re using some Javascript library like React to generate your HTML.”

Cassey introduced me to Michael Fairchild, who works in accessibility and uses Glitch for work. Currently, Michael is an Accessibility Consultant at Deque Systems where he works with and teaches software development teams “how to make accessibility a core part of their development process, from design to deployment.” In his spare time he’s also the co-chair for the W3C ARIA-AT community group and works on a11ysupport.io. Michael lives and breathes accessibility and was kind of enough to answer a few questions about his projects on Glitch.

How have you used Glitch in your accessibility work?

In my development and accessibility work, I frequently use Glitch to build and test reduced use-cases. I commonly find myself asking these questions:

If I code this thing this specific way, will it work with screen reader software?

Why isn't this thing working with screen reader software or voice control software?

In those situations, I will spin up a simple page and write a reduced test case for the specific situation. I'll use only minimal styles, JS, and code to reduce the number of variables and the chance of interference. Then I'll test it on multiple platforms (Windows, MacOS, iOS, and Android) for example. This lets me quickly determine if something should work, or give me a hint as to why something is not working.

Is there something that makes Glitch unique when creating accessibility focused project?

I find Glitch very simple and easy to use which makes development a breeze. Additionally, it's easy to add multiple pages, changes are saved automatically, and I can open a full page view with ease. This makes it very easy to test on multiple devices without my test being in something like an iframe and wrapped by another site, which can in theory, interfere with the tests.

What’s different about using Glitch for teaching vs using it to test out an idea for something yourself?

Glitch is great for both tasks, and I think there is a lot of overlap between the two. One thing that sets Glitch apart for teaching is that the memorable URLs make it easy to share projects verbally. For example, I recently gave a talk on Accessibility at a developer conference and wanted to provide an easy way for the audience to keep learning after the talk was over, so I created factual-raven.glitch.me and shared it during the talk.

Additionally, I recently used Glitch to create and deliver a screen reader workshop for a local Accessibility meetup that I help run. It was easy to create different pages for each module and share it with everyone that attended the workshop. Having the instructions in their browsers made it easy for attendees to follow links to more reading and reference the workshop materials in the future.

You can follow Michael on Twitter and learn more about accessibility with this Glitch app. Plus check out Cassey's Community Health Resources.

Every step to tell a story with the new Interactive Storytelling template

Mapbox's new Interactive Storytelling template is out and available on Glitch, ready for you to dive in creating guided narratives with a fully interactive map and little to no code. Major media outlets have used this format to tell powerful stories, like the San Francisco Chronicle's deeper look at the origin of the Kincade fire, or the Washington Post's exploration of the 1968 Riots in Washington, DC.

At its core, the template can be used as is, without ever having to style or upload any data to Mapbox Studio. All you need is a Mapbox account, an access token, and Glitch to start building a simple but very effective "scrollytelling" story that showcases different places in a region or a timeline of events with chapters and basic markers on an animated map.

First, navigate to Mapbox's Glitch team page and click on our template collection. All you have to do from here is click on "remix This" on the featured template called "mapbox-storytelling-template."

This will take you to the Glitch code editor. There you will see 3 files:

config.js: This is the file you will be working in.

index.html: This contains the code for the template. You can open it in your browser to see your progress as you build your story.

An asset folder for all your images and other assets.

To get started, open config.js. Below is the configuration file untouched:

First, add your access token (which you can find here after creating an account.) Mapbox Streets is included in the template by default, but if you'd like to use a different base map, just swap in the style url of your choice.

We're going to the include the default markers by leaving showMarkers value to true.

Now, we're going to start filling in the location information for each chapter. You can use this tool to help you identify the coordinates (map center), zoom level, pitch, and bearing, to dial in the exact map view for each chapter. To do this, open the tool, type in an address in the geocoder in the top right corner, and move the map around to adjust the view. Control+click and tilt and rotates the map if you'd like. And don't forget to take into account the placement of the block of text in the template (default value is left-aligned). Finally, copy the location information code and paste into the configuration file.

Repeat this process until you have all your locations set up. You can then start adding a title, a description, and url or paths to images.

To check out the results, click on "Show" > "In a New Window" and scroll. That's it! You now have a fully functional map with markers. You can view a demo here.

Now, we're going to take this template to the next level by styling custom layers in Studio and using the template to control the layers' visibility. The following example uses 2016 Presidential election data from Tony McGovern who scraped the data from Townhall.com and US Census data.

To include custom layers in your story, you first have to upload and style your data using Mapbox Studio.

After uploading and styling your data, you should toggle the layer's opacity (NOT the visibility) based on the layer's desired behavior when scrolling. The rule of thumb is that the opacity of a layer set in Studio will remain the same until it is changed in the template configuration files.

In the example above, I want all my layers to be invisible on my first chapter so I will toggle the layer opacity to 0. You can keep all the layers that won't require any adjustments, such as background, water, etc at opacity 1.

For all layers that you don't want to be visible in the first chapter, turn the opacity down. Those will be adjusted in the configuration file later. Once you're done, you can publish your style and click on the "Share" button to get the style url and your access token.

The url and the token will go at the top of your configuration file. You should also turn off the showMarkers option since we won't be needing the default markers for this story. Pick a theme (dark , light) and an alignment (left, right, center).

Now, we're going to update our chapters and adjust layers based on the story we'd like to tell.

Similar to our basic example, you can update the title, the description, and even add an image. But we're also going to specify the opacity of each layer, which will be triggered as the user scrolls through the different chapters of your story. This means that when a text block enters or exits the screen, the layers' opacity will adjust based on your settings.

Note that a chapter can enter the screen twice: once while the user scrolls down and once while the user scrolls back up. This means that if you want certain layers to return to their initial state, you will want to specify the settings in onChapterEnter.

With onChapterEnter and onChapterExit you can control the opacity of each layer using a number from 0 to 1. Below, I am adjusting winnerto be 0.8 on enter and 0 on exit.

For each chapter, you can repeat those steps, adjusting your layers as needed. The result should be a story that takes the reader through relevant data points, contextualized by location and visualizations. You can experience the full demo here.

Glitch saves your code automatically so all you need to do to view your demo is to go to the url provided. With very little code, you now have a fully functioning interactive live on the web!

Share what you build on twitter with #builtwithmapbox and if you have any questions, reach out at @lobenichou.

How to build a “scrollytelling” map using Mapbox and Glitch

The holiday season has officially begun, which means it’s the time of year where you may likely attend one awkward get-together, at the very least, whether that’s a family dinner, a work party or gathering at a friend of a friend’s house. Phew. Whatever situation, is anything a greater challenge than having to come up with small talk for hours on end? Don’t worry because these apps all make for easy conversation starters and will disrupt any unwanted awkwardness.

Do you have that one relative who likes talking about the history of a battle you’ve never heard of over dinner? The “library-of-places” is the perfect app to keep them occupied and educated while everyone else grabs a second slice of pumpkin pie. This project lets you explore the Library of Congress digital archives through an interactive map.

Just click any two spots on the United States and a route will form filled with photos from the archives that were taken in places along the route — scroll down to see the results. The creator, Aidan Nelson, is a New York-based artist and educator who is currently a researcher in residence at NYU’s Interactive Telecommunications Program.

Your cousin is a first-year college student and spends a ton of time on TikTok, but (quick!) you need a way to show that you’re quite aware of what should and should not be acceptable in this economy. Luckily, the “twin-peaks-password-generator” is here for you. Show that you watch prestige television and that you care about cybersecurity with this clever little project from Josh Mock.

This project is easily remixable, so if you don’t like Twin Peaks, create the password app of your dreams using keywords from any of your favorite prestige TV shows: Breaking Bad, The Wire, etc.

Finally, the kids. If you’ve been to any big gathering ever, you know that worrying about keeping the kids entertained is of the most importance. Enter “tenori-off” from Monica Dinculescu, a project inspired by the Tenori-on, a 2007 electronic musical instrument from Yamaha. “I LOVE the Tenori-on,” Monica writes in the project description. “Since they're rare and mad expensive, I've never seen one, so I made a JavaScript version of what I think it looks like.”

To use the instrument, click any one of the blocks to add either a synth or drum sound — the higher on the block, the higher the note. You can also enter a new number into the “Step 100 ms” block to adjust the speed.

The holidays can be a busy, expensive and exhausting time, but hopefully with a little help from some fun and interesting apps, it can go smoothly. Add these apps to your My Stuff with this collection.

Fun Apps for Long Weekends

Last week, on Monday the 4th of November, 2019, we had a seven hour outage. At 2.14pm UTC we deployed a new release to Glitch. Normally, we do this many times a day, without incident. In this instance the code we released broke our API servers, which resulted in the sitewide outage. During this time you were unable to access the Glitch site (which is a Glitch application) and your own Glitch projects.

As part of our normal process, we reverted the change and re-deployed. This however, did not result in the API servers recovering. After several hours of diagnosis, we identified a combination of issues, principally a crash loop in our API servers and very aggressive retry logic in our API proxies, that prevented the API servers from recovering. We re-deployed a new fleet of API proxies to address the issue, the API servers returned to normal, and the site began serving traffic again at 9.00pm UTC.

We’re deeply sorry this occurred and are grateful for the patience and support shown by the community. To help understand this outage and what we’ve learned from it, we want to share with you what happened and why it took so long to resolve.

At 2.14pm (UTC) a release was deployed to production. The triggering release was a substantial redesign of our logging module, called ng-logger. This is used by all of our backend services to aggregate and send telemetry to a number of different services. We believe the accidental introduction of a “deep clone” led to a depletion of memory, leaving our API servers unable to process requests.

The release led to an immediate drop in requests being served by our API proxies. Automated alerting via Pingdom and DataDog triggered a page at approximately 2.15pm, 1 minute into the deploy. The page triggers were Pingdom being unable to access the Glitch site, the API and a number of other key sites. After we detected the drop in requests, we began tracking the incident. Our Infrastructure team began to diagnose the problem and our Support team notified users of an issue.

We identified the immediate culprit as the recent deploy. We reverted the deploy and this new deployment was completed by 2.41pm. Unfortunately, rather than resolving the issue as we had hoped, the API servers remained locked.

We then began to diagnose the issue further. Our diagnosis was hindered by some gaps in our telemetry. We have begun to instrument all of our services with distributed traces but the process is not yet complete. This means we have limits in what we can observe outside of our API and API proxy servers.

During our diagnosis we identified several problems that we believe either contributed to the outage, or exacerbated the underlying outage. These included:

We have very aggressive retry logic in the API proxies, and some related systems, resulting in repeated attempts to contact the API servers. This kept the API servers overloaded and locked up.

A possible crash loop in the API service due to how we handle Sequelize timeouts.

Once we’d identified these issues, we determined that we needed some way of diverting traffic away from our API proxies to us to spin up a new fleet that wouldn’t become locked down immediately due to the flood of traffic. We made some changes to our load balancing configuration to allow us to create new fleets of API proxies that would start without receiving traffic.

Once these fleets were created, we began to send traffic to them and the API servers began to recover. This recovery began at 8.45pm and full site recovery was achieved at 9.00pm.

We learned a lot during this process. We conducted a blameless retrospective with the involvement of most of the product, design, and engineering teams, as well as a few other folks from across the company. It was our first “at scale” retrospective and we learned a lot about how we will run retrospectives in the future.

We also learned a lot about our incident management process. We’ve grown dramatically in the last year and some people on the team weren’t familiar with the process. Their first exposure to it was during this incident. We’ll be updating and familiarising everyone with the process going forward.

Our existing incident process also didn’t call for a “scribe” role to record the timeline of events concurrently with the incident, meaning we had to reconstruct events and data after the incident. We’ve updated our process to clearly define this role and ensure a timeline is initiated when the incident is declared.

We’re pleased our monitoring identified the issue in a very short period of time and that our (only partially-implemented) observability framework through Honeycomb allowed us to rapidly test and resolve many theories. It’s a priority for us to complete the deployment of this observability framework to additional parts of the platform to remove the current gaps in coverage. This lack of complete coverage resulted in some time as we followed false leads and leading to some frustration as we occasionally ran out of ideas of what to investigate.

We’ve identified we need to work on the retry logic and the timeouts of the API servers and proxies. We’re also going to be looking at our deploy process, which could be more resilient and better designed, especially to allow us to deploy in more staged manner. This might have allowed us to mitigate the issue before it hit the whole fleet.

This outage was upsetting and stressful for our users and for the team. But we learned a great deal and are correcting the issues that allowed the outage to occur and going deeper on our incident management, deploy and other processes to ensure future incidents are handled more efficiently. Thank you for your patience during the outage and we'll see you on Glitch.com!

Details of the Glitch outage on November 4th, 2019

Anil Dash is CEO of Glitch, and the host of Function, a podcast exploring how technology shapes culture and society. 

He also serves on the boards of Stack Overflow and the Data & Society Research Institute, and was formerly a contributing editor of WIRED. He’s been blogging since 1999 and won’t stop telling people about that one time he was on the Desus and Mero late-night show.

Maurice Cherry is Glitch's creative strategist.

Incidents

The Workers Who Keep the Web Safe in an Unsafe World

Animal Crossing Helped Me Get Back to Real Life

New Rules of the Classroom: Remote Teaching in the Age of Covid-19

Better Tech: Why We Need Calm Technology Now More Than Ever

Function with Anil Dash

Open World is a fiction anthology podcast about how science and technology serve as a backdrop for imagining a better future, helping us envision the positive influence that science and tech can play in a real future. 

Open World is produced by Keisha "TK" Dutes and hosted by critically-acclaimed podcaster Rose Eveleth, for Glitch. By speaking to the pressing concerns of our times through love, friendship, community, compassion, and sustainability, the host will help listeners imagine new ways forward into a future that is less scary and more positive, and something we can start working on constructively.  

Open World launches weekly episodes in Spring 2020. Want to contact the show? Email youropenworld@glitch.com.

Open World

Shift Shift Forward is a podcast that showcases everything that makes Glitch the best place to create on the web. We’ll talk to some of our top creators, explore cool apps, and even hear from the people who work at Glitch!

Shift Shift Forward

Celebrating the makers, creators and coders who shine.

Recent Posts