Horror Web Dev Stories - 2021

For episode 400, Scott and Wes talk about web dev horror stories - 2021 edition! LogRocket - Sponsor LogRocket lets you replay what users do on your site, helping you reproduce bugs and fix issues faster. It’s an exception tracker, a session re-player and a performance monitor. Get 14 days free at logrocket.com/syntax. Mux - Sponsor Mux Video is an API-first platform that makes it easy for any developer to build beautiful video. Powered by data and designed by video experts, your video will work perfectly on every device, every time. Mux Video handles storage, encoding, and delivery so you can focus on building your product. Live streaming is just as easy and Mux will scale with you as you grow, whether you’re serving a few dozen streams or a few million. Visit mux.com/syntax. Linode - Sponsor Whether you’re working on a personal project or managing enterprise infrastructure, you deserve simple, affordable, and accessible cloud computing solutions that allow you to take your project to the next level. Simplify your cloud infrastructure with Linode’s Linux virtual machines and develop, deploy, and scale your modern applications faster and easier. Get started on Linode today with a $100 in free credit for listeners of Syntax. You can find all the details at linode.com/syntax. Linode has 11 global data centers and provides 24/7/365 human support with no tiers or hand-offs regardless of your plan size. In addition to shared and dedicated compute instances, you can use your $100 in credit on S3-compatible object storage, Managed Kubernetes, and more. Visit linode.com/syntax and click on the “Create Free Account” button to get started. Show Notes 02:54 - Hi guys, love the show. I wanted to share with you something that happened just the other day (Oct 4th), I was starting my new job today at a large tech company. They use React for everything (even DNS!, don’t ask me how, it’s complicated). I figured I’d celebrate my first day and push some code to prod, (how hard could useEffect be right?) Next thing you know, they ended up bringing in a guy with an angle grinder to get access to the server cage. 04:15 - No one from Denver can buy 06:38 - Bug accidentally gives $90 million to users https://www.cnbc.com/2021/10/01/defi-protocol-compound-mistakenly-gives-away-millions-to-users.html 08:34 - Share Pointy Knives Hi! I’m a developer at a consulting firm in Sweden, writing C# on the backend and using React with either JavaScript or TypeScript and hosting things in Azure 99% of the time (and 1% in SharePoint). I was in my last week at my last job before I was due to start my new job. Worked 12 h/day to keep up with all the handovers etc. to colleagues so they would have a chance to continue working on the solutions I have taken care of. One project was a process tool hosted in SharePoint Online. The guy who would oversee it had -1% experience with SharePoint (which I pointed out to my bosses). But to make things a bit easier, I wrote a deploy script to ease things a bit. Starts the terminal and runs the script towards the acceptance environment. Umpteen million errors appear… Which is strange, because there would only be about 20 commands (which can cause errors like these). I log into the environment to double check if I now accidentally entered the wrong values in the script (which looked okay according to me). But I get a 404 error when I try to reach the environment… I log into the admin interface; I discover that the site is gone… Also checking the trash can, there are no things there. Very strange. I find that I’m in a different folder than the one where I saved my script… In that folder there is an old deploy script that was used when the project was started a thousand years ago (which was not used after the project was “finished”). The first thing the script does is force delete the site and then try to create a new empty site… The site is gone with lists and everything (lists are a SharePoint thing, think of it as sql-lite), there are no backups of the acceptance environment (although it is very important). I just feel a little panicked about how I’m going to solve this. However, I remember testing a tool six months ago to copy entire environments. Where the first attempt was made on the acceptance environment. Finds the cloned environment and can use the same tool to clone it back. It took only 8-12 hours of work to create all the new things done in the environment in the last 6 months instead of X number of hours to build everything from scratch. Once I updated a feature that saves accessories on orders (same solution). However, I failed to add all the new fields to the production environment. Which meant that accessories were not saved at all… Which was discovered after a week… I fixed the error in 5 minutes and the sellers had to contact x number of customers to double check what kind of accessories they would have for their orders… 11:22 - External HD One time I needed to format a server. It was an outdated Windows server. I selected all the files and copied and pasted to an external hard drive. My drive was pretty fast and it took like a minute. I was like: “Wow! That’s a great external hd”. Formatted the server and, as soon as I realized it didn’t copy 10% of the files, I had that face. We all know that face. Anyways. Tried to restore the files using some HD recovery tools but they were all corrupted, not by the formatting itself but for the installation of the new OS. My boss was pissed! I was very young so I blame it on the server. I’m not proud of it. But why the heck they would ask a developer to format a server in the first place? By the way, my birthday is on Halloween. Spoooky. 13:07 - Hey Loser I was testing new code to automate mass-mailings to our customers. Who knows what demon drove me but I wrote the “test” mailings like ransom notes: “Dear loser! Fork over all your $$$ or else!” Well, all was looking great and I wa s feeling pretty pleased with myself. Progress bars were sliding and counters were spinning. But I could hear a rising commotion from the marketing guys behind me. Phones ringing, voices raised. Turns out I had moronically wired myself to the production database! Even worse for me, I’d only been at the company a month or two. I thought my goose was cooked and the Big Boss was plenty mad, but I owned up right away and apologized. We put out a cover story that we’d been hacked and all was forgiven. 15:01 - HE HATE ME I was part of the developer team that accidentally leaked the 8 cities the XFL, an alternate football league, a week before their press conference. ewrestling.com/article/wwe-ac… We were using Contentful and Gatsby. A junior dev entered the information into the prod space instead of the UAT space and when we released some bug fixes, it picked up the contact us content update. I found out after seeing stories pop up in Google News when I was about to go to sleep. Was taking the content down when we started getting calls from the CIO of the WWE. The league went bust because of COVID. 19:23 - I Don’t Have Memory of This I had two pretty bad code changes that only showed their problems when they went live in production. Around 6 years ago, I was running into a large performance issue with some of our queries running slowly against this giant DB. We were using JPA/Hibernate and we had a bunch of joins that were done lazily. I switched a few of them to eager so that they would create a single SQL statement instead of a bunch (or thousands). The change worked fine on my dev environment, QA, and staging. Staging was supposed to be representative of production. So we went live and within minutes the entire system went down because of out of memory errors. We quickly switched back to the lazy joins. We found out that staging had more memory and fewer DB records than production though they were supposed to be exactly the same. 21:05 - Your Performance is Slowing us down