Welcome back to my series where I am breaking down application security so that anyone can do it without hiring expensive teams of employees. Today we finally get to the topic of security testing aka penetration testing (“legal”) aka hacking (“illegal”).

Avengers, Assemble…?

In our previous posts, we discussed the value of knowing your application inside and out, how to analyze your application and understanding what could go wrong with it, assembling all those theoretical vulnerabilities, and combining it altogether to form ways to break your system through threat modeling.

At this point, each of those steps above has culminated in a solid understanding of a bunch of potential flaws with your application.

So why not just fix all of what you found? The answer is because of time.

Imagine, you are running an extremely lean startup trying to increase your revenue. Everything with you and your schedule is a tradeoff.

You’re trading off:

• time spent on building more features with time spent on obtaining more clients

• time spent on events with time spent on cold emailing

• time spent on selling to a specific potential lead with time spent on marketing+advertisement

• time spent on fixing (or even analyzing for that matter) ALL of your potential vulnerabilities with all of the other stuff above

You don’t have the luxury of unlimited resources to just hire away these problems, you want to fix only things that are relevant and necessary to fix.

So what defines “relevant and necessary”?

Proof Is In The Pudding

There’s a well known adage that goes a little something like: “Health is not valued until sickness comes”.

What this really boils down to is that no one cares about taking care of their body until someone (aka doctor) discusses scary test results or something drastic happens. (BONUS LIFE TIP: health is wealth y’all. keeping yourself healthy will be your biggest ROI in life.)

But there’s a whole process to becoming healthy from a sick state, and it looks a lot like what we've been doing in this series.

When you walk in with a rash, a fever and a headache, a doctor doesn't just start throwing a bunch of pills at you. They write out a list of all the things it could be (this is called a differential diagnosis) based on the description of your symptoms and measurements of your body (heart rate, blood pressure, etc.), and then they order tests to figure out which one it actually is. Bloodwork, imaging, a biopsy, whatever it takes to rule things in or rule things out.

Doctors test and confirm before they treat. Nobody wants to be put on antibiotics, antifungals and steroids all at the same time "just in case". Every treatment costs money and every treatment comes with side effects.

Does all this sound familiar? Your threat model is your differential diagnosis. It's the list of everything that could be wrong with your application, and testing is the lab work that narrows that list down to what can actually go wrong, so you know what to focus on.

Remember at the end of the last post I said we'd talk about making sure your threat model is actually right, because after all it's just a model of what could go wrong? This is how. Your threat model is really a bunch of guesses that look like: "if an attacker does X, then Y bad thing happens to Z". Testing is where you actually go do X and see if Y really happens.

Let's take the restaurant ordering app from Part 5. On paper we came up with 4 different ways people could abuse the discount codes. Sounds scary! But until you make a test account, stack two codes on an order and actually watch the total drop below what you meant to charge, it's still just a theory. Maybe your checkout already caps the discount and you never knew. Maybe it doesn't and you're about to be giving away free food. Either way, now you know, and knowing is what lets you decide what's actually worth your time.

That's all proof really is: you made the bad thing happen, and you (or anyone else) can make it happen again the exact same way. (BONUS LIFE TIP: only do this on things you own. That's the whole difference between the "legal" and "illegal" I mentioned at the start)

Don't Burn Down the Kitchen

Since it's your app, permission to test is covered. The bigger risk when you test your own stuff is hurting yourself (or your customers) by accident. So a few ground rules that are more than just good to know:

1. Use test accounts you own. Make at least two so you can play both attacker and victim. Never prove a bug with a real customer's data. That turns a test into a real incident.

2. Test on a separate environment if you can (aka "staging"). If you don't have one, you can test in production, just carefully: do it when traffic is low, give your team a heads up, and know how to undo what you change.

3. Fake money, fake orders. Use your payment provider's test mode (Stripe has one) and keep test orders away from the real kitchen. Nothing like explaining to your line cooks why they just made 40 burritos for $0.12.

4. Don't attack the stuff your app runs on. Testing your app is fine. Hammering your cloud provider's infrastructure isn't. AWS for example bans things like flooding their network with traffic. Skim your providers' testing policies first.

Pick Your Battles

So does proving things first actually save you time? Yeah, and the security industry's own data backs this up.

Researchers looked at over 237,000 publicly known vulnerabilities in common software and found that only about 6% of them have ever been seen used by real attackers. That's roughly 6 out of every 100. They also compared two ways of deciding what to fix. Fixing everything rated "high severity" took about 6 times the work of focusing on the ones most likely to actually get exploited, for the same amount of protection.

That data is about bugs in popular software, not the code you wrote, but the lesson carries over: scary on paper doesn't mean real.

Testing isn't free either, though. If a fix takes 5 minutes (say, adding a missing login check to one route), don't bother proving it first. Just fix it, and save your testing time for the findings that would take days to fix. And if you test something and can't make it happen, that doesn't mean you're safe. It might just mean you haven't found the way in yet (remember, attackers have way more time than you do).

Remember that list of tradeoffs at the start? Features vs clients, events vs cold emails, selling vs marketing. Every hour you don't spend fixing a bug that was never real is an hour you get back for all of that.

We’ll talk about what it means to fix correctly in a future post, but the main takeaway today?

Prove it, then fix it.