Ahh indeed, that was kind of my initial question for the top-commentor. I'm interested in whether Shopify explored getting security consultants in to review this area before going for a bug bounty on it, or went straight to the bug bounty.
My personal feeling is that the order of play should be
as you can use the first two stages to catch all the basic stuff and some of the advanced stuff then leave the bug bounty to pay out for things the first two elements missed, but that you still want to know about.
So I'm interested in data that suggests that companies are either going for that route, or have decided to cut out the external consultant review and go straight to bug bounty. In this case I'd guess part of that would be whether Shopify did indeed expect a $300k+ bug bounty programme or whether that was a surprise to them.
I think companies should get standard compliance stuff done first (they're in payments so they certainly have lots of these to have) + a set of standard vulnerability scans (Nesus / FireEye). These things are "cheap" and easy to get, it's standard package 1-week-audit-for-XYZ.
Then get custom pen testing and bug bounty programs later. They're a lot of work to get done and get right. pen testing is a lot of investment and preparation upfront[1], bug bounty is on a longer term.
[1] Don't bring people at $2k a day if you didn't think through what they're gonna do.
I think what 'xal is trying to say is that this bounty had more to do with security marketing than with accomplishing a particular tactical security goal. Their comment even concludes with a note that most or all the findings were accounted for with a sandboxing design they'd already planned.
It's security marketing, in particular: they're trying to increase engagement with their bug bounty program. A big problem bug bounty programs that run without promotion run into is that the median submission is of terrible quality, but the best submissions are so good it's hard to get them through any other vector.
If you're looking to run a bug bounty for a specific feature and want to maximize quality while minimizing effort triaging terrible submissions, I think there are much more cost-effective ways to accomplish that by structuring the bounty program (for instance: I might not run it on a platform like Hackeroni at all).
But if you're looking to run bounties for all your stuff in the future and want to maximize the likelihood that the good bounty hunters will pay attention to you to begin with, this might be a pretty cost effective way to do that.
My personal feeling is that the order of play should be
Internal Security Review --> External Security Review --> Bug Bounty
as you can use the first two stages to catch all the basic stuff and some of the advanced stuff then leave the bug bounty to pay out for things the first two elements missed, but that you still want to know about.
So I'm interested in data that suggests that companies are either going for that route, or have decided to cut out the external consultant review and go straight to bug bounty. In this case I'd guess part of that would be whether Shopify did indeed expect a $300k+ bug bounty programme or whether that was a surprise to them.