Honestly, I like the cucumber format, but what you ought to do is publish an API of phrases so that monitoring providers can target a standard set of monitoring operations. Right know, I don't see how they can execute a generic spec as described.

Cucumber devs might not like this, but at the moment I'm taking cues for the API phrases from cucumber-rails-training-wheels (web_steps.rb for capybara) - I see why the devs wanted it removed, but it works great for something like this that needs wide support.

Something Javascript-based might work. Perhaps the site-wanting-monitoring supplies a collection of JS scripts to run, each of which will be provided a standard object providing assertions/conclusions/reporting about specific features/URLs/scenarios.

Wouldn't sending raw Javascript upstream limit adoption due to the increased burden of securing the provider against malicious or buggy scripts? For this use case, a declarative solution seems like a requirement.

It's a burden, sure, but is the burden that large? Accepting and sandboxing remote Javascript seems relatively well understood, with lots of open reusable code already available. And perhaps the burden is offset by the familiarity and flexibility.