Would Statistical Rethinking help me interpret web app metrics? E.g. if I have a canary out and the response times are longer after x requests, is that significant?
I've found that Statistics is one of those topics that changes your world view about everything. You can consider pretty much any issue statistically, and that will enrich your perspective significantly. In that sense, Statistical Rethinking will help. However, it's a book on Bayesian stats, it's quite dense, and examples are coded in R. It may be overkill for web app metrics interpretation. For that you may be better served with basic stats & inference, frequencies, descriptive statistics, percentiles, basic distributions, data visualization (e.g., trend lines, scatter plots, boxplots, histograms), etc.
To be clear though, Statistical Rethinking is a beautiful piece of work. You can check out the author's lectures[0] and see how much they suit your needs.
The book is a bit more foundational than that. It teaches you about Bayesian statistics, and discusses (among other things) why the concept of binary yes/no statistical significance is usually not the best way of evaluating a hypothesis with data.
However, for your question specifically, the choice of prior is less meaningful when you have lots of data, and presumably a web app seeing hundreds or thousands of requests per second can gather enough data to determine if the canary has a different latency profile than the deployed version within a few seconds. Also, presumably you would use an uninformed prior for a test like that. If I were trying to prevent latency regressions in an automated deployment pipeline I would just compare latency samples after 1 minute with a t-test or something simple like that.
Bike commuting during winter is fine as long as you have the right clothes. I have commuted in a -55F wind chill in perfect comfort. I will concede that it takes a few weeks to calibrate clothing if you're not used to it, but it's not some huge barrier if it's something you're interested in
Snow cover can make it impossible. Frozen rain means you need somewhere to store your wet clothes, and they will not be nice to put on when going back. Somebody who isn't a huge biking freak will prefer the car.
Disclaimer: My Epic experience is around a decade old.
I never saw a riot, but I definitely saw providers who hated Epic when it was installed. As in people literally screaming at the top of their lungs at IT nerds. I saw a few contributing factors:
1. Epic supports making all the technically-mandatory-but-not-mandatory-if-you're-in-a-hurry data actually mandatory, so providers have more typing and less ability to just get stuff done when needed (which minimizes agency).
2. Related to above, Epic will make the updates for any new regulation, which increases the amount of data providers need to collect.
3. The customizability of the Epic UI and workflows for each type of visit was completely insane; many permutations of steps in a visit had to be supported which meant a huge maintenance surface with ok coverage rather than an opinionated workflow for each visit that could be optimized and improved over time.
In my experience, the software development profession could spend a long, long time doing some self-reflection about this one. It's eloquently stated, and something a lot of developers could learn. Too many times, I've seen overly restrictive inputs cause users to hate and distrust the software. Ironically, overly restrictive inputs cause users to think that the software doesn't properly understand the domain, which is the root of mistrust.
We should be very liberal with accepted inputs. I call them "Fuck It Buttons." There are lots of cases where you want a "Fuck It" button to just go around all the data entry and get an answer or move on with minimum info. Warn that the data isn't complete and we're using defaults, and don't just make output look the same as a complete workflow, but let them go through, nonetheless. Health care is just one example, but "Fuck It" comes up in every industry.
This is the UI/UX equivalent of knowing which hills to die on.
I highly doubt the developers have anything to do with this in this case. The people requiring the inputs are managers who are responding to regulators and insurers. They're not going to buy software that has a Fuck It Button that allows workers to skip data entry that could cost them money, time, or lawsuits.
At Epic there aren't really product managers. The devs mostly set the projects, design, scope with input from clinicals/sales (a very small group).
So it is interesting to me because most of the design choices - both good and bad - are made by the devs themselves with input from area experts in the aforementioned group, QA, and customer implementation/support.
Some might argue there would be better results with someone who is not the dev managing the product more. But there are pros as well as cons.
> Ironically, overly restrictive inputs cause users to think that the software doesn't properly understand the domain, which is the root of mistrust.
Well, they're quite likely right! I freely admit that as a developer, I don't understand the domain remotely as well as the people working in it (construction workers in my case). The reason there's still a decent market for custom software and that the likes of Epic haven't gobbled up everything, is that every so often a construction worker, dentist, nurse or whatever picks up enough programming to actually make something that suits them, and they manage to bypass the administrators addicted to sales dazzle.
At our company we came to the same conclusion. We have a DSL that is essentially a programmable schema to describe the shape of the data (more specifically a contract) you want to capture as well as how answers and decisions are derived from it. The only hard validation we have are types, eg. you can't put letters in a box that captures a number type. Other than types the rest is soft-validation which means that you can input anything, even if it is partial or not quite correct, and the system will do its best with what it has. In tandem, at any point in time you can ask the system to tell you what is missing and/or incorrect. All this then affects the lifecycle of the information, ie. you can't move past certain checkpoints in the workflow if the information is not in the required shape. In the context of a medical software imagine you can fill in just the things that will get you back meaningful answers to help you treat someone and you can deal with the rest after to make the case complete.
Like with most things, it depends! Maybe it's a default value; maybe it's a null; maybe it's a special value that triggers a workflow on insert. I'm an evangelist for RDBMSes, and they can do so much, so let them help you!
Maybe you have a state column that's derived that you cannot move to another step in the workflow until all nulls are filled in, but you've let the UI save what data it knows about and move on. It totally depends on what the user is doing and why we're skipping steps/data.
Honestly this sort of deferred validation exists as a standard feature of certain modes of data intake that are often criticized for same, for example paper forms (the "required" fields can be left blank), or creating support tickets via email (required fields stay null until an agent updates the ticket via web UI). At some point additional round trips may occur to pay back this debt, but debt is a powerful tool for that person in the field who needs to move onto other things until the dust settles and they can pay it back.
Paper forms can be filled in by different people with different roles at different times. Customers can fill in their personal data and possibly leave blank something they don't understand. A clerk can review the form, ask the right question and fill in the missing field plus the remaining ones.
"Fill in what you know and leave the rest to us" is a simple and cheap to implement GUI compared to a full fledged workflow. It could bootstrap a process quickly at the cost of some extra labor in the customer facing department. Maybe they have that extra bandwidth and the sw developers don't.
It happens in hospitals. Treat the human first, treat the chart once they're stable. Speedbumps in the process are fine, because we shouldn't be bypassing protocol all the time, but flat-out roadblocks aren't, because sometimes we really don't have time to worry about all that.
My wife and I have good insurance (like $200 out of pocket for surgery good) and I still hate going to the doctor because I know I'm going to get stuck with an involuntary side hustle as a medical billing clerk if I want the benefits we're entitled to to actually be honored. Our insurance paid maybe ~6k in charges one time, then 6 months later retroactively denied the claims and billed us for them. There's just no repose ever at any time, and you have to deal with a system that is always stacked against you.
No joke. If we'd been compensated at any kind of reasonable rate (not even our day-job rates) for time spent dealing with billing and insurance crap, it would've covered a good chunk of our (substantial) bills.
