I'm working on Vedro, a Python testing framework as a pytest alternative without the magic and with clear output.
The main idea is that tests should just be Python: plain `assert` statements instead of custom matchers, no fixture magic, and when tests fail you get readable diffs that actually show what went wrong. Tests can be simple functions or structured with steps that self-document in the output.
I like the promise, and it looks nice. But I'm not sure what are the selling points.
- pytest already works with assert. Why brag about something that is already commonplace?
- It could help if your docs explained the alternative to using fixtures. I assume it would be done by re-using givens, but you could make it clearer what is the preferred way to do it, and what is gained, or lost, but doing it that way.
- Can you explain how your diff is better than the pytest diff? (I'm asking as someone who hates the pytest diff)
Thanks for the feedback, it helps me see things from a different perspective.
These are excellent questions, and you're absolutely right that they should be clear from the landing page. I'll work on fixing that.
Short answers:
1. Good point about asserts. When writing the benefits, I was targeting a broader audience (unittest users, people coming from other languages like JS), but the reality is most visitors are probably "pytest escapers" who already know pytest uses assert. I'll reorganize the selling points to focus on what actually differentiates Vedro.
2. The main philosophy is "all you need is functions and their compositions", no special decorators or dependency injection magic. But this is indeed missing from the index page. Will definitely add clear examples showing how to handle common fixture use cases with plain functions.
3. One diff example on the landing page clearly isn't enough. I'll add more comparisons. Since you hate pytest's diff output too, I'd love to hear what specifically bothers you about it, your pain points would be incredibly valuable for improving how I present Vedro's approach.
Because my hate for pytest is ardent and dear to my heart, I will try to explain.
I wrote a very simple function:
def test_whatever():
a = range(10)
b = (10 / x for x in a)
c = list(b)
When I run it with normal Python, this is the exception:
Traceback (most recent call last):
File "test_test.py", line 10, in <module>
test_whatever()
File "test_test.py", line 6, in test_whatever
c = list(b)
^^^^^^^
File "test_test.py", line 5, in <genexpr>
b = (10 / x for x in a)
~~~^~~
ZeroDivisionError: division by zero
It's compact and simple to understand. It pinpoints the exact location of the error, and I easily scan the text to find the function call-stack.
Now here's the pytest error:
___________________________________ test_whatever ___________________________________
def test_whatever():
a = range(10)
b = (10 / x for x in a)
> c = list(b)
test_test.py:6:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
.0 = <range_iterator object at 0x107695e90>
> b = (10 / x for x in a)
E ZeroDivisionError: division by zero
test_test.py:5: ZeroDivisionError
It doesn't pinpoint the error, it adds code lines that might be irrelevant, and extra information I don't care about.
I will say using `--tb=short` fixes most of it, at least in this example, and sometimes it's even preferable, because it's shorter. But it still doesn't pinpoint the error like normal Python exceptions do.
Thank you for sharing this example! You've perfectly captured one of my main frustrations: pytest tries to be "helpful" by adding extra context, but ends up obscuring the actual error.
Thanks for the support! It means a lot, especially from someone who shares the pytest frustration.
About bare `assert`s. Vedro is actually flexible enough to use any matchers you prefer, but let me share why I stick with plain asserts:
1. In most editor themes, `assert` jumps out with distinct syntax highlighting. When scanning tests, I can quickly spot the assertions and understand what's being tested.
2. The expressions feel cleaner to me:
assert error_code not in [400, 500]
# vs
assert_that(error_code, is_not(any_of(400, 500))) # hamcrest
3. I like that there's nothing new to learn, the expressions work exactly like they do in any Python code, with no special test behavior or surprises.
Would love to hear what specifically bothers you about bare asserts, always looking to understand different perspectives on testing ergonomics!
Your first and second points makes sense. They don't matter much to me, but I see how others could value those things.
Aside: I also don't like the hamcrest syntax. I also don't love unittest's syntax but it's OK and it's pervasive (i.e., available in the stdlib).
The third point is where I start to disagree more strongly.
> I like that there's nothing new to learn, the expressions work exactly like they do in any Python code, with no special test behavior or surprises.
This doesn't seem true to me.
> the expressions work exactly like they do in any Python code
Not to my mind. In normal Python, an assertion communicates something that is unequivocally believed to be true, not something that may or may not be true (a test). Let me see if I can explain it this way, I often use asserts in tests to show (and enforce) something that I believe to be true and must be true, before the test can have any meaning. E.g.,
The "assert" communicates that this is a precondition, the "self.AssertTrue" communicates that this is a test.
I can 100% see that others might not see/care about the distinction, but I think it is important.
> no special test behavior
Well, that's not quite true. You have to handle the AssertionError specially and do some fairly magical work to figure out the details of the expression that failed. The unittest-style assertions just report the values passed into them.
I don't really like that magic, both from an aesthetic standpoint and from a least-complexity-in-my-tooling standpoint. Again, I can understand others making different tradeoffs.
Thank you for taking the time to explain your perspective, this is exactly the kind of thoughtful feedback that helps me understand different testing philosophies.
Your distinction between assertions (preconditions/invariants) vs tests (things being verified) is really interesting. I can absolutely see how using different syntax helps communicate intent: "this MUST be true for the test to even make sense" vs "this is what we're actually testing". That semantic clarity is valuable, it reminds me of contract programming in languages like D where preconditions and postconditions have distinct roles.
You're absolutely right about the special assertion handling, that's definitely special behavior and I should have been clearer. What I meant was that the expressions themselves evaluate using Python's normal rules without hidden transformations. For example, Playwright's `expect(loc).to_have_text("text")` silently normalizes whitespace, so "hello world" might match "hello world". With plain asserts, `assert element.text == "hello world"` means exactly that: no normalization, no special matching rules. The expression evaluates the same way it would in a REPL.
But yes, extracting the failure details does require machinery that I'm not thrilled about either. It's the one compromise I made to support the plain assert syntax that many Python developers expect. But actually, you can use the `asserts` helper which does exactly what you described without any special handling:
from vedro import scenario, asserts as _
@scenario
def calculate_discount():
product = ...
discounted = apply_discount(product, 0.2)
_.assert_equal(discounted.price, 80.0)
I appreciate the consideration. I'll be watching Vedro with interest.
I do like the nicer vedro.asserts mechanism. If that works with unittest and pytest it would be really nice. You might get a few converts that way, too.
Each time a new LLM version comes out, I give it another try at generating tests. However, even with the latest models, tailored GPTs, and well-crafted prompts with code examples, the same issues keep surfacing:
- The models often create several tests within the same equivalence class, which barely expands test coverage
- They either skip parameterization, creating multiple redundant tests, or go overboard with 5+ parameters that make tests hard to read and maintain
- The model seems focused on "writing a test at any cost" often resorting to excessive mocking or monkey-patching without much thought
- The models don’t leverage existing helper functions or classes in the project, requiring me to upload the whole project context each time or customize GPTs for every individual project
Given these limitations, I primarily use LLMs for refactoring tests where IDE isn’t as efficient:
- Extracting repetitive code in tests into helpers or fixtures
- Merging multiple tests into a single parameterized test
- Breaking up overly complex parameterized tests for readability
- Renaming tests to maintain a consistent style across a module, without getting stuck on names
The main idea is that tests should just be Python: plain `assert` statements instead of custom matchers, no fixture magic, and when tests fail you get readable diffs that actually show what went wrong. Tests can be simple functions or structured with steps that self-document in the output.
https://vedro.io
I would be very happy to receive any feedback!