Feature: Or-Operator

[colin-kiegel]

Just another random idea. :-)

Maybe you could introduce an 'or' operator, where 'or' would have lowest precedence:

Assert::command(&["foo"])
    .succeeds().and().prints("42")
    .or().fails().and().prints("not ready yet..");

Internally this could be achieved by splitting Assert

pub struct Assert {
    cmd: Vec<String>,
    expect: Vec<Expect>, // <-- delegate to a sub component,
                         //     which we can now put in vec
                         //     `.or` would simply push a fresh layer
}

struct Expect {
    success: Option<bool>,
    exit_code: Option<i32>,
    stdout: Option<OutputAssertion>,
    stderr: Option<OutputAssertion>,
}

The biggest challenge would be displaying failures. Here is a suggestion, that should be more or less readable even for large chunks or in the multi-line case - each or-branch will report only the first assertion that failed

'CLI assertion failed: `([1/2] stdout of `echo 42` expected to contain `"41"`)` (output was: `"42\n"`)'
or `([2/2] stdout of `echo 42` expected to contain `"1337"`)` (output was: `"42\n"`)'

[killercup]

Hmm, interesting. Quite difficult, though. What would be a use-case for that?

Can you accomplish a similar thing by matching on the error from .execute()?

[colin-kiegel]

Maybe testing a binary with non-deterministic behaviour?

I personally don't have a use case yet. When I saw and, I was looking for or and then I started thinking. That's all. :-)

[behnam]

I would really like to see the syntax getting more powerful to support more complex expectations. One solution that comes to mind is to use a Predicate model, to be the input into verifiers, like stdout() and stderr().

So, instead of

    assert_cmd!(wc "README.md")
        .stdout().is("1337 README.md")
        .unwrap();

we would have:

    assert_cmd!(wc "README.md")
        .stdout(P::is("1337 README.md"))
        .unwrap();

and we can have more complex features, like:

    assert_cmd!(wc "README.md")
        .stdout(P::any(
            P::is("1337 README.md"),
            P::is("42 README.md"),
        ))
        .unwrap();

and:

    assert_cmd!(wc "README.md")
        .stdout(P::all(
            P::contains("1337"),
            P::contains("README.md"),
        ))
        .unwrap();

.

One benefit of moving expectation into the argument of the verifier is to not have the type of the object changes along the chain of calls, which is much better for UX and ease of use.

Also, it would result in a better formatting by rustfmt, as stdou() and is() won't be considered in the same level and broken into separate lines, on a long chain.

[behnam]

Btw, looks like the only existing crate is predicates one that only supports integers. I think a generic Predicate<T> class can be developed to support various applications like this case.

[killercup]

FYI we discussed this a bit in the 1.0 issue and recent PRs Behnam Esfahbod ❄ <notifications@github.com> schrieb am Sa. 4. Nov. 2017 um 00:25:

…

Btw, looks like the only existing crate is predicates <https://docs.rs/predicates/> one that only supports integers. I think a generic Predicate<T> class can be developed to support various applications like this case. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#19 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABOXyJ5H9w0jcPVv2l5QSiSxN-MheMpks5sy6D9gaJpZM4MkhFG> .

[epage]

Also, it would result in a better formatting by rustfmt, as stdou() and is() won't be considered in the same level and broken into separate lines, on a long chain.

This is #70 and is being discussed in #74 and #75 which is also moving into more general predicate forms that would be great to split off into a separate library. I know I'll be wanting them for a filesystem assert library I'm considering making.

[epage]

Btw, looks like the only existing crate is predicates one that only supports integers. I think a generic Predicate class can be developed to support various applications like this case.

Just looked at predicates. The one thing we'd need to add on top of what they do is generating a Result that describes why the predicate failed.

[killercup]

One more issue to link here: #55. We should consolidate this stuff somewhere… it's not specific to add or…

I see the following points here:

1. We want a nice, fluent API
2. We want the ability to express complex assertions
3. We want to show amazing error messages

2 will most likely lead to implementing an interpreter for higher-order logic, which the predicate crate most likely already is. On the practical side, this will mean we need to implement a bunch of predicates aside from any and all (DNF and CNF) and the already existing not/is/contains stuff.

Do you have a good idea how to implement such a predicates crate without writing a huge unmaintainable mess or predicates? My original idea (see #41 (comment)):

assert_cli::Assert::main_binary()
.stderr(|x: OutputAssertThingy| -> bool {
    x.is("bar") || (x.contains("baz") || !x.is("bazinga"))
})

where the OutputAssertThingy created by the call to stderr implements all the predicates as methods. It also contains a log that each predicate execution adds to, and that we can print when the assertion result is false. Tricky thing is to make sense of that log, as we don't know the context of the predicate (!x.is("bazinga") would log something like { assertion: Is, state: Failed, content: "bazinga" } and print it like "expected content to be bazinga").

[behnam]

Well, if you want to have free-form bool expression as an assertion, I would say it's better to follow the lead of Rust's assert!() and go with a macro, which on error can print out the whole logic being failed. Maybe even more like the new RFC: rust-lang/rfcs#2011

[epage]

RE "We want the ability to express complex assertions"

I agree that we should question how complex of predicates we allow in the API.

I just updated #55 with my thoughts on how the Fn predicates can work. I suspect with that kind of backdoor, we don't need logic predicates or even OutputAssertThingy.

If we do decide on OutputAssertThingy, an alternative idea is for each predicate to return an error

assert_cli::Assert::main_binary()
.stderr(|x: OutputAssertThingy| -> Result<(), OutputError> {
    x.is("bar")? || (x.contains("baz")? || x.isnt("bazinga")?)
})

Hmm, kind of ugly.

[epage]

#98 is exploring using the predicates crate which will offer us or and other complex expressions.

[epage]

This is now the responsibility of assert_cmd and predicate-rs which supports this.

assert_cli is being morphed into a higher-level crate that makes up at least assert_cmd and assert_fs. See #41