ryan.dens
2020-02-19 00:22
This is definitely an area of interest for my team. We fit the paradigm of ?a service [that] is used by a lot of consumers.?. We haven?t experienced any real problems here, but I?m sure there?s some benefit for us. Our typical workflow today is:
1. Endpoint is specified in a shared repo with the OpenAPI spec. teams delibate on what should be included in the endpoint (body, headers, path, etc) and then the result is merged.
2. Provider implements the endpoint
3. Consumer(s) write client code with pact tests for the interactions. Contracts are published on feature branches and verified by the consumer, and then the client code is shipped.
4. Consumer code which exercises the client code is built and shipped
often steps 2 + 3 happen in tandem. Bugs usually get found in step 3, but occasionally step 4. If its found, the consumer team typically figures out who implemented the spec wrong, so the advantage of having cases generated from the spec would be the clarity about who was wrong. that being said, it?s usually pretty clear and not a major issue for us.
detert
2020-02-19 06:32
Hi all. Great idea. We use Swagger to document our endpoints. I think it would be a great benefit, if Swagger could be combined with pact
matt.fellows
2020-02-19 11:14
Thanks Ryan. Did you read the blog post mentioned in the canny feature request? It articulates some of the key challenges with the ?Consumer driven? aspect fairly well
matt.fellows
2020-02-19 11:15
I suspect it also translates to other protocols too - e.g. gRPC/Protobufs where the ?remote procedure call? is provider driven
matt.fellows
2020-02-19 11:15
This thread is about getting a conversation around all of the things though - so thanks
bernardoguerr
2020-02-19 11:29
Regarding the issues with OpenAPI Swagger, one thing I've seen work for companies that kind of auto-solves some of the issues that Ron listed in his blog post (https://pactflow.io/blog/the-curious-case-for-the-provider-driven-contract/) is this:
The service uses some sort of middleware (e.g. for Express https://www.npmjs.com/package/swagger-express-validator ) that verifies the requests (and responses if you want) against the OpenAPI/Swagger specification. Now, they get direct feedback if they forgot to update the OpenAPI/Swagger docs (for example, their tests should in theory break), and therefore the OpenAPI docs are kept more up to date.
Now of course this isn't on Pact's side to implement, but it is food for thought and could be considered an idea for best practices.
ryan.dens
2020-02-19 13:45
yes i did, it was very helpful to formalize/generalize some of the challenges we are facing! Ultimately, even when using code gen tools which generate client code from the spec, we wouldn?t have the confidence that everything was implemented correctly without a real test. Pact has provided us a great compromise, allowing us to confidently collaborate together on a spec before moving to implementation. Consumer driven contract tests also represent a paradigm shift for our product and engineering teams, and our product development lifecycle hasn?t yet changed to take advantage of some of these optimizations.
matt.fellows
2020-08-07 01:24
Keen to get feedback from anyone using OAS/Swagger and would like to see it working better with Pact https://github.com/pact-foundation/pact-specification/issues/76
heytaco
2020-08-25 03:57
Hi there! My name is HeyTaco!, and you can use me to give people tacos to show your appreciation. My tacos will spread joy through Slack!
matthew.schaad
2021-07-13 21:40
hello Swagger-loving Pact practitioners! I've got an idea for combining OAS and Pact, and I want a sanity check. Is this the right room to discuss it?
matthew.schaad
2021-07-13 21:52
(assuming internal-to-my-company services, so Pact would actually be useful)
matthew.schaad
2021-07-13 21:55
1. provider services publish an OAS spec.
2. consumer tooling downloads the OAS spec and presents it to the user for down-selection (filtering). Tool outputs the filtered OAS spec.
3. additional consumer tooling generates client code from the filtered OAS spec, and also translates the OAS to a Pact. Now you don't have to write the pacts manually, and it's impossible for the client and the pact to get out of sync.
4. Auto-generated Pacts are tracked in Pact-Broker.
matthew.schaad
2021-07-13 21:56
now providers can leverage Pact-Broker for reasoning about whether a given API change is OK, or find out who they need to talk to to make it OK.
matt.fellows
2021-07-14 06:20
I'm not seeing how it would help the provider though, because if I understand correctly, the premise is that the OAS the provider generates is their contract and the consumer's must abide by it. Would you still have the provider verify the pacts? If not, how do you ensure they don't drift?
I.e. is it possible for a provider to release a new OAS without the consumer yet pulling in the new one?
matthew.schaad
2021-07-14 13:50
fair question! The paradigm is still Pact-style, which is Consumer-driven. OAS as a technology is a means to an end.
matthew.schaad
2021-07-14 13:51
since the OAS ecosystem is more mature, it's convenient to use it for codegen and documentation.
matthew.schaad
2021-07-14 13:52
Since we're dealing with internal services, though, we can insert pact-broker and mandate that providers are _required_ to make sure that API changes are backward-compatible with all existing consumers.
matthew.schaad
2021-07-14 13:52
Do the providers verify the pacts? Absolutely. It's part of our CI process, as you'd expect.
matthew.schaad
2021-07-14 14:23
sounds like there's a lot of overlap between "bi-directional" approach and what I want to do.
matthew.schaad
2021-07-14 14:23
oh, I forgot to answer this question:
`I.e. is it possible for a provider to release a new OAS without the consumer yet pulling in the new one?`
matthew.schaad
2021-07-14 14:27
absolutely. And that's desirable. For API additions, we don't want consumers to consume the new parts just because they're _there_. Consumers should depend on the minimum they need to give providers maximum flexibility for change.
For API removals, the Providers are expected to have verified their proposed changes against the published pacts to guarantee that the changes are safe. The consumer has no need to regenerate their clients.
matthew.schaad
2021-07-14 14:28
So again, I did read up on bi-directional, and there's a lot of overlap with what I want to do.
matthew.schaad
2021-07-14 14:30
I looked at the trade-off matrix and thought it was interesting. The page doesn't give the _reasoning_ for each of the scores it gives, so I don't quite get it yet. In particular, the "guarantees" box for bi-di only gives a rating of "+".
matthew.schaad
2021-07-14 14:30
I wonder if you would say the same about my proposed approach? And if so, why?
matt.fellows
2021-07-14 22:21
your approach would have better guarantees, it would have the same guarantees that Pact has
matt.fellows
2021-07-14 22:22
The upside of your approach is the OAS is more tightly integrated into the consumer side
matt.fellows
2021-07-14 22:22
bi-directional doesn?t use Pact, it essentially compares schemas - which loses some level of fidelity - hence why the trade off matrix marks it down
matt.fellows
2021-07-14 22:22
(good point about reasoning, i?ll add that to the list of things to improve on)
matt.fellows
2021-07-14 22:23
> For API additions, we don?t want consumers to consume the new parts just because they?re _there_. Consumers should depend on the minimum they need to give providers maximum flexibility for change.
:100:
matt.fellows
2021-07-14 22:23
I was just fishing to work out where the mechanism to prevent drift came in - you clarified that (Pact)
matt.fellows
2021-07-14 22:26
So in your case, given that you?re still using Pact, is the main objective just to get better visibility of the OAS?
matt.fellows
2021-07-14 22:27
One option is to leave Pact as is, and use a tool like https://bitbucket.org/atlassian/swagger-mock-validator/ to ensure that the consumer contracts don?t drift from the Pact implementation. Not a perfect fit, but might strike the balance you?re after
matthew.schaad
2021-07-15 14:03
I realize now that there are so many assumptions (especially of motivation) that I've made in the design of this thing, it is difficult to grok! So, thank you for all the questions.
As a group (in my org) we decided we wanted to do spec-first API development. There are several reasons for that: (1) we actually want teams thinking about what's going across the wire, since that's the _actual_ interface to the application; (2) so far the design of our APIs are inconsistent and devs don't have visibility into what already exists; (3) I need to review the API designs myself (see #2), and specs make that easier, especially when they're designed _first_; (4) specs are amenable to code generation (especially for clients, but for servers too), which is not only faster than manual development, but makes it easy to put standard practices in place (such as the DTO pattern, especially on the server side; and the "it's just a Java adapter to the HTTP API" principle of client authoring).
(Whew! That's a lot of motivations.)
matthew.schaad
2021-07-15 14:04
OAS is the shoo-in technology for provider-side specs, so that's what we've chosen.
matthew.schaad
2021-07-15 14:05
(And just to say that we've chosen OAS as our technology for service _description_ does not mean we've chosen a _classic contracts_ approach. On the contrary, we've selected CDC instead.)
matthew.schaad
2021-07-15 14:07
As for compatibility strategy, we've selected CDC over classic provider-driven contracts. The shoo-in technology for that is Pact. (And pact-broker does a ton of stuff I want, including making provider verification of CDCs possible.)
matthew.schaad
2021-07-15 14:08
So here I am, motivated to choose _two_ contract technologies, with an uncomfortable amount of overlap between them.
matthew.schaad
2021-07-15 14:08
I've already told you my solution, though: I choose both, and I'm constructing an adapter between them to make it all work.
matthew.schaad
2021-07-15 14:12
thank you so much for poking at my idea! I felt like I needed someone with greater expertise in the CDC paradigm to provide scrutiny.
matthew.schaad
2021-07-15 14:13
As for _swagger-mock-validator_: I was unaware of this tool, but I see a place for it in my tech strategy as services begin to proliferate. Thanks for the pointer!
matt.fellows
2021-07-16 01:44
Thanks, that all makes sense to me. Combining technologies to get the best out of both (and dealing with overlap/trade-offs of using both)
matt.fellows
2021-07-16 01:45
I think you could get a lot from dropping the extra work of trying to get the subset of the OAS into the hands of the consumer, and just generating clients from the OAS itself, then using Pact to test the actual consumed interface
matt.fellows
2021-07-16 01:46
the swagger mock validator will then give you extra confidence things are working albeit if you are generating contracts from an OAS file itself, I?m not sure what chances are they?ll drift?
tjones
2021-07-18 14:22
My personal view is that spec-first API development is kind of an anti-pattern, because I strongly agree with this point you made:
> we actually want teams thinking about what's going across the wire, since that's the actual interface to the application
This is why I prefer consumer-driven API development - servers exist to serve, so why not let the client decide what interface is most convenient?
tjones
2021-07-18 14:23
(of course, plenty of teams find spec first works for them, and clearly CDC isn't for everyone, because what do you do if you don't know who your consumers are?)
tjones
2021-07-18 14:24
> you could get a lot from dropping the extra work of trying to get the subset of the OAS into the hands of the consumer, and just generating clients from the OAS itself, then using Pact to test the actual consumed interface
This is an excellent approach. Most teams who aren't using tools like Pact are providing the same value manually, by (eg) creating postman collections that contain example request / response pairs.
matthew.schaad
2021-07-19 14:54
> dropping the extra work of trying to get the subset of the OAS into the hands of the consumer
Hmm, are we talking past each other? Not sure I follow this comment. I want to get the original OAS into the hands of the consumers, and let them filter the OAS on their own and generate clients from it. At that rate, the OAS given by the provider is like a menu of what's available, and then consumers advertise "I'll have the `POST /widgets` please, with a side of `GET`."
Anyway, maybe you understood what I meant already, and you were just trying to point out that we could skip the part where each team has to generate its own client; rather, there can be one client to rule them all given by the provider, and we can just let Pact do its work on the consumer side.
Technologically, that will totally work. Culturally--for me--it won't. Why not? Because my teams don't have the discipline already of maintaining Pacts, and the disparity between what's in the Pact and what's actually being consumed is a constant problem to be managed. Basically, I don't think they'll do it. But if I push the work of client generation to consumers, I can tie actual consumption to the Pact directly, and my people-problem goes away.
Another way to think of it is this: I need architectural guardrails that enforce the guarantee that Pacts match actual consumption patterns, because I can't rely on culture or process yet.
matthew.schaad
2021-07-19 14:54
Also: as a bonus, no teams will be tempted to write clients by hand anymore, because everyone will have embraced codegen. :slightly_smiling_face:
matthew.schaad
2021-07-19 14:56
Thank you again for the scrutiny. I appreciate the technical scrutiny and being forced to articulate my actual problem-set.
connor.beck
2021-09-02 20:47
Hi! I'm trying to figure out if bi-directional contracts will add anything to our system confidence when used in a particular situation or if we've already kind of covered everything and so it won't add much.
We have a typescript/angular client and a java server. We take an API first approach where we define all of our models and APIs in an openAPI spec yaml file, then we use that file to generate models and interfaces (which must be implemented) for the server, as well as models and fully-functioning services for the client. The client can use these services and models to make requests to the server without any extra configuration.
On the client side, we use Cypress. Currently, these tests do not replace the server with a test double (and so are effectively E2E tests), but we're looking to change that so that some tests will continue to be E2E and the rest will focus exclusively on the client, stubbing out the server. The easy way to do this, it would seem, would be to use cy.intercept and such, probably making use of the same generated models that the implementation code uses.
Lets say we wanted to use Pact Bi-directional contracts to verify the integration between our client and server (and lets assume there was an adaptor of some sort to allow using cypress stubs instead of making pact tests). What exactly is this feature adding in this scenario? We have compile-time verification that both the implementation and the tests are compatible with the provider schema, since they use code generated directly from the schema. Thus, `can-i-deploy` would never fail because we would know well before it's called that the two were incompatible.
Is there still some value I'm not seeing in bi-directional contracts in a setup like this? I see some value in the standard Pact approach, since you're replaying the consumer requests against the server and so getting a better guarantee, but then we have to go back to writing a full pact-based consumer unit test suite instead of harnessing our Cypress stubbing (or we reverse it and have our unit tests drive the stub server, but that raises it's own issues).
matt.fellows
2021-09-03 06:27
IIRC this might be you https://stackoverflow.com/questions/68760203/using-pact-with-a-generated-client-api-service ?
matt.fellows
2021-09-03 06:29
In addition to the comments there, this statement was curious to me
> as we already have deployment pipelines in place to deploy simultaneously
This reminds me of the SOAP days. The API provider generated a SOAP client (axis/xfire with Java in our case) and gave it to the consumers (us). Version control of that was usually iffy, but that has more solutions these days.
but the problem was that at deployment time, you had to coordinate a release (potentially going into ?maintenance? mode), else all hell broke lose
matt.fellows
2021-09-03 06:30
That?s enough for me personally to want to have protections, because now I know if it?s safe to roll forward back, and can do it *independently* of the other collaborator(s).
connor.beck
2021-09-07 12:55
We kind of have steps in place to address that as well. We have a configuration file that basically lists all of the dependencies of each microservice, including it's consumers. Then, when you push to a branch and open a PR, we set up a Canary environment specifically for that PR. In the canary environment, we deploy the microservice under test, all of it's dependencies, and everything else is just a copy/reference to services running in our standard development environment, but reconfigured at the API Gateway level (I think? not sure about this part) to redirect requests that would normally go to the development version of the microservice to the canary version.
connor.beck
2021-09-07 12:56
Then, when we merge, we basically just promote canary to be the new development version (with a merge queue to prevent issues with overlapping merges)
connor.beck
2021-09-07 12:58
I'll be honest, I probably got at least some, if not most of that wrong, I'm only familiar with the process at a pretty high level. However, at this point, I just sort of need to trust that it's working as intended. This is all a bit of a work in progress as well, but the intention is not to have any downtime or "maintanance mode" worries.
matt.fellows
2021-12-16 20:32
Can you please elaborate on what you are trying to do? There are plenty of OSS tools that turn an OAS into a stub if that what you're asking?
bernard
2022-02-15 17:08
:wave: hi I've just joined... Great to see a growing community here. I'd like some advice and help to build a POC. I'd like to generate a Pact file using OAS (Swagger). And I was told to reach out for help here on the Pact docs website.
matt.fellows
2022-02-15 21:31
Hi Bernard! We don?t have a way to generate a pact file from an OAS (that would be a https://docs.pact.io/faq/#can-i-generate-my-pact-file-from-something-like-swagger)
matt.fellows
2022-02-15 21:32
In any case, Pactflow has a new feature (in dev preview) called https://docs.pactflow.io/docs/workshops/bi-directional. It allows you to use the OAS as a provider contract, but the consumer still needs to publish a consumer contract. This can be via a standard Pact test, or by converting existing mocks into a pact file. This way, you still know what the consumer needs, and Pactflow can ensure the provider meets those needs.
diva.pant1
2022-02-15 21:52
Hi @matt.fellows I have a question, so basically we create the pact at the consumer side publish it to the pact broker, and the provider just verifies that contract right? (ofcourse we create the provider states) but only one contract is created which is verified by the provider.
And in bi-directional flows we create contracts at both sides?
matt.fellows
2022-02-16 03:15
bi-directional is more decoupled, each side produces their own contract and we just ensure they are compatible
bernard
2022-02-16 09:25
Hi @matt.fellows referring to the *bad idea* link I actually meant to "_generate skeleton Pact test code from a Swagger document_".
matt.fellows
2022-02-16 10:17
I guess it could be done, but then we?d need to do that for each language, and then for each test framework. I?m not convinced of its utility, personally.
matt.fellows
2022-02-16 10:18
Also, you then would need to remove any parts of the OAS you?re not using, so I think it?s likely to lead to consumers inadvertently requesting more of what they need of a provider
matt.fellows
2022-02-16 10:19
But I?m open to a PR or something that could do it - I don?t think it would be terribly difficult. We?re always very conscious of creating more tools, because that means:
1. More code to support and bugs to fix, across (currently) 10+ languages
2. More documentation/tutorials/workshops etc. needs to be written, managed and updated
3. More components to consider when adding new major features to Pact
So it?s something we?d want to be really sure about!
matt.fellows
2022-02-16 10:20
What are you using at the moment to mock the provider APIs in your consumer code bases? There is potentially _another_ way
bernard
2022-02-16 13:26
Hi @matt.fellows we're at POC stage right now. I spoke to your colleague earlier this morning (GMT) and I think all the above can be avoided if I look into https://docs.pactflow.io/docs/workshops/bi-directional. Is that right from my understanding?
I'm currently using the Pact consumer/provider GitHub example. I've completed the CI/CD workshop. And I've added Cypress.
I've applied to your developer program and I'm waiting on a response to my application.
To my understanding, if I'm accepted... There will be information about creating a provider contract using OAS which I can employ to further our POC.
bernard
2022-02-16 13:38
I was looking for information about bi-directional contracts and ended up on the https://docs.pact.io/help/how_to_ask_for_help/ after a Google search.
