|
Re: Stopping tests
by hippo (Archbishop) on Sep 13, 2023 at 10:59 UTC
|
| [reply] |
testing/release advice [was: Stopping tests]
by pryrt (Abbot) on Sep 13, 2023 at 16:39 UTC
|
Sending a dev version to CPAN before there's a released version makes it really hard for us to find your project and the cpantesters results for those dev versions, as kcott mentioned, because the distribution isn't in the publicly-searchable distros. For other readers of this thread, this link will get you to v0.01_4 in metacpan, and if Bod releases further dev releases before a full release, you can use the pulldown from there to look for other dev releases. Similarly, this gets you to the cpantesters matrix for 0.01_4, from which you can get to the other dev-release test results or the log.txt results for tests that have been submitted but not merged into the main interface.
In my personal opinion, doing a dev release to CPAN without an initial release without having at least done your own suite of multi-environment testing borders on abusing the cpantesters network -- if you've don't your own multi-environment testing, and have previously had a successful initial release, then if cpantesters show a strange error in an environment that you don't have access to testing, then I can understand doing a dev release to CPAN before my next release (and in fact I've done that on my projects in the past), but it shouldn't be your first line of defense.
With free Continuous Integration testing available through GitHub Actions and places like Appveyor, there's no reason for not doing a suite of tests long before any release or dev-release to CPAN, on at least Ubuntu and Windows (I haven't done any macos testing, though I forget whether that's just because I haven't tried, or because I've tried but failed; I've still got a personal TODO for freebsd and similar through a GitHub action, because those platforms are often the edge cases for my modules, but I ran into some difficulties, and it's been a low priority for me).
And as a final small piece of advice: always include the META_MERGE in your Makefile.PL (removing it for older versions of MakeMaker that don't understand that syntax), so that people can easily navigate from the metacpan page for your distribution to the repo and issues -- once I did find your results and metacpan entry, I then had to come back here to find your link to your repo.
Coming soon: links to the issues and PR I have to help you understand/implement CI and META_MERGE. The CI PR will come complete with showing your tests failing, presumably for the same mismatches that the cpantesters have been reporting.
update 1 : add links:
update 2 : added "first line of defense" phrase. | [reply] |
|
|
In my personal opinion, doing a dev release to CPAN without an initial release without having at least done your own suite of multi-environment testing borders on abusing the cpantesters network
My understanding was that we should always upload a dev release first!
I thought I got that from Re: What do I use to release a module to CPAN for the first time? but rereading it, that's not where it came from...
The point of this thread was to try to lessen the load on cpan testers, certainly not to abuse them.
Surely, if we upload a production release before a dev release, people can start using it before it is properly tested - or am I missing something here?
there's no reason for not doing a suite of tests long before any release or dev-release to CPAN, on at least Ubuntu and Windows
The tests were created ahead of writing the code. However, one test I couldn't write without the help from Testing image output. I had planned what I wanted to test but couldn't convert that visual test into a programmatic test. v0.01_1 omitted that test, subsequent versions included it and tried to get it to work in different environments. The tests were all run locally using prove and passed...I wouldn't have uploaded a module that failed local tests.
As a very recent user of GitHub, I had no idea I could test modules with it!
And as a final small piece of advice: always include the META_MERGE in your Makefile.PL
It's on my list of things to do...to work out what all these META files are in a distribution...
It strikes me as very odd that we have to run gmake disttest, then go and copy files from one place to another. Surely, the toolchain should do that for us if it always needs doing, shouldn't it? I always forget if I'm supposed to be copying META.yml or MYMETA.yml. I guess that if I understand the significance of these files, it will all be clearer.
Question...
Given where I am with 4 dev releases, what would you advise I do next?
Is it sensible to remove the tests that fail and upload a production release next and then develop it further from there?
| [reply]
[d/l]
[select] |
|
|
My understanding was that we should always upload a dev release first!
The meaning of "first" is my quibble. If by "first", you mean "before I've done a initial public release", I have no problem with that (though I've never done that on my handful of CPAN modules, and don't remember seeing that advice). If by "first" you mean "before I've thoroughly tested my distribution", I would vehemently disagree with that understanding. :-)
if we upload a production release before a dev release, people can start using it before it is properly tested
Those who want a pre-release can install a pre-production module from GitHub, it doesn't have to be on CPAN for that.
However, one test I couldn't write without the help from Testing image output. I had planned what I wanted to test but couldn't convert that visual test into a programmatic test.
And CI would have helped you figure that out, even if your own machine isn't showing the error. Another vote in favor.
As a very recent user of GitHub, I had no idea I could test modules with it
Hence my post. :-) And hence my issue and PR, to help you get started with that. I got my start with stevieb's excellent Travis-CI test structure and extended that to AppVeyor (though I don't remember where I cribbed my first .appveyor.yml, sorry), then much later used his GitHub Actions test structure to start converting over to GitHub Actions instead. One usually needs a starting point for such things, unless one is very good at starting from raw documentation.
It strikes me as very odd that we have to run gmake disttest, then go and copy files from one place to another.
I've never had to do that with my gmake dist / gmake disttest . I vaguely remember being highly curious why you needed to do that, the last time you mentioned it, but didn't have time at that point to look into it, and I had probably hoped that someone else would disabuse you of that need. It might not be today, but I'll try to put that on my mental back-burner again. If no one else pipes up in the next couple hours, maybe you should create a new issue on your repo and @mention me over there (maybe even assign me, if it lets you), and that might serve to poke me over time.
Given where I am with 4 dev releases, what would you advise I do next?
Merge the CI environment I gave in the PR, and watch it fail. Then start debugging your test and/or module code until those CI tests are passing. You might want to create an issue in GitHub to track it, and if you're having problems getting it to pass, you could even show your exact steps that you have tests passing on your personal machine in a comment on that issue, so they can be compared to what the CI does.
I would suggest not doing another dev release to CPAN until you have a suite of tests that are working both for you and in the CI. Only after getting those tests passing would I say you should do another CPAN dev release. And given the number of problems you've had, dev release is still probably the best bet, until you've got a wider array of passing results. (I'm normally willing to do an initial full release after my personal tests are all passing, and my remote CI tests are passing, and I've done enough to convince myself that my distribution works well enough that someone in the wild would be able to get it to install and pass and do what my docs say it does -- having 100% test coverage helps give me those warm fuzzies, though it's not required in my personal workflow.)
Is it sensible to remove the tests that fail and upload a production release next and then develop it further from there?
IMO: Only if you are confident that the core features in your distribution actually work right, and that your documentation explains which features are not
fully tested or might not work for others, even though they work on your machine.
I feel really weird giving advice like this. I'm just an amateur with a few CPAN modules that, as far as I know, no one but me uses (okay, I know of one other user of one of my modules, but one real user is statistically zero compared to CPAN norms). I'm not sure why anyone would trust any of my advice. And I definitely make no claim that it's "best practices".
| [reply]
[d/l]
[select] |
|
|
|
|
|
|
|
|
|
|
|
|
|
My understanding was that we should always upload a dev release first!
I thought I got that from Re: What do I use to release a module to CPAN for the first time? but rereading it, that's not where it came from...
Bod you're right, you didn't get it from the most excellent Re: What do I use to release a module to CPAN for the first time? by davido.
And you certainly didn't get it from my Writing Solid CPAN Modules ... because I'm a Dev release ignoramus :)
To remedy that I've just added some extra references to Writing Solid CPAN Modules:
- perlmodstyle (perldoc) - Perl module style guide (re making a dev release: if you want to release a 'beta' or 'alpha' version of a module but don't want CPAN.pm to list it as most recent use an '_' after the regular version number followed by at least 2 digits, e.g. 1.20_01)
- CPAN (perldoc) (FAQ 12) - How do I install a "DEVELOPER RELEASE" of a module? By default, CPAN will install the latest non-developer release of a module. If you want to install a dev release, you have to specify the partial path starting with the author id to the tarball you wish to install, like so: cpan> install KWILLIAMS/Module-Build-0.27_07.tar.gz
- Update (thanks hippo): CPAN::Meta::Spec (perldoc) - specification for CPAN distribution metadata (for release_status, if the version field contains an underscore character, then release_status must not be "stable")
Bod: note the leading zero in Ken's Module-Build-0.27_07.tar.gz ...
I think you meant to use _01, _02, ... instead of _1, _2 in your Dev releases :)
More definitive references on CPAN Dev releases welcome.
| [reply]
[d/l]
[select] |
|
|
|
|
Re: Stopping tests
by Anonymous Monk on Sep 13, 2023 at 11:40 UTC
|
?? I thought GD::png() was shown to be unsuitable. Unrelated to trouble with tests: I understand the desire to use real-life image, neither too small nor on-the-fly generated gradient, etc. As much as "what else did you expect from Microsoft?" are cheap kind of jokes, consider this: e.g.GD::png() and IM's convert (+ many others I'm sure) produce ~1.2 MB PNG from "CoventryCathedral.jpg", why would your users need to download a ~2,3 MB version for what's really a rather simple module, only because you chose Microsoft Paint for conversion? :-) Actually, this extra megabyte for lossless, decades old codec is quite a "facepalm" thing.
| [reply]
[d/l]
[select] |
|
|
I thought GD::png() was shown to be unsuitable
My understanding was that GD::jpeg was the unsuitable method due to the format...
But we are well beyond my understanding of images here...I mostly use GIMP to manually manipulate images and GD to programmatically deal with them. When it gets to the black box of the image itself, my knowledge ends.
Given that Image::Square produces images that are visually correct, I am wondering if it needs the actual image content to be tested. Perhaps just testing the size and aspect ratio of 1:1 is sufficient for this module...
Edit:
Changed link to GitHub repo
| [reply]
[d/l]
[select] |
|
|
ken@titan ~/tmp/bod_tmp/Image-Square-0.01_3/t
$ ls -l
total 2601
...
-rw-r--r-- 1 ken None 2315428 Sep 13 07:29 CoventryCathedral.png
-rw-r--r-- 1 ken None 330858 Sep 13 07:29 decoration.png
...
I'm fairly certain that the point the AM was making was in relation to the size of these test files.
The CoventryCathedral.png image is 1584×625 pixels.
I'd suggest creating an image of maybe 15x6 pixels (or similar small size) for testing;
this would reduce the size of your test data, and hence your module tarball, by some orders of magnitude.
| [reply]
[d/l]
[select] |
|
|
|
|
|
|
| [reply] |