Lagotto is under active development with frequent releases that add new features. The upgrade process is sometimes more complicated than it should be. What can be done to make upgrades easier?
- more automated tests to catch regressions and other hard to detect errors.
- error reporting
- a supported reference platform
- automated updates using Docker or similar tools
- paid support
I am in the middle of transitioning the acceptance tests from
rspec, planned for the next release. This should improve test coverage and will identify some of the nastier bugs.
The built-in error reporting is usually very informative. We could make it easier to report these errors back to me, e.g. with a
create Github issue button next to the error.
The various users of the lagotto software use slightly different platforms: Ubuntu, CentOS or RHEL, physical server vs. VM vs. Cloud, slightly different Ruby versions, etc. Harmonising the platform could make it easier to avoid some hard to detect errors. My preference would be Ubuntu 14.04 because of the support for newer Ruby and Passenger versions via PPA.
Installation via Docker should simplify installation, but would also limit the deployment choices.
Individual user support in an open source project can hit limits. Would a paid support option be something that people would be willing to use?
Thanks for starting this conversation and apologies for being so slow to respond.
Reading through your suggestions I'm thinking a self-contained Docker instance would be an excellent start.
A large problem with installing Lagotto (I've found) is the bleeding-edge dependencies and conflicts. If an official Docker container existed (and it's dependant containers documented), I could remove almost all of the config I've built to get Lagotto up and running and keep it running. It would tie the Docker installation mechanism to Linux servers only - this isn't much of a drawback in my opinion. It may also serve as a reference implementation.
The next big problem (again, for me) is keeping the default configuration of the sources up to date. We have a version of lagotto that we're prepping for release using a fully migrated ALM database, but so many sources are broken because of stale configuration (I presume, from a quick comparison to a stock lagotto). I'm going to have invest at least a day, probably more, going through each one individually, debug it's issues, update with any modified urls, etc ... just writing it sounds exhausting.
Two possible solutions to this:
If the sources could be separated from the main code base they could receive more attention and, so long as they adhered to an interface, could be updated (or not) as often as the real world requires them to be (cool URI's may not change, but not all URI's are cool enough for school). This would allow us to remain on a stable version of lagotto using up to date sources.
A means to load fixture data in json/xml/yaml/whatever format, overriding what is in the database. RoR might already have this feature, otherwise it might be a bit too onerous to develop. This would allow me to load up a clean version of Lagotto, dump the fixtures, exclude the ones I don't need, tweak the others with the right credentials and then just use these to restore configuration to fresh Lagotto instances.
More tests are always welcome. The test commands for rake complete suspiciously quickly. Something that would check the environment and report errors/warning would be welcome. You could require a complete and verified installation before submitting a bug.
While paid support wouldn't be off the books, what would be more convenient is a paid-for hosted solution. However building software so complex and difficult to maintain that it requires a dedicated team to run and manage it (I often feel like this is what my job is becoming) suggests something about the software. I don't think this domain is that complex, however I can't boast your experience in this regard.
Thanks for the lengthy reply. Here are some initial thoughts:
- good to hear that you like the idea of using Docker for Lagotto. I will need to do more thinking as Docker is a fast-moving target, and we are focussing on deployment rather than development. One of several options is to use https://packer.io/ to create a docker container (and possibly other outputs, e.g. an AMI) for every release. Or I move away from Vagrant/Chef and fully embrace the Docker ecosystem.
- I have started to separate out the sources from the rest of the codebase as part of the work on the Push API. This should be done in April, and sources will then talk to the rest of Lagotto via an API call. Among other things, this will allow sources to be updated separately, or to be written in another language.
- the upcoming version 4.0 will no longer use CouchDB. This will make it easier to migrate data from one system to the next, load fixture data, etc. We are already using fixture/seed data for many tables, located in
- let me think about a smarter system to load source configurations that make it easier to upgrade.
- the next release will see a lot of new rspec acceptance tests, bringing the total number of tests to over 1,000. I find tests helpful to quickly see regression bugs, and to cover some of the more obscure use cases.
- a paid-for hosted solution is a consideration, but this requires both a business model behind it, and a better way to get data in and out of the system. I like to see a hosted solution as a service where the publisher has unrestricted access to all data.
Hey Martin, thanks for getting back to me. Discourse didn't alert me you had so I kind of forgot about it.
Separation of sources makes me really happy.
Your point about Docker is something that really bothered me as well. I have solid build code that I couldn't reuse for Docker. I have many projects to maintain and some are migrating to Docker and even then only some environments are migrating to Docker (test, prod) and some will remain as-is (dev). I'm not sure if this is possible within Puppet/Chef, but my way around this using Salt was to build another, drastically simplified state tree for docker environments with each container it's own master with simple tests to determine if other containers have been linked so a base container can be built and then updated when dependant services are detected (db, web server, etc).
It sucks having to maintain two types of installation, but I was able to keep the Dockerfile to just boilerplate, keep the build config within the same project and have the two state branches share the same configuration.
I agree that Docker is both exciting and painful because it is moving so fast. But my guess is that the momentum is just too big to leave much room for provisioning tools such as Chef or Salt - even though you can of course combine them.
My current thinking is to have an optional Docker install ready for Lagotto in the next few months, and to keep the Vagrant/Chef workflow around until I feel comfortable with the Docker workflow for Lagotto.
I agree, and from what I understand my approach isn't terribly idiomatic, but the moment you jump into a container and do an
apt-get update or whatever, that feeling of a pristine ecosystem with unwritten rules of conduct goes out the window and all you want to do is just make it work.
I honestly think applications should be built with containerization in mind. Nix has some interesting ideas: http://nixos.org/