Day 3: Upgrades and Stack Traces

Today, I bought a virtual private server for everything plantdata.io that I’m not planning on running out of a container, including this blog. I had been running it all on shared hosting. It took a bit to move everything over to the VPS, but things should be moving noticeably faster as a result. I had hoped it would be a bit longer until I had to upgrade my hosting, but I noticed it was occasionally taking longer than 10 seconds to load simple pages or contact the mediawiki api. Not only was it getting on my nerves, with response times like that, it would certainly cruise straight past updater script timeouts. This should be far less of a problem with the new VPS.

Once that was completed, I got back into the minor explosion I created at the end of Day 2 (which was not technically yesterday). I have so far been unable to get the containerized wikidata query service to pull data correctly from plantdata’s wikibase instance, but I think I know why that is now: Docker is doing some things I wasn’t expecting. This makes sense, as my expectations for what Docker does and does not do are roughly three days old at this point. It would be pretty weird if I already knew all the surprises after three days.

I did manage to catch a pretty good ‘gotcha’ on behalf of the crowd hoping to do similar things with a containerized wdqs pointing to an existing wikibase instance. Running runUpdate.sh in verbose mode revealed that the script assumes you’ve configured your wikibase instance to run at [domain]/w/. So, if your mediawiki api is configured to live somewhere other than [domain]/w/api.php, you will either need to do a redirect on the wikibase end, or hack your configured directory structure into runUpdate.sh. After reading some documented reasons why you probably don’t want to run mediawiki straight from the web root (it was), I opted to change the mediawiki configuration to match the script.

Tomorrow’s Plan is identical to yesterday’s plan in every way, including the part where it’s clearly a multi-day plan that couldn’t fit into a single day, no matter how great of a day it ends up being:

  • Continue learning things about docker, presumably by exploding and unexploding all the test containers until I stop being surprised by its behavior.
  • Finish writing my own compose file to forego the containerized wikibase instance, and instead point all the query service containers to my real pre-existing plantdata wikibase install
  • Verify that the instances are communicating the way I think they should be, or learn enough to alter my expectations
  • Start populating data in the real plantdata wikibase instance (Data In)
  • Get comfortable with SPARQL and write some sample queries (Data Out)

Day 2: Test-driving the Tools

Today was spent largely playing with docker again, trying to debug the issues I was having with the wikibase and wikidata query service test instances I started playing with yesterday. I am pleased to report that I did manage to fix the problems I was having yesterday in getting the query service to import data from the adjacent wikibase install. Turns out the data wasn’t loading, because the timestamp of the newest data was older than a timestamp on the updater system. I got a lot of this in the updater container output:

20:51:47.216 [main] ERROR org.wikidata.query.rdf.tool.Update - Error during initialization.

wdqs-updater_1   | java.lang.IllegalStateException: RDF store reports the last update time is before the minimum safe poll time.  You will have to reload from scratch or you might have missing data.

Thanks again to the same very helpful expert from yesterday (Yay again, Stas!), I was able to fix this problem by running the runUpdate.sh script once with an additional -s parameter to reset the timestamp to something older. That way, it would rebuild more of the data, get a new timestamp from the latest update, and stop complaining in general.

To further complicate things, I quickly ran into another wall: I couldn’t reliably run a docker command on the updater container. Docker exec commands kept complaining that the container was in the process of restarting and advising me to try again, despite docker-compose ps telling me consistently that the container was up. I stopped and restarted that container and the whole set several times, and kept getting the same results. Instead of continuing to fight that one container, I ran the following command on the main wdqs container instance, which was stable and based on the same image as the updater container:

docker exec wikibasedocker_wdqs_1 ./runUpdate.sh -h http://wdqs.svc:9999 -- --wikibaseHost wikibase.svc --wikibaseScheme http --entityNamespaces 120,122 -s 20180301000000

Fixed! This enabled me to spend some time today basking in the utter confusion of trying to understand what on Earth SPARQL is all about, but with data behind it this time. The fact that this is only Day 2 means I am pretty far ahead of the general schedule of events I had vaguely guessed at before I actually started poking around with these tools. I may even be in a place where I can usefully start importing data to the production wikibase instance before this week is out.

It’s probably worth noting, though, that the updater container being flaky and restarting all the time is probably something I’ll have to revisit later. I can certainly imagine that the instability in that container was what caused the timestamps to go funny in the first place. Then again, this may be completely expected behavior for that container, but it certainly does seem suspicious.

I finished out the day starting to convince the compose file to have the query service look at the production location of the plantdata wikibase, and got just far enough to generate several pages of errors to dig into tomorrow. Nothing like finishing with a small explosion.

Tomorrow’s Plan:

  • Continue learning things about docker, presumably by exploding and unexploding all the test containers until I stop being surprised by its behavior.
  • Finish writing my own compose file to forego the containerized wikibase instance, and instead point all the query service containers to my real pre-existing plantdata wikibase install
  • Verify that the instances are communicating the way I think they should be, or learn enough to alter my expectations
  • Start populating data in the real plantdata wikibase instance (Data In)
  • Get comfortable with SPARQL and write some sample queries (Data Out)

Same as yesterday: This plan is clearly too big for one day. If I can manage to successfully import data from outside the docker setup to the containerized wikidata service, I will be positively delighted.

Today’s most useful link:

https://github.com/phusion/baseimage-docker/issues/319 – handily explained some warnings I kept running into while building things in docker. Looks like we’re extending a base image with the same issue being described in that thread, but they’re just warnings (which I believe strongly in addressing vs ignoring), they are pretty easy to fix with one line of code, and they didn’t seem to be breaking anything on their own. It’s just… you know. Less red in the compose output means fewer immediate mysteries to follow up on.

Day 1: The Beginning

Today was the first day of a two-month sabbatical that I am taking, in order to get this plantdata project off the ground. I have decided that every day I work on this, I will write a short blog post outlining what I accomplished during that day, things I learned, and next steps. I’m doing this for two reasons:

  1. While I have spent most of my adult life as a full-time coder, at some point not too far away that majority share will be in *managing* full-time coders. It’s been a long time since I’ve had the opportunity to focus on building something, and to put it bluntly, I am so rusty I can actually hear creaking noises sometimes. I’m hoping that keeping decent notes on a schedule will help me get back into the game.
  2. Shame, really. Shame as a motivator. I fully expect to spend about two weeks flailing wildly with very little to show for it, and having to tell everyone about the whole thing should force me to document mental progress better than I would if left entirely to my own unobserved flailing.

I have decided to start this journey with an investigation of the relatively recent docker images and compose work that’s been going on. I knew it wasn’t going to be entirely smooth sailing for me, as I have never before used docker, or successfully installed blazegraph on anything. Nevertheless, I was able to use docker-compose to spin up some instances in a couple hours.

One early takeaway: Good grief, the Wikidata Query Service needs a lot of memory to start up! It wouldn’t run cleanly until I upgraded my docker box from 4GB memory to 8GB. This also doubled the monthly cost of running this little experiment with my web host, but… /me shrugs

Once my test box had a sufficient amount of RAM, the compose command ran cleanly, and I could load the frontends of both the containerized wikibase instance, and the containerized wikidata query service. I did not expect to get that far before lunch. Unfortunately, after lunch it rapidly became clear to me that the wikidata query service wasn’t *quite* connected up to the wikibase instance: Confusingly, the typeahead in the query helper UI could get objects and properties, but no data was ever returned upon running an actual query.

SPARQL isn’t exactly something I’m comfortable with either at this exact moment. Not knowing if it was misconfigured machines or my own inability to write a well-formed SPARQL query, I called in some expert help who was very helpfully watching his email (thanks, Stas!).

For the readers also uncomfortable with SPARQL, here’s an easy query to test if your wikidata query service is actually talking to anything or not:

SELECT * WHERE { ?x ?y ?z } Limit 10

Turns out that even though the query helper typeaheads work like everything is wired up correctly, my containerized wdqs instance isn’t loading data updates from the adjacent wikibase container. I destroyed those containers and remade them just for fun (isn’t that what containers are for?), and while it did not magically fix the issue, it was genuinely entertaining for a minute.

I did clear up a misconception I’d been carrying with me for a while: The query service gets its data by monitoring the recent changes feed on your target wiki. And here I’d been thinking there was some kind of db dump and import on a cron job I’d have to set up eventually. I’m honestly a little surprised that’s not the case, and now I’m wondering what options exist for recovery/rebuilding if your wdqs instance walks off into outer space…

Tomorrow’s Plan:

  • Learn everything about docker. Particularly, if there is a nice way to have containers land their logs somewhere easily accessible. But also everything else.
  • Get the test containers in the wikibase group to talk to eachother the way they are supposed to
  • Write my own compose file to forego the containerized wikibase instance, and instead point all the query service containers to my real pre-existing plantdata wikibase install
  • Verify that the instances are communicating the way you think they are
  • Start populating data in the real plantdata wikibase instance (Data In)
  • Get comfortable with SPARQL and write some sample queries (Data Out)

I’ll be delighted if I accomplish two of those six things tomorrow.

Today’s most useful links: