Well, crud. Overdrive changed their site and hosed me. Previously, they had a page on their site that allowed you to see all of their libary sites/urls. I wrote a quick snippet of code that would go through all of those pages and grab all the library urls (the last run of the code showed 2286 library sites). My plan was to run that code once a week or so to get the current, up-to-date list of Overdrive sites that OverReader could query against.
Now, Overdrive changed their libraries page to be a map and you have to search for libraries near you. So there’s no obvious way to get a list of ALL libraries. Luckily, I have the list from the last time I ran it and I can seed the DB with that, but now I have no way of getting the current list of valid Overdrive libraries. So, if a library goes away (does that happen?) or a new one comes online, my code won’t be able to know and can’t query against it. The only solution I can think of is to make it easy for the users to let me know when their Overdrive library doesn’t work on OverReader. That’s a terrible long-term solution, but it’s all I have for now. Maybe something will come to me later.
In the last blog post, I found that the queries on the server were slow. I figured out why… for some reason, on the Linode server, some of the connections to the Overdrive server just don’t connect and the default in the apache http client is 2 minutes so it sits there for 2 minutes, fails, then retries and makes the connection. So that’s why it’s slow. Not sure why the connection fails so frequently on the Linode server when it hasn’t ever happened locally. The good news is, it seems to connect fine when it does the retry. Maybe I can just set the timeout to something lower (10 seconds?) and not worry about it too much? Maybe it’s worth contacting Linode though and see if they know what’s going on.
I got the API code working and in place. Works great on my local machine. Takes about 20 seconds to query 100 books (and I cache those results in the local DB for a month so subsequent searches of the same book at the same library will be fast). But… when I deployed it to my server at overreader.com, it’s really SLOW. Takes 2-4 minutes to do the queries. (but if you immediately do the same search again, it’s all local data so it takes ~2 seconds) I’m going to put in some more debug timing in it to see if I can figure out where the slowdown is on the server. I haven’t multithreaded it, that could help but I want to understand why it’s so slow when deployed before I multithread.
I’m half tempted to just do a weekly (or monthly) query of their entire DB for every library and just store that locally then all user queries would just be local queries and super fast. Not sure if Overdrive would appreciate me grabbing their entire DB though.
Maybe the timing info will help me figure out where the slowdown is though. I’m using linode.com as my host and it says I have “125 Mbit Network Out” but maybe that’s too slow? I wouldn’t think so.
On the bright side, it now queries for the book in both ebook and audiobook format and shows icons and links to each type.
We’re up to 11 testers now. WooHoo! Right now I’m just running the searches for them and sending them a link to the results. Soon I’ll let them loose on the site so they can try it out. I cleaned up the pages a bit the other day so they look better. I still need to do some backend optimmization. I also need to setup a new host. I decided to pay for some hosting, at least at first, because any amount of traffic would probably take down my free host I am currently using and I don’t want people’s first exposure to the site to be errors.
We’ve got a nice handful of testers now that I have shown the site to and what it does/how it works. Feedback has been very positive. I’ll be letting them try out the site for themselves soon (instead of running it for them and sending them the results).
OverReader needs some people to try out the site and give us some feedback. If you use Goodreads to track books you want to read and you check out ebooks from your local library via Overdrive, you’d be the perfect candidate to help test.
All you need is your Goodreads id and your local library’s Overdrive URL to use OverReader. No login or signup is needed. If you’d like to try it out and give some feedback, let us know.