Well, crud. Overdrive changed their site and hosed me. Previously, they had a page on their site that allowed you to see all of their libary sites/urls. I wrote a quick snippet of code that would go through all of those pages and grab all the library urls (the last run of the code showed 2286 library sites). My plan was to run that code once a week or so to get the current, up-to-date list of Overdrive sites that OverReader could query against.
Now, Overdrive changed their libraries page to be a map and you have to search for libraries near you. So there’s no obvious way to get a list of ALL libraries. Luckily, I have the list from the last time I ran it and I can seed the DB with that, but now I have no way of getting the current list of valid Overdrive libraries. So, if a library goes away (does that happen?) or a new one comes online, my code won’t be able to know and can’t query against it. The only solution I can think of is to make it easy for the users to let me know when their Overdrive library doesn’t work on OverReader. That’s a terrible long-term solution, but it’s all I have for now. Maybe something will come to me later.
In the last blog post, I found that the queries on the server were slow. I figured out why… for some reason, on the Linode server, some of the connections to the Overdrive server just don’t connect and the default in the apache http client is 2 minutes so it sits there for 2 minutes, fails, then retries and makes the connection. So that’s why it’s slow. Not sure why the connection fails so frequently on the Linode server when it hasn’t ever happened locally. The good news is, it seems to connect fine when it does the retry. Maybe I can just set the timeout to something lower (10 seconds?) and not worry about it too much? Maybe it’s worth contacting Linode though and see if they know what’s going on.
I got the API code working and in place. Works great on my local machine. Takes about 20 seconds to query 100 books (and I cache those results in the local DB for a month so subsequent searches of the same book at the same library will be fast). But… when I deployed it to my server at overreader.com, it’s really SLOW. Takes 2-4 minutes to do the queries. (but if you immediately do the same search again, it’s all local data so it takes ~2 seconds) I’m going to put in some more debug timing in it to see if I can figure out where the slowdown is on the server. I haven’t multithreaded it, that could help but I want to understand why it’s so slow when deployed before I multithread.
I’m half tempted to just do a weekly (or monthly) query of their entire DB for every library and just store that locally then all user queries would just be local queries and super fast. Not sure if Overdrive would appreciate me grabbing their entire DB though.
Maybe the timing info will help me figure out where the slowdown is though. I’m using linode.com as my host and it says I have “125 Mbit Network Out” but maybe that’s too slow? I wouldn’t think so.
On the bright side, it now queries for the book in both ebook and audiobook format and shows icons and links to each type.
We’re up to 11 testers now. WooHoo! Right now I’m just running the searches for them and sending them a link to the results. Soon I’ll let them loose on the site so they can try it out. I cleaned up the pages a bit the other day so they look better. I still need to do some backend optimmization. I also need to setup a new host. I decided to pay for some hosting, at least at first, because any amount of traffic would probably take down my free host I am currently using and I don’t want people’s first exposure to the site to be errors.
When I first started coding OverReader, I saw that Overdrive had an API I could use. For those that don’t code, an API is just an easy way to get info from them and it’s made to handle lots of traffic. I applied for access to their API and shortly after read that access is only given to official developers and partners (I am neither). So, I assumed I wouldn’t get access and I made the code work another way (without API access).
Fast forward to today and I get an email from Overdrive saying I was accepted into their developer program and now have API access! I’m not sure if it was a mistake or if they really liked my application or what. I’m still hesitant to totally embrace it since they could revoke access at any time. I’m going to keep my code that does things “the hard way” around just in case they do revoke the API access.
I poked around a bit in the APIs and I think I know how I can leverage them to make OverReader better/faster. Stay tuned!