Wednesday, October 28, 2009

Using Mendeley effectively on multiple systems using an external storage drive

If you do not already know, Mendeley is soon to become the defacto standard for storing, indexing, searching, ordering and sharing all your academic research papers. Its free download and easy to setup. Check the website to get a feel for it. I will not describe Mendeley here but will get to my point.

WARNINGS

The solution presented below worked for me. It may NOT work for you. There is destruction of state involved so please go through the complete post and decide for yourself. I am making certain assumptions about this solution, so make sure your assumptions match mine before trying this out.

Assumptions:
  • My environment is Mendeley Desktop v 0.9.4.1 running on Ubuntu 9.04.
  • I assume good familiarity with Linux and esp. Ubuntu.
  • I assume you have 2 working Mendeley setups on both systems. Will still work if you dont have but all steps may not apply.
  • I assume that you have access to the original data that you indexed with Mendeley.
  • The solution may break with future releases of Mendeley esp. if they change some of the document paths.
  • This solution may not easily port to Windows/ MAC though i believe the line of reasoning should still apply.
  • I have used Mendeley possibly since its first release and i am quite familiar with its interface / settings etc.

My Problem

I have 2 Ubuntu 9.04 systems (home and work) that i use for my research and i spend equal time on both of them. So i end up storing a lot of papers (my total collection is about 5000) on both. Once i discovered Mendeley (about 5-6 months back), i would index documents on both my systems separately and then sync up the bibliographies (and corresponding metadata like notes) with Mendeley Web (which is the online component of the Mendeley system). Now, though i had access to the full bibliography on both my systems, i could not access the corresponding documents on both systems i.e. i could only access documents on the system on which they were indexed.

Mendeley currently gives an online account with 500MB storage and allows storing documents in it. My collection is somewhere about 5GB and its not worth syncing that much anyway esp. since some of it is of proprietary nature. But i desperately needed a solution since i am so used to using Mendeley now and need it whenever i am doing my work.

So this is what i did:

Initial Setup
  • I got a 120GB external harddrive and formatted it with ext4 (filesystem doesnt matter).
  • Then I set the properties of the external drive to always mount as /media/extstor2. This ensures that we always have a constant path prefix whenever you attach the drive. This can be easily done by clicking properties of the drive, selecting Volume tab and fixing the mount point settings (works on Gnome).
  • Created two folders called db and papers on the new drive.
mkdir /media/extstor2/db
mkdir /media/extstor2/papers
On my home system
  • I first reset my complete database. WARNING: This will completely destroy the database (i.e. your bibliography and notes but NOT your documents). I hate this step but this was necessary because mendeley stores absolute paths to all its documents (you can dig into their sqlite3 database and see for yourself) and so if you just shift the database onto a new folder all references to documents get messed up. Even using the repair option of Mendeley doesnt fix this. This should be a good feature for them to add soon. 
mendeleydesktop --reset
    • After this i logged into my online account and deleted the entire collection from there. Note that if you have any notes attached to the document, this is the time to save them. There is no easy way to do that except cut and paste into some text editor.Note that the above step is essential because i think Mendeley has a bug where if it synchronizes again with the online account after you have created a new local database things get extremely messy and it crashes. Talking with experience here.
    • Now come the tricks:
    cd /media/extstor2/db
    mkdir Mendeley\ Ltd.
    cd ~/.local/share/data
    rm -rf Mendeley\ Ltd.

    ln -s /media/extstor2/db/Mendeley\ Ltd./ Mendeley\ Ltd.
    • What i am doing above is essentially repointing the database location to the external drive. Note that this also means that you will always require the external harddrive to use Mendeley.
    • Opened up Mendeley again. This time it should start with nothing in it and offer you to login into your online account. DO NOT LOGIN YET.
    • Opened Tools->Options and clicked on File Organizer.
    • Enabled the 'Organize My Files' tab and set the path to /media/extstor2/papers
    • Enabled other options as desired.
    • Now logged in to my online account and let it sync. Nothing should sync as nothing exists but it is good to let the desktop handshake with the web account.
    • I then added all my folders where i keep my collection. Luckily i had not deleted them.
    • After this i let Mendeley index my complete collection (takes time proportional to your collection and system speed). Took about 5-6 hours.
    • Once done, coped all the saved notes to the corresponding papers. Had to do manually :(.
    • Then i synchronized again with the online account.
    At this point we have a working database stored on the external harddrive. Now take this drive to your other system and proceed as follows:

    On the work system
    • Connected the drive to the system and mount it as /media/extstor2 to begin with.
    • Then set the properties of the external drive to always mount as /media/extstor2. This ensures that we always have a constant path prefix whenever you attach the drive. This can be easily done by clicking properties of the drive, selecting Volume tab and fixing the mount point settings.
    • Reset current database. 
    mendeleydesktop --reset
      • Tricks Again
      cd ~/.local/share/data
      rm -rf Mendeley\ Ltd.

      ln -s /media/extstor2/db/Mendeley\ Ltd./ Mendeley\ Ltd.
        • Opened up Mendeley.
        • Opened Tools->Options and click on File Organizer.
        • Enabled the 'Organize My Files' tab and set the path to /media/extstor2/papers
        • Enabled other options as desired.
        • Now logged in to my online account and let it sync. Now i saw everything just the same way as my home system.
        • Added your desired folders for watching files on my work system.

        That's it!!! Problem solved. Enjoy. Let me know if this solution solves somebody else's pain.

        26 comments:

        Vijayaditya said...

        Hi,
        Any ideas on syncing large databases across operating systems. ..?

        M said...

        Thank you! Very helpful. Works great if you keep files in dropbox too.

        Arun Viswanathan aka n30bli7z said...

        @M Yes, dropbox does work great if you have a small collection of files (unless you buy their subscription) and are willing to put it all in the cloud. I have a large collection of about 6000+ documents and containing some personal documents that I wouldn't want to put online.

        I did try the same technique with Ubuntu One but found the initial versions unreliable and slow. Thus experimented with the hard-drive.

        Thanks for the suggestion.

        Debtor Paul said...

        This was very helpful, since I use Ubuntu almost everywhere. However, I have a couple of places where I must use Windows. Is it possible to do something similar on Windows?

        Arun Viswanathan aka n30bli7z said...

        @Paul..

        Thanks for the comments.

        As for windows, the problems arise primarily due to Windows not natively recognizing filesystems other than the MS ones. So your ext3 formatted disk is invisible to Windows.

        That said, I think it should be possible to format your external drive as NTFS and mount it on Ubuntu since it has good NTFS support. You can then apply the steps as in my post.

        Additionally, you would need a way to create a symlink on Windows. The following tutorial http://www.howtogeek.com/howto/windows-vista/using-symlinks-in-windows-vista/ should help.

        I would advice you to exercise caution since I have not tried out this solution.

        Debtor Paul said...

        OK, I ran into a problem. Do to several circumstances I had to completely reinstall Mendeley on my primary laptop, an Ubuntu box.

        When I started everything back up, following the instructions in this post, everything was in place perfectly, except one thing. While all my tags, and bibliographical data were intact, none of the entries show the file attached (all of them should be linked to actual files on my stick). Do you know what is happening and how to fix it?

        Thank you in advance.

        Arun Viswanathan aka n30bli7z said...

        @Paul

        Mendeley stores the full file paths to your files in its database. It is possible that the file paths in the database are pointing to the old location.

        Can you do the following (this wont change anything on the system):

        1. Open a terminal.
        2. Install sqlite (if you dont have it) using
        sudo apt-get install sqlite3
        3. Go to the mendeley database directory
        4. You will find your database file stored as your @www.mendeley.com.sqlite
        5. sqlite3 @www.mendeley.com.sqlite
        6. select * from Files;

        This should list the full paths of all your files. Are they pointing to their new location? Or are they showing the old path?

        Arun Viswanathan aka n30bli7z said...

        @Paul:

        Small error in my response.

        The sqlite filename is youremail@www.mendeley.com.sqlite.

        Debtor Paul said...

        OK, I did steps 1-5 in the terminal, but not exactly sure what to do with step 6. Right now it says "Enter SQL statements terminated with a ";"
        sqlite>

        What exactly should I type now?

        Arun Viswanathan aka n30bli7z said...

        Great! At the terminal, copy paste the exact statement in line 6 i.e.

        select * from Files;

        This command basically asks sqlite to list all the entries it has in the table called 'Files'.

        Debtor Paul said...

        It is difficult for me to tell exactly what I am looking at, but it is in fact listing files paths that go to the correct place on my flash drive.

        Basically, each item listed reads like this:

        (long list of numbers and letters) | (a good file path)

        Arun Viswanathan aka n30bli7z said...

        Hmmm... atleast the file paths are correct. Thats good,

        Can you confirm the following:
        In the Mendeley application, under Tools->Options, click on File Organizer.
        Is the 'Organize My Files' tab enabled?
        Is the path to your papers directory on the USB stick listed correctly?

        Debtor Paul said...

        Yes, everything looks right there.

        Debtor Paul said...

        OK, I have gone back through the instructions on this post, deleting everything online, resetting Mendeleydesktop,etc.However, I am now trying to use the version installed from the .deb package rather than the generic linux version. I noticed that when I try to open the program after the reset I don't have the option of opening without logging in.

        I have backed up my database and library folders previously. Just hoping I can figure something out soon. I need full access to my library.

        Thank you for the extra help.

        Debtor Paul said...

        Ha, nevermind. I forgot that if I deleted everything online that I would have to re-enter all of my tags and bibliographic data (not an exciting prospect). It looks like I can go through and tell Mendeley where the files are one by one from within the program. That seems like the least painful option. As much as I would like everything kept on my external drive, it looks like I'll just have to find some other way to sync everything at a price???

        Debtor Paul said...

        Perhaps Ubuntu One syncing may be my best bet, though I would love to be able to sync with my Windows computers as well. Oh well.

        Debtor Paul said...

        OK, let's see how many times I can comment here!

        Here is my plan - I have paid for an upgraded mendeley account, and will sync the normal way. Do you know of any way that I can restore the file paths (or whatever needs to be done) without doing it one by one within the program?

        Arun Viswanathan aka n30bli7z said...

        @Paul

        Sorry that I could not be of much help but these workaround solutions are rather painful as they are hacks.

        About Ubuntu One like solution: I prefer using DropBox which works like a charm across Windows, Linux and Mac. You get 2GB of free storage and additional for a fee. I have tried both Ubuntu One and Dropbox but DropBox is by far the best out there. I have about 7 machines connected to my DropBox account.

        Let me think a little more about the filepath issue you are facing before I respond. In the meantime can you tell me your exact setup:
        1. Ubuntu version
        2. Mendeley version
        Anything else you feel relevant?

        Debtor Paul said...

        Ubuntu 11.04, Mendeley 0.9.9.2

        I can't think of anything else relevant right now.

        As far as Drop Box goes. I would like that solution if they had a more moderate plan for me. I need more than 2gb, but I don't need 50gb (and don't wish to pay the price).

        supah sushi said...

        Hi! I've been looking for a solution for using Mendeley on multiple systems using an external hard drive. I have the same situation as you did, wherein I need to be able to access or open my files/pdfs from Mendeley, not just have synced bibliographies. However, I am using 2 Windows machines. Honestly, I don't know where to start. I tried organizing my database in Mendeley and setting the directory on my portable hard drive on both systems but can only access the files where I added them. :( Sorry for being such a n00b.

        Arun Viswanathan aka n30bli7z said...

        @supan I have never tried this solution on Windows and cannot give you one immediately since the linking etc. on windows is not natively supported (AFAIK). I will need to try this out first. I will post my findings and update you as soon as I get some time.

        supah sushi said...

        Thanks for the reply Mr. Arun. Meanwhile, I tried to something about it. I posted about it here:

        http://feedback.mendeley.com/forums/4941-mendeley-feedback/suggestions/301592-pdfs-should-be-attached-with-paths-relative-to-a-u

        I am "supahsushi". So far it's okay, but I'm still thinking if there is a more elegant and correct solution ^^; Thanks sir, really appreciate it.

        Ruby said...

        I’m no techie, so please forgive my question. What if the hard drive crashes? Will it work in another OS? Have you ever considered using online storage services to protect your files? Having one is pretty essential, but having many would be so beneficial, right?
        - Ruby Badcoe

        Arunuikas said...

        Thanks, this worked wonders on ubuntu 12.10 and I have my mendeley entirely set up on my hard disk. But i am now trying to figure out how to get thos going on a windows system.

        I have been looking around to see what could go wrong and these points stood out.

        symlink on windows 7 is supported but file paths may be a problem (drive letter and not the same as ubuntu media/stor/mendeley/db in my case)

        do you have any suggestions or workarounds?

        Also i wanted to check, will updating mendeley have any issues with the symlink? I just noticed they have an update available for ubuntu but am not sure if i should go for it.

        thanks,
        Another Arun

        mahasiswa teladan said...

        hi..Im student from Informatics engineering, this article is very informative, thanks for sharing :)

        Coyote pradereƱo said...

        Hi thanks for the post!! do u have some code to allow the sqlite library to be accessed by more than one user at the same time??

        Best!!