User:Ckeene/ep3 upgrade

Note: For some reason: "sudo /etc/init.d/apache2 stop" and "sudo /etc/init.d/apache2 start" not working, use sudo /usr/sbin/apache2ctl start instead.

Eprints 3 upgrade

A journal of our upgrade process from eprints2 to eprints3. This will include my own stupid mistakes and errors, many will be laughable and show my true incompetence. I keep them here so others do not do the same!

Current situation

Current Version: EPrints 2.3.12 (Southern Chicken) [Born on 2005-05-24] Running on Ubuntu 6.06 LTS (based on Debian) Visual look and quite a bit of the wording has been changed, and endnote/BibTeX import functionality has been added (via the downloads from the eprints.org site).

The plan

1. Apache 2 installed, with mod perl and php
2. Other dependencies
3. Install eprints3
4. Create test archive (ep3v) - v for vanilla
5. Install and run migration tool. Document results.
6. Look in to problems, use mailing list for support
7. Modify wording of user interface (both for depositors and end users) to follow changes made on eprints2 (where they still apply and where changes were not bought across)
8. Apply Sussex Templates to eprints3
9. Install additional modules to bring in line with extra functionality of our eprints2 setup, see table below
10. Repeat migration process of data to test and address issues.
11. Follow steps to add additional functionality, such as auto-complete features, which are not out-of-the-box, but core features of eprints3.
12. Update documentation and screenshots ready for eprints3
13. Demonstrate to others, solicit feedback
14. Make changes based on user feedback
15. Migrate data one final time, update apache so that eprints.sussex.ac.uk points to eprints3, not eprints2. This will make eprints3 live. woohoo!

Steps to bring eprint3 (out of the box) in-line with our current eprints2

1. Import from Endnote
2. Import from Bibtex
3. LDAP authentication
4. Import/update users from text file
5. Fields created to hold record id (from Uni pub db) and personnel id (for Sussex authors)

Install Apache 2, mod_perl, etc

1. key doc http://wiki.eprints.org/w/Installing_EPrints_3_on_Ubuntu_6.10
2. sudo apt-get install apache2 libapache2-mod-perl2
3. eprints2 is on port 80. vi /etc/apache2/ports.cfg, changed from 80 to 8082
4. /usr/sbin/apache2ctl start
5. http://localhost:8082/ http://researchonline.lib.sussex.ac.uk:8082/ http://eprints.sussex.ac.uk:8082/ all work!
6. [we already have mysql installed - skip step]
7. sudo apt-get install libxml-libxml-perl libunicode-string-perl \

 libterm-readkey-perl libmime-lite-perl libdbd-mysql-perl

1. most of these were already installed. but at least

libterm-readkey-perl libmime-lite-perl were not.

1. /etc/init.d/httpd stop start - eprints2 is still working!
2. sudo cpan install Readonly

Eprints 3 install

1. Follow http://wiki.eprints.org/w/Installing_EPrints_3_on_Ubuntu_6.10
2. We already have an 'eprints' user so I'm going to try and not create one now, the only issue might be that the instructions state the users home should be /opt/eprints3 but I'm going to leave it as /home/eprints/ and see what happens
3. sudo adduser www-data eprints
4. tar xzvf eprints-3.0.2-beta-3.tar.gz (yup I'm installing a beta!)
5. cd eprints-3.0.2-beta-3
6. Remind myself of our smtp server: grep smtp /opt/eprints2/perl_lib/EPrints/SystemSettings.pm
7. ./configure --with-smtp-server=smtp.sussex.ac.uk
8. sudo ./install.pl

Apache setup

1. sudo vi /etc/apache2/sites-available/eprints
2. add to this new file: Include /opt/eprints3/cfg/apache.conf
3. sudo a2ensite eprints

1. sudo ln -s /opt/eprints3/bin/epindexer /etc/init.d/epindexer
2. but: "ln: creating symbolic link `/etc/init.d/epindexer' to `/opt/eprints3/bin/epindexer': File exists"
3. looks like it will overwrite the eprints 2 indexer: ls -l shows... /etc/init.d/epindexer -> /opt/eprints2/bin/epindexer But this is ok, we run the indexer as a job from cron "/opt/eprints2/bin/indexer start --rollcount 6 --once" and don't use this symlink, which is for running it as a service, should be ok to overwrite (I hope)...
4. sudo rm /etc/init.d/epindexer
5. sudo ln -s /opt/eprints3/bin/epindexer /etc/init.d/epindexer
6. sudo update-rc.d epindexer defaults 99 99  : produced an error message that it already existed, so should be fine to skip this.

Configure an archive

1. Documentation a little unclear here, the Ubuntu docs tell us to follow the 'main eprints3 documentation', so from the main Eprints3 page I clicked on 'installation' and then followed Getting Started on the right, this seems half updated for eprints3 but also referes to eprints2.
2. However I also found on the main Eprints 3 docs page a link to [[Getting Started with EPrints 3] which I shall follow here. I also found the bottom of Installing EPrints 3 on RedHat Enterprise 4] useful. (I've updated the Ubuntu page to link to these two pages).
3. cd /opt/eprints3/
4. bin/epadmin create
5. (at this point I create a cname in our dns for ep3v.lib.sussex.ac.uk as an alias for eprints.lib.sussex.ac.uk, the v is for vanilla as this will be a eprints3 install with no changes made to it for reference)
6. archiveID is ep3v
7. continue to enter details as prompted, at db username i accept the default it suggests. Get stuck with db password, reports 'bad input'. I quit out of this, and then played with epadmin config_db. In the end had a database but no user and it just didn't seem happy. Easy option, delete directory and start again!
8. epadmin create, again, good until, complains "EPrints can create the database, and grant the correct permissions." when doing the db stuff, this was because I had forgotten to delete the ep3v db (which I should have done when I deleted the directory).
9. because the create process terminated prematurely I need to run some commands manually
10. run bin/epadmin config_db ep3v
11. run bin/create_user ep3v (seems to have a fairly strict password criteria, a 7 digit combination of numbers and letters produced 'bad input')
12. bin/generate_static ep3v
13. bin/import_subjects ep3v
14. bin/generate_apacheconf
15. sudo /usr/sbin/apache2ctl stop
16. It works!
17. http://ep3v.lib.sussex.ac.uk:8082/

22 June 2007

First attempt to migrate data

1. create dns/dhcp record for ep3m.lib.sussex.ac.uk (m for migration)
2. create new archive:
   1. /opt/eprints3
   2. bin/epadmin create
   3. follow creation process (db bit works fine this time, but requires a decent password!).
3. started to follow http://wiki.eprints.org/w/Migration
4. downloaded and extracted toolkit
5. run mkconfig.pl sussex
6. cd /opt/eprints3/archives/ep3m
7. cp -rp cfg cfg.orig
8. copy cfg from toolkit to ep3m
9. it works. But do we want this. The nice submission process of ep3 has been turned in to the uglyness that is ep2.
10. Though... how can we migrate just our changes, without migrating the ep2 default stuff?
11. cd migration/mtoolkit
12. ./export3data.pl sussex eprints > eprints.xml
13. ./export3data.pl sussex users > users.xml
14. ./export3data.pl sussex subjects > subjects.xml
15. /opt/eprints3/bin/epadmin erase_data ep3m
16. /opt/eprints3/bin/import_subjects --verbose --force --xml ep3m subjects.xml
17. /opt/eprints3/bin/import --verbose --migration ep3m user XML users.xml
18. a number of warnings appear:
    • Can't exec "NOTFOUND": No such file or directory at /opt/eprints3/perl_lib/EPrints/Plugin/Convert/ImageMagick/ThumbnailDocuments.pm line 73.
    • I suspect these should be ok
    • note http://www.eprints.org/tech.php/7265.html
19. use ubuntu package manager to install imagemagick
20. sudo /usr/sbin/apache2ctl stop
21. sudo /usr/sbin/apache2ctl start
22. /opt/eprints3/bin/epadmin erase_eprints ep3m
    • this did not resolve the issue (though /usr/bin/convert exists).
23. sudo /etc/init.d/epindexer
24. generate_views etc

This has produce a system with records from the ep2 imported. I have not yet tested to ensure they have migrated correctly, but progress has been made!

Migrating config: Migrating the config and wording (mkconfig.pl) looks useful for those who have made major changes. However we made a few changes, to wording, to allow LDAP user types, to add wording for bibtex import etc. So much of this has been over taken by EP3, and add to that the fact the migration basically reverts much of the superior ep3 setup back to a EP2 (e.g. the deposit process). At this stage I suspect I will not overwrite the ep3 cfg files with those from the migration tool. Instead customising ep3 as required. The tool is still useful however, as I can use the cfg files it produces (in its own directory away from the actual eprints installation) to cut and paste segments of code as required.

Ubuntu / system packages Part 2

Since I started this process (which has come in very small bursts with big gaps in-between) it looks like http://wiki.eprints.org/w/Installing_EPrints_3_on_Ubuntu_6.10 has been updated. Excellent, looks like it is recommending some packages which I will install now.

1. http://wiki.eprints.org/w/Installing_EPrints_3_on_Ubuntu_6.10
2. sudo aptitude install pdftk tetex-bin psutils imagemagick
   • oddly this results in the British version of firefox being uninstalled!
   • The following packages are unused and will be REMOVED: mozilla-firefox-locale-en-gb
3. sudo aptitude install gs xpdf xv antiword elinks

Back to migration

Thumbnails

1. /opt/eprints3/bin/epadmin redo_thumbnails ep3m
2. error message still appearing ("Can't exec "NOTFOUND": No such file or directory at /opt/eprints3/perl_lib/EPrints/Plugin/Convert/ImageMagick/ThumbnailDocuments.pm line 73"
3. find email message: http://www.eprints.org/tech.php/6484.html
4. this points to /opt/eprints3/perl_lib/EPrints/systemSettings.pm
5. this contains lines such as "'convert' => 'NOTFOUND'"
   • so even though convert - and others - are now installed eprints thinks they are not
   • I can either manually edit this file or rerun the installer.
   • I'll try reinstalling, useful for future reference.
6. cd /home/eprints/eprints-3.0.2-beta-3
7. ./configure --with-smtp-server=smtp.sussex.ac.uk
8. ./install
   • oddly the systemsettings file is still showing the same
   • after renaming the file and running ./configure, ./install.pl again systemsettings.pm now shows the locations and paths for the executables!
9. stop start eprints
10. /opt/eprints3/bin/epadmin redo_thumbnails ep3m
    • various warning messages as it finds problems with PDF files but these can probably be ignored.
11. generate_abstracts
12. move mouse over a pdf icon in the web interface, thumbnail appears! it now works.

upgrade to 3.1

• in June 2008 it was upgrade to 3.1rc1 then rc2. This was quick and straight forward.
• changed port from 8082 to 8080...
  • sudo vi /etc/apache2/ports.conf
  • /opt/eprints3/bin/epadmin config_core ep3v (had to re-enter a load of stuff again)
  • /opt/eprints3/bin/generate_apacheconf
  • edited firewall to allow 8080 (I hate linux firewalls!)

ldap

Things to do

• Existing users; will be imported via the import script from ep2
• set up ldap authentication
• use script i created for ep2 for new users