OxCLIC MDIDimporting

Contents

  1. Importing Images and the associated Portfolio catalogue into MDID as a new collection
    1. Export from Portfolio the catalogue information as catalogue.txt
    2. Importing a catalogue from portfolio to MDID ...more fun and games
    3. Checking in Excel the .csv file
    4. Convert the .csv to XML using helper tool
    5. Create in MDID an empty collection
    6. Import images into MDID using Imagemanager tool
    7. Customising the collection in MDID
  2. Draft early version of importing steps
    1. Open catalogue.txt in Excel
    2. Dealing with special characters not supported in the field names
    3. Move the .csv file to the MDID server
    4. Setup the collection in MDID
    5. Loading the images into MDID

Importing Images and the associated Portfolio catalogue into MDID as a new collection

Export from Portfolio the catalogue information as catalogue.txt

File > Export

Importing a catalogue from portfolio to MDID ...more fun and games

In Portfolio you should select the records for which you would like to export the metadata. Choose File -> Export Field Values. You will then get to choose which field values you would like to export to a tab-separated text file.

You will need to open the file up and convert it to CSV. This is most conveniently done using a spreadsheet such as excel.

Then you will need to manipulate the first row of the file, which contains the headings from the Portfolio catalogue. You will need to replace certain characters (in the first line only) for the inport filter to be able to proceed (see Section 5.2 above ).

If you do not remove these characters, any column that contains such a character will be discarded silently by the convert.exe program, and will not be imported into MDID. It is arduous to add the data in later, so you must get this operation correct right now.

The file: [MDID2 Curator's Workshop .pdf http://mdid.org/mdidwiki/images/b/bb/MDID2CuratorWorkshop.pdf] gives instructions from this point onwards, and is quite good.

Checking in Excel the .csv file

Identifier Field

the records. These values will be used to match input records to existing records in the collection. This column is referred to as the Identifier Field.

Resource Field

usually the file names of the associated image files. This column is referred to as the Resource Field.

All other columns must match exactly one field in the target collection.Not all collection fields need to have a column in the spreadsheet or CSV file, but all columns in the spreadsheet or CSV file must match a field in the collection. In general, each row in the spreadsheet or line in the CSV file represents one data record. The only exception is for fields that have multiple values, in which case the following rows or lines can contain additional values, as long as the Identifier Field and Resource Field for these additional rows are blank.

Convert the .csv to XML using helper tool

Locate the program convert.exe, which comes with the MDID2 Tools download (avaliable from sourceforge with the MDID2 server software). Run the program (does not seem to run over remote desktop on Windows server 2003) and point it at the csv file that you have just created. Apparently it is possible to work from an Excel worksheet instead of a CSV file, but I have not tested how well this works.

There are a number of settings that you need to make before you generate the XML file.

Choose the Identifier field (this should be the accession number e.g. arth_aa1994) and Resource field (this should be the filename e.g. arth_aa1994.JPG ) from the drop down menus.

Under Options you should choose to split field values at newline characters rather than semicolons.

Click Start Conversion and you will be prompted for the name and destination of the XML file.

Create in MDID an empty collection

The next step is to log into the MDID2 application and create a new collection to hold the images and metadata. Fill out the name, Description and image path (which you will presumably have to know by this stage) Then click on Field Definitions at the top of the page. Choose to get your field definitions from the XML file you have just created, and navigate to locate it.

Next you choose Import Data

Choose your XML file again, and click the import data button.

The page will refresh, showing you the progress it has made in importing your data.

{{{Import Status

The last data import for this collection successfully finished. 1107 of 1107 records processed: 1107 records added 0 records replaced 0 records already existed and were skipped 0 records did not have an identifier and were skipped }}} Assuming this completes satisfactorily you now need to deal with the images in your collection.

Import images into MDID using Imagemanager tool

On the MDID server, run Imagemanager.

Connect to the server on the non-webauthed port (8081 in our case) e.g. 

http://oxclic.oucs.ox.ac.uk:8081/

Select the collection you have just created (or possibly the collection for which you want to update images).

Missing images are shown in red in the image manager. Initially you should click 'Select Problems'

Then you need to point the program at a single directory that contains the images associated with the collection. This is done by Selecting 'Collection' -> Assign Local resources and navigating to locate the image directory. Next you press the Action Upload button, then Start. You can either repeat this process for individual folders of images, or bung all your images into a single folder.

The program will import the images to the MDID2 Application.

Check that the images are now available to be browsed in the online MDID application.

Check all images are there and all metadata fields are there. If fields are missing then it is likely that the field name contains XML unfriendly characters.

Customising the collection in MDID

Collection > Field order

Collection > fields > modify

Tip: Tidy the field name order to be the most suitable for presentation on screen, the most important field first, perhaps hiding the more obscure field names from browses and searches. In MDID as collection administrator to modify field definitions so they .-

Management > Collections > Manage Collection X > Field Definitions > Modify

Management > Collections > Manage Collection X > Field Definitions > Move

For compatibility with MDID's default settings you should mark one of your fields to be a 'Title' field. You do this here:

Management > Collections > Manage Collection X > Field Definitions > Modify > Label > Title

In order to facilitate cross searching you also need to make a field to be the dc:title field i.e. "Dublin Core Title" that appears in Lightbox views etc

Management > Collections > Manage Collection X > Field Definitions > Modify > DC Elements > Title

Mapping fields to Dublin Core for cross-searching

It is also possible to map these fields to Dublin Core Elements.

From the Curators Workshop Manual:

"DC Element: The Dublin core element, which is used to map fields in different collections to each other. In order to perform cross collection searches, all involved collections must have Dublin core elements set for at least some of their fields. This property is optional. Multiple fields can have the same Dublin core element entry. "

Management > Collections > Manage Collection X > Field Definitions > Modify > DC Elements

Draft early version of importing steps

Open catalogue.txt in Excel

The Curators Workshop manual takes you through this process very thoroughly:

"The instructions in this section assume that your cataloging data is available in a Microsoft Excel spreadsheet, with the first row containing column headers and each row below representing an image record.

Preparing a CSV file

The only format in which cataloging data can be imported into MDID2 is specially formatted XML. Using the data conversion tool, Microsoft Excel spreadsheets and CSV (Comma Separated Value) files can be converted to correctly formatted XML files.

in a meaningless XML file:

headings. The headings themselves must match the field names in the target collection exactly.

the records. These values will be used to match input records to existing records in the collection. This column is referred to as the Identifier Field.

usually the file names of the associated image files. This column is referred to as the Resource Field.

collection fields need to have a column in the spreadsheet or CSV file, but all columns in the spreadsheet or CSV file must match a field in the collection.

record. The only exception is for fields that have multiple values, in which case the following rows or lines can contain additional values, as long as the Identifier Field and Resource Field for these additional rows are blank.

below the data records.

Dealing with special characters not supported in the field names

Portfolio has many default field names with spaces and it is common to use a colon in DC and VRA field names. It is necessary to modify all field names that contain spaces e.g. "Short Filename" with "Short_Filename". It is straightforward to open the catalogue.csv file in Excel and change any top row field name that contain spaces to have underscores instead.

MDID needs a field which is a unique identifier, called Identifier and a field that refers to the resource image name called Resource. It is sensible to have these two fields as the first two fields in the .csv file. As we have a unique identifier from the OxCLIC naming conventions in the form of the name of the image, this field can if duplicated to serve both purposes.

Steps to do this in Excel

Move the .csv file to the MDID server

Check there aren't any odd characters appearing if moved from Mac to PC by looking at the file in a text editor. Check again that the field names don't contain spaces or unusual characters. If they do replace with underscores using find and replace in a text editor.

Download the set of Curators tools from the MDID Source Forge site http://sourceforge.net/project/showfiles.php?group_id=115144

The steps below are clearly outlined in the essential document, The Curator's Workshop Handbook. The Curator's workshop manual is the key document that talks you through this process step by step. The notes below are brief comments that are relevant to the OxCLIC workflow. http://mdid.org/mdidwiki/images/b/bb/MDID2CuratorWorkshop.pdf

You could also get the tools from http://mdid.org, clicking on downloads and then clicking on "MDID2 package downloads at SourceForge.net" on the down load page. Both tools require Windows XP with .net installed

Use the "Convert.exe" XML generator tool to convert the .csv into the required XML

In the application, open the catalogue.csv file

Click - "Start Conversion"

You should now have an xml version called catalogue.xml

Setup the collection in MDID

Log into MDID

Go

Management > Collections > Create New Collection

Setup a new empty collection in MDID and

Go

Field Definitions > Import Data > catalogue.xml

This takes around 30 seconds for 350 records. Please wait while it imports the records.

Check to see the field names are the correct ones and that none are missing. If missing check that the original .csv file hasn't got spaces, brackets or commas in any field titles. A simple batch "find and replace" of these characters in a text editor can help tidy up the column field names. Once the cataogue information is imported you can search and broswe the information in the online application, it is not necessary to import the images immediately as MDID recognises that the images are not available.

Loading the images into MDID

Run the Imagemanager.exe application

It should say all images are missing !

Select All - Browse to the folder where the images are located

Select the records you want to assign to images and import. Repeat with folders of images until all are imported.