Tuesday, April 20, 2016
PROTOTYPE DESIGN PROPOSAL
“Book of Fame" Project Team
University of Toronto
Background: The "Book of Fame" project team, a subset of the Old Books, New Science lab at the University of Toronto, is integrating the IIIF (International Image Interoperability Framework) application programming interface into a video game development engine with the intention of broadening and simplifying access to digital art repositories among the international community while presenting scholars with a new and innovative method of interacting with these otherwise inaccessible artefacts.
Prototype Proposal: The purpose of the project, henceforth referred to as the "prototype," is to create an edutainment tool that will enable users to interact with a fully rendered virtual book in three dimensional space. This virtual book will consist of two elements: (1) a single-piece model of a book (binding and pages) that opens to a predetermined place in the book, in which (2) an interactive gathering of individual, blank pages is embedded. The pages in the gathering will be organized according to the book's codicological formula and utilize the IIIF protocol to display the corresponding manuscript images. The prototype will also interact with metadata retrieved from the library server such as image area definitions, annotations, and text transcriptions.
Figure 1. Sketch distinguishing the two elements of the virtual book.
Project Goals:
-
Expand the range of function and service for IIIF technologies
By integrating the IIIF protocol into a video game development engine and rendering separated manuscript images as a virtual codex, the prototype positions and presents these artefacts within a controlled environment that is governed by conventions familiar to users of the information age. Creating such a virtual environment further overcomes logistical problems such as library access and copyright laws, offering a simple and fun means of interactive with digital repositories available to undergraduates and mature scholars alike. -
Recapturing the lost dimensions of digitized manuscripts
The prototype aims to model the features of the physical book that have been ignored within the traditional web or e-publishing interface. Such features include (but are not limited to): the physical size of the object, binding, composition, texture, evidence of rebinding, damage sustained throughout its provenance, its codicological construction, a recto image’s direct relationship with its verso, paper composition, watermarks and other details only visible in certain circumstances (for example, underneath infrared lighting). -
Recreating artefacts that (can) no longer exist
Because a book’s provenance witnesses the separation, redistribution, and rebinding of its leaves, medieval books in particular have taken several different forms throughout their lifetimes. One advantage of a virtual environment is to gather manuscript leaves from different libraries and reshuffle them into different orders, recreating the lost forms of a book or creating entirely new forms altogether. Using the prototype, all of this can be done and studied without ever disturbing the existing physical artefact. -
Open new ways in which glosses, annotations, and metadata can interact
The prototype aims to place three levels of annotations in conversation with one another: (1) glosses inscribed on the physical text and present on the scanned digital image, (2) annotations by contemporary scholars that have been recorded on the library’s server and retrieved using the IIIF protocol, and (3) communal annotations hosted on a local server that do not alter the metadata on the library server. -
Make a contribution to the scholarly discussion of digital books
By creating a virtual codex as a digital surrogate to the physical book, this prototype will serve as a point of discussion for the relationship between physical and digital books as well as raise questions regarding notions of authority that scholars traditionally invest in the physical artefact.
Target Audience: IIIF community, MoMA attendees, manuscript scholars (including, eventually,
undergraduate students)
Target Hardware: PC, Web
Engine: Unity (with Mirador integration)
Scripting Language: C#
Figure 2. Sketch of "Book Mode" displayed with generic toolset and user interface.
Interactive Features:
- Book Mode: IIIF Viewer
Render a gathering of individual pages embedded within a single-piece model of a book.- The single-piece model consists of a cover, its binding, and two non-interactive blocks of pages (one appearing on each side of the book when it is laid open). Pages within the interactive gathering should snap to the non-interactive blocks when they are not being turned or manipulated.
- Individual pages within the gathering will be represented by thin models that use the IIIF protocol to retrieve images from the library server for their textures.
- The engine will always load the two adjacent pages on each side of the open pages; hence only a maximum of six pages should be downloaded into the engine’s cache at a single moment in time.
- Pages should animate flexibly (as opposed to an automatic page flipping animation). The user should be able to point and drag pages with realistic physics, perhaps using Unity’s “cloth” modelling template.
- Pages can only be turned one at a time, though there will be an option to auto-turn the book to a specific page. This feature can use an automatic page flipping animation to speed things up.
- The prototype should track the user’s position within the gathering of pages and display it on some sort of tracking meter, if not on the single-piece model itself.
- Click and Drag
When the book is closed, the user should be able to click and drag to rotate the book in three dimensional space. Double-clicking on the book will open it up to a predetermined page. While open, neither the user’s nor the book’s position in space can be changed. Instead, the user can click and drag pages in the interactive gathering. Once a page is dragged past a certain point (likely 90° relative to the base), the page is considered “turned” and will fall face-down if the click is released. The reader can close the book by double clicking on the edges of the cover, visible on either side of the book.
- Metadata: Hover and Identification
The prototype will use the metadata to identify spaces of interest upon the page and label them with corresponding annotations. For example, the coordinates (x,y,w,h) may identify a region of a manuscript image containing an illumination and carry descriptive information regarding that illumination. When the user hovers over the region defined by the coordinates, the corresponding descriptive metadata will be displayed.
- Metadata: Selection
Similarly, if the user activates the selection tool, he or she can click and drag a box around a specific region of the open page to highlight its contents. The user may then add his or her own annotation to the highlighted coordinates, which is then converted into the IIIF protocol and stored for retrieval upon the prototype’s local server.
- Metadata: Display Annotations
The user will be able to click a button on the UI that will display three different levels of annotations: library, local, and global (i.e. library and local). These annotations appear as small windows with manicules pointing to the corresponding region. Library and local annotations will appear in different colours to signify that they are not from the same records, despite appearing within and in times occupying the same virtual space.
- Metadata: Transcription
Using the IIIF retrieval protocol and the coordinate positioning system, the prototype will place a transcription of the open pages’ text on an invisible layer that hovers above the displayed image. When the user activates the magnifying glass tool, a fully modelled magnifying glass appears on screen and moves according to the mouse’s movements. Rather than distorting the image, the magnifying glass reveals the transcription text laid atop a generic manuscript page texture.
- Middle English Dictionary Integration
If the user activates both the magnifying glass tool and the MED tool, double-clicking on a word will retrieve its corresponding entries from the MED web site and display them in a separate window.
User Interface
- Main Menu
When the prototype application is first loaded, a splash screen will be displayed featuring a talking bird that will explain via dialogue boxes the purpose of the tool, its feature set, and the basic controls. This introduction can be skipped by pressing “SKIP” in the corner of the splash screen.
- Camera
The camera represents the user’s scope of vision. If the book is positioned at chest
height, the camera is positioned at the head. The user can zoom in and out on the book
without his or her position within the virtual world.
- Controls
- Navigation Mode
When the book is closed, the user can move around three dimensional space as
well as move the book around.
- Arrow Keys
The user can use the arrow keys to move his or her (and hence, the
camera’s x and y) position in three dimensional space.
- Click
The user can click on active buttons of the feature palette that appear at
the bottom of the screen. However, the only buttons that are active during
Navigation Mode are “Help,” “Reset,” and “Bird.”
- Click and Drag
Clicking on the closed book and dragging the mouse (horizontal only) will
move the book’s position in three dimensional space.
- Click and Drag while holding Shift
Clicking on the closed book and dragging the mouse while holding shift
will allow the user to rotate the book in three dimensional space so that
he or she may have a better look at the book’s binding.
- Hover
Hovering over the book will reveal its name and provide any descriptive
details pertaining to the book as whole (as opposed to individual pages).
- Double-click
Double-clicking on the closed book will reset the book to its default
rotational orientation (horizontally parallel to the ground) and open the
book up to a predetermined section. This will also fix the user in position
before middle of the book and send the application into Book Mode.
- Arrow Keys
- Book Mode (Default Tool)
- Arrow Keys
Limited camera movement to simulate looking at the book from different
angles without ever moving the user’s position in space.
- Click
The user can click on active buttons in the feature palette to activate
different point-and-click toolsets. All buttons are active during Book Mode.
- Click and Drag
Clicking on a page and dragging the mouse will manipulate the page as if
it were being pinched and pulled from that point. Dragging a page beyond
90 degrees relative to the book’s horizontal axis will recognize the page
as turned and cause it to fall face-down as opposed to face-up.
- Hover
Hovering over a region of the page with its region coordinates specified in the metadata will cause any corresponding descriptive information to be displayed.
- Double-click
Double-clicking on the protruding edge of the book cover on either side
will cause the book to close in the corresponding direction and send the
user back into Navigation Mode.
- Arrow Keys
- Navigation Mode
- Feature Palette
There are a number of buttons on the menu bar at the bottom of the screen that can be clicked on to activate and deactivate. With the exception of assistance tools (Help, MED, Bird), clicking any tool will unclick all other tools.
- Hand Tool (Navigation and Book Mode)
Activated by default, this tool allows the user to physically manipulate objects
- Click and Drag
Navigation Mode: move book around in three dimensional space
Book Mode: manipulate and turn pages
- Hover
Navigation Mode: hover over the book to reveal a window containing
descriptive information about the book
Book Mode: hover over metadata-encoded regions to reveal a window
containing descriptive information specific to that region
- Double-click
Navigation Mode: double-click the book to open the book
Book Mode: double-click the cover edge to close the book
- Click and Drag
- Annotation/Selection Tool (Book Mode only)
When clicked, this tool allows the user to create his or her own annotations
- Click
Book Mode: clicking without dragging clears any existing selection
- Click and Drag
Book Mode: create a new set of region coordinates, which will then
prompt the user to add corresponding descriptive metadata to be stored
in the local server
- Double-click
Book Mode: double-clicking on a previously created set of region
coordinates will highlight that region and display the recorded descriptive
metadata in a window
- Delete
Book Mode: if the user has double-clicked and highlighted a previously
created region, pressing delete will erase both the coordinates and the
corresponding metadata from the local server
- Click
- Display Annotation Tool (Book Mode only)
Clicking this button cycles through various levels of annotation display
- Click
Book Mode: clicking multiple times on the button makes different layers of annotations appear in a predetermined order (library, local, global, off = cyclical)
- Click
- Lens of Authority (Book Mode only)
- Hover
Book Mode: hovering with the magnifying glass tool reveals the invisible
transcription layer within the confines of the glass
- Double-Click
Book Mode: double-clicking on a word (i.e. the word at the precise center
of the magnifying glass) will cause the dictionary window to open and
retrieve its corresponding entries from the MED web site
- Hover
- Dictionary/MED Tool (Book Mode only)
Retrieves entries for a desired word from the Middle English Dictionary web site
- Click
Book Mode: clicking on the dictionary button opens a window in which
entries will be listed for words that the user double-clicks
- Double-Click
Book Mode: double-clicking the region of a word on the image will search
the transcribed word on the MED web site and display a list of entries in
the previously revealed window
- Click
- Help Tool (Navigation and Book Mode)
Opens a help window to assist the user
- Click
Navigation and Book Mode: clicking on the Help button opens up a
window listing the controls and other information
- Click
- Bird Tool (Navigation and Book Mode)
Demonstrates twitter integration by displaying the most recent tweet
- Click
Navigation and Book Mode: clicking on the bird in the bottom right corner
will cause it to squawk and display the most recent tweet on the project’s
twitter page
- Click
- Pencil Tool (Book Mode only)
Allows users to mark up and deface the manuscript image on a layer that does
not alter the library’s original image
- Click and Drag
Book Mode: clicking and dragging allows the user to draw and
markup/deface the page (perhaps with constraints), the data of which is
then saved to the local server and can be displayed along with local
annotations
- Click and Drag
- Reading Tracker (Book Mode only)
Displayed at the center of the bottom menu in an arc-like shape, this clock would
track the user’s position within the book.
- Hand Tool (Navigation and Book Mode)
Figure 3. Toolset icons for user interface.
Aesthetics:
- Artstyle
A virtual, faux Fisher library with a grid-like digital floor and pixelated sky. There is no furniture in this place: the lone book hovers at chest height wherever the user places it.
- Art Assets
- Panoramic background of the Fisher library
- Single-piece model of book
- Individual pages
- Animations for opening the book and manipulating pages
- User interface buttons and windows
- Bird (possibly 2D) with simple animations (i.e. turning head, opening beak)
- Sound Assets
- Opening book
- Turning page
- Reveal annotation
- Open window
- Bird squawk
Project Scope: One gathering of 17 leaves embedded as a gathering within a single-piece model of a book