WO2014178039A1 - Scrolling electronic documents with a smartphone - Google Patents

Abstract

The invention comprises novel input means for portable devices such as smartphones, PDAs, tablets, and the like. One or more sensors of the device are adapted to allow movements of the device itself to represent user actions to some degree of precision. These actions may comprise inter alia translating, rotating, accelerating (linearly or rotationally), shaking, flipping, sliding, various gestures, or the like. Other movements are further possible as will be clear to one skilled in the art. These movements may be used to control such actions as scrolling, paging, zooming, scaling, rotating, translating, opening, closing, and other operations of use in a GUI or other environment, for purposes of manipulating digital objects.

Description

Scrolling electronic documents with a smartphone

This application claims the benefit of US patent application No. 61816998 filed on 29 April 2013.

Field of the Invention

[1] The present invention relates to the field of man-machine interaction (MMI). In particular it relates to input means for computers, such as control of computer games, control over display of content such as documents on portable electronic devices like smartphones, and generally speaking any set of analog or digital inputs, suitable for replacement of and/or in addition to standard input means such as the mouse, touch-screen and the like.

Background of the Invention

[2] Input means for computers are of central importance for MMI. In the earliest days of computing the input means comprised punch cards, switches, and knobs; since then input means have progressed to include the keyboard, mouse, touch-screen, microphone, and camera. Generally these devices are intended to allow the human user to fix the value of one or more variables to some degree of accuracy, such that the computer may make use thereof.

[3] European patent application EP0701220 discloses a reader for displaying an electronic document in a manner which promotes comfortable viewing, displaying at least a portion of the article on an output device, and using an input device of the digital computer to cause a display of additional portions of the article to be displayed on the output device in the article viewing mode. The specific input device is chosen from amongst standard hardware including mouse and keyboard.

[4] EP2402851A1 discloses a type of pinch-to-zoom operation, which allows for use of a touch-screen as input means for document zooming. However as will be appreciated, device sensors such as IMU and the like are not exploited.

[5] Similarly application WO2010071630A1 provides a system for document editing using multiple touch points; like EP2402851A1, no provision is made for use of further novel input means. [6] US20120260220 provides for gesture recognition and control by use thereof, including sensing of distortion of a device from a neutral state, and force measurement. However the means of control is not necessarily an intuitive process in the embodiments listed, and fusion of information from several sensors is not disclosed.

[7] US patent application 8321428B discloses methods allowing a user to navigate through previously viewed documents; however as in US20120260220 navigation is provided through standard means such as a button on a toolbar or through a menu listing.

[8] It will furthermore be appreciated that in recent years, portable and hand held devices have begun to impose new requirements on input means, as the form of MMI has changed. The keyboard/mouse is less important and new modalities have become popular: touch-pad, touch-screen, voice recognition and processing, gesture control, and eye tracking have all become more or less commonplace. These input means are often cumbersome, in some cases requiring use of both hands, and in certain cases being difficult to use, ergonomically compromised, unintuitive, and/or hard to incorporate into everyday life. There is thus a long-felt need for improvements in input means for MMI.

Summary of the Invention

[9] The invention comprises systems and methods for improved input means. The input means generally comprise a set of sensors located on or within a portable computing platform such as a smartphone, tablet, personal digital assistant (PDA) or the like. The sensors may be adapted to provide indications related to the position of the platform in space, i.e. location and orientation. The sensors may be affected by user actions such as shifting, translating, rotating, accelerating, shaking or otherwise moving the platform in space. By reading the sensors periodically (either continuously or intermittently), the computing platform senses its own position and movement in space. The computing platform may use this information as input from the user for control over various graphical user interfaces (GUIs), maps or document viewers, pictures, or other elements.

[10] In one embodiment, the device may read at least one sensor value that depends on its position and/or its motion. The user may intentionally move the device in a predefined manner, detectable by the device sensors. The device then interprets the movement and uses it for positioning a pointer, moving a selection tool, scrolling and paging through a document, moving a character in a game, controlling properties of media player, interacting with a user interface, or otherwise controlling values used by the computing platform. [11] Analysis of device movement may take the form of integrating sensed accelerations and rotations to perform dead reckoning, to determine velocities, to determine orientation, to determine time derivatives of sensed or derived quantities, Fourier transforms thereof, and the like. Derived quantities based upon multiple sensors may be computed using sensor fusion, for example by means of a Kalman filter. Device position and orientation may also be determined by means of image analysis; for example, successive video frames may be analyzed and registered to determine device position, orientation, and changes thereto.

[12] As described above the input means may be used to control video games, for navigating through documents, observing pictures, scanning maps, watching movies, or for any other interaction with the computing platform. Again this control is obtained by means of positioning and/or moving the computing platform, where 'moving' may comprise actions such as translating, rotating, accelerating and other motions as will be described below.

[13] For purposes of illustration, one embodiment of the invention provides a control action reminiscent to the action of reading a newspaper with a magnifying glass. The smartphone or other computing platform implementing the invention replaces the magnifying glass. The document to be read is stored within the smartphone. To read the document, the user scans the smartphone above a table or other surface, just as one might scan over an object with a magnifying glass. The smartphone displays a portion of the document on its screen, and the portion that is displayed moves within the document in response to or according to the smartphone's movement in space. The system may be so engineered as to give the user the impression that he is observing a real paper or document that has been laid on the table; moving the smartphone will cause it to display that part of the document that would have been seen had the document been physically laid on the table below it.

[14] The foregoing embodiments of the invention have been described and illustrated in conjunction with systems and methods thereof, which are meant to be merely illustrative, and not limiting. Furthermore just as every particular reference may embody particular methods/systems, yet not require such, ultimately such teaching is meant for all expressions notwithstanding the use of particular embodiments.

Brief Description of the Drawings

[15] In the figures and/or description herein, the following reference numerals have been utilized throughout the figures:

[16] Fig. 1 shows an exemplary embodiment of the invention. [17] Figs. 2 A - 2E show an exemplary embodiment of the invention.

[18] Fig. 3 shows a block diagram for an exemplary embodiment of the invention.

[19] Fig. 4 shows another block diagram for an exemplary embodiment of the invention.

[20] Fig. 5 shows yet another block diagram for an exemplary embodiment of the invention.

[21] It should be understood that the drawings are not necessarily drawn to scale.

[22] The term 'movement' hereinafter refers to shifts, translations, panning, rotations, accelerations (linear or angular), gestures comprising several primitive movements, and combinations thereof.

[23] The terms 'device' and 'smartphone' hereinafter refer to a programmable cellular phone, PDA, tablet computer, laptop computer, computing platform, smart watch, or other portable and/ or hand held and/or wearable computing device equipped with one or more sensors.

[24] The term 'sensor' refers hereinafter to any sensing unit suitable for use on or within a smartphone, including accelerometer, magnetometer, inclinometer, gyro, IMU (inertial measurement unit), CCD image sensor, CMOS image sensor, camera (on the front or rear side of the device), microphone, proximity sensor, light sensor, capacitive sensor, touch sensor, vibration sensor, and the like.

[25] The term 'document' refers hereinafter to any electronic content including text document, word-processing application, graphics object, picture, map, roadmap, graphical- user-interface (GUI), operating system screen, electronic game, movie, menu and the like.

[26] The term 'environment' refers hereinafter to parameters such as orientation, position, velocity, acceleration, and the like.

Detailed Description of Preferred Embodiments

[27] The present invention will be understood from the following detailed description of preferred embodiments, which are meant to be descriptive and not limiting. For the sake of brevity, some well-known features, methods, systems, procedures, components, circuits, and so on, are not described in detail.

[28] The invention uses the various sensors of a smartphone or other portable computing platform to allow the use of its own position in space as novel input means. For example, the smartphone may be translated to navigate through a document being read. [29] The various sensors of the device that may be used for purposes of the inventive method include but are not limited to the camera(s) (front camera and/or back camera), image and video sensors such as CCD and CMOS, light sensors, acceleration sensors, GPS, magnetometer, IMU, inclinometer, proximity sensor, capacitive sensor, touch sensor, optical sensors, and other sensors as may be found on or in the portable computing device being used to implement the inventive method.

[30] As will be appreciated by one skilled in the art these or other possible sensors of the invention are adapted to return values associated with position, orientation, linear velocity, angular velocity, linear acceleration, angular acceleration, and the like.

[31] The portable computing device used for implementing the method will generally have electronic data storage and display means as well as sensors such as those listed above. For example cellular phones, smartphones, iPhone™, iPad™, tablets, Samsung Galaxy™ and the like may all be suitable for implementing the invention. The device will generally comprise a hand held and/or portable and/or wearable device, and generally includes a screen, projector, or other display means.

[32] Generally speaking the invention makes use of smartphone sensors to implement novel input means. The various sensors are used to sense position, velocity, acceleration and general motion of the smartphone in space. These movements are detected by the sensors and then processed and interpreted in order to control various parameters of the computing device, for example the position of a pointer such as the mouse pointer, the position of a character in a video game, the firing of shots in a video game, navigation through a digital document, selection of areas, clicking of mouse buttons, dragging operations, operation of virtual keyboards, manipulation of 2- or 3-dimensional objects being modeled or otherwise handled virtually, and any other parameter potentially useful for man-machine interaction.

[33] In an embodiment of the invention, device sensors are used to navigate documents by means of moving the device (either by translating, rotating, or other movements). When the device moves (for example by translating the device above a table or other flat surface) or is otherwise manipulated, the relative position of the document portion displayed on the device screen changes as well. Thus the user can scan over the document by moving the smartphone as though he were scanning the smartphone over a newspaper or other real object. In other words the document may appear to remain substantially fixed with respect to real world coordinates, while the smartphone scans over it.

[34] Fig. 1 shows one possible exemplary embodiment of the invention. Here the smartphone or other portable device 101 has a screen 102. The smartphone is held above table 103. The display 102 shows a virtual object that does not exist outside of the device, namely (in this example) text 104. The displayed text 104 is, in general, a portion of a larger object, for example a document. When the user moves the device, the displayed text changes accordingly, such that the larger object (the document) appears to remain substantially fixed with respect to table 103. Thus, the user may get the illusion that the document is in fact overlaid upon the table 103, while the screen 102 is continuously displaying only that portion of the document that virtually presents below the current location of the device 101.

[35] Another feature within provision of the invention allows the user to zoom in or out of the document by moving the device towards or away from a surface, respectively, or vice versa. For example, zooming into or out of a page can be accomplished by approaching the table (for a zoom -in effect) or moving away from the table (for a zoom -out effect). Changing the distance of the device from the background, can be interpreted by the inventive method as a command for zooming-in or zooming-out. This will magnify or minify the displayed portion of the document. In general, when the displayed portion gets smaller, the displayed details (e.g. letters, numbers etc.) will correspondingly become larger, and vice versa.

[36] Predefined or user-defined specific movements of the device in space may be interpreted as indicating turning pages, for example to the next or previous page of a document, electronic book and the like. Also, this feature may be found useful for navigating among operating-system screens and forms, GUI objects, games, applications, menus, maps and the like. As a further example, rapidly turning the device to the left can be used to define a gesture for paging forward; upon executing this gesture, the system will replace the currently displayed page with the next. Rapidly turning the device to the right similarly may be used for paging backwards.

[37] The camera may be used as a position sensor. It is within provision of the invention to use at least one camera as a position sensor, for instance by use of image registration algorithms, simultaneous localization and mapping (SLAM), or other image processing means. The data provided from one or more cameras, for example the back side camera, the front side camera or both may be analyzed and processed to determine position, absolute position, relative position and general movements of the device in space.

[38] Figs. 2A - 2E show one possible exemplary embodiment of the invention. The device 200 is, in general, a device capable of displaying electronic data; for this purpose, the device 200 contains a display means ('screen') on its front side (the side facing the user). The device 200 may be able to connect to an external screen or projector for purposes of display. The device 200 contains a sensor or a plurality of sensors. Such sensors may be useful to detect the environment and variations thereof. For example, the device 200 may contain at least one camera, on its back side or front side or both. The device 200 may be a smartphone, laptop, tablet, portable personal computer and the like.

[39] In Figs. 2A - 2E, background 210 may be a table, desk, floor, or any object above which device 200 is carried. Background 210 may be a table or a desk or any surface that device 200 is placed on. Background 210 may be the scene or a view or an image as captured by at least one camera or other sensor in device 200. Background 210 may represent the environment in general, but also may represent a virtual object.

[40] In Figs. 2A - 2E, device 200 stores a document 220. Here, for illustration purposes the document contains a set of letters and numbers. The letters and numbers in document 220 may be understood as representing areas in document 220. Document 220 should be understood as representing any electronic content. For a convenient and readable display of document 220, device 200 displays only a partial section 230 of the full document 220 at a given time. Section 230 that is currently being displayed on the screen of device 200 is varied as the user moves device 200, thus enabling the user to navigate through the document 220 by physically shifting (translating, rotating or the like) device 200 in space. The user can move device 200 in his hand, for example leftwards or rightwards, and the displayed portion 230 of document 220 will change accordingly, for example displaying more leftward or rightward portions of the text, respectively. By scanning with device 200 (for example above table 210) the user may be given the impression that he is viewing a physical document 240 resting on the table. This 'virtual reality' effect may be useful for convenient and intuitive navigation through a document or other electronic content.

[41] For clarity, the user, the user's hand that carries and moves the device 200, the internal structure of device 200 and the sensors inside it are not shown in Figs 2A - 2E.

[42] Fig. 2A shows a situation that will be treated here for illustration purposes as a reference situation. In this reference situation, the user carries in his hand a device 200 above a table 210. Electronic content e.g. a document 220 is stored in device 200. Device 200 displays some partial section 230 of document 220. The displayed portion 230 can, in general, be read or viewed by the user from the screen of device 200.

[43] Fig. 2B shows device 200 after it was shifted by the user to the left relative to the reference situation (see Fig. 2A). This shift was detected by sensors in device 200; for example, the camera on the back side of device 200 captured a sequence of images of table 210 during the shift (in general, the captured images are not displayed on screen, but only serve for determining position and/or movement of the device). Consequently, device 200 analyzed the sequence of images and was able to derive at least one input value related to the physical shift of the device in space using any of a number of known algorithms adapted for such purposes. The derived input value(s) may be equal, proportional, indicative, or otherwise related to the physical shift of the device in space. Alternatively, if provided with such, a front camera of device 200 captured a sequence of images during the shift, from which the shift-related input value may be determined in a similar way. As a further example, one or more acceleration, orientation, gyroscopic, or other sensors inside the device 200 detect the spatial shift and provide the data from which shift-related input value(s) is determined. Furthermore, a combination of more than one sensor may be employed to detect the shift and provide the data from which dedicated algorithms will determine at least one shift-related input value. As a result, device 200 uses the set of at least one shift-related input value to update the displayed part 230 of document 220; it is now displaying a portion of the document 220 that was on the left of the original displayed portion. This update of the display is preferably performed continuously along with the leftward movement of device 200.

[44] Fig. 2C shows device 200 after downward movement from the reference situation (shown in Fig. 2A). This movement is essentially horizontal, i.e. the level of device 200 above table 210 remains approximately constant; the user only shifts the device "down the page" i.e. brings the device closer to his body. The physical movement of device 200 is detected by at least one sensor contained in device 200. Consequently, dedicated algorithms process the data from the sensors and derive a set of at least one shift-related input value. As a result, the display on the screen of device 200 is updated in accordance with the set of shift-related input values; the displayed portion 230 is now of a lower part of the document 220 relative to the reference situation (see Fig. 2A).

[45] Fig. 2D shows device 200 after zooming-in movement relative to the reference situation (see Fig. 2A). The zooming operation may be performed by a predefined movement, e.g. changing the distance from device 200 to table 210; for example, the user may zoom -in by bringing device 200 closer to table 210. The physical movement of device 200 is detected by at least one sensor contained in device 200. Consequently, dedicated algorithms process the data from the sensors and derive a set of at least one shift-related input value. As a result, the display on the screen of device 200 is updated in accordance with the set of shift-related input values; a smaller portion from the document is displayed in a different scale (magnified) with larger details, larger letters, etc. This is illustrated by comparison to Fig. 2A. In addition, zooming may be performed by interacting with controls, touch-screen, voice recognition, knobs, buttons, scroll bars or the like on device 200. [46] Fig. 2E shows device 200 after it was taken by the user downwards and rightwards relative to the reference situation (see Fig. 2A). Here, the level of the device remains approximately constant and the movement is substantially a horizontal shift. The physical movement of device 200 is detected by at least one sensor contained in device 200. Consequently, dedicated algorithms process the data from the sensors and derive a set of at least one shift-related input value. As a result, the display on the screen of device 200 is updated in accordance with the set of shift-related input values; the lower right corner of the document is displayed on the screen of device 200.

[47] In an embodiment of the invention, algorithms of the inventive method may be used to determine similarity of device movements to predefined gestures. These gestures may be selected from a menu and assigned to various actions (similar to the assignment of various actions to left and right mouse buttons). The classification of a given movement to one or more predefined gestures may be accomplished for example using a classification algorithm. The gestures may furthermore be defined by the user, for example by having the user perform a given action a number of times until a threshold variance (for instance) is achieved.

[48] A gesture may in some embodiments be defined in terms of a sequence of some set of position, orientation and acceleration values over time, and/or a sequence of changes in such values over time. These sequences may be characterized for example by polynomial or other mathematical functions, or by predefined or 'taught' (user generated) tables of orientation/acceleration values over time.

[49] Also displayed in Fig. 2E is the combined display mode. Here, if the shift exceeds the limits of the virtual object 240, the screen of device 200 combines the display of the displayed part 230 of virtual object 240 and a second display of an image or a live video of table 210 underlying the virtual object 240. As stated above, the image data acquired by the camera is primarily used for motion detection. However, in some circumstances the image data of the underlying surface (or any other real or virtual background in general) may be incorporated, at least partially into the display. For example, it is within provision of the invention to identify the end of a virtual document, and to display the real background beyond the limits of said virtual document.

[50] It should be understood that the specific movements described above for Figs 2A - 2E are exemplary and not limiting cases. They intend to describe some features of the invention, and its actual implementation is a generalization of the described specific embodiments. Thus for example the leftward movement should be understood as possibly being rightward movement as well, the downward movement should be understood as upward as well, the zoom-in movement should be understood as zoom-out as well, and the choice to describe a combined display of the background (the table) beyond the limits of the virtual object should be understood as a non-limiting suggestion that can be replaced with a different choice as well, all with the necessary modifications.

[51] We now discuss the content that may be displayed in Fig. 2. The electronic content displayed by the device may take any form suitable, for example including video, movies, TV broadcasts such as sports, and the like. The displayed content is not necessarily loaded from the storage (HD etc). In general, the content is displayed within a frame.

[52] The user can use the inventive method in order to focus on a specific element or event within the frame, for example magnifying and tracking a specific element (such as a football player, the ball, or the like) even when that element changes its relative position in the frame. For example, if a user wants to track the movement of a single player in a game, the user can shift/ tilt the inventive device in space and follow this player as he runs across the field, thus changing the position of the player in the broadcast frame or maintaining its position with certain bounds. It is within provision of the invention that the event of interest remains substantially in the center of the display, and optionally magnified. Thus a subframe of the entire original image frame is displayed to the user, the subframe containing the element or event of interest. The subframe can be moved by the user with respect to the frame, in order to better track the event or object of interest.

[53] Optionally, an element in a scene can be selected (for instance by means of a double tap or a special predefined gesture) and the selected object will be tracked automatically without need for the user to rotate or shift the device.

[54] It is within provision of the invention that a "continuous scrolling" function be provided. In continuous scrolling mode, the shift-related input value described above starts a movement of the displayed portion of the document within the entire document, but here, the displayed portion continues to move until a second input value is accepted, for example a second shift-related input value, or until the end of the document is reached. The movement of the displayed portion of the document occurs in some velocity, or rate of change, and some direction in the entire document, determined largely by the shift-related input value. This movement of the displayed portion may occur in ID, 2D or 3D. Scrolling velocity may be substantially constant, variable, decreasing or increasing. Said velocity and the direction of the movement of the displayed portion of document within the entire document may be sensitive to user inputs on-line; the velocity and the direction of scrolling may change in response to the movement of the device in space even during the movement. This may give the user a smooth and intuitive feeling, and a good sense of orientation while navigating through a document.

[55] In an illustrative example for the continuous scrolling mode, the document that is being displayed and navigated may be any electronic content object, for example an electronic road-map, newspaper, and the like. In this example the sensors used to detect and track the position of the device in space include at least one tilt sensor. The map is invoked in a standard way, and displayed on the screen of the smartphone. In general, only a small portion of the map is displayed, in some predefined and optionally user selectable scale. To navigate and scan the map in the traditional way, the user may interact with the touchscreen, dragging his finger across the screen surface. This will generate a signal that will be interpreted by the device as a set of input values. This set of input values will be used by the device to change the displayed portion of the map. For example, if the user wants to scroll the display rightwards, i.e. to move the displayed portion of the map to the right, the user may touch the screen with his finger at a point on the right side of the screen, and drag his finger towards a second point on the left side of the screen. As a result, the displayed portion of the map will move rightwards, displaying parts of the map, like streets and cities that were on the right to the previous display. The user may launch an application previously installed on his device, and which is based on the invention herein disclosed. This will allow another method to control the display in addition to or instead of the touchscreen interaction or any other traditional navigation method. The application may set up and configure the required sensors. For example the application may set up and configure at least one tilt sensor. The application may give an indication to the user that it is ready to operate. To navigate the map and change the displayed portion of the map the user may tilt the device in space; one or more of the yaw, pitch, and roll of the device may all be employed to control scrolling rate and direction. A tilt may be performed, for example, by rotating the device around an axis parallel to its long side that passes through its geometric center, in an angle of between 5 to 15 degrees clockwise or counter clockwise. A general tilt is detected by the appropriate sensors and consequently a set of at least one shift-related input value is determined. In an embodiment of the invention, the general tilt is identified by 3 parameters, Tx, Ty and Tz representing the rotation angle, for example in units of degrees, around each of the mutually orthogonal axes x, y and z correspondingly. Axes x, y, z are fixed to the device, orthogonal to each other, passing through its geometric center, with x and y axes parallel to the screen and z axis orthogonal to the screen. For example, the x axis may be parallel to the width of the device, the y axis may be parallel to the length of the device and the z axis may be parallel to the height (thickness) of the device. Thus, rotation by an angle Tx around the x axis will produce a tilt that may be denoted by the term 'pitch', with positive and negative values of Tx representing counter clockwise and clockwise rotations around the axis, correspondingly or vice-versa. In the same manner rotation by an angle Ty around the y axis will produce a tilt that may be denoted by the term 'roll', and rotation by an angle Tz around the z axis will produce a tilt that may be denoted by the term 'yaw'. A general tilt of the device may be represented by the three values Tx, Ty and Tz, or the pitch, roll and yaw. In an embodiment of the invention, the values Tx, Ty and Tz will be generated by appropriate sensors of the device. The device then interprets these values and uses them to control the display; for example to change the displayed portion within the entire document, map and the like. For example, a Tx value greater in absolute value than a predefined value will generate scrolling along one direction, while Ty value greater in absolute value than a predefined value will generate scrolling along second direction. Said Tx value is generated by the pitch of the device and it may generate forward and backward scrolling, while said Ty value is generated by the roll of the device and it may generate rightwards and leftwards scrolling. The speed of the scrolling and its orientation ('orientation' here refers to the selection between forwards and backwards along one direction of scrolling, and between rightwards and leftwards along second direction of scrolling) may be determined by the value of the rotation, Tx and Ty. In an embodiment of the invention, a specific tilt, for example the Tz value representing the yaw of the device may generate a zoom-in and zoom-out change in the displayed portion of the document, map and the like. For example, counter clockwise rotation of the device about the z axis will be detected by the appropriate sensors within the device. Data from the sensors will be read and processed to provide a positive Tz value representing the yaw angle. Consequently, the display of the device will zoom-out in some relation to Tz. According to this example, clockwise rotation about the z axis will produce negative Tz value, and a consequent zoom- in operation will be performed.

[56] It is within provision of the invention that this method of controlling the display of a device by tilting it in space may be combined with other methods for display control, for example by shifting the device in space as described above. Also, tilting may be detectable by image processing algorithms applied to the image data provided by device's camera, image sensors, or the combination of multiple image sensors.

[57] In the continuous scrolling mode this set of shift-related input values generates a continuous change in the displayed part of the map, in a direction and speed that are substantially determined by the tilt. In this mode the user may continuously change the tilt of the device and the velocity and direction of the scrolling will be controlled accordingly. This may be useful, for example, to follow roads or paths on the map using one hand only and without the need to repetitively drag fingers on the screen. The scrolling will continue in a direction and velocity determined by the tilt, until (for example) the end of the file is reached. To stop scrolling the user may return the device to substantially horizontal disposition or turn-off the inventive method.

[58] Fig. 3 shows a block diagram for an embodiment of the invention. The electronic device implementing the invention comprises a set of sensors 380, processor unit 370, storage unit 375, and a screen 360 with a dedicated display driver.

[59] In Fig. 3, sensors section 380 of the device may comprise a camera, a plurality of cameras e.g. back camera and front camera, a gyroscope (gyro) or a plurality of gyroscopes for orientation detection and motion detection (accelerometer), a photodetector or a plurality of photodetectors, magnetometer, proximity sensor, light sensor, vibration sensor, and possibly other sensors. It is within provision of the invention to use various device sensors to infer or derive values of interest such as: position, orientation, linear velocity, angular velocity, linear acceleration, angular acceleration, further derivatives of position and orientation (e.g. 'jerk'), vibration level, and the like. Sensors section 380 may also comprise a touch-screen control unit.

[60] In Fig. 3, the front/ back camera of sensors section 380 is an image sensor, employing a CCD or CMOS detector, for example. The camera may be equipped with an optical setup comprising optical components, e.g. lens, prism, filter, iris pinhole and the like. The camera controller (not shown in the diagram) may be integrated into the camera unit, the processor unit 370, or implemented as a separate unit between camera unit and processor unit 370. Camera controller functionality may be distributed over more than one unit. The camera controller unit is an optional unit serving as an interface between camera and processor 370, digitizing the acquired analog data at sensor level and transmitting it in appropriate digital format to processor 370. Camera controller may comprise an A/D converter (analog to digital). Camera controller may receive commands from processor unit 370 and adjust camera settings accordingly. Camera controller may perform preliminary processing on the acquired data.

[61] In Fig. 3, a photo-detector may be included in sensors section 380, measuring ambient light.

[62] In Fig. 3, a gyro unit may be incorporated in sensors section 380; for example, a gyroscope based device such as a MEMS gyroscope, which may be sensitive to variations in orientation and to angular accelerations. [63] In Fig. 3, in sensors section 380, a touch-screen controller may be employed, if the screen unit is of a type suitable for such use. The touch-screen is adapted to convert user's touching and finger movements on the screen's surface (or at some distance from the screen surface) into detectable signals, for example of finger position, velocity, and the like. Signals from the touch-screen are transmitted to the touch-screen controller unit.

[64] In Fig. 3, storage unit 375 stores digital data, software, computer programs, files, operating system files, user data, documents, maps and media files. Storage unit 375 may comprise a disk, hard disk, drive, hard drive, memory, flash card, SDRAM, DDR, DDR2, DDR3, and any other methods used to store digital data.

[65] In Fig. 3, processor unit 370 is adapted to execute software applications, run computer programs, perform calculations, interact with the storage unit 375, perform read and write operations to and from storage unit 375, receive data from sensors 380, interpret data from sensors 380, receive image data from cameras, implement data processing and image processing algorithms and much more. Processor unit 370 may comprise a CPU (central processing unit). Processor unit 370 may comprise an application processor e.g. CORTEX A8, ARM 9, and the like.

[66] In Fig. 3, screen unit 360 is a visual output means of the device. Common technologies for the screen panels, among others are: 1. LCD (liquid crystal display), 2. LED (light emitting diode), 3. CSTN (color super twist nematic), 4. TFT (thin film transistor), 5. OLED (organic light emitting diode). Screen 360 is adapted to display pictures, movies, documents, games, video and the like to the user. In some cases, screen 360 is a 'touch-screen'. With a touch-screen, the screen surface also serves as input means to the device; the user may influence the operation or behavior of the device by touching the screen surface with his finger, or by moving his finger across the screen surface.

[67] In Fig. 3, the display driver unit is an optional unit located in general in the interface between processor 370 and the screen 360. For example, Samsung DDI S6D6AA0 may be employed.

[68] In Fig. 3, in an embodiment of the invention, electronic content such as an electronic document is retrieved by processor 370 from storage 375 and displayed on screen 360. In general, for a convenient and readable display, only a portion of the document is displayed on the screen. Therefore, the user may want to navigate through the entire document; while reading, he may want to change the portion of the document that is currently being displayed. [69] In Fig. 3, the back camera from sensors section 380 captures a continuous sequence of images (e.g. video stream) of the background, for example the surface of a table located below the device. The captured sequence of images is processed by processor 370, but in general is not displayed on screen 360. The screen 360 primarily displays a portion of the document.

[70] In Fig. 3, at least one sensor from sensors section 380 detects position and movements of the device in space. For example, while moving the device relative to the background, a sequence of images is captured by the back camera. The data from the camera is transferred to processor 370. Processor 370 analyzes this data and derives values related to the movement performed. Values derived by the processor 370 may be indicative, proportional, equal or otherwise related to the movement. This set of at least one value derived by processor 370 will be treated by the processor 370 as shift-related input values, or movement-related input values. For the analysis of the data, in order to derive movement- related input values from the sequence of images captured, processor 370 may employ an algorithm, e.g. tracking algorithm, feature recognition algorithm, image processing algorithm, registration algorithm, or other algorithm. There are many suitable standard algorithms for tracking a video sequence, e.g. auto-correlation, phase -correlation, optical flow and many others.

[71] In Fig. 3, once processor 370 determines the movement-related input values form the incoming video sequence, it updates the display on screen 360 according to said input values. The display update may be performed via the display driver unit. Thus, processor 370 is adapted to interpret a set of predefined movements of the device in space, performed by the user who carries the device. Furthermore, processor 370 is adapted to update the display via the display driver unit.

[72] In the example above we described the use of a single camera as a sensor for sensing movement of the device in space; this was for clarity only. It is within provision of the invention to use other sensors, a combination of more than one sensor, or a plurality of sensors serving jointly or separately, continuously or intermittently, to sense and detect position, orientation and/or motion of the device in space. The other camera (front camera), the gyro including the accelerometer, as well as other sensors, are all potentially employed to detect position, orientation and/or motion of the device in space, in 3D, 2D or ID.

[73] It is within provision of the invention to limit sensitivity, such that the inventive method is unaffected by movements below some threshold level. This may prove useful to prevent unwanted movements due to inadvertent tremors or vibrations of the hand, unintentional drift of the hand, vibration due to movement in a vehicle or walking, and the like. It is within provision of the invention that the device be sensitive to 'sharp' changes in position/orientation only, namely changes occurring after large accelerations (angular or linear) or large jerk (third derivative of position with respect to time). The sensitivity of response may in some embodiments be tuned just as the sensitivity of the pc-mouse to user movement can often be controlled in software. This feature may be found useful to provide stable behavior of the device, ignoring minor, unintended changes in position, e.g. vibration of the hand, slow drift, unintended instability, etc.

[74] It is within provision of the invention to use adaptive algorithms for determining movement-related input values. For example, in one embodiment of the invention, the device is sensitive to relative movement. In this case, change of position in time will be exploited to determine movement-related input values. Evaluation of first derivatives, second derivatives and higher derivatives of position with respect to time may be incorporated to determine movement-related input values, where position may be lateral position, angular position, orientation and the like. In addition to the relative position values derived above, in an embodiment of the invention the device may be sensitive to absolute position which may be derived for example by processing image data acquired by at least one camera.

[75] Fig. 4 presents a block diagram representing one possible embodiment of the invention. The diagram presents the relations and mutual interactions between the user, the environment and the device (the device includes 'Storage', 'Sensors', 'CPU' and 'Screen').

[76] In Fig. 4, the device contains, among other parts and subsystems, a storage device, CPU, sensors and a screen. The storage device may hold electronic data, files, documents, and other digital objects. The sensors of the device may include at least one camera, infrared sensor, optical sensor, accelerometer, and the like. The screen is the display means. The device uses it to display electronic content e.g. documents to the user.

[77] In Fig. 4, the 'Environment' block represents any information that may be acquired by the sensors of the device. In this diagram, the environment as captured by the sensors enables the device to 'experience' or sense the world.

[78] In Fig. 4, the 'User' block represents any person that uses the device in order to invoke and display an electronic content. In general, the user carries the device in his hand. In other words, the user controls at least partially the position and orientation of the device in space. [79] In Fig. 4, connection 450 represents the effect of the user on the environment as sensed by the device. Connection 460 represents the effect of the environment on the device, mainly through the sensors section. For example, the user may move the device above a table. Consequently, the image captured by the back side camera is changed. Thus, the environment as sensed by the device has changed.

[80] For another example, the user accelerates the device horizontally. The accelerometer and/or image processing of camera information detect this acceleration. Thus, the environment (as defined above) seen by the device has changed. In this manner, by influencing the environment as detected from the device perspective, the user interacts with the device. The device may process and interpret this change of environment to change the display to the user, represented by connection 470. This loop of connections 450-460-470 enables the user to control the display of the device by physical interaction with the device, namely carrying and moving it around, rotating it, accelerating it, and the like.

[81] It is within provision of the invention that a user-control method will be incorporated into the device, for turning on and turning off the inventive methods, and for controlling the mode of operation of the inventive methods. Launching an application associated with the invention, controlling its mode of operation and turning said application off, may be accomplished by user interaction with any standard and non-standard user- control means of the device. User-control means of the device may be for example a dedicated hardware button, switch, knob, slider, touch-screen, icon on a touch-screen, voice recognition input channel, device's sensors and the like. User interaction with user-control means may be executed by pressing a button, touching a touch-screen or an icon on touchscreen, speaking to the device's microphone, performing some predefined movement of the device in space, e.g. rapid shaking from side to side (or other gestures), and the like. The interaction of the user with the user-control means of the device may generate a detectable user-control signal for said device. The device may detect a specific user-control signal and interpret it according to predefined procedures. The interpretation of the user-control signal and the successive procedure executed by the device is the user-control command. Various standard and non-standard techniques and methods for establishing the interaction between the user and the device may be employed in order to turn-on, turn-off and control the mode of operation of the inventive methods.

[82] A single user-control means may be adapted to generate more than one user- control signal. A single user-control signal may be adapted to generate more than one user- control commands. The interpretation of a user-control signal and the selection of the appropriate user-control command may be context-sensitive; it may be dependent on device's status and/ or application's status. For example, pressing a single button may turn- on or turn-off the inventive application, depending on application's status; if the application is turned-off, then pressing the button will turn it on, while if the application is turned-on, then pressing the same button will turn it off. This may be denoted as a 'Toggling' button.

[83] The application may turn on by default with the device. The application may turn on automatically. For example, the device may be configured to automatically launch the application depending on the occurrence of a predefined event. For example, each time a document is opened in the device, the application is also turned on.

[84] Fig. 5 presents a block diagram for the operation of the device in an embodiment of the invention. Four blocks in the diagram represent the operation of the device in regular, traditional mode, before the application based on the disclosed invention is turned on by the user; these are "Load document data" ('Load'), "Select portion of document" ('Select'), "Display portion of document" ('Display') and 'Touchscreen'.

[85] In Fig. 5, the 'Load' block represents the retrieval of a document or other electronic content from the storage means of the device, for example from the hard-disk, Flash card and the like. This 'Load' procedure may occur consequent and in response to user's interaction with the device. The 'Select' block represents the preparation of the document data for display; some portion of the document is selected to be shown on the screen. To select the proper portion for the display, the device may use default values, user preferences, screen properties and configuration, and other parameters. In the 'Select' block the device may also adjust screen configuration to the displayed content and format. The 'Display' block represents the presentation of the selected portion of the document on the screen.

[86] In Fig. 5, the user, in general, reads the document-portion from the screen of the device, and may interact with the device via the touch-screen in order to change/ update the displayed portion of the document. The block 'Touchscreen' represents the input as provided from the touch-screen or any other sensor to the device. Touch-screen input data may be generated by the user, for example by touching the screen in a predefined manner. Consequent to user interaction with the touch-screen, a physical measurement occurs, and is subsequently processed and interpreted. The information is then transferred to the 'Select' block. The 'Select' block uses the information to change/ update the selection of document portion. To summarize this part of the diagram in Fig. 5, MMI enables the user to control the display of a document and thus navigate through a document, by interacting with the touch-screen. [87] In Fig. 5, an embodiment of the invention herein disclosed ("the application") is delineated by the blocks 'Start', "Turn on camera" and "Image processing".

[88] In Fig. 5, the block 'Start' represents the user command to activate the application. This command may be performed by pressing a hardware button, touching an icon on the touch-screen, or any other interaction with the device.

[89] In Fig. 5, the "Turn on camera" block represents the configuration of the camera or any other sensor into a predefined state. Also, the block "Turn on camera" represents the configuration of possible data acquisition capabilities, digitizers, A/D converters and the like. Also, the block "Turn on camera" represents the configuration of possible camera control interface or general sensor control interface.

[90] In Fig. 5, the "Image processing" block represents the analysis performed by the device on the data supplied by the camera (or any other sensor). Data are transferred from the camera to the "Image processing" block in digital form. The "Image processing" block uses various algorithms to analyze the data and interpret it. Image processing, tracking, feature recognition, registration, correlation, phase -correlation and other algorithms and mathematical methods may be found useful to analyze the data from the camera. The purpose of the data analysis in the "Image processing" block is to identify position and movements of the device in space; for example, the sequence of images captured by the camera while shifting the device with reference to the background will potentially reflect the shift or movement that occurred. At least one movement-related input value can be determined from the images data by suitable and well known algorithms. This input value will be used to control the display.

[91] In Fig. 5, the output of the "Image processing" block serves as an input for the 'Select' block. The 'Select' block may use the information provided by the "Image processing" block in order to change/ update the document-portion selected for display. For example, the 'Select' block may use the information provided by the "Image processing" block in a way which is similar to some extent to the way it uses the information from the regular 'Touchscreen' block in order to change/ update the display.

[92] In Fig. 5, block 'Select' may receive input data from both block 'Touchscreen' and block "Image processing" . In principle, both input channels may influence the functionality of the 'Select' block. However, the two channels may be defined to work in exclusive mode; in this mode of operation, only one input channel is active, the other one is shut down or ignored. For example, if the application is turned off, the regular input channel ('Touchscreen' block) is active. When the application is turned on, the regular input channel ('Touchscreen' block) is turned off, or simply ignored, and the input from "Image processing" block remains the only input channel for the 'Select' block. In this situation, the display will not change with regular user operations like touching the touch-screen, mouse movements, etc. and the control of the display will be performed solely by moving the device in space, as required by the application. Alternatively, the device may be designed to work in non-exclusive mode, or the combined mode. In this mode of operation, both input channels "Image processing" and 'Touchscreen' are active and the 'Select' block process the information from both blocks. In this situation, the user may affect the display of the device by touching its screen (or any regular interaction with the device) and/ or by moving it in space as facilitated by the application.

[93] As will be appreciated, the device may be of special usefulness to the vision impaired and other handicapped persons; for example allowing documents to be scanned over at great magnification to allow the vision-impaired to read more easily.

[94] The foregoing description and illustrations of the embodiments of the invention has been presented for the purposes of illustration. It is not intended to be exhaustive or to limit the invention to the above description in any form.

[95] Any term that has been defined above and used in the claims, should be interpreted according to this definition.

[96] The reference numbers in the claims are not a part of the claims, but rather used for facilitating the reading thereof. These reference numbers should not be interpreted as limiting the claims in any form.

Claims

1. A method for controlling aspects of a portable computerized device having a display comprising steps of:

displaying a digital object on said display of said device using a first set of display parameters;

sensing the environment including position, velocity, acceleration and jerk of said device by reading a set of values provided by sensors contained in said device; obtaining a second set of display parameters associated with said values;

displaying said digital object on said display of said device using said second set of display parameters,

wherein a user controls aspects of said display of said digital object by moving, translating, tilting and rotating said device in space.

2. The method of claim 1 wherein said digital object is selected from the group consisting of: electronic content, document, text document, text file, word- processing application, picture, electronic road-map, newspaper, graphics object, movie, video, video frame, electronic game, animation, screen, window, graphical-user-interface (GUI), operating system screen, menu.

3. The method of claim 2 wherein said environment comprises physical parameters of said device selected from the group consisting of: linear position, linear velocity, linear acceleration, orientation, angular velocity, angular acceleration, functions of these parameters, and time derivatives of these parameters; said environment being influenced by user gestures selected from the group consisting of: translating, rotating, tilting, accelerating linearly, accelerating rotationally, shaking, flipping, and sliding, said gestures being used for control over aspects of said portable device.

4. The method of claim 3 where said sensors are selected from the group consisting of: camera, CMOS, CCD, image sensor, accelerometer, magnetometer, inclinometer, gyro, JMU, microphone, proximity sensor, light sensor, capacitive sensor, touch sensor, and vibration sensor.

5. The method of claim 4 wherein said second set of display parameters is

determined by algorithms adapted to process said sensor values.

6. The method of claim 5 wherein said aspects comprise scrolling the display of said digital object by use of said set of sensor values, wherein scrolling the display by moving said portable computing device is provided.

7. The method of claim 6 providing scrolling by means of tilting and/or translating and/or rotating said portable computing device.

8. The method of claim 7 providing for continuous scrolling.

9. The method of claim 5 wherein said aspects comprise zooming the display of a digital object by use of said set of sensor values.

10. The method of claim 9 providing zooming by means of tilting and/or translating and/or rotating said portable computing device.

11. The method of claim 10 providing for continuous zooming.

12. The method of claim 5 where said aspects comprise scrolling and zooming the display of a digital object by use of said set of sensor values such that said digital object appears to remain fixed in real world coordinates, in the form of a combined display.

13. The method of claim 5 wherein the sensitivity of said sensors to the environment of said device may be tuned.

14. The method of claim 5 allowing a user to define, view and manually track a subframe of a digital object wherein said subframe contains a selected image event or element.

15. The method of claim 5 allowing a user to scroll a digital roadmap by means of tilting said device in a direction and magnitude associated with the direction and speed in which one desires to scroll.

16. The method of claim 5 wherein said sensors comprise at least one camera and algorithms adapted to track the position and orientation of said portable computing device by means of image processing of image data from said camera to determine said position and orientation.

17. The method of claim 16 allowing a user to scroll, zoom and turn pages in a

document viewer, word processing application and text files, by physically manipulating position and/ or orientation of said device in space.

18. The method of claim 5 wherein said digital object comprises video information from one or multiple cameras.

19. The method of claim 5 adapted to facilitate MMI for movement-impaired and/ or vision-impaired users.