JP2021135911A - Display device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a display device which outputs an image in which a handwritten figure and an image of a content including the handwritten figure are superimposed as comment information.
SOLUTION: In a display device including a display unit for displaying contents, an input unit, and a control unit, the control unit accepts input of a handwritten figure for the displayed content via the input unit. , The image of the content in the range including the handwritten figure and the image obtained by superimposing the handwritten figure are output as comment information.
[Selection diagram] Fig. 4

Description

The present invention relates to a display device.

In recent years, personal computers and tablets that support touch and pen input have become widespread, and various techniques related to handwriting input have been proposed.

For example, if multiple handwritten strokes are text inputs, one or more handwritten word blocks are generated, positioned according to a first predetermined layout criterion, and if multiple handwritten strokes are sketches, a sketch content object is generated. , A technique has been proposed that responds to receiving a plurality of handwritten strokes in an input area by positioning according to a second predetermined layout standard different from the first predetermined layout standard (see Patent Document 1).

Further, a technique for deleting image data by inputting a predetermined command (gesture) to the image data input immediately before has been proposed (see Patent Document 2).

Special Table 2018-530042 Japanese Unexamined Patent Publication No. 6-309093

Reviews by others are indispensable for brushing up content such as document files, and applications such as applications that allow handwritten comments to be entered are also being used. Some such applications have the ability to display a list of comments added to the content. However, the display of the comment list alone does not accurately show the points and areas that are being talked about, and the user cannot grasp which part of the content the handwritten comment was input to. There was a challenge.

In order to solve the above-mentioned problems, an object of the display device of the present invention is to provide a display device that outputs an image in which a handwritten figure and an image of a content including the handwritten figure are superimposed as comment information.

The first embodiment for solving the above-mentioned problems is a display unit (for example, the display unit 110 of FIG. 2), an input unit (for example, the input unit 120 of FIG. 2), and a control unit (for example, the display unit 120 of FIG. 2) for displaying the contents. For example, in a display device including the control unit 100) of FIG. 2, the control unit accepts input of a handwritten figure for the displayed content via the input unit, and the range including the handwritten figure. It is a display device characterized by outputting an image in which an image of a content and the handwritten figure are superimposed as comment information.

In the second embodiment, a display unit for displaying the content (for example, the display unit 110 in FIG. 2), an input unit (for example, the input unit 120 in FIG. 2), and a control unit (for example, the control unit 100 in FIG. 2) ), The control unit accepts the input of the handwritten figure for the displayed content via the input unit, and when the handwritten figure that receives the input is an arrow, the arrow is The display device is characterized in that an image of contents in a range included in the indicated area and an image obtained by superimposing the handwritten figure included in the range included in the area indicated by the arrow are output as comment information.

According to the display device of the present invention, it is possible to output an image in which a handwritten figure and an image of a content including the handwritten figure are superimposed as comment information.

It is a figure for demonstrating the outline of the system in 1st Embodiment. It is a figure for demonstrating the functional structure of the display device in 1st Embodiment. It is a figure for demonstrating content information and comment information in 1st Embodiment. It is a figure for demonstrating the process in 1st Embodiment. It is a figure for demonstrating the operation example in 1st Embodiment. It is a figure for demonstrating the operation example in 1st Embodiment. It is a figure for demonstrating the process in 2nd Embodiment. It is a figure for demonstrating the operation example in 2nd Embodiment. It is a figure for demonstrating the process in 3rd Embodiment. It is a figure for demonstrating the operation example in 3rd Embodiment.

Hereinafter, embodiments when the display device of the present invention is applied will be described. It should be noted that each embodiment is described for convenience of explaining the present invention, and the technical scope of the present invention is not limited to the following embodiments.

[1. First Embodiment]
[1.1 Overall configuration]
FIG. 1A is a diagram showing the appearance of the display device 10. The display device 10 may be, for example, an information processing device such as a tablet-type terminal device or a notebook computer, or a large display device used in an electronic whiteboard or an electronic blackboard (IWB: Interactive Whiteboard). , It may be a table type display device. The user's operation on the display device 10 is input by, for example, detecting a touch on the touch panel or detecting an operation using the input device 15.

Although the display device 10 will be described as a single device in the present embodiment, it may be configured to include a plurality of display devices or a configuration in which the display device and the server are linked.

For example, as shown in FIG. 1B, the system may be configured to include a display device 10H used by a user who is an administrator and a plurality of display devices used by other users. The outline of the operation of the entire system will be described below.

(1) In the display device 10H used by the user H who is the administrator, the user H selects the content (F10). At this time, the user H may add a comment to the content by himself / herself. The comment may be stored as a file (data) separate from the content.

(2) The user A reads out the content and the comment from the display device 10H in the display device 10A used by the user A (F12).

(3) User A executes a process of adding a comment to the read content (F14).

(4) The comment added to the content by the user A is transmitted to the display device 10H (F16). The display device 10H updates the comment corresponding to the content based on the received comment, so that the comment added by the user A is reflected in the content.

(5) Similarly, the user B and the user C also execute the process of adding a comment to the content selected by the user H.

In this way, by using this system, each user can add a comment to the content.

Further, the display device may be a system that can be connected to the server device. For example, as shown in FIG. 1 (c), the server device 20 that the display device 10 can communicate with is connected to the system. The server device 20 may be in the same network or in the cloud. As shown in FIG. 1 (c), each of the display devices 10 is configured to be able to communicate with the server device 20. That is, the contents stored in the display device 10H and the data related to the comments in the configuration of FIG. 1B are stored in the server device 20. The display device 10H, the display device 10A, and the display device 10B can each realize the same operation as the above-described configuration by referring to the server device 20.

The following description describes a case where one display device is used, but a person skilled in the art can also apply the second configuration and the third configuration.

[1.2 Functional configuration]
Subsequently, the functional configuration of the display device 10 will be described with reference to FIG. It is sufficient for each configuration to have what is necessary for operation, and it is not an essential configuration.

The control unit 100 is a functional unit for controlling the entire display device 10. The control unit 100 realizes various functions by reading and executing various programs stored in the storage unit 150, and is composed of one or a plurality of arithmetic units (for example, a CPU (Central Processing Unit)). ..

Further, the control unit 100 operates as a comment processing unit 102, a UI (User Interface) processing unit 104, and a user authentication unit 106 by executing the program.

The comment processing unit 102 executes an editing process such as inputting, changing, or deleting a comment according to an operation input by the user. The comment processing unit 102 adds necessary information to the comment input by the user and stores it in the comment information 156 of the storage unit 150. When the comment information is stored in the other display device 10 or the server device 20, the comment processing unit 102 may send and receive comments to and from the other device via the communication unit 160.

The comment processing unit 102 processes the object input by the user as a comment. Objects entered by the user include, for example, graphics, text (characters), and symbols.

For example, the comment processing unit 102 accepts the input of a text comment when the user inputs it from the software keyboard or the hardware keyboard. Further, when the figure is drawn by the touch panel or the figure is drawn by the operation pen, the comment processing unit 102 accepts the input of the comment of the input figure. In the following description, a figure input by a touch operation on the touch panel or an operation with an operation pen is referred to as a handwritten figure. The handwritten figure includes, for example, a figure constituting a character or a symbol, and a figure such as a point, a line, an arrow, or a rectangle. Further, the comment processing unit 102 may switch between a text comment and a graphic comment by a touch button or an operation to accept the input.

Further, the comment processing unit 102 may convert the handwritten figure into text by recognizing the character, or may convert the text into an image. In the present embodiment, the user can add a comment without being aware of the text input and the handwriting input.

Further, the comment processing unit 102 may process an image such as a stamp as a comment added by the user.

The comment processing unit 102 includes other general editing processes such as insertion, deletion, change, replacement, and movement, change of character type, movement of cursor, change of font, and line color / thickness. Known editing processes such as modification and line deformation can be performed.

The UI processing unit 104 generates a screen of the user interface and displays it on the display unit 110. In the present embodiment, the UI processing unit 104 performs a process of displaying a comment and a list of comments on the display unit 110. For example, as shown in FIG. 5A, the content is displayed on the entire display screen W100, and the comments are displayed vertically arranged in the area R100. The comments may be arranged in chronological order or may be displayed for each user. For example, a user icon is displayed in the area R102 of FIG. 5A. When the user icon is selected by the user, the UI processing unit 104 reads the comment corresponding to the selected user from the comment information 156 and displays it. The UI processing unit 104 may display the list of comments superimposed on the content, or divides the display area of the display unit 110 and displays the list of comments in an area different from the area where the content is displayed. You may. Further, the UI processing unit 104 may switch between displaying and hiding the list of comments.

The user authentication unit 106 authenticates the user. Specifically, the user authentication unit 106 refers to the user information 158, for example, authenticates the user who makes a comment, or authenticates the owner (user) of the content. The user authentication unit 106 may use an external authentication server. When the authentication server is used, the user authentication unit 106 and the user information 158 may be stored in the authentication server.

The display unit 110 displays contents and comments, and displays various states of the display device 10 and operation input states. For example, it is composed of a liquid crystal display (LCD), an organic EL panel, electronic paper using an electrophoresis method, and the like.

The input unit 120 receives an operation input from the user. For example, it is composed of a capacitance type touch panel and a pressure sensitive type touch panel. The input unit 120 may be a combination of a touch panel and an operation pen, or may be an input device such as a keyboard or a mouse. In addition, for example, the input unit 120 may input information by the user, such as performing voice input in combination with a microphone or the like.

Further, the input unit 120 detects a position where the user has operated the input and outputs the input position to the control unit 100. The input position is preferably, for example, the coordinates on the display screen displayed on the display unit 110 as the position detected on the touch panel. Other than that, it may be a position such as a cursor position (row or column position in a sentence) or layout information (for example, on a button or a scroll bar).

The storage unit 150 is a functional unit that stores various programs and various data necessary for the operation of the display device 10. The storage unit 150 is composed of, for example, a semiconductor memory SSD (Solid State Drive), an HDD (Hard Disk Drive), or the like. Further, the storage unit 150 stores the content information 152, the handwritten graphic information 154, the comment information 156, and the user information 158.

The content information 152 stores the content and the information about the content. For example, as shown in FIG. 3A, the content information 152 includes the following information.

-Content ID: Information for identifying the content.
-Content: Information about the content. The content is, for example, a text file, an image file, a PDF (Portable Document Format) file, an office file, or the like. As these contents, the actual data may be stored, or the stored location (for example, the location of the folder, the URL (uniform resource locator)) may be stored. Further, the content may be composed of one or a plurality of pages.
-Comment ID list: A list of comment IDs. A comment ID corresponding to the comment added to the content is included.

The handwritten figure information 154 stores information related to the handwritten figure input for the content. The handwritten figure input to the content means a handwritten figure directly input to the content by being handwritten by the user with a finger, an operation pen, or the like in the range in which the content is displayed. The handwritten figure information 154 stores the handwritten figure and the content ID of the content in which the handwritten figure is input. Further, the position of the handwritten figure on the content may be stored. Further, the handwritten figure may be stored as a raster format image or may be stored as a vector format image.

The comment information 156 stores a comment added by the user to the content and information about the content. For example, as shown in FIG. 3B, the comment information 156 includes the following information.

-Comment ID: Information for identifying a comment.
-Content ID: Information for identifying the content corresponding to the comment.
-Position: Information on the position where a comment is added in the content. It is the content or the position of the content in the page, and is indicated by, for example, coordinates (XY coordinates). Further, when the content is text data such as text data, it may be represented by rows and columns.
-Creator: Information about the creator of the comment.
-Reference ID: Information for identifying the comment (parent comment) referred to by the comment. When the user adds a comment hierarchically, the parent comment ID is stored in order to identify the parent comment. In addition, related comments that are comments of parents and children are expressed as comment groups.
-Comment: Information indicating the content of the comment actually entered (added) by the user. For example, text data, image data, data related to an attached file, and the like are stored in the content.

The user information 158 stores information about the user. For example, as the information for identifying the user, the login ID, password, name, terminal used, biometric information of the user, ID of the input pen to be used, and the like for each user may be stored. Further, the user information 158 is stored by associating an icon with each user.

The communication unit 160 communicates with another device. For example, it connects to a LAN (Local Area Network) to send and receive information about comments and send and receive documents to and from other devices. As the communication method, communication such as LTE / 4G / 5G may be used in addition to LAN which is a general Ethernet (registered trademark).

In each of the above-described configurations, the display device 10 may have a function as required. For example, when each data is stored in the display device 10 or the server device 20 used by a user (general user) who is not an administrator, the content information 152 and the comment information 156 do not need to be stored in the storage unit 150. Further, various information may be stored in a server on the cloud as needed.

The configuration shown in FIG. 2 may include at least a control unit 100 and a storage unit 150, and may include one capable of realizing the operation in the present embodiment. In addition, some functions may be realized by an external device. For example, the display unit 110 may be configured as an external device such as a display. That is, the display device 10 may be composed of a display device and a terminal device, and the control unit 100 and the storage unit 150 may be provided in the terminal device.

[1.3 Processing flow]
The main processing flow of this embodiment will be described with reference to the flow of FIG. First, the control unit 100 displays the content on the display unit 110 (step S102). For example, the control unit 100 may read the content from the content information 152 or may receive the content via the communication unit 160.

The control unit 100 determines whether or not there is a touch input (step S104). Specifically, the control unit 100 determines that there is a touch input when the touchdown or touch-up to the input unit 120 (touch panel) is detected, or when the input by the operation pen to the input unit 120 is detected.

When there is a touch input, the control unit 100 transitions to the handwritten comment input state (mode) and executes the handwritten figure input process (step S104; Yes → step S106). The handwritten comment input state (mode) is a state (mode) in which a handwritten figure can be directly input to the displayed content. When the handwritten figure input process is executed, the control unit 100 outputs the handwritten figure as a handwritten figure based on the trajectory of the input to the touch panel. The control unit 100 may temporarily output the handwritten figure to the storage unit 150 or output the handwritten figure information 154 to the handwritten figure information 154 until the input is completed.

Subsequently, the control unit 100 determines whether or not the input of the handwritten figure is completed (step S108). Here, as a case where the input of the handwritten figure is completed, the following cases can be mentioned.

(1) When a predetermined time has elapsed since the touch-up When the input unit 120 (for example, the touch panel) has detected the touch-up of the user's touch input and a predetermined time has elapsed, the control unit 100 has a handwritten figure. It is determined that the input is completed. The predetermined time may be preset or may be set by the user. The predetermined time is preferably about 100 milliseconds to 3 seconds, and more preferably about 1 second. As a result, even if touchdown and touchup are detected a plurality of times by inputting a plurality of handwritten figures in succession, the control unit 100 touches again before the predetermined time elapses. When down is detected, the input of the handwritten figure can be continued.

(2) When an operation other than the operation in the handwritten figure input is detected The operation in the handwritten figure input is, for example, an operation of moving the touched position while touching the touch panel or moving while touching the operation pen (drag). Operation). Here, when an operation other than the normal handwritten figure input operation (for example, double tap, long press, etc.) is detected, the control unit 100 determines that the handwritten figure input is completed.

(3) When an operation indicating that the input of the handwritten figure is completed When the user performs an operation indicating that the input of the handwritten figure is completed, the control unit 100 determines that the input of the handwritten figure is completed. For example, the control unit 100 causes the display unit 110 to display a button such as "input end", and when the button is selected by the user, determines that the input of the handwritten figure is completed.

When the control unit 100 determines that the input of the handwritten figure by the user is completed, the control unit 100 ends the handwritten comment input state (mode) and displays the menu on the display unit 110 (step S108; Yes → step S110). Specifically, the control unit 100 displays a menu at the upper left or lower right of a rectangular area in contact with the outer circumference of the handwritten figure, or displays a menu around the rectangular area. Further, the control unit 100 may display a thick colored frame on the menu or blink the menu after a predetermined time has elapsed after displaying the menu. By displaying the menu in such a display mode, the control unit 100 can easily display the menu so that the user can recognize the menu.

The displayed menu includes an item indicating that the input handwritten figure is added (commented) to the content as a comment. By making it possible to select whether to comment out via the menu, the user adds a series of handwritten figures entered from touchdown to touchup to the displayed content to the displayed content. It can be confirmed whether or not.

Subsequently, when commenting is selected from the menu (step S112; Yes), the control unit 100 outputs the input handwritten figure as a comment (step S114). By doing so, the control unit 100 targets the handwritten figure to be commented while the handwritten figure input process is being executed (while the handwritten comment input state (mode) is in progress). Can be decided as and commented.

In addition, the control unit 100 can determine the handwritten figure to be commented in the case where all the handwritten figures input before the predetermined time elapses after detecting the touchdown after detecting the touchdown are commented. If not, the user may be asked to specify a handwritten figure to be commented. In this case, the control unit 100 displays a screen for allowing the user to specify the handwritten figure to be commented (for example, a screen for designating the range of the area including the handwritten figure) on the display unit 110, and makes a comment based on the user's specification. Determine the handwritten figure to be used.

In step S114, first, the control unit 100 determines the handwritten figure to be commented out. Then, among the contents displayed on the display unit 110, a range including the handwritten figure to be commented is specified, and the image of the contents in the range is captured to acquire the content. The image of the content acquired in this way serves as the background of the handwritten figure to be commented out.

Here, the control unit 100 specifies, for example, an area (for example, a rectangular area or a circular area) in which the outer circumference of the handwritten figure touches as a range including the handwritten figure. The control unit 100 may specify a region in which the circumference is expanded by a predetermined size (for example, 10 pixels) from the region where the outer circumference of the handwritten figure touches as a range including the handwritten figure.

The control unit 100 may capture a range in which the characteristics of the image of the content as the background can be grasped by specifying the area to be captured based on the characteristics of the image of the content. For example, the control unit 100 specifies the background color of the content image, and divides the content image into a plurality of areas according to the specified background color. Further, the control unit 100 specifies as an area to capture one or a plurality of areas including the handwritten figure to be commented out among the plurality of areas in the image of the content. By doing so, the control unit 100 can acquire the image of the content in the area including the handwritten figure and the certain content included in the image of the content among the images of the content.

When the area to be captured includes an object other than the handwritten figure to be commented (for example, a handwritten figure already input), the control unit 100 superimposes the image of the content and the object. Images may be acquired.

Further, the control unit 100 may specify an area to be captured based on the characteristics of the handwritten figure to be commented out. For example, when the control unit 100 recognizes a figure including a line extended in a certain direction as a handwritten figure, the area indicated by the arrow or line from the direction indicated by the arrow or line (one end direction). It may be captured so as to include an image of the content included in (target area). For example, in the case of an arrow, the direction at one end is the direction in which the arrowhead is located, and in the case of a line, the direction is the direction in which the end (the position where touch-up is detected) is located. The method of specifying the target area will be described later.

Subsequently, the control unit 100 outputs to the storage unit 150 an image in which the acquired image of the content and the handwritten figure input by the user are superimposed.

By the above-described processing, the control unit 100 superimposes the handwritten figure input by the user and the image of the content (that is, the image that becomes the background of the handwritten figure) and outputs it as a comment.

At this time, the control unit 100 adds necessary information to the comment and outputs (stores) it to the comment information 156. For example, the control unit 100 stores the comment, the ID of the content to which the comment is added, and the position to which the comment is added in the comment information 156. Here, the control unit 100 sets the position where the comment is added to, for example, a predetermined position on the boundary line of the range including the commented handwritten figure (any of the four corners if it is a rectangular area). The control unit 100 may set the position to which the comment is added as the center position of the range in which the content is captured, or may be the position where the input of the handwritten figure is started or the position around the center. When the handwritten figure is temporarily output to the storage unit 150, the control unit 100 may output (store) the handwritten figure to the handwritten figure information 154.

When the user selects to execute another process instead of commenting, the control unit 100 executes the selected process (step S112; No → step S116; Yes).

If neither process is selected, the control unit 100 may transition the process to step S106 so as to execute the handwritten figure input process again (step S116; No → step S106).

[1.4 Operation example]
An operation example of this embodiment will be described with reference to FIGS. 5 and 6. 5 (a), 5 (b), and 5 (c) are views showing an example of a display screen displayed on the display unit 110. The display screen W100 shown in FIG. 5A is an example of a display screen when the content is displayed as a whole and the comments are displayed in a list in the area R100. The area R100 may be located at any position on the top, bottom, left, or right of the screen edge. Further, the area R100 may not be located at the edge of the screen, and may be superimposed and displayed on the content in a window format, for example.

In the area R100, comments added to the content are displayed in a list. For example, the comment of the user A is displayed as the comment C100, and the comment of the user B is displayed as the comment C102.

When displaying comments in a list, the parent comment / child comment may be displayed so that they can be confirmed. For example, the comment C102, which is a child comment of the comment C100, may be displayed (displayed so as to be nested) so as to hang from the comment C100, which is the parent comment.

Further, on the display screen W100, the handwritten figure C104 is displayed as the handwritten figure input by the user on the content.

After the input of the handwritten figure is completed, the display screen displayed on the display unit 110 transitions from the display screen W100 to the display screen W110 shown in FIG. 5B. On the display screen W110, the menu M110 is displayed in the vicinity of the handwritten figure C114. The user can comment out the handwritten figure by selecting an item (“comment” in the example of FIG. 5B) indicating that the handwritten figure is commented out from the menu M110. When it is selected to comment the handwritten figure, an image obtained by superimposing the handwritten figure and the image of the content in the area including the handwritten figure is output as a comment. After the comment is output, the display screen displayed on the display unit 110 transitions to the display screen W120 shown in FIG. 5 (c).

In the display screen W120 shown in FIG. 5C, in the area R120 for displaying comments in a list, as a comment, for example, an image in which a handwritten figure and an image of a content including the handwritten figure are combined, such as a comment C120, are displayed. Is displayed as a comment. Therefore, the user can grasp the input comment and the image of the content targeted by the comment only by checking the area R120.

Further, FIG. 6 is another diagram showing an operation example. FIG. 6A is an example of a display screen W130 displayed on the display unit 110 when a handwritten figure is input to a portion where the text “○○ steamer” is displayed as an image of the content. .. In FIG. 6A, as a handwritten figure, a line is input under the text of "○○ steamer", and further, an arrow and a figure constituting a character indicating "Title change !!" are input. show. Further, the area E130 indicates a rectangular range in which the outer circumference of the input handwritten figure touches.

In this case, the image of the content in the range indicated by the area E130 is captured, and the image obtained by superimposing the captured image and the handwritten figure is output. Further, comment information including the output image as a comment is output. As a result, in the area R130 for displaying the comments in a list, as shown in the comment C130, an image in which the content image (text "○○ steamer") and the handwritten figure are superimposed is displayed as a comment. The user can grasp that the comment "Title change !!" is for the text "○○ steamer" just by looking at the comment C130.

FIG. 6B is an example of the display screen W140 displayed on the display unit 110 when an arrow is input as a handwritten figure to be commented out. The area E140 included in the display screen W140 indicates a rectangular range in which the outer circumference of the arrow, which is a handwritten figure to be commented out, touches.

The display screen W140 includes the area 142 and the area 148 as areas including the handwritten figure (handwritten character) input before the arrow, which is the handwritten figure to be commented, is input. show. The area 142 and the area 148 are located in the direction indicated by the arrow input as a handwritten figure.

In this case, the display device 10 specifies, for example, the area E142 as the target area among the area 142 and the area 148 which are the areas located in the direction indicated by the arrow.

Here, the display device 10 may specify the area including the target image as the target area after determining the image (target image) indicated by the arrow. For example, the display device 10 may search for an image existing around the position of the arrow among the content images, and determine the target image from the search result. Further, the display device 10 measures the distance from the character or image closest to the arrow to another character or image adjacent to the character or image, and sets the distance from the character or image to a place where the distance becomes wider than a predetermined value. You may recognize it. The character searched by the display device 10 may be the text (character) of the object, the character composed of the handwritten figure, or the text (character) included in the image of the content. You may. Further, the display device 10 may determine a group image including an image (aggregate) adjacent to the target image as a background. Subsequently, the display device 10 specifies an area including the target image and the group image (for example, an area where the outer periphery of the target image and the group image touches) as the target area.

The display device 10 may use a method of obtaining the distance between the objects and determining the target area based on the distance. Further, when the target area cannot be determined only by the distance between the objects, the display device 10 may use a method of executing character recognition, meaning recognition, and figure recognition to determine the range in which the text can be communicated as the target area. .. Further, if the determination cannot be made automatically, the display device 10 may display a GUI image (for example, a screen on which the target area can be specified) in order to allow the user to specify the target area. ..

The method for specifying the target area may be a combination of a plurality of the above-mentioned methods, or may be a method other than the above-mentioned methods. Further, the method for specifying the target area may be predetermined or may be selected by the user.

A specific operation example will be described with reference to FIG. 6 (b). First, the display device 10 searches for one character (handwritten graphic object) existing around the position of the arrow. As a result, for example, one character among the characters included in the area 142 is searched by the display device 10. Next, the display device 10 searches for a character adjacent to the searched character. Here, since the characters included in the area 142 are not widely spaced from each other, the display device 10 can recognize the characters included in the area E 142 as the target image by setting an appropriate predetermined value. On the other hand, the display device 10 determines that the distance between the characters included in the area E142 and the characters included in the area E148 is wider than a predetermined value, so that the target image includes the characters included in the area E142. Can be recognized as. As a result, the display device 10 specifies the area 142 as the target area among the area 142 and the area 148 located in the direction indicated by the arrow.

After specifying the target area, the display device 10 subsequently specifies the area to be the range for capturing the image of the content by any of the following methods.

(A) Method of Identifying an Area Containing an Arrow and an Area Containing a Target Area The display device 10 identifies an area including an area including an arrow and an area in a direction indicated by the arrow, and captures an image of the content. The range. For example, the display device 10 specifies a rectangular area in contact with the outer periphery of the area including the arrow and the area in the direction indicated by the arrow.

In the example of FIG. 6B, the display device 10 specifies an area E144, which is a rectangular area in which the outer circumferences of the area E140 and the area E142 are in contact with each other, as a handwritten figure to be commented as a range for capturing an image of the content. do.

(B) Method of Specifying Only the Target Area The display device 10 specifies only the area in the direction indicated by the arrow and sets it as a range for capturing the image of the content.

In the example of FIG. 6B, the display device 10 specifies only the area E142 as the range for capturing the image of the content.

(C) Method of specifying including the direction of the other end of the arrow The display device 10 specifies a region in the other end direction that is not the direction indicated by the arrow, in addition to the area including the arrow and the target area. Then, the display device 10 sets a region including three regions of the region including the arrow, the target region, and the region in the other end direction other than the direction indicated by the arrow as the range for capturing the image of the content. The display device 10 may specify the area in the other end direction by the same method as the method for specifying the target area.

In the example of FIG. 6B, the case where the area E146 is specified by the display device 10 as the area in the other end direction is shown. In this case, the display device 10 touches the outer periphery of the area E140, which is the area including the arrow, the area E142, which is the target area, and the area 146, which is the area in the other end direction, as the range for capturing the image of the content. Identify the rectangular area. As a result, as shown in the comment C140 of FIG. 6B, the arrow of the handwritten figure, the characters located in the direction of one end and the other end of the arrow, and the arrow of the handwritten figure and the content of the area including the character are included. The image superimposed with the image is output as a comment.

In this way, the display device 10 specifies the target area based on the image or object of the content, so that the target area includes a group of images or objects. Further, the display device 10 can acquire the target image that can be understood as the meaning of the direction indicated by the arrow for the user by acquiring the image of the range including the target area, and can clarify the meaning of the comment. can.

Note that FIG. 6B has described the case where an arrow is input as a handwritten figure, but even when a line is input as a handwritten figure, an image of the content in the direction indicated by the line may be captured.

In this embodiment, the condition for outputting the handwritten figure as a comment (output condition) is a condition that the item for commenting the handwritten figure is selected from the menu displayed after a lapse of a predetermined time after touching up. However, other conditions may be used. For example, the condition may be that the menu is not displayed and a predetermined time has elapsed since the touch-up was performed.

If the content is composed of a plurality of pages, comments are added for each page. In this case, the information of the corresponding page is stored in the handwritten graphic information 154 and the comment information 156. Further, the UI processing unit 104 displays the comment added to the page of the displayed content on the display unit 110 by superimposing it on the content, or displays it as a list of comments.

According to the present embodiment, when the display device adds a handwritten figure input by the user to the content as a comment, the display device outputs as a comment an image of the content of the area including the handwritten figure and an image obtained by superimposing the handwritten figure. Can be done. In addition, comment information including such an output image can be output. As a result, the user can easily grasp which part of the content the handwritten comment is entered by simply browsing the list of comments.

[2. Second Embodiment]
[2.1 Processing flow]
The second embodiment is an embodiment in which a file can be attached to the comment. In this embodiment, in addition to the first embodiment, the control unit 100 executes the file attachment process shown in FIG. 7. In the present embodiment, the UI processing unit 104 displays for accepting from the user an operation of attaching a file to the individual comments displayed in the list or the comments selected by the user among the comments displayed in the list ( For example, the clip button) is included and displayed. In the present embodiment, only the points different from the first embodiment will be described below.

First, the control unit 100 determines whether or not there has been an operation of attaching a file (step S202). The operation of attaching a file is, for example, an operation of selecting a clip button by the user, which is a display for accepting an operation of attaching a file displayed in a comment displayed in a list. In this case, the comment including the clip button selected by the user is the comment to which the file is attached. The operation of attaching a file may be an operation of dragging the file to be attached and dropping it on a comment to attach the file (drag and drop).

When there is an operation of attaching a file, the control unit 100 controls the display unit 110 to display a file selection screen for selecting a file to be attached to the comment (step S202; Yes → step S204).

The control unit 100 displays the file selection screen until the file is selected by the user (step S206; No → step S204). On the other hand, when the file is selected by the user, the control unit 100 attaches the file selected by the user to the comment (step S206; Yes → step S208).

Specifically, the control unit 100 associates the comment information 156 corresponding to the comment to which the file is attached with the information of the file selected by the user in step S206. For example, the control unit 100 stores the file itself to be attached and the storage location of the file (for example, a file path or URL) in the comment information 156 corresponding to the comment to which the file is attached. Any method may be used as long as the comment and the file can be associated with each other, and the control unit 100 associates the comment ID of the comment to which the file is attached with the file selected by the user. A method of storing in the storage unit 150 may be used.

When the user performs an operation to display the file attached to the comment, the control unit 100 displays the file attached to the comment on the display unit 110 based on the user's operation.

[2.2 Operation example]
An operation example of this embodiment will be described with reference to FIG. FIG. 8A is a diagram showing an example of the display screen W200 on which the content is displayed. At the right end of the display screen W200, an area R200 for displaying a list of comments is secured.

In the area R200, comments added to the content are displayed in a list. Here, in the comments displayed in the list in the area R200, the clip button M200 is displayed, for example, like the comment C200. Note that FIG. 8A is a diagram showing a case where the clip button M200 is displayed in the comment selected by the user, but the clip button is displayed in each comment displayed in the area R200. May be good.

When the user selects the clip button M200, the display screen W210 including the file selection screen R210 is displayed as shown in FIG. 8B. The user selects a file to be attached to the comment from the file selection screen R210.

FIG. 8C is a diagram showing an example of the display screen W220 when a file to be attached to the comment is selected by the user. In the area R220 for displaying the list of comments secured at the right end of the display screen W220, the comment to which the file is attached is displayed to indicate that the file is attached. For example, like the comment C220 to which the file is attached, the clip fastening button M220 is displayed as a display indicating that the file is attached to the comment. Further, in this case, the clip fastening button M220 may be used as a button for accepting an operation of displaying a file attached to the comment.

Here, when the clip-on button M220 is selected by the user, the file attached to the comment is opened by the default application and the contents of the file are displayed as shown in FIG. 8 (d). The screen W230 is displayed.

According to this embodiment, the user can attach a related file to the comment. In addition, the attached file can display the contents of the file by operating the comment displayed as a list. In this way, the user can grasp the comment added to the content and the file related to the comment only by browsing the list of comments.

[3. Third Embodiment]
The third embodiment is an embodiment in which the control unit 100 controls to switch the display or non-display of the handwritten figure displayed superimposed on the content. In this embodiment, in addition to the first embodiment and the second embodiment, the control unit 100 executes the display switching process shown in FIG. In the first embodiment and the second embodiment, the display switching process is executed by the control unit 100 after the handwritten figure is stored in the handwritten figure information 154.

In the present embodiment, the display or non-display of the handwritten figure will be switched for each user. Further, when the control unit 100 outputs the handwritten figure to the handwritten figure information 154, the control unit 100 is handwritten with the user who input the handwritten figure by, for example, executing a process of outputting the handwritten figure information 154 for each user. Output in association with the figure. The determination of the user who has input the handwritten figure is, for example, determined by using the input device 15, the determination based on the display device in which the comment input operation is performed, or the determination by performing the switching operation at the time of input. It can be realized by doing.

Further, in the present embodiment, when the handwritten figure is superimposed and displayed on the content, the control unit 100 displays an identification display indicating the user who has input the handwritten figure in the vicinity of the handwritten figure. The displayed identification display is, for example, an icon indicating a user, a label or a button displaying a user name, or the like. In the present embodiment, an icon indicating a user is displayed as an identification display.

[3.1 Processing flow]
The display switching process will be described with reference to FIG. First, when the handwritten figure is not displayed on the content, the control unit 100 reads the handwritten figure information 154 and superimposes and displays the read handwritten figure on the content (step S302; No → step S304). The control unit 100 also displays an icon indicating the user who input the handwritten figure in the vicinity of the handwritten figure (for example, a position where a comment is added).

Subsequently, the UI processing unit 104 determines whether or not a touch input has been made to the icon indicating the user displayed on the content (step S306). When it is determined that the touch input for the icon on the content has been performed, the control unit 100 hides the handwritten figure input by the user indicated by the selected icon (step S306; Yes → step S308).

Specifically, the control unit 100 identifies the user indicated by the touch-input icon, and prevents the handwritten graphic information input by the specified user from being read from the handwritten graphic information 154. Then, the control unit 100 redisplays the handwritten figure so that the handwritten figure input by the user indicated by the selected icon is not superimposed on the content and displayed. The control unit 100 keeps the icon on the content displayed.

Subsequently, the control unit 100 determines whether or not a touch input has been made to the icon on the content selected by the user in step S306 (step S310). When it is determined that the touch input, which is an operation of selecting an icon on the content, has been performed, the control unit 100 displays the handwritten figure input by the user indicated by the selected icon (step S310; Yes → step). S312). Specifically, the control unit 100 reads from the handwritten graphic information 154 corresponding to the user indicated by the touch-input icon, and superimposes and displays the read handwritten graphic on the content displayed on the display unit 110. To do.

Note that the control unit 100 may transition the process to step S306 so as to execute the process of hiding the handwritten figure again after executing the process of step S312.

[3.2 Operation example]
An operation example of this embodiment will be described with reference to FIG. FIG. 10A is a diagram showing an example of a display screen W300 in which a handwritten figure is superimposed on the content. The area R300 shown in FIG. 10A is an area including a handwritten figure input by the user C. Further, an icon M300 indicating the user C is displayed as an identification display in the vicinity of the handwritten figure input by the user C. For example, as shown in FIG. 10A, the icon M300 is displayed around the area R300 including the handwritten figure input by the user C.

When the touch input, which is an operation for selecting the icon M300, is detected, the display screen W310 in which the handwritten figure input by the user C is hidden is displayed as shown in FIG. 10B. The icon M310 indicating the user C is displayed.

Further, when the touch input which is the operation of selecting the icon M310 is detected on the display screen W310, the handwritten figure input by the user C is displayed, and again as shown in the display screen W300 of FIG. 10A. The display screen is displayed. In this way, each time the icon indicating the user C is selected, the display / non-display of the handwritten figure input by the user C can be switched.

According to the present embodiment, the user generally selects the identification display for switching the display / non-display of the handwritten figure which cannot be hidden or requires a complicated operation to hide it. It can be done by operation. As a result, the user can always secure an area for inputting the handwritten figure.

In the present embodiment, it has been described that the display / non-display of the handwritten figure is switched for each user, but the display / non-display of the handwritten figure may be switched for each comment, and the handwritten figure may be switched. You may be able to switch between showing and hiding all comments at once. When switching the display / non-display of the handwritten figure for each comment, the storage unit 150 may store the handwritten figure information 154 for each comment. Further, the UI processing unit 104 reads out the handwritten figure information 154 corresponding to the comment other than the comment selected to be hidden, and superimposes the handwritten figure stored in the read handwritten figure information 154 on the content. It should be displayed.

[4. Modification example]
The present invention is not limited to the above-described embodiments, and various modifications can be made. That is, the technical scope of the present invention also includes embodiments obtained by combining technical means appropriately modified within a range that does not deviate from the gist of the present invention.

Further, each function may be configured by a program or may be configured by hardware. Then, when it is realized by a program, it is recorded on a recording medium, and the program may be read from the recording medium and executed, or stored in a network, downloaded and executed.

In addition, although the above-described embodiments are described separately for convenience of explanation, it is needless to say that they may be combined and executed within a technically possible range.

Further, the program that operates in each device in the embodiment is a program that controls a CPU or the like (a program that causes a computer to function) so as to realize the functions of the above-described embodiment. Then, the information handled by these devices is temporarily stored in a temporary storage device (for example, RAM) at the time of processing, then stored in various ROM, HDD, and SSD storage devices, and read by the CPU as needed. , Correction / writing is performed.

Further, in the case of distribution to the market, the program can be stored and distributed in a portable recording medium, or transferred to a server computer connected via a network such as the Internet. In this case, it goes without saying that the storage device of the server computer is also included in the present invention.

Further, an application having each function may be realized by installing and executing it on various devices such as a smartphone, a tablet, and an image forming device.

10 Display device 100 Control unit 102 Comment processing unit 104 UI processing unit 106 User authentication unit 110 Display unit 120 Input unit 150 Storage unit 152 Content information 154 Handwritten graphic information 156 Comment information 158 User information 160 Communication unit

Claims (10)

In a display device including a display unit for displaying contents, an input unit, and a control unit,
The control unit
The input of the handwritten figure for the displayed content is accepted via the input unit, and the input is received.
An image of the content in the range including the handwritten figure and an image obtained by superimposing the handwritten figure are output as comment information.
A display device characterized by that. The control unit
The display device according to claim 1, wherein when a predetermined output condition is met, an image of contents in a range including the handwritten figure and an image obtained by superimposing the handwritten figure are output as comment information. .. Equipped with a touch panel as an input unit
The output condition is characterized in that the menu is displayed after a predetermined time has elapsed after the handwritten figure is input and touched up, and output as comment information is selected from the menu. The display device according to claim 2. The display device according to any one of claims 1 to 3, wherein the control unit captures an image of a region including the outer circumference of the handwritten figure of the content to obtain an image of the content. The display device according to claim 4, wherein when the handwritten figure is an arrow, the control unit captures an image of a region indicated by the arrow in the content to obtain an image of the content. The display device according to any one of claims 1 to 5, wherein the control unit stores the comment information in association with a file. The display device according to any one of claims 1 to 6, wherein the control unit controls to switch the display or non-display of the handwritten figure. The control unit
The display device according to any one of claims 1 to 6, wherein an identification display indicating a user who has input the handwritten figure is performed in the vicinity of the handwritten figure. The display device according to claim 8, wherein the control unit controls to switch the display or non-display of the handwritten figure by selecting the identification display. In a display device including a display unit for displaying contents, an input unit, and a control unit,
The control unit
The input of the handwritten figure for the displayed content is accepted via the input unit, and the input is received.
When the handwritten figure that has received the input is an arrow, the image of the content in the range included in the area indicated by the arrow and the image obtained by superimposing the handwritten figure included in the range included in the area indicated by the arrow are commented. Output as information,
A display device characterized by that.

Images