JP4895016B2 - Imaging device - Google Patents

Description

  The present invention relates to an image encoding device for encoding a captured moving image.

  Digital video cameras and the like have become widespread, and general users can easily pick up moving images. In addition, the performance of digital video cameras has been improving year by year, and some have a CMOS (Complementary Metal-Oxide Semiconductor) sensor capable of imaging at 300 frames per second. When such a camera is used, an encoded image stream capable of special reproduction such as high-quality slow reproduction can be generated even when a moving image with a large subject movement such as sports is captured.

Patent Document 1 discloses a technique for setting a frame speed in accordance with the stationary and movement of a subject with a camera equipped with an automatic focus adjustment device.
JP-A-5-5930

  The data amount of the encoded image stream imaged at a high frame rate, such as 300 frames per second, becomes larger than the data amount of the encoded image stream imaged at a normal frame rate. In particular, a portable and small digital video camera has a strong demand for reducing the amount of data. Therefore, it is desirable to reduce the data amount as much as possible in a portion where a high quality image is not required in the encoded image stream. For example, when the user performs high-speed panning in order to switch subjects and scenes, an image captured during panning can be said to be an image that does not require a high-quality image.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide an image encoding device capable of reducing the amount of data while suppressing a decrease in image quality.

  In order to solve the above-described problems, an image coding apparatus according to an aspect of the present invention detects that the apparatus is moving with a movement amount, speed, or acceleration exceeding a predetermined threshold during moving image capturing. By doing so, the frame rate is lowered. The frame rate may be decreased during a period in which it is detected that the apparatus is moving at a movement amount, speed or acceleration exceeding a predetermined threshold.

  According to this aspect, by reducing the frame rate during a period that does not significantly affect the quality of the moving image, such as during high-speed panning, the amount of data can be reduced while suppressing a decrease in image quality. Can do.

  Another aspect of the present invention is also an image encoding device. This apparatus stops recording a captured picture by detecting that the apparatus is moving at a movement amount, speed, or acceleration exceeding a predetermined threshold during moving image capturing. During the period when it is detected that the apparatus is moving at a movement amount, speed or acceleration exceeding a predetermined threshold, recording of pictures taken during that period may be stopped. “Picture” is a unit of encoding, and the concept may include a frame, a field, a VOP (Video Object Plane), and the like. The frame rate may be reduced while recording of the captured picture is stopped.

  According to this aspect, the amount of data can be reduced while suppressing the deterioration of the image quality by stopping the recording of the picture of the part that does not greatly affect the quality of the moving image, such as during high-speed panning. be able to.

  It may be determined whether or not the apparatus is moving with a movement amount, speed, or acceleration exceeding a predetermined threshold with reference to a change in motion vector over a plurality of captured images. With reference to “change in motion vector”, it may be determined whether or not the movement amount of the macroblock between pictures exceeds a set movement amount. When the set movement amount is exceeded, it may be determined whether or not the degree of coincidence in the movement direction between the macroblocks of each picture exceeds the set degree of coincidence.

  The determination may be made with reference to the change in the motion vector of the region of interest across the plurality of captured images. The “attention area” may be an area including at least a face. The “attention area” may be set based on an area focused by autofocus control.

  With reference to the output of the sensor for detecting the movement of the apparatus, it may be determined whether or not the apparatus is moving at a movement amount, speed or acceleration exceeding a predetermined threshold. The “sensor” may output a change in posture of the apparatus at an angular velocity.

  The moving image may be captured at a frame rate higher than the basic frame rate for a period not exceeding the predetermined threshold. The “basic frame rate” may be a rate corresponding to non-shifted reproduction. Even a moving image with a fast moving subject can be captured at a high frame rate to support high-quality slow playback.

  Refer to the change in motion vector over multiple captured images for a period that does not exceed the predetermined threshold.If the predetermined upper limit value is exceeded, the frame rate is increased, and if the predetermined lower limit value is exceeded, the frame rate is decreased. Also good. While the predetermined threshold value is not exceeded, the frame rate is adaptively switched according to the movement of the subject and the like, so that the data amount can be further reduced while suppressing the deterioration of the image quality.

  It should be noted that any combination of the above-described constituent elements and a conversion of the expression of the present invention between a method, an apparatus, a system, a recording medium, a computer program, and the like are also effective as an aspect of the present invention.

  According to the present invention, it is possible to reduce the amount of data while suppressing deterioration in image quality.

  Before describing the embodiments of the present invention in detail, an outline will be described. When a user is capturing a moving image using an imaging device, the imaging device may be moved greatly in order to switch a subject or a scene. An image captured during the period from the subject or scene before switching to the subject or scene after switching is not an image that the user intends to capture actively. Therefore, an image captured during that period need not be captured at a frame rate that is high enough to enable high-quality slow playback. Therefore, in the embodiment of the present invention, the data amount is reduced by reducing the frame rate during that period. Also, the amount of data is reduced by stopping the encoding or recording during that period.

  FIG. 1 is a diagram showing a general transition of the number of frames when recording a moving image including a scene change. FIG. 1 shows a moving image obtained by capturing a baseball game. The image a shows a scene just before the batter hits the bat but the ball does not fly forward and is caught by the catcher. During this time, the first runner is stealing. To catch the first runner out on a second base, the catcher sent the ball to the second when he caught the ball. In order to capture whether or not the theft was successful, the user who captured this situation panned the imaging device at high speed and switched from the scene near the home base to the scene near the second base.

  The image b shows an image captured while the user is panning the imaging device at high speed in order to switch to a scene near the second base. The image c shows the scene near the switched second base. As shown in FIG. 1, in a general process, the frame rate is constant and the number of recorded frames is constant regardless of scene switching.

  FIG. 2 is a diagram showing frame number transition 1 according to the embodiment of the present invention when a moving image including a scene change is recorded. The image shown in FIG. 2 is also an image of the same situation as in FIG. The image b picked up during the period of panning at high speed is not recorded. Therefore, an image captured during this period may not be encoded. When estimating the start or end of high-speed panning with reference to changes in features such as motion vectors between captured frames, it is necessary to continue imaging itself. At that time, the frame rate may be lowered or maintained. Note that when determining the start or end of high-speed panning using a gyro sensor or the like, the imaging itself may be stopped. Image recording is resumed when it is determined that high-speed panning has been completed.

  As shown in FIG. 2, the image b picked up during the period of panning at high speed is greatly blurred. Even if the image during this period is reproduced, an image of high quality cannot be obtained originally. Further, even if the user's intention of taking an image is taken into consideration, the image is not picked up as a particularly necessary scene. Therefore, the image b of the period need not be recorded, and in that case, the data amount is reduced.

  FIG. 3 is a diagram showing frame number transition 2 according to the embodiment of the present invention when a moving image including a scene change is recorded. The image shown in FIG. 3 is also taken of the same situation as in FIG. The number of frames when recording the image b picked up during the period of high-speed panning is reduced. At that time, the frame rate may be reduced. In the example shown in FIG. 3 as well, the amount of data is reduced by reducing the number of frames when recording the image b captured during the period. Compared to the example shown in FIG. 2, the image at the time of the scene change is smoother. Compared with the moving image shown in FIG. 1, the quality of the entire moving image is almost the same, but the data amount is reduced.

Hereinafter, specific embodiments for realizing the above-described processing will be described in detail.
FIG. 4 is a configuration diagram of an imaging apparatus 500 equipped with the encoding apparatus 100 according to the first embodiment. These configurations can be realized in hardware by a CPU, memory, or other LSI of an arbitrary computer, and in terms of software, it is realized by a program having an image encoding function loaded in the memory. So, functional blocks that are realized by their cooperation are drawn. Therefore, those skilled in the art will understand that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof.

  The imaging device 500 includes an imaging unit 400 and an encoding device 100. The imaging unit 400 includes a solid-state imaging device such as a CCD (Charge Coupled Devices) sensor or a CMOS sensor and a driver that drives the solid-state imaging device, converts the captured image into an electrical signal, and outputs the electrical signal to the encoding device 100. The encoding apparatus 100 receives an input of a moving image in units of frames, encodes the moving image, and outputs an encoded stream.

  The encoding apparatus 100 according to the present embodiment includes MPEG (Moving Picture Experts Group) series standards (MPEG-1, MPEG-) standardized by ISO (International Organization for Standardization) / IEC (International Electrotechnical Commission). 2 and MPEG-4), standardized by ITU-T (International Telecommunication Union-Telecommunication Standardization Sector) which is an international standard organization for telecommunications. 26x series standards (H.261, H.262 and H.263), or H.264, the latest video compression coding standard standardized jointly by both standards organizations. H.264 / AVC (official recommendation names in both organizations are MPEG-4 Part 10: Advanced Video Coding and H.264 respectively).

  In the MPEG series standard, an image frame for intra-frame encoding is an I (Intra) frame, an image frame for forward inter-frame predictive encoding with a past frame as a reference image, a P (Predictive) frame, and past and future An image frame that performs bidirectional inter-frame predictive coding using this frame as a reference image is called a B frame.

  On the other hand, H. In H.264 / AVC, a frame that can be used as a reference image may be a past two frames as a reference image or a future two frames as a reference image regardless of the time. Further, three or more frames can be used as the reference image regardless of the number of frames that can be used as the reference image. Therefore, in MPEG-1 / 2/4, the B frame refers to a Bi-directional prediction frame. Note that in H.264 / AVC, the B frame refers to a bi-predictive prediction frame because the time of the reference image does not matter before and after.

In the specification of the present application, a frame and a picture are used in the same meaning, and an I frame, a P frame, and a B frame are also called an I picture, a P picture, and a B picture, respectively.
In this specification, a frame is used as an example of the encoding unit. However, the encoding unit may be a field. The unit of encoding may be a VOP in MPEG-4.

  The block generation unit 10 divides the image frame input from the imaging unit 400 into macro blocks. Macroblocks are formed in order from the upper left to the lower right of the image frame. The block generation unit 10 supplies the generated macroblock to the differentiator 12 and the motion compensation unit 60.

  If the image frame supplied from the block generation unit 10 is an I frame, the differentiator 12 outputs it to the DCT unit 20 as it is, but if it is a P frame or a B frame, the difference image 12 is supplied from the motion compensation unit 60. Is calculated and supplied to the DCT unit 20.

  The motion compensation unit 60 uses a past or future image frame stored in the frame memory 80 as a reference image, and has the smallest error for each macroblock of the P frame or the B frame input from the block generation unit 10. A prediction area is searched from the reference image, and a motion vector indicating a deviation from the macroblock to the prediction area is obtained. The motion compensation unit 60 performs motion compensation for each macroblock using the motion vector, and generates a predicted image. The motion compensation unit 60 supplies the generated motion vector to the frame rate control unit 70 and the variable length encoding unit 90, and supplies the prediction image to the difference unit 12 and the adder 14.

  The motion compensation unit 60 can apply both bidirectional prediction and unidirectional prediction. In the unidirectional prediction, the motion compensation unit 60 generates a forward motion vector indicating the motion with respect to the forward reference frame. In the bi-directional prediction, in addition to the forward motion vector, two motion vectors of a backward motion vector indicating motion with respect to the backward reference frame are generated.

  The differentiator 12 obtains a difference between the current image output from the block generation unit 10, that is, the image to be encoded, and the predicted image output from the motion compensation unit 60, and outputs the difference to the DCT unit 20. The DCT unit 20 performs a discrete cosine transform (DCT) on the difference image given from the differentiator 12 and gives a DCT coefficient to the quantization unit 30.

  The quantization unit 30 quantizes the DCT coefficient and provides it to the variable length coding unit 90. The variable length coding unit 90 performs variable length coding on the quantized DCT coefficient of the difference image together with the motion vector supplied from the motion compensation unit 60, and generates a coded stream CS. The encoded stream CS is recorded on a recording medium such as a memory card, a hard disk, or a DVD, or is streamed.

  The quantization unit 30 supplies the quantized DCT coefficient of the image frame to the inverse quantization unit 40. The inverse quantization unit 40 inversely quantizes the supplied quantized data and supplies the quantized data to the inverse DCT unit 50. The inverse DCT unit 50 performs inverse discrete cosine transform on the supplied inverse quantized data. Thereby, the encoded image frame is restored. The restored image frame is input to the adder 14.

  If the image frame supplied from the inverse DCT unit 50 is an I frame, the adder 14 stores it in the frame memory 80 as it is. If the image frame supplied from the inverse DCT unit 50 is a P frame or a B frame, the adder 14 is a difference image, and thus is supplied from the difference image supplied from the inverse DCT unit 50 and the motion compensation unit 60. By adding the predicted image, the original image frame is reconstructed and stored in the frame memory 80.

  In the case of P frame or B frame encoding processing, the motion compensation unit 60 operates as described above. However, in the case of I frame encoding processing, the motion compensation unit 60 does not operate and is not shown here. The I frame is supplied to the DCT unit 20 after intra prediction.

  The frame rate control unit 70 changes the frame rate according to a predetermined condition. It may be changed adaptively. For example, when the user moves the imaging apparatus 500 with a movement amount, speed, or acceleration exceeding a predetermined first threshold in order to change a subject or a scene, the frame rate is decreased. The first threshold value is a threshold value for determining whether or not the movement is made to change the subject or the scene, and can be set to a value obtained by the designer through experiments or simulations. Also, the user may set it.

  The frame rate control unit 70 can stop encoding or recording as the frame rate is changed. Also, encoding or recording can be stopped independently of the change in frame rate. Further, the frame rate control unit 70 can reduce the number of frames subjected to the encoding process after the block generation unit 10 and can also reduce the number of frames to be recorded. These number of frames can also be controlled independently of the frame rate. Prior to these processes, the frame rate control unit 70 estimates the motion applied to the imaging apparatus 500 with reference to the motion vector provided from the motion compensation unit 60. This will be specifically described below.

  FIG. 5 is a diagram illustrating Example 1 of the frame rate control unit 70 according to the first embodiment. The frame rate control unit 70 according to the first embodiment includes a motion vector analysis unit 72 and a frame rate setting unit 78. The motion vector analysis unit 72 includes a movement amount detection unit 73 and a movement direction detection unit 74.

  The motion vector analysis unit 72 acquires and analyzes the motion vector of each macroblock from the motion compensation unit 60. The movement amount detection unit 73 detects the movement amount of each macroblock from the acquired motion vector. For example, the amount of movement of each macroblock in the x direction and the y direction is expressed as an absolute value, and a value obtained by adding up the amounts of movement of all the macroblocks for each frame (hereinafter referred to as total movement amount) is detected. . The movement direction detection unit 74 detects the movement direction of each macroblock from the acquired motion vector.

  The motion vector analysis unit 72 determines whether or not the total movement amount exceeds a predetermined second threshold value. The predetermined second threshold value can also be set to a value obtained by the designer through experiments or simulations. When a different mode is installed for each photographing application, the second threshold value may be changed for each mode. This is because the assumed moving speed of the subject differs for each use. This second threshold is associated with the first threshold. That is, the relationship between the total movement amount and the movement applied to the imaging apparatus 500 is modeled, and the second threshold value is determined when the first threshold value is set. When the total movement amount is very large, it can be modeled that the imaging device 500 itself is moved at high speed, not the subject has moved.

  However, even when the total movement amount is very large, there is a possibility that a plurality of subjects are moving at high speed. Therefore, the motion vector analysis unit 72 determines the degree of coincidence in the moving direction of all macroblocks in each frame. When the degree of coincidence exceeds a predetermined third threshold, it is estimated that the imaging device 500 is panned. This third threshold value can also be set to a value obtained by the designer through experiments and simulations, and is associated with the first threshold value. When most macroblocks are moving in the same direction, it is estimated that the imaging apparatus 500 itself is panned in the opposite direction.

  The motion vector analysis unit 72 performs the determination as described above over a plurality of frames. By observing over a plurality of frames, it is possible to grasp the tendency of various parameters such as the total movement amount of macroblocks and the degree of coincidence of movement directions of macroblocks. For example, the increase / decrease tendency of the parameter, the period during which the parameter value is maintained, the number of times the parameter exceeds the second threshold or the third threshold, the frequency, and the like can be obtained.

  The motion vector analysis unit 72 can also perform statistical processing on various parameters. For example, an abnormal value can be eliminated by eliminating a maximum value and a minimum value or performing a moving average process on a parameter over a plurality of frames. Based on these analyses, the motion vector analysis unit 72 instructs the frame rate setting unit 78 to change the frame rate when it is estimated that the imaging apparatus 500 has moved at a movement amount, speed, or acceleration that exceeds the first threshold. To do.

  The frame rate setting unit 78 changes the frame rate in response to an instruction from the motion vector analysis unit 72. For example, it is assumed that the frame rate in the high image quality mode suitable for sports photography and the like and capable of high-quality slow reproduction is 300, and the frame rate in the normal mode is 60. When the motion vector analysis unit 72 determines that the imaging apparatus 500 has moved with a movement amount, speed, or acceleration exceeding the first threshold during shooting in the high image quality mode, the frame rate setting unit 78 sets the frame rate to 300. To 60. If the data amount is to be further reduced, it may be changed to less than 60. For example, it may be 1. Moreover, you may instruct | indicate to the component after the block generation part 10 to stop encoding, changing a frame rate to 60.

  FIG. 6 is a flowchart illustrating the operation of the imaging apparatus 500 according to the first embodiment in which the frame rate control unit 70 according to the first embodiment is mounted. As a premise, it is assumed that the image is shot in the above-described high image quality mode. It is assumed that when a high-speed panning operation or the like by the user is detected, the mode is changed to the normal mode or the low image quality mode. In FIG. 6, the frame rate in the high image quality mode is expressed as a basic value, and the frame rate in the shifted normal mode or low image quality mode is expressed as a reduction value.

  First, when there is a parameter that can be set by the user, the parameter is set (S10). For example, the frame rate after the transition is set. Further, when the sensitivity for invoking the processing for reducing the frame rate can be set by selecting from the weak, medium, and strong modes, one of the modes is selected. The second threshold value and the third threshold value corresponding to the selected mode are set.

  When an instruction to start imaging is input from the user to the imaging apparatus 500 (Y in S12), the imaging apparatus 500 starts imaging (S14). When an instruction to end imaging is input from the user to the imaging apparatus 500 (Y in S16), the process ends. While an instruction to end imaging is not input (N in S16), the following processing is performed.

  The motion compensation unit 60 extracts the motion vector of each macroblock for each frame and gives it to the motion vector analysis unit 72 (S22). The motion vector analysis unit 72 analyzes the given motion vector (S24). Specifically, the total movement amount of the macroblock and the movement direction of the macroblock are detected for each frame.

  The motion vector analysis unit 72 detects whether or not the total movement amount of the macroblock has continued to exceed the second threshold for a predetermined time (S26). This time can also be set to a value obtained by the designer through experiments and simulations. During this time, even if there are one to several frames that do not exceed the second threshold, it may be considered that the second threshold has been continuously exceeded.

  If the second threshold is not exceeded (N in S26), the frame rate is set to the basic value (S28). If the current frame rate is set to the basic value, that state is maintained. Thereafter, the process proceeds to step S16, and the process is continued. When the second threshold value is continuously exceeded (Y in S26), the motion vector analysis unit 72 detects whether or not the degree of coincidence in the moving direction of the frame analyzed during that time has continued to exceed the third threshold value. (S29). During the time period, even if there are about one to several frames having a matching degree lower than the third threshold value, it may be considered that the third threshold value is continuously exceeded.

  If the third threshold value is continuously exceeded (Y in S29), the frame rate is set to the reduction value (S30). If the current frame rate value is set to the reduction value, that state is maintained. Thereafter, the process proceeds to step S16, and the process is continued. If the third threshold is not continuously exceeded (N in S29), the frame rate is set to the basic value (S28). If the current frame rate is set to the basic value, that state is maintained. Thereafter, the process proceeds to step S16, and the process is continued.

  The process of returning from the normal mode or the low image quality mode, that is, the state in which the frame rate is set to the reduction value, to the high image quality mode is also possible by the processes after step S26. In this case, at least one of the second threshold and the third threshold may be different from the threshold used when shifting to the normal mode or the low image quality mode. For example, in order to avoid a situation where the required scene does not return to the high image quality mode, the second threshold value used for determining the return to the high image quality mode is shifted to the normal mode or the low image quality mode. It may be smaller than the second threshold value used for determining.

  In the above description, the process of shifting to the normal mode or the low image quality mode when the high-speed panning operation or the like by the user is detected is described assuming that the image is captured in the high image quality mode. In this regard, assuming that the image is shot in the normal mode, when a high-speed panning operation or the like by the user is detected, the present invention can also be applied to processing for shifting to the low image quality mode. In this case, the frame rate in the normal mode is set as a basic value, and the frame rate in the low image quality mode is set as a reduction value.

  As described above, according to Example 1 of the present embodiment, by referring to the motion vector of the macroblock over the entire frame and changing the frame rate, the data amount can be reduced while suppressing deterioration in image quality. Can be reduced. In addition, when shooting in the high image quality mode, it is possible to suppress the enlargement of the data amount while recording a high quality image. That is, the amount of unnecessary scene data can be reduced.

  FIG. 7 is a diagram illustrating Example 2 of the frame rate control unit 70 according to the first embodiment. The frame rate control unit 70 according to the second embodiment includes an attention area setting unit 76 in addition to the components of the frame rate control unit 70 according to the first embodiment. Here, the attention area is an area including a subject that the user is mainly shooting. For example, humans and animals are applicable.

  The attention area setting unit 76 uses the high frequency component of the DCT coefficient to detect the contour component of the subject, thereby specifying the position of the subject in the image and setting the area including the subject as the attention area. it can. Further, in the person photographing mode or the like, the position of the face in the image may be specified by pattern matching and set as the attention area.

  In the case of the imaging apparatus 500 equipped with an autofocus function, an autofocus control unit 110 is provided. The autofocus control unit 110 automatically controls the subject in focus with reference to the contrast in the image. The user can focus on the subject by moving the imaging apparatus 500 so that the subject is located in the center of the viewfinder. In the case of a configuration in which a position to be focused can be selected from a plurality of points as well as the center of the finder, it is possible to focus by selecting the points.

  The attention area setting section 76 may acquire an in-focus area from the autofocus control section 110 and set the area or an area including the area as the attention area. Note that if the attention area is set to a rectangle, the amount of calculation can be reduced. The attention area setting section 76 sets the set attention area in the motion compensation section 60.

  In addition to the processing described above, the motion compensation unit 60 according to the present embodiment also performs the following processing. That is, the prediction area with the smallest error from the attention area is searched from the reference image stored in the frame memory 80, and a motion vector indicating a shift from the attention area to the prediction area is obtained. The motion compensation unit 60 supplies this motion vector to the motion vector analysis unit 72.

  The motion vector analysis unit 72 acquires the motion vector of the attention area from the motion compensation unit 60 and analyzes it. The movement amount detection unit 73 (not shown in FIG. 7) shown in FIG. 5 detects the movement amount of the attention area from the acquired motion vector. Since it is not necessary to determine the degree of coincidence in the movement direction of the macroblock, the movement direction of the attention area may not be detected.

  The motion vector analysis unit 72 determines whether or not the movement amount of the attention area has exceeded a predetermined fourth threshold value. The fourth threshold value can also be set to a value obtained by the designer through experiments and simulations, and is associated with the first threshold value.

  When shooting a moving subject, such as when shooting a child running at an athletic meet, the user may pan the imaging device 500 to follow the subject. It is necessary to distinguish this panning from panning for switching scenes. In order to allow the former panning, the fourth threshold value may be set to be larger than the second threshold value adjusted by a ratio corresponding to the movement amount detection region.

  The motion vector analysis unit 72 changes the frame rate to the frame rate setting unit 78 when it is estimated that the imaging apparatus 500 has moved at a movement amount, speed, or acceleration exceeding the first threshold based on the movement amount of the region of interest. Instruct. The frame rate setting unit 78 changes the frame rate in response to an instruction from the motion vector analysis unit 72.

  FIG. 8 is a flowchart illustrating the operation of the imaging apparatus 500 according to the first embodiment in which the frame rate control unit 70 according to the second embodiment is mounted. FIG. 8 is based on the flowchart of FIG. 6, and the same reference numerals are given to the same processes. Hereinafter, description of the same processing as in FIG. 6 will be omitted as appropriate, and different processing will be mainly described.

  First, since the process from step S10 to step S16 is the same as that of FIG. 6, description is abbreviate | omitted. In FIG. 8, while the imaging end instruction is not input (N in S16), the following processing is performed.

  When the attention area setting section 76 sets the attention area (Y in S18), the motion compensation section 60 extracts the motion vector of the attention area for each frame and gives it to the motion vector analysis section 72 (S20). The motion vector analysis unit 72 analyzes the given motion vector (S24). Specifically, the movement amount of the attention area is detected for each frame.

  The motion vector analysis unit 72 detects whether or not the movement amount of the region of interest continues to exceed the fourth threshold value for a predetermined time (S26). This time can also be set to a value obtained by the designer through experiments and simulations.

  If the fourth threshold value is not exceeded (N in S26), the frame rate is set to the basic value (S28). If the current frame rate is set to the basic value, that state is maintained. Thereafter, the process proceeds to step S16, and the process is continued. When the fourth threshold value is continuously exceeded (Y in S26), the frame rate is set to the reduction value (S30). If the current frame rate is set to the reduction value, that state is maintained. When the frame rate is set to the reduction value, the attention area setting section 76 releases the attention area (S31). Further, the attention area is not set while the frame rate is the reduction value. This is because this period is not a period in which the user is photographing a specific subject, but is a period in which a scene or the like is switched. Thereafter, the process proceeds to step S16, and the process is continued.

  When the attention area setting unit 76 does not set the attention area in Step S18 (N in S18), the processing is the same as the processing after Step S22 described in FIG. For convenience of explanation, the process of step S29 is not shown in FIG. 8, but the process of determining the degree of coincidence in the movement direction is also performed.

  Also in the second embodiment, as in the first embodiment, the process of returning from the normal mode or the low image quality mode to the high image quality mode can be performed by the processes after step S26. Since the attention area is canceled in the normal mode or the low image quality mode, the total movement amount of the macroblock over the entire frame is compared with the second threshold value. The region of interest may be reset after returning to the high image quality mode.

  As described above, according to Example 2 of the present embodiment, the data amount is reduced while suppressing the deterioration of the image quality by changing the frame rate with reference to the motion vector of the attention area including the subject. can do. In addition, since the determination is made using the motion vector of the attention area including the subject, the user's intention of shooting can be accurately estimated, and erroneous determination can be reduced.

  Next, Example 3 will be described. In the third embodiment, the frame rate is adaptively changed according to the change in the movement of the subject or the like during execution of the processing according to the first embodiment and the mode in which high-speed imaging is performed at a frame rate higher than the normal frame rate. This is a combination of processing. The configuration according to the third embodiment is the same as that shown in FIGS. 4 and 5, but the motion vector analysis unit 72 and the frame rate setting unit 78 also perform the following processing.

  The frame rate setting unit 78 is set with a plurality of basic frame rates, transitionable frame rates, that is, frame rate changes. The basic frame rate refers to a frame rate corresponding to shooting in the normal mode. The change point of the frame rate can be set to a frame rate that is an integral multiple of the basic frame rate.

  The motion vector analysis unit 72 determines whether or not the total movement amount of the macroblock has exceeded a predetermined fifth threshold and whether or not the macroblock has moved below a predetermined sixth threshold. The fifth threshold value is an upper limit value for determining whether to increase the current frame rate, and the sixth threshold value is a lower limit value for determining whether to decrease the current frame rate. is there. Any threshold value can be set to a value obtained by the designer through experiments or simulations.

  FIG. 9 is a flowchart illustrating an operation of the imaging apparatus 500 according to the first embodiment in which the frame rate control unit 70 according to the third embodiment is mounted. FIG. 9 is based on the flowchart of FIG. 6, and the same reference numerals are given to the same processes. Hereinafter, description of the same processing as in FIG. 6 will be omitted as appropriate, and different processing will be mainly described.

  First, the processing from step S10 to step S30 is basically the same as that shown in FIG. In the third embodiment, after the frame rate is set or maintained at the basic value in step S28, the process proceeds to step S32.

  The motion vector analysis unit 72 determines whether or not the number of times that the total movement amount of the macroblock exceeds the fifth threshold within the unit analysis period exceeds a predetermined number of times set in advance (S32). When the number of times the fifth threshold value has been exceeded exceeds a predetermined number (Y in S32), the current frame rate is changed in the high direction (S34). Thereafter, the process proceeds to step S16 to continue the processing.

  In step S32, when the number of times exceeding the fifth threshold does not exceed the predetermined number (N in S32), the motion vector analyzing unit 72 determines that the total movement amount of the macroblock is within the unit analysis period. It is determined whether or not the number of times less than the number exceeds a predetermined number set in advance (S36). When the number of times that the value falls below the sixth threshold exceeds a predetermined number (Y in S36), the current frame rate is changed to a lower direction (S38). Thereafter, the process proceeds to step S16 to continue the processing. If the number of times that the value falls below the sixth threshold does not exceed the predetermined number (N in S36), the process proceeds to step S16 without changing the frame rate. The predetermined number of times used for the processing in step S32 and step S36 can also be set to a value obtained by the designer through experiments and simulations.

  Note that when the change point of the frame rate to be changed in the low direction or the high direction no longer exists in the low direction or the high direction, the frame rate is not changed even when the predetermined number of times described above is exceeded. The change in the frame rate is basically a stepwise transition to adjacent change points, but exceeds the fifth threshold or falls below the sixth threshold, for example, exceeding a predetermined number of times or more. When the frequency is very high, the frame rate may be changed to a change point that is two or more steps away from the current operation point. The fifth threshold value and the sixth threshold value may be set in advance for each change point of each frame rate.

  As described above, according to Example 3 of the present embodiment, by referring to the motion vector of the macroblock over the entire frame and changing the frame rate, the data amount can be reduced while suppressing the deterioration of the image quality. Can be reduced. Further, by adaptively changing the frame rate within a range that does not exceed the second threshold, it is possible to reduce the amount of data while recording a high-quality image. Within this range, the frame rate is increased to accommodate high-quality slow playback as the movement amount of the motion vector increases. However, if the movement amount of the motion vector increases beyond this range, it is determined that panning has been performed at high speed, and the frame rate is lowered.

  FIG. 10 is a configuration diagram of an imaging apparatus 500 according to the second embodiment. An imaging apparatus 500 according to the second embodiment has a configuration in which a camera shake detection unit 120 is added to the imaging apparatus 500 according to the first embodiment. Therefore, since the components other than those related to the camera shake detection unit 120 are the same as those in the first embodiment, the description thereof is omitted.

  The camera shake detection unit 120 includes a gyro sensor and the like. When the camera shake is detected by the output signal of the sensor, the camera shake detection unit 120 moves the correction lens or the solid-state imaging device in a direction to cancel the camera shake. The gyro sensor outputs the movement of the imaging device 500 in the yaw direction and the pitch direction as an angular velocity.

  The frame rate control unit 70 according to the present embodiment refers to the value obtained from the camera shake detection unit 120 and corresponding to the movement of the imaging device 500 in the yaw angle and pitch direction, and the motion applied to the imaging device 500 Guess. Therefore, in this embodiment, it is not necessary to acquire a motion vector from the motion compensation unit 60. The frame rate control unit 70 determines whether or not the value corresponding to the above-described movement in the yaw angle and pitch direction exceeds a predetermined sixth threshold value. The sixth threshold value can also be set to a value obtained by the designer through experiments and simulations, and is associated with the first threshold value.

  FIG. 11 is a flowchart illustrating the operation of the imaging apparatus 500 according to the second embodiment. FIG. 11 is based on the flowchart of FIG. 6, and the same reference numerals are assigned to the same processes. Hereinafter, description of the same processing as in FIG. 6 will be omitted as appropriate, and different processing will be mainly described.

  First, since the process from step S10 to step S16 is the same as that of FIG. 6, description is abbreviate | omitted. In FIG. 11, while the imaging end instruction is not input (N in S16), the following processing is performed.

  The frame rate control unit 70 acquires an output value of a gyro sensor or the like from the camera shake detection unit 120 (S40). The frame rate control unit 70 determines whether or not the output value exceeds the sixth threshold value (S42). If the sixth threshold is not exceeded (N in S42), the frame rate is set to the basic value (S44). If the current frame rate is set to the basic value, that state is maintained. Thereafter, the process proceeds to step S16, and the process is continued.

  If the sixth threshold is exceeded (Y in S42), the frame rate is set to a reduction value (S46). If the current frame rate is set to the reduction value, that state is maintained. Thereafter, the process proceeds to step S16, and the process is continued. Also in the second embodiment, as in the first embodiment, the process of returning from the normal mode or the low image quality mode to the high image quality mode can be performed by the processes after step S42. The sixth threshold value may also be set to a different value when determining whether or not to shift to the normal mode or the low image quality mode and when determining whether or not to return to the high image quality mode.

  As described above, according to the present embodiment, the amount of data can be reduced while suppressing a decrease in image quality by detecting the movement of the imaging apparatus 500 with a sensor and changing the frame rate. In addition, since the movement of the imaging apparatus 500 is directly detected by the sensor, it is possible to avoid a situation in which the frame rate is lowered due to a large movement of the subject or the like against the user's intention. In addition, since it is not necessary to estimate the movement of the imaging apparatus 500 from the captured frame, the imaging itself is performed in a period in which it is determined that the imaging apparatus 500 is moved with a movement amount, speed, or acceleration exceeding the first threshold. It is also possible to stop the process.

  The present invention has been described based on some embodiments. This embodiment is an exemplification, and it will be understood by those skilled in the art that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are also within the scope of the present invention. is there.

  For example, in the embodiment, a message indicating a shooting mode, a frame rate, and how many times the image is captured may be displayed on a display unit such as a finder (not shown) of the imaging apparatus 500. Thereby, the change of the frame rate can be notified to the user in real time.

It is a figure which shows the transition of the general frame number at the time of recording the moving image containing a scene change. It is a figure which shows the transition 1 of the frame number which concerns on embodiment of this invention at the time of recording the moving image containing a scene change. It is a figure which shows the transition 2 of the frame number which concerns on embodiment of this invention at the time of recording the moving image containing a scene change. It is a block diagram of the imaging device carrying the encoding device which concerns on Embodiment 1. FIG. 6 is a diagram illustrating Example 1 of the frame rate control unit according to Embodiment 1. FIG. 6 is a flowchart illustrating an operation of the imaging apparatus according to the first embodiment in which the frame rate control unit according to the first embodiment is mounted. 6 is a diagram illustrating Example 2 of the frame rate control unit according to the first embodiment. FIG. 12 is a flowchart illustrating an operation of the imaging apparatus according to the first embodiment in which the frame rate control unit according to the second embodiment is mounted. 12 is a flowchart illustrating an operation of the imaging apparatus according to the first embodiment in which the frame rate control unit according to the third embodiment is mounted. 6 is a configuration diagram of an imaging apparatus according to Embodiment 2. FIG. 6 is a flowchart illustrating an operation of the imaging apparatus according to the second embodiment.

Explanation of symbols

  10 block generation units, 12 differentiators, 14 adders, 20 DCT units, 30 quantization units, 40 inverse quantization units, 50 inverse DCT units, 60 motion compensation units, 70 frame rate control units, 72 motion vector analysis units, 73 movement amount detection unit, 74 movement direction detection unit, 76 attention area setting unit, 78 frame rate setting unit, 80 frame memory, 90 variable length encoding unit, 100 encoding device, 110 autofocus control unit, 120 camera shake detection unit 500 Imaging device.

Claims (4)

An imaging unit that acquires a moving image to be imaged by shooting;
A motion detector that determines whether or not the imaging apparatus is moving at a movement amount or speed exceeding a predetermined threshold;
A frame rate setting unit for setting a frame rate of the imaging unit;
With
The motion detection unit detects a motion vector of a frame constituting the moving image acquired by the imaging unit, determines whether the detected motion vector exceeds a predetermined first threshold value, or of the imaging device By determining whether or not the output of the sensor for detecting movement exceeds a predetermined second threshold, it is determined whether or not the imaging apparatus is moving at a movement amount or speed exceeding the threshold,
The frame rate setting unit reduces the frame rate of the imaging unit when it is determined that the imaging unit is moving at a movement amount or speed exceeding the threshold. An imaging unit that acquires a moving image to be imaged by shooting;
A motion detector that determines whether or not the imaging apparatus is moving at a movement amount or speed exceeding a predetermined threshold;
An encoding unit that encodes the moving image acquired by the imaging unit and records the encoded image on a recording medium;
With
The motion detection unit detects a motion vector of a frame constituting the moving image acquired by the imaging unit, determines whether the detected motion vector exceeds a predetermined first threshold value, or of the imaging device By determining whether or not the output of the sensor for detecting movement exceeds a predetermined second threshold, it is determined whether or not the imaging apparatus is moving at a movement amount or speed exceeding the threshold,
The encoding unit stops encoding a moving image or recording on a recording medium when it is determined that the imaging apparatus is moving at a movement amount or speed exceeding the threshold. An imaging device. The motion detection unit determines whether the detected motion vector exceeds the first threshold over a plurality of frames, or whether the output of the sensor exceeds the second threshold over a predetermined period. The imaging apparatus according to claim 1, wherein the imaging apparatus determines whether or not the imaging apparatus is moving at a movement amount or speed exceeding the threshold.   The imaging apparatus according to claim 1, wherein the motion detection unit detects a motion vector of a region of interest in a frame constituting the moving image acquired by the imaging unit.

Images