JP6794854B2 - Arithmetic processing unit and control method of arithmetic processing unit - Google Patents

Description

The present invention relates to an arithmetic processing unit and a control method for the arithmetic processing unit.

The GPU (Graphic Processing Unit) used in the arithmetic processing unit is originally a processor for image processing, but since it is optimized for matrix calculation by having a large number of sum-product arithmetic units, it is a process for machine learning. It is often used as a processor that performs. The GPU is also generally used in the process of performing deep learning.

In deep learning, processing is often performed using a neural network. For example, in the case of deep learning of image recognition, it has two processes, a forward process for determining what a given image is and a backward process for updating the parameters of the neural network for determining. The arithmetic processing unit that performs deep learning performs backward processing using the difference between the calculation result in the forward processing and the expected value, and updates the parameters of the neural network. Then, the arithmetic processing unit improves the accuracy of the forward processing by using the updated parameters.

A neural network may consist of multiple layers. In the forward propagation in which the forward processing is performed, arithmetic processing such as extraction of features is performed on the input data in each layer, and the output result is obtained. Then, in the back propagation in which the backward processing is performed, learning to update each parameter using the difference between the result of the forward propagation and the expected value is repeated in the opposite direction to the forward propagation in each layer. As described above, the neural network has a multi-layer structure in which different arithmetic processes performed in each layer are performed. Since it has such a structure, in order to update the parameters for each layer, the difference between the calculation result of the subsequent layer and the expected value is obtained, and the difference is set to the previous layer and the difference calculation of that layer is performed. Learning is performed while propagating the result of the above to the previous layer. The one before and one ahead in the explanation here are based on the direction of forward propagation.

Further, as an arithmetic process mainly used in image recognition in deep learning, there is a process called a convolutional neural network. In a convolutional neural network, an operation called convolution is often used. Hereinafter, it is referred to as "convolution operation". For example, in the case of image recognition, a filter having predetermined parameters as each element is arranged in the area on the input image. In forward propagation, the input side is called the bottom and the output side is called the top. Even in back propagation, the positional relationship does not change, and the output side is called the bottom and the input side is called the top. The input data of each layer in the forward propagation direction including the original image is called "bottom data". When the convolution operation is performed in the image recognition of deep learning, the input data is in a bitmap format, and when the data are arranged in order and stacked, the input data becomes the same as the apparent image. Further, each element data constituting the input data represents a shade in the case of gray scale, and represents data for three colors in the case of RGB (Read Green Blue). The filter is also called "weight data".

Then, by multiplying each element of the input data in which the filter is arranged and each element of the filter, the feature amount of the area in which the filter is arranged in the input data is calculated. The arrangement of the filter on the original image is performed on the entire input data using the predetermined movement width of the filter, and the calculated feature amount is summarized as the output data output as the result of the convolution operation. The output data that is the result of the convolution operation in this forward processing is called "top data".

There are two operations in the convolution operation in backward processing. One is an operation of calculating the difference parameter using the difference between the top data and the expected value, which is the calculation result of the forward processing, and the original image. The difference between the top data and the expected value, which is the calculation result of the forward processing, is called "top difference data". Further, the calculated difference parameter is called "weight difference data". This weight difference data is used to update the weight data to improve the calculation accuracy in the forward processing. The other is an operation of calculating the difference for the operation of the previous backward processing by using the top difference data and the weight data. The difference for the calculation of the previous backward processing is called "bottom difference data". This bottom difference data is used as the top difference data in the previous layer.

Japanese Unexamined Patent Publication No. 2011-13168

However, the total number of convolution operations can be calculated as follows. For example, the number of element data of bottom data is C'× C', the number of bottom data is N, the number of element data of weight data is K × K, and the number of elements of top difference data is C ×. Consider the case where C is and the number of top data is P. Further, it is assumed that one convolution operation in the forward processing is one multiplication and one addition. In this case, the total number of operations in the forward processing is P × C × C × N × K × K × 2. For example, in the case of C = 13, N = 256, K = 3, C = 13 and P = 256, the total number of operations in the forward processing is 256 × 13 × 13 × 256 × 3 × 3 × 2 = 1990360512. Here, when the size of the weight data is large, the speed-up method by Fast Fourier Transform (FFT) is effective, but when the condition is not satisfied, the effect of calculation constraint by FFT is obtained. It is difficult. Therefore, it is difficult to reduce the number of operations while maintaining the image recognition accuracy system in the convolution operation that is not bound by a specific condition.

The disclosed technique has been made in view of the above, and an object of the present invention is to provide an arithmetic processing unit and a control method of the arithmetic processing unit that reduce the number of arithmetic operations while maintaining an image recognition accuracy system.

In one embodiment of the arithmetic processing apparatus and the control method of the arithmetic processing apparatus disclosed in the present application, the storage unit excludes a predetermined number of element data from the first data having the element data forming the matrix and the element data forming the matrix. The second data having the arranged shape is stored. The conversion unit converts the first data based on the arrangement shape of the second data. The convolution calculation unit performs a convolution calculation on the first data converted by the conversion unit using the second data as a filter.

In one aspect, the present invention can reduce the number of operations while maintaining the image recognition accuracy system.

FIG. 1 is a diagram for explaining the overall flow of processing in a convolutional neural network. FIG. 2 is a diagram for explaining a forward convolution operation and a backward convolution operation. FIG. 3 is a block diagram showing details of the arithmetic processing layer. FIG. 4 is a block diagram showing details of a convolution calculation unit that performs a forward convolution operation according to the first embodiment. FIG. 5 is a diagram showing an example of a filter definition. FIG. 6 is a diagram for explaining an example of conversion of bottom data. FIG. 7 is a diagram showing the appearance of the bottom data after conversion. FIG. 8 is a diagram showing an example of conversion of bottom data. FIG. 9 is a diagram showing another example of bottom data conversion. FIG. 10 is a diagram for explaining a forward convolution operation when the new filter definition is used. FIG. 11 is a diagram for explaining the backward convolution bottom difference operation when the new filter definition is used. FIG. 12 is a diagram for explaining the backward convolution weight difference calculation when the new filter definition is used. FIG. 13 is a flowchart of processing in the arithmetic processing layer when the new filter definition is used. FIG. 14 is a flowchart of the forward convolution calculation by the convolution calculation unit according to the first embodiment. FIG. 15 is a flowchart of a backward convolution calculation by the convolution calculation unit according to the first embodiment. FIG. 16 is a diagram for explaining a pooling process when the number of strides by the pooling process unit according to the second embodiment is 2. FIG. 17 is a diagram for explaining a pooling process when the number of strides by the pooling process unit according to the second embodiment is 1. FIG. 18 is a diagram for explaining a forward convolution operation by the convolution operation unit according to the third embodiment. FIG. 19 is a diagram for explaining an example of a forward convolution operation using a new filter definition by the convolution operation unit according to the fourth embodiment. FIG. 20 is a diagram for explaining another example of the forward convolution operation using the new filter definition by the convolution operation unit according to the fourth embodiment. FIG. 21 is a diagram for explaining a description example of a program for forward convolution operation. FIG. 22 is a diagram for explaining a description example of a program for backward convolution weight difference calculation. FIG. 23 is a diagram for explaining a description example of a program for backward convolution bottom difference calculation. FIG. 24 is a hardware configuration diagram of the arithmetic processing unit.

Hereinafter, examples of the arithmetic processing unit and the control method of the arithmetic processing unit disclosed in the present application will be described in detail with reference to the drawings. The following examples do not limit the arithmetic processing unit and the control method of the arithmetic processing unit disclosed in the present application.

FIG. 1 is a diagram for explaining the overall flow of processing in a convolutional neural network (CNN). Here, in this embodiment, the processing in CNN for image recognition will be described. As shown in FIG. 1, the arithmetic processing unit 1 receives the input of the input data 2. The arithmetic processing unit 1 executes processing by a plurality of arithmetic processing layers 11 to 13 in the CNN. In the following, when the arithmetic processing layers 11 to 13 are not distinguished, they are simply referred to as “arithmetic processing layer 10”.

In each arithmetic processing layer 10, arithmetic processing such as extraction of feature points is performed in the propagation direction which is the direction of arrow P1. In the following, the arithmetic processing in the direction toward the arrow P1 by the arithmetic processing unit 1 may be referred to as “forward arithmetic”. Further, in each arithmetic processing layer 10, two types of arithmetic processing are performed in the back propagation direction in the arrow P2 direction in order to improve the accuracy of extraction of feature points in each layer in the back propagation direction in the arrow P2 direction. I do. In the following, the arithmetic processing in the direction toward the arrow P2 by the arithmetic processing unit 1 may be referred to as “backward arithmetic”.

Each arithmetic processing layer 10 acquires weight data, which is a filter used for extracting a feature amount, from a storage device such as a memory. Further, the arithmetic processing layer 11, which is the first layer, acquires the input data 2 from a storage device such as a memory. Then, the arithmetic processing layer 11 executes the convolution operation using the input data 2 as the bottom data and the weight data for the bottom data. Next, the arithmetic processing layer 12, which is the second layer, uses the output data from the arithmetic processing layer 11 as bottom data, and performs the convolution operation using the bottom data and the weight data. In this way, the arithmetic processing device 1 sequentially performs arithmetic processing in each arithmetic processing layer 10, and normalizes the arithmetic result of the convolution operation using the weight data in the nth arithmetic processing layer 13. The data representing the feature amount subjected to the above is output as the output data 3. Hereinafter, the convolution operation using the bottom data and the weight data in the forward operation is referred to as a “forward convolution operation”.

Further, each arithmetic processing layer 10 obtains the weight difference data by using the top difference data which is the difference between the expected value and the output data 3 as one of the convolution operations in the backward operation. For example, the arithmetic processing layer 13, which is the nth layer, has a predetermined expected value, and compares the output data 3 with the expected value. Then, the arithmetic processing layer 13 obtains the top difference data which is the difference between the output data 3 and the expected value, and acquires the obtained top difference data as input data. Next, the arithmetic processing layer 13 obtains the weight difference data which is the difference from the expected value of the weight data of the weight data by using the input data and the bottom data used in the forward convolution operation in the nth layer. Then, the arithmetic processing layer 13 corrects the weight data in the nth layer by using the obtained weight difference data. Further, the arithmetic processing layer 13 uses the difference between the corrected weight data, the output data 3, and the expected value as a convolution operation in another backward arithmetic, and uses the difference between the bottom data and the expected value of the bottom data. Calculate some bottom difference data.

Next, the arithmetic processing layer 10 of the n-1th layer acquires data obtained by subjecting the bottom difference data calculated in the arithmetic processing layer 13 to reverse pooling processing or denormalization processing as top difference data. Next, the arithmetic processing layer 10 of the n-1th layer calculates the weight difference data using the bottom data and the top difference data used in the forward convolution operation in the n-1th layer. Then, the arithmetic processing layer 10 of the n-1th layer corrects the weight data in the n-1th layer by using the obtained weight difference data. Further, the arithmetic processing layer 10 of the n-1th layer calculates the bottom difference data in the n-1th layer by using the modified weight data and the top difference data. The arithmetic processing unit 1 repeats the convolution operation in the backward operation described above up to the first layer. Hereinafter, the convolution operation in the backward operation is referred to as "backword convolution operation".

That is, with the arrow P1 direction as the arrangement direction of each layer, the arithmetic processing unit 1 calculates the top difference data in the specific arithmetic processing layer 10 in the arithmetic processing layer 10 which is one layer ahead of the specific arithmetic processing layer 10. Then, the arithmetic processing unit 1 obtains the weight difference data in the specific arithmetic processing layer 10 by using the calculated top difference data and the bottom data which is the output data of the previous arithmetic processing layer 10. Then, the arithmetic processing unit 1 corrects the weight data used by the specific arithmetic processing layer 10 by using the weight difference data in the specific arithmetic processing layer 10 obtained. Further, the arithmetic processing unit 1 calculates the top difference data and the bottom difference data in the specific arithmetic processing layer 10.

In the following, in the backward convolution operation, the operation of obtaining the weight difference data by using the top difference data and the bottom data is referred to as "backward convolution weight difference operation". Further, the operation of calculating the bottom difference data using the corrected weight data and the top difference data is called "backward convolution bottom difference calculation".

The arithmetic processing unit 1 sequentially repeats the correction of the weight data in each arithmetic processing layer 10 and the calculation of the top difference data in the previous arithmetic processing layer 10 to obtain the weight data of all the layers of each arithmetic processing layer 10. It is modified according to the expected value of the output data 3 of the arithmetic processing layer 13.

The arithmetic processing unit 1 can improve the accuracy of image recognition and perform highly accurate image recognition by learning to repeatedly update parameters using the feature amounts acquired in each layer. Further, for example, in the case of voice recognition, the input data 2 becomes voice data, and in the case of text mining, the input data 2 becomes a word.

Here, in this embodiment, the case where the bottom data, which is the image data, has the element data arranged as a matrix in a square will be described. In the following, the amount of movement of weight data at one time in the forward convolution operation may be referred to as “the number of strides”.

Here, with reference to FIG. 2, the forward convolution operation and the backward operation will be further described. FIG. 2 is a diagram for explaining a forward convolution operation and a backward convolution operation. FIG. 2 shows from the first layer that starts arithmetic processing using the input data 2 to the nth layer that generates the top difference data 203 from the output data 206 and the expected value 207. Here, the arithmetic processing layer 11 is the first layer, the arithmetic processing layer 14 is the n-1th layer, the arithmetic processing layer 13 is the nth layer, and the arithmetic operations in the arithmetic processing layers 11 to 14 up to the nth layer are examples. Described in. Further, the processing described by the circle in FIG. 2 represents an arithmetic processing. The arithmetic processing F1 represents a forward convolution operation. The calculation process F2 represents a backward convolution weight difference calculation. Further, the calculation process F3 represents a backward convolution bottom difference calculation.

The arithmetic processing unit 1 performs a forward convolution operation represented by the arithmetic processing F1 on the input data 2 and the weight data 202 in the first layer in the arithmetic processing layer 11, and calculates the top data 209. After that, although not shown, similarly, in the next second layer, the bottom data 201 acquired from the top data 209 calculated in the previous layer and the weight data 202 in the second layer are similarly subjected to the arithmetic processing F1. Perform the represented forward convolution operation. Each arithmetic processing layer 10 repeats the same forward arithmetic. Then, the final nth layer, the arithmetic processing layer 13, is subjected to the arithmetic processing F1 with respect to the bottom data 201 acquired from the top data 209 calculated in the arithmetic processing layer 14 and the weight data 202 in the nth layer. Perform the represented forward convolution operation.

Further, the arithmetic processing layer 13 compares the output data 3 with the expected value 207 to calculate the top difference data 203. Here, since the input data 2 corresponds to the bottom data 201 in the second layer to the nth layer, it is treated as the bottom data 201 of the first layer below. Further, the output data 3 of the nth layer corresponds to the top data 209 in the first layer to the n-1th layer.

In the case of the backward calculation, the arithmetic processing layer 13 performs the weight difference calculation of the convolution backward represented by the arithmetic processing F2 on the top difference data 203 and the bottom data 201, and calculates the weight difference data 204. Further, the arithmetic processing layer 13 updates the weight data 202 by using the weight difference data 204. Here, the arrow of the alternate long and short dash line in FIG. 2 represents the process of updating the weight data 202. Specifically, the arithmetic processing unit 1 multiplies the weight difference data 204 by the learning rate to calculate new weight data 202. Further, the arithmetic processing layer 13 performs the backward convolution bottom difference calculation represented by the arithmetic processing F3 on the weight data 202 and the top difference data 203 used in the forward convolution operation, and calculates the bottom difference data 205. ..

The arithmetic processing layer 14 performs a weight difference calculation of the convolution backward represented by the arithmetic processing F2 on the top difference data 203 and the bottom data 201 acquired from the bottom difference data 205 output by the arithmetic processing layer 13, and the weight difference. Data 204 is calculated. Further, the arithmetic processing layer 14 updates the weight data 202 by using the weight difference data 204. Further, the arithmetic processing layer 14 performs the backward convolution bottom difference calculation represented by the arithmetic processing F3 on the weight data 202 and the top difference data 203 used in the forward convolution operation, and calculates the bottom difference data 205. .. Each arithmetic processing layer 10 repeats the same backward arithmetic. Then, the arithmetic processing layer 11, which is the last first layer, uses the top difference data 203 acquired from the bottom difference data 205 similarly calculated in the second layer, and uses the backward convolution weight difference calculation and the backward tatami. Performs included bottom difference calculation.

FIG. 3 is a block diagram showing details of the arithmetic processing layer. The arithmetic processing layer 10 has a convolution arithmetic unit 101, an activation processing unit 102, and a pooling processing unit 103 as functional units that execute forward operations. In addition, the arithmetic processing layer 10 has a pooling processing unit 104, an activation processing unit 105, and a convolution operation unit 106 as functional units for executing backward operations.

The convolution calculation unit 101 performs a convolution calculation described later using the output data from the calculation processing layer 10 in the previous stage. Here, the convolution calculation unit 101 will be described in more detail with reference to FIG. FIG. 4 is a block diagram showing details of a convolution calculation unit that performs a forward convolution operation according to the first embodiment. As shown in FIG. 4, the convolution calculation unit 101 includes an input data processing unit 111, a multiplication unit 112, an addition unit 113, an output data creation unit 114, and a weight data storage unit 115.

The weight data storage unit 115 stores weight data 202 corresponding to a plurality of types of filter definitions used in the forward convolution operation. In this embodiment, the weight data storage unit 115 stores the weight data 202 created by using the new filter definition 301 and the filter definition 302 shown in FIG. FIG. 5 is a diagram showing an example of a filter definition. The filter definition 302 is a conventional filter definition having a size of 3 × 3. The new filter definition 301 is a new filter definition corresponding to the filter definition 302.

The new filter definition 301 has symmetry with respect to the center with respect to axes 31 to 314. That is, the new filter definition 301 has symmetry in the vertical, horizontal, and diagonal directions, and can accurately recognize the image in the vertical, horizontal, and diagonal directions. Therefore, the new filter definition 301 has less decrease in image recognition accuracy than the case where the filter definition 302 is used, and can sufficiently perform image recognition.

In this embodiment, 3 × 3 weight data 202 is used, but the weight data storage unit 115 may store weight data 202 having different sizes. For example, the weight data storage unit 115 may store the new filter definition 303 and the filter definition 304. The filter definition 304 is a conventional filter definition having a size of 5 × 5. The new filter definition 303 is a new filter definition corresponding to the filter definition 304. The new filter definition 303 also has center symmetry with respect to axes 331-334. That is, the new filter definition 303 has less decrease in image recognition accuracy than the case where the filter definition 304 is used, and can sufficiently perform image recognition. In the new filter definitions 301 and 303, the element data including the row is peeked one by one according to the distance from the middle row from the state where the same number of element data are arranged in the row direction and the column direction. Further, in the new filter definitions 301 and 303, the rows are shifted so that the position of half of the row excluding the element data and the position of half of the row before excluding the element data match.

Further, in this embodiment, two types of filter definitions, the new filter definitions 301 and 303, have been described, but if the number of element data is smaller than that of the conventional filter definitions such as the filter definitions 302 and 304, the new filter definition is defined. Is not limited to this. However, it is preferable that the new filter definition has symmetry with respect to the center in the vertical, horizontal, and diagonal directions. In the following, the weight data 202 created by using the new filter definition 301 will be referred to as “weight data 221”.

The input data processing unit 111 receives the input of the bottom data 201 from the arithmetic processing layer 10 in the previous stage in the forward operation. This bottom data 201 corresponds to an example of "first data". Then, the input data processing unit 111 acquires the weight data 202 from the weight data storage unit 115. Next, the input data processing unit 111 determines whether or not to use the new filter definition 301 for image determination based on an instruction from an operator input from an input device (not shown). When the new filter definition 301 is not used, the input data processing unit 111 informs the multiplication unit 112 that the weight data 202 created by using the filter definition 302 is used, and outputs bottom data.

On the other hand, when the new filter definition 301 is used, the input data processing unit 111 determines whether or not the input bottom data 201 is data corresponding to the new filter definition 301. When the bottom data 201 is data corresponding to the new filter definition 301, the input data processing unit 111 informs the multiplication unit 112 that the weight data 221 created by using the new filter definition 301 is used, and also transmits the bottom data. Output. This weight data 221 corresponds to an example of "second data".

On the other hand, when the bottom data 201 does not correspond to the new filter definition 301, the input data processing unit 111 converts the bottom data 201 according to the new filter definition 301. FIG. 6 is a diagram for explaining an example of conversion of bottom data.

In this embodiment, the input data processing unit 111 calculates the average with the adjacent element data for the interval of the bottom data 201 and stores it at the position of the element data. For example, the case of bottom data 201 having 8 × 8 element data b00 to b63 shown in FIG. 6 will be described. The input data processing unit 111 skips the first line and sets the second line as the first line to change the interval.

First, the input data processing unit 111 calculates the element data nb08, which is the average of the element data b08 and the element data b09 in the second line, and stores the element data nb08 at the position of the element data b08. Next, the input data processing unit 111 calculates the element data nbv09, which is the average of the element data b09 and the element data b10, and stores it at the position of the element data b09. In this way, the input data processing unit 111 repeats storing the average value of the two adjacent element data at the position of the younger element data from the element data b08 to b15. However, regarding the element data b15, the next element data b16 does not exist on the right side. Therefore, on the right side of the element data b15, the calculation is performed assuming that the element data having a value of 0 exists next to the element data for calculating the average. That is, the input data processing unit 111 calculates the element data nb15, which is the average of the element data b15 and the element data of 0, and stores the element data nb15 at the position of the element data b15. In this way, the input data processing unit 111 calculates the element data nb08 to nb15 of the second line after conversion.

Similarly, the input data processing unit 111 calculates the element data nb24 to nb31, nb40 to nb47, and nb56 to nb63 on the fourth, sixth, and eighth lines. As a result, the input data processing unit 111 creates the bottom data 211 obtained by converting the bottom data 201. In the following, when all the element data of the bottom data 211 is represented, it is expressed as element data b00 to nb63.

FIG. 7 is a diagram showing the appearance of the bottom data after conversion. The element data b00 to nb63 of the bottom data 211 are arranged in a state of being assigned to each dot. That is, the arithmetic processing unit 1 performs a forward convolution operation using the converted bottom data 211. However, the actual appearance as an image is the bottom data 210 in which the element data nb08 to nb15, nb24 to nb31, nb40 to nb47, and nb56 to nb63 of each converted line are shifted to the right by half a dot. That is, as shown in FIG. 7, the appearance of the bottom data 211 can be represented as the bottom data 210. In the following, for the sake of clarity, the converted bottom data 211 will be described using the apparent bottom data 210.

Here, the conversion of the bottom data 201 will be described more specifically with reference to FIGS. 8 and 9. FIG. 8 is a diagram showing an example of conversion of bottom data. Further, FIG. 9 is a diagram showing another example of conversion of bottom data.

For example, as shown in FIG. 8, the case where the Chinese numeral three is input to the input data processing unit 111 as the bottom data 201 will be described. In this case, the top line of the three exists in the second line of the bottom data 201, the middle line of the three exists in the fifth line, and the bottom line of the three exists in the eighth line. Each element data b00 to b63 has a value represented by the shade information 30. The element data other than the element data representing 3 in the bottom data 201 has 0 representing white as a value. Further, the element data representing three in the bottom data 201 has a value 255 representing black.

The input data processing unit 111 calculates the average of the adjacent data of the element data b08 to b15 of the second line, and calculates the element data nb08 to nb15 after conversion. In this case, the element data nb08 has a value of 127. Further, the element data nb09 to nb13 have a value of 255. Further, the element data nb14 has a value 127. Further, the element data nb15 has 0 as a value.

Further, the input data processing unit 111 calculates the average of adjacent data of the element data b24 to b31 and b40 to b47 in the 4th and 6th lines, and calculates the converted element data nb24 to nb31 and nb40 to nb47. .. In this case, since the values of the element data b24 to b31 and b40 to b47 are all 0 in the 4th and 6th lines, the values of the converted element data nb24 to nb31 and nb40 to nb47 are also 0.

Further, the input data processing unit 111 calculates the average of the adjacent data of the element data b56 to b63 on the eighth line, and calculates the converted element data nb56 to nb63. In this case, the element data nb56 to nb62 have a value of 255. Further, the element data nb64 has a value 127.

The input data processing unit 111 converts the bottom data 201, which is an image representing the three Chinese numerals. In that case, as shown in FIG. 8, the converted bottom data 210 is an image representing the three Chinese numerals having a slight difference in shading.

Next, as shown in FIG. 9, a case where a diagonal image is input to the input data processing unit 111 as bottom data 201 will be described. In this case, a line exists on the diagonal line of the bottom data 201. Also in this case, each element data b00 to b63 has a value represented by the shade information 30 in FIG. The element data b00, b09, b18, b27, b36, b45, b54 and b63 representing the diagonal line have a value representing gray, and the other element data has a value of 0.

Then, the input data processing unit 111 calculates the average of the adjacent data of the element data b08 to b15, b24 to b31, b40 to b47 and b56 to b63, and the element data nb08 to nb15, nb24 to nb31, nb40 to nb47 and Find nb56 to nb63. In this case, the element data nb08, nb09, nb26, nb27nb44, nb45, nb62 and nb63 have half the values of the element data b08 to b15, b24 to b31, b40 to b47 and b56 to b63. Further, the element data nb10 to nb15, nb24 to nb25, nb28 to nb31, nb40 to nb43, nb46 to nb47 and nb56 to nb61 have 0 as a value.

In this case, the input data processing unit 111 converts the bottom data 201, which is an image representing the diagonal line. In that case, the converted bottom data 210 is an image representing a diagonal line in which there is a slight difference in shading, as shown in FIG.

In this way, the bottom data 210 created by being converted by the input data processing unit 111 becomes an image that can be used as the same image as the bottom data 201 before conversion in the vertical and horizontal directions and the diagonal direction. Since the image can be substantially represented by a combination of vertical lines, horizontal lines, and diagonal lines, the converted bottom data 210 can be used as an image similar to the unconverted bottom data 201.

Then, the input data processing unit 111 outputs the converted bottom data 210 to the multiplication unit 112. Further, the input data processing unit 111 notifies the multiplication unit 112 that the weight data 221 is used.

Here, in the arithmetic processing layer 11 of the first layer in FIG. 1, the input data processing unit 111 uses the input data 2 input from the outside as the bottom data 201, so that the bottom data 201 becomes the new filter definition 301. It may not be supported. In that case, the input data processing unit 111 converts the bottom data 201 to match the new filter definition 301. On the other hand, in the arithmetic processing layers 12 to 13 of the second and subsequent layers in FIG. 1, the top data 209 output from the arithmetic processing layer 10 in the previous stage already corresponds to the new filter definition 301, so that the input data processing The unit 111 can output the bottom data 201 to the multiplication unit 112 as it is without performing conversion. This input data processing unit 111 corresponds to an example of a “conversion unit”.

When the new filter definition 301 is not used, the multiplication unit 112 receives a notification from the input data processing unit 111 of the use of the weight data 202 created by using the filter definition 302. Further, the multiplication unit 112 receives the input of the bottom data 201 that has not been converted.

The multiplication unit 112 multiplies each element data in the normal forward convolution operation by using the weight data 202 and the bottom data 201 created by using the filter definition 302. Then, the multiplication unit 112 outputs the multiplication result to the addition unit 113.

Further, when the new filter definition 301 is used, the multiplication unit 112 receives a notification of the use of the weight data 221 from the input data processing unit 111. Further, the multiplication unit 112 receives the input of the bottom data 201 corresponding to the new filter definition 301 or the bottom data 210 converted so as to correspond to the new filter definition 301. Then, the multiplication unit 112 multiplies each element data in the forward convolution operation by using the input bottom data 201 or 210 and the weight data 221.

For example, the multiplication method after conversion and when the bottom data 210 is used will be described with reference to FIG. FIG. 10 is a diagram for explaining a forward convolution operation when the new filter definition is used. Here, the case where the number of strides, which is the amount of movement of the weight data 221 at one time, is 1, will be described. Further, in the following, the direction in which the columns of the bottom data 210 in FIG. 10 extend, that is, the vertical direction is referred to as "column direction", and the direction in which the rows extend, that is, the horizontal direction is referred to as "row direction".

The multiplication unit 112 receives the input of the bottom data 210 shown in FIG. Further, the multiplication unit 112 acquires the weight data 221 shown in FIG. 10 from the weight data storage unit 115. Then, the multiplication unit 112 first matches the first column of the weight data 221 with the first column of the bottom data 210, and makes each element data of the weight data 221 overlap with the element data of the lower number of the bottom data 210. The weight data 221 is arranged in. For example, in the case of FIG. 10, the multiplication unit 112 sets the weight data 221 so that the element data w00 matches the element data b01, the element data w02 matches the element data nb09, and the element data w05 matches the element data b17. Deploy. Then, the multiplication unit 112 multiplies each overlapping element data of the bottom data 210 and the weight data 221 and outputs each multiplication result to the addition unit 113. In the following, the calculation of arranging the weight data 221 at a predetermined position on the bottom data 210 and multiplying each overlapping element data is referred to as "multiplication of one element data of the top data 209".

Next, the multiplication unit 112 moves the weight data 221 in the row direction on the bottom data 210 by one element data which is the number of strides. Then, the multiplication unit 112 performs multiplication on one element data of the top data 209 at the moved position, and outputs each multiplication result to the addition unit 113. In this way, after the calculation is completed, the multiplication unit 112 moves the weight data 221 in the row direction by the number of strides, and repeats the multiplication on one element data of the top data 209. Then, when the weight data 221 moves to the end in the row direction, in the next calculation, the multiplication unit 112 moves the weight data 221 in the column direction by one element data which is the number of strides, and further, in the row direction. The weight data 221 is returned to the head position. Then, the multiplication unit 112 moves the weight data 221 in the row direction and repeats the multiplication on one element data of the top data 209. The multiplication unit 112 multiplies one element data of the top data 209 until the bottom line of the weight data 221 matches the bottom line of the bottom data 210 and the weight data 221 moves to the end of the bottom data 210. Repeat.

For example, a case where the weight data 221 is arranged at the position surrounded by the thick line frame of the bottom data 210 in FIG. 10 and the calculation is performed will be described. Here, the multiplication of each element data is represented only by a sign. The multiplication unit 112 performs w00 × nb09, w01 × nb10, w02 × b17, w03 × b18, w04 × b19, w05 × nb25 and w06 × nb26 as multiplications for one top data 209. Then, the multiplication unit 112 outputs each multiplication result to the addition unit 113.

The addition unit 113 receives the input of the multiplication result from the multiplication unit 112. Then, the addition unit 113 adds each of the multiplication results of multiplication to one top data 209 to calculate the total. Hereinafter, the addition of the multiplication result of multiplication to one top data 209 is referred to as "addition to one element data of top data 209". Then, the addition unit 113 outputs the addition result to the output data creation unit 114. The addition unit 113 repeats the addition to one element data of the top data 209 for all the multiplications performed by the multiplication unit 112 to one element data of the top data 209, and outputs the addition result to the output data creation unit 114. To do.

For example, a case where the weight data 221 is arranged at the position surrounded by the thick line frame of the bottom data 210 in FIG. 10 will be described. The addition unit 113 receives inputs of w00 × nb09, w01 × nb10, w02 × n17, w03 × b18, w04 × b19, w05 × nb25, and w06 × nb26 from the multiplication unit 112. Then, the addition unit 113 calculates w00 × nb09 + w01 × nb10 + w02 × b17 + w03 × b18 + w04 × b19 + w05 × nb25 + w06 × nb26.

The output data creation unit 114 receives the input of the addition result of the addition for one element data of the top data 209 from the addition unit 113. Then, the output data creation unit 114 repeats the allocation of the acquired addition results in order from the beginning of the top data 209. For example, when the weight data 221 is arranged at the position surrounded by the thick line frame of the bottom data 210 in FIG. 10, the output data creation unit 114 uses the acquired addition result as the element data t18. That is, w00 × nb09 + w01 × nb10 + w02 × n17 + w03 × n18 + w04 × n19 + w05 × nb25 + w06 × nb26 corresponds to the element data t18 of the top data 209. The output data creation unit 114 repeatedly assigns the top data 209 of the addition result acquired in this way to each element data to generate the top data 209. Then, the output data creation unit 114 outputs the generated top data 209 to the activation processing unit 102. Hereinafter, the multiplication and addition of the top data 209 to one element data and the allocation of the element data of the top data 209 of the addition result are collectively referred to as "sum product operation for one element data of the top data 209". The multiplication unit 112, the addition unit 113, and the output data creation unit 114 correspond to an example of the “convolution calculation unit”.

Here, when the conventional filter definition 302 is used, the convolution calculation unit 101 performs 9 multiplications and addition of 9 multiplication results in the sum product calculation for one element data of the top data 209. On the other hand, when the new filter definition 301 is used, the convolution calculation unit 101 performs 7 multiplications and addition of 7 multiplication results in the sum product calculation for one element data of the top data 209. Therefore, both the number of multiplications and the number of values to be added are smaller when the new filter definition 301 is used than when the conventional filter definition 302 is used. That is, in the case of the forward convolution operation, the storage area used can be smaller and the calculation efficiency can be improved when the new filter definition 301 is used as compared with the case where the conventional filter definition 302 is used. be able to.

The explanation will be continued by returning to FIG. The convolution calculation unit 106 performs a backward convolution operation on the data that has been denormalized by the activation processing unit 105. Here, the backward convolution operation by the convolution operation unit 106 will be described in more detail. First, the backward convolution bottom difference calculation will be described with reference to FIG. FIG. 11 is a diagram for explaining the backward convolution bottom difference operation when the new filter definition is used.

Here, the case where the forward convolution operation is performed using the bottom data 210 obtained by converting the 8 × 8 bottom data 201 used in FIG. 10 and the weight data 221 will be described. In this case, the convolution calculation unit 106 has the same arrangement shape as the arrangement shape of the top data 209 of FIG. 10 obtained by the forward convolution operation, that is, the appearance of the arrangement shape of the data shifted by one element at intervals. The input of the top difference data 203 is received from the activation processing unit 105. Here, the top difference data 203 also has the same arrangement shape as the arrangement shape of the top data 209. Further, the bottom difference data 205 calculated by the backward convolution bottom difference calculation has the same arrangement shape as the bottom data 201. The top difference data 203 has element data td00 to td63. Further, the bottom difference data 205 has element data bd00 to nbd63.

The convolution calculation unit 106 receives the input of the top difference data 203 shown in FIG. Then, the convolution calculation unit 106 first matches the first column of the weight data 221 with the first column of the top difference data 203, and each element data of the weight data 221 is an element having a lower number than the top difference data 203. The weight data 221 is arranged so as to overlap the data. For example, in the case of FIG. 11, the convolution calculation unit 106 uses weight data such that the element data w00 matches the element data td01, the element data w02 matches the element data td09, and the element data w05 matches the element data td17. Place 221. Then, the convolution calculation unit 106 multiplies each overlapping element data of the top difference data 203 and the weight data 221. Further, the convolution calculation unit 106 adds each of the multiplication results to calculate the total. Then, the convolution calculation unit 106 sets the calculated addition result as the element data bd00 of the bottom difference data 205.

Next, the convolution calculation unit 106 moves the weight data 221 in the row direction on the top difference data 203 by the amount of one element data which is the number of strides. Then, the convolution calculation unit 106 multiplies one bottom difference data 205 at the moved position, adds the multiplication results, and calculates the total. In this way, the convolution calculation unit 106 moves the weight data 221 in the row direction by the number of strides after the calculation is completed, and repeats multiplication and addition. Then, when the weight data 221 moves to the end in the row direction, in the next calculation, the convolution calculation unit 106 moves the weight data 221 in the column direction by one element data which is the number of strides, and further, the row The weight data 221 is returned to the position at the beginning of the direction. Then, the convolution calculation unit 106 repeats multiplication and addition while moving the weight data 221 in the row direction. The convolution calculation unit 106 repeats multiplication and addition until the bottom line of the weight data 221 matches the bottom line of the top difference data 203 and the weight data 221 moves to the end of the top difference data 203. Then, the convolution calculation unit 106 allocates the multiplication and addition results in the numerical order of the element data b01 to nb63 of the bottom difference data 205. In the following, multiplication and addition in a state where the weight data 221 is arranged at a predetermined position on the top difference data 203, and operations assigned to the element data b00 to nb63 of the bottom difference data 205 are collectively referred to as "bottom difference data". It is called "sum product operation for one element data of 205".

For example, a case where the weight data 221 is arranged at the position surrounded by the thick line frame of the top difference data 203 in FIG. 11 and the calculation is performed will be described. Here, the multiplication of each element data is represented only by a sign. The convolution calculation unit 106 sets w00 × td09 + w01 × td10 + w02 × td17 + w03 × td18 + w04 × td19 + w05 × td25 + w06 × td26 as element data bd18 as a sum product operation for one element data of the bottom difference data 205.

Here, when the conventional filter definition 302 is used, the convolution calculation unit 101 performs 9 multiplications and addition of 9 multiplication results in the sum product calculation for one element data of the bottom difference data 205. On the other hand, when the new filter definition 301 is used, the convolution calculation unit 101 performs 7 multiplications and addition of 7 multiplication results in the sum product calculation for one element data of the bottom difference data 205. .. Therefore, both the number of multiplications and the number of values to be added are smaller when the new filter definition 301 is used than when the conventional filter definition 302 is used. That is, even in the case of the backward convolution bottom difference calculation, the storage area used can be smaller when the new filter definition 301 is used than when the conventional filter definition 302 is used, and the calculation can be performed. Efficiency can also be improved.

Next, the backward convolution weight difference calculation will be described with reference to FIG. FIG. 12 is a diagram for explaining the backward convolution weight difference calculation when the new filter definition is used. The weight difference data 204 calculated by the forward convolution weight difference calculation has the same arrangement shape as the arrangement shape of the weight data 221. The weight difference data 204 has element data wd00 to wd07.

The convolution calculation unit 106 acquires the bottom data 210 used in the forward convolution calculation. Further, the convolution calculation unit 106 receives the input of the top difference data 203 shown in FIG. Next, the convolution calculation unit 106 determines whether or not the bottom data 210 has a size for calculating the weight difference data 204. When the size is small, the convolution calculation unit 106 adds element data 212 having a value of 0 around the bottom data 210. In the following, the bottom data 210 to which the element data 212 is added is simply referred to as the bottom data 210.

Next, the convolution calculation unit 106 first matches the first column of the top difference data 203 with the first column of the bottom data 210, and each element data of the top difference data 203 has a lower number of the bottom data 210. The top difference data 203 is arranged so as to overlap the element data. For example, the convolution calculation unit 106 arranges the top difference data 203 so as to match the thick line frame of the bottom data 210. Then, the convolution calculation unit 106 multiplies each overlapping element data of the bottom data 210 and the top difference data 203. Further, the convolution calculation unit 106 adds each of the multiplication results to calculate the total. Then, the convolution calculation unit 106 sets the calculated addition result as the element data w00 of the weight difference data 204.

Next, the convolution calculation unit 106 moves the top difference data 203 in the row direction on the bottom data 210 by one element data which is the number of strides. Then, the convolution calculation unit 106 multiplies the element data at the moved position, adds the multiplication results, and calculates the total. In this way, the convolution calculation unit 106 moves the top difference data 203 in the row direction by the number of strides after the calculation is completed, and repeats multiplication and addition. Then, when the top difference data 203 moves to the end in the row direction, in the next calculation, the convolution calculation unit 106 moves the top difference data 203 in the column direction by one element data which is the number of strides, and further. , The top difference data 203 is returned to the first position in the row direction. Then, the convolution calculation unit 106 repeats multiplication and addition while moving the top difference data 203 in the row direction. The convolution calculation unit 106 repeats multiplication and addition until the bottom line of the top difference data 203 matches the bottom line of the bottom data 210 and the top difference data 203 moves to the end of the bottom data 210. Then, the convolution calculation unit 106 allocates the multiplication and addition results in the numerical order of the element data w01 to w07 of the weight difference data 204. In the following, multiplication and addition, and operations assigned to the element data w00 to w07 of the weight difference data 204 in a state where the top difference data 203 is arranged at a predetermined position on the bottom data 210 are collectively referred to as "weight difference data". It is called "sum product operation for one element data of 204".

For example, a case where the top difference data 203 is arranged at the position surrounded by the thick line frame of the bottom data 210 in FIG. 12 and the calculation is performed will be described. Here, the multiplication of each element data is represented only by a sign. The convolution calculation unit 106 performs the following calculation as a sum product calculation for one element data of the weight difference data 204. The convolution calculation unit 106 calculates td00 × 0 + ... + td07 × 0 + td08 × b00 + ... + td15 × b07 + td16 × 0 + td17 × nb08 + ・ ・ ・ + td23 × nb14 + ・ ・ ・ + td56 × b48 + ... Then, the convolution calculation unit 106 sets the calculation result as the element data wd00.

Here, when the conventional filter definition 302 is used, the convolution calculation unit 101 performs the sum product calculation on one element data of the weight difference data 204 nine times. On the other hand, when the new filter definition 301 is used, the convolution calculation unit 101 only needs to perform the sum product operation for one element data of the weight difference data 204 seven times. Therefore, the number of calculations can be reduced when the new filter definition 301 is used as compared with the case where the conventional filter definition 302 is used. That is, even in the case of the backward convolution weight difference calculation, the storage area used can be made smaller when the new filter definition 301 is used than when the conventional filter definition 302 is used, and the calculation can be performed. Efficiency can also be improved.

The explanation will be continued by returning to FIG. The activation processing unit 102 normalizes the top data output from the convolution calculation unit 101. The pooling processing unit 103 does not change the response to a minute position change by thinning out or integrating the element data with respect to the top data normalized by the activation processing unit 102. The process performed by the pooling process unit 103 is called a pooling process. Then, the pooling processing unit 103 outputs the top data subjected to the pooling processing to the arithmetic processing layer 10 of the next stage. The pooling processing unit 103 outputs a tag 151 representing the processing added to the data to the pooling processing unit 104.

The pooling processing unit 104 receives an input of a tag 151 representing the pooling processing of the executed response from the pooling processing unit 103. Further, the pooling processing unit 104 receives input of bottom difference data from the arithmetic processing layer 10 in the subsequent stage. Then, the pooling processing unit 104 performs the reverse processing of the pooling processing specified by the tag 151 on the acquired bottom difference data. The process performed by the pooling process unit 104 is called a reverse pooling process. The activation processing unit 105 performs activation processing on the data that has been reverse pooled by the pooling processing unit 104.

Further, although the operation of the arithmetic processing unit 1 at the time of learning has been described above, the arithmetic processing unit 1 recognizes the input data 2 by using the weight data 202 acquired by the learning. Therefore, the recognition process in each arithmetic processing layer 10 will be described.

The convolution calculation unit 101 receives the input of bottom data. Then, the forward convolution operation is performed using the weight data acquired in the learning. Then, the activation processing unit 102 and the pooling processing unit 103 perform pooling processing such as normalization on the top data. After that, the pooling processing unit 103 outputs the processed top data to the next arithmetic processing layer 10. By repeating such a forward convolution operation in each arithmetic processing layer 10, the arithmetic processing unit 1 finally acquires the output data 3 for recognition.

Next, with reference to FIG. 13, the processing flow in the arithmetic processing layer when the new filter definition 301 is used will be described. FIG. 13 is a flowchart of processing in the arithmetic processing layer when the new filter definition is used.

The input data processing unit 111 in the arithmetic processing layer 11 of the first layer calculates the average of adjacent element data in intervals with respect to the input data 2, and uses the bottom data 210 that matches the new filter definition 301 as one element data. Generate (step S1). Then, the input data processing unit 111 outputs the bottom data 210 to the multiplication unit 112.

The multiplication unit 112 receives the input of the bottom data 210 from the input data processing unit 111. Then, the multiplication unit 112, the addition unit 113, and the output data creation unit 114 repeat the sum product operation on one element data of the top data 209 to perform the forward convolution operation (step S2). Then, the output data creation unit 114 outputs the top data 209 which is the calculation result.

The activation processing unit 102 and the pooling processing unit 103 perform forward and other processing operations such as a pooling process for normalizing the top data 209 output from the output data creation unit 114 (step S3). Then, the pooling processing unit 103 outputs the processed data to the arithmetic processing layer 12 of the second layer.

The arithmetic processing layer 10 of the second to second n-1 layers and the arithmetic processing layer 13 of the nth layer execute the same processing including the forward convolution operation and the forward other processing operation (step S4).

Next, the arithmetic processing layer 13 of the nth layer compares the output data 3 with the expected value 207 (step S5).

Next, the pooling processing unit 104 and the activation processing unit 105 of the arithmetic processing layer 13 of the nth layer perform backward other processing operations including the reverse pooling processing on the comparison result (step S6). Then, the activation processing unit 105 outputs the processed data as the top difference data 203 to the convolution calculation unit 106.

Next, the convolution calculation unit 106 of the calculation processing layer 13 of the nth layer receives the input of the top difference data 203 from the activation processing unit 105. Then, the convolution calculation unit 106 performs a backward convolution calculation using the top difference data 203, the weight data 202, and the bottom data 210 (step S7). The convolution calculation unit 106 updates the weight data 202. Further, the convolution calculation unit 106 outputs the obtained bottom difference data 205 to the calculation processing layer 10 of the n-1th layer.

The arithmetic processing layer 10 of the first to third layers, the arithmetic processing layer 12 of the second layer, and the arithmetic processing layer 11 of the first layer execute the same processing including the backward other processing arithmetic and the backward convolution operation. (Step S8). As a result, the weight data 202 of the arithmetic processing layer 10 of the first to third layers, the arithmetic processing layer 12 of the second layer, and the arithmetic processing layer 11 of the first layer is updated.

Next, with reference to FIG. 14, the flow of the forward convolution calculation by the convolution calculation unit 101 will be described. FIG. 14 is a flowchart of the forward convolution calculation by the convolution calculation unit according to the first embodiment.

The input data processing unit 111 determines whether or not to use the new filter definition 301 (step S101). When the new filter definition 301 is not used (step S101: negation), the input data processing unit 111 outputs the input data as it is to the multiplication unit 112 as bottom data 201. The multiplication unit 112, the addition unit 113, and the output data creation unit 114 execute a normal forward convolution operation (step S102), and end the forward convolution operation.

On the other hand, when the new filter definition 301 is used (step S101: affirmative), the input data processing unit 111 acquires the weight data 221 corresponding to the new filter definition 301 from the weight data storage unit 115 (step S103). ..

Next, the input data processing unit 111 determines whether or not the input data corresponds to the new filter definition 301 (step S104). When the input data does not correspond to the new filter definition 301 (step S104: negation), the input data processing unit 111 uses the data obtained by averaging the input data of the processing results in the previous layer at intervals as the bottom data 201 used for the calculation. (Step S105).

When the input data corresponds to the new filter definition 301 (step S104: affirmative), the input data processing unit 111 uses the input data of the processing result in the previous layer as the bottom data 201 (step S106).

The input data processing unit 111 outputs the bottom data 201 corresponding to the new filter definition 301 to the multiplication unit 112. The multiplication unit 112, the addition unit 113, and the output data creation unit 114 execute a forward convolution operation using the input bottom data 201 and the weight data 221 corresponding to the new filter definition 301 (step S107).

Next, with reference to FIG. 15, the flow of the backward convolution operation by the convolution operation unit 106 will be described. FIG. 15 is a flowchart of a backward convolution calculation by the convolution calculation unit according to the first embodiment.

The convolution calculation unit 106 determines whether or not to use the new filter definition 301 (step S201). When the new filter definition 301 is not used (step S201: negation), the convolution calculation unit 106 executes a normal backward convolution operation (step S202) using the input data as the bottom data 201 as it is, and performs a forward convolution operation. To finish.

On the other hand, when the new filter definition 301 is used (step S201: affirmative), the convolution calculation unit 106 determines whether or not it is the first layer in the back propagation direction (step S203). In the case of the first layer in the back propagation direction (step S203: affirmative), the convolution calculation unit 106 tops the data to which backward processing or the like is applied to the difference between the output data 3 by the forward calculation and the expected value 207. Acquired as difference data 203 (step S204).

On the other hand, in the case of a layer other than the first layer in the back propagation direction (step S203: negation), the convolution calculation unit 106 performs backward and other processing on the bottom difference data 205 output from the previous layer. The obtained data is acquired as the top difference data 203 (step S205).

Then, the convolution calculation unit 106 executes the backward weight difference calculation and the backward bottom difference calculation using the bottom data 201, the weight data 221 using the new filter definition 301, and the top difference data 203 (step S206).

As described above, the arithmetic processing apparatus according to the present embodiment performs forward convolution operations and backward convolution operations using a new filter definition in which the number of element data is smaller than that of the conventional filter definition of a square matrix. Do. The following table shows the ratio of the amount of calculation between the conventional filter definition and the new filter definition according to the filter size. Here, the new filter definition is to reduce the element data one by one from the center row to the edge row, and shift the half position of each row so that it matches the half position before reducing the element data. Is the definition generated by.

As described above, the arithmetic processing unit according to the present embodiment can reduce the amount of arithmetic operations in the forward convolution operation and the backward convolution operation. Here, the arithmetic processing apparatus according to the present embodiment performs an operation for converting the input data, but the number of operations for converting the input data is smaller than the number of operations reduced in the convolution operation, so that the amount of calculation is reduced. be able to. In addition, the arithmetic processing unit according to this embodiment can also contribute to the reduction of memory throughput by reducing the amount of data. Even when a filter that does not satisfy the conditions for using the high-speed method by the fast Fourier transform is used, the arithmetic processing unit according to the present embodiment can reduce the amount of arithmetic operations in the forward convolution operation and the backward convolution operation. .. Therefore, the arithmetic processing unit according to the present embodiment can improve the arithmetic efficiency while suppressing the capacity of the storage device used in the arithmetic of deep learning.

In particular, a filter having a size of 3 × 3 is a filter often used in deep learning, and a filter having a size of 3 × 3 also reduces the number of operations as described in the embodiment.

Further, in the arithmetic processing unit according to the present embodiment, the weight data can be reduced, and the amount of data in the forward convolution operation and the backward convolution operation can be suppressed to be small.

Next, Example 2 will be described. When the bottom data is converted according to the new filter definition, the arithmetic processing unit according to the present embodiment performs the pooling process by using the data calculated using the bottom data as it is. The arithmetic processing unit according to this example is also shown in FIGS. 1 and 2. In the following, description of the functions of the same parts as in the first embodiment will be omitted.

FIG. 16 is a diagram for explaining a pooling process when the number of strides by the pooling process unit according to the second embodiment is 2.

The pooling processing unit 103 receives the input of the data normalized by the activation processing unit 102 to the top data 209 output by the convolution calculation unit 101. Here, the case where the forward convolution operation is performed using the 8 × 8 bottom data 201 and the new filter definition 301 will be described. That is, the pooling processing unit 103 receives the input of the data 401 shown in FIG. Here, the data 401 has element data i00 to i63. The element data i00 to i63 correspond to the element data t00 to t63 of the top data 209, respectively.

The pooling processing unit 103 stores the thick line frame 411 shown on the data 401 in FIG. 16 as the pooling size. Then, the pooling processing unit 103 is first arranged so that the line above the thick line frame 411 matches the youngest element data in the first line of the data 401. Then, the element data i00, i01, i08 and i09 included in the thick line frame 411 are acquired, and the pooling process such as selection of the average or maximum value of the acquired element data is performed to acquire the value. Then, the pooling processing unit 103 sets the element data p00 of the data 402 that outputs the acquired value.

Next, the pooling processing unit 103 performs pooling processing and acquires a value while advancing the thick line frame 411 in the row direction by two element data. Then, when the thick line frame 411 reaches the end of the row of the data 410, the pooling processing unit 103 moves the thick line frame 411 in the column direction by two element data and returns the thick line frame 411 to the beginning of the row. .. After that, the pooling processing unit 103 repeats the same pooling process while advancing the thick line frame 411 in the row direction by two element data, and the line below the thick line frame 411 is one of the data 401. Repeat until the end of the bottom line is reached. Then, the pooling processing unit 103 sets the acquired values as the element data p01 to p15 of the data 402 for outputting each.

For example, when the thick line frame 411 is arranged at the position on the data 401 shown in FIG. 16, the pooling processing unit 103 acquires the element data i18, i19, i26 and i27. Then, the pooling processing unit 103 performs pooling processing using the element data i18, i19, i26 and i27, and acquires a value. After that, the pooling processing unit 103 sets the acquired value as the element data p05 of the data 402.

The pooling processing unit 103 acquires the element data p00 to p15 and completes the data 402. After that, the pooling processing unit 103 outputs the data 402 to the next arithmetic processing layer 10.

FIG. 17 is a diagram for explaining a pooling process when the number of strides by the pooling process unit according to the second embodiment is 1. When the number of strides is 1, the target of pooling is lowered line by line. Therefore, in the case of an arrangement shape such as data 401, two different pooling sizes are used.

The pooling processing unit 103 stores the thick line frames 412 and 413 shown on the data 401 in FIG. 17 as the pooling size. Then, the pooling processing unit 103 uses the pooling size of the thick line frame 413 when calculating the element data of the odd-numbered rows of the data 402. Further, when calculating the element data of the odd-numbered rows of the data 402, the pooling size of the thick line frame 412 is used.

Specifically, the pooling processing unit 103 is first arranged so that the line above the thick line frame 413 matches the youngest element data in the first line of the data 401. Then, the element data i00, i01, i08 and i09 included in the thick line frame 413 are acquired, and the pooling process is performed by selecting the average or maximum value of the acquired element data to acquire the value. Then, the pooling processing unit 103 sets the element data p00 of the data 402 that outputs the acquired value. After that, the pooling processing unit 103 acquires the value by the pooling process while advancing the thick line frame 413 in the row direction by the element data until the thick line frame 413 reaches the end of the line of the data 401. Then, the pooling processing unit 103 sets the element data p01 to p06 of the data 402 to output the acquired values, respectively.

Next, the pooling processing unit 103 is arranged so that the line above the thick line frame 412 matches the element data of the youngest number in the second line of the data 401. Then, the element data i08, i09, i16 and i17 included in the thick line frame 411 are acquired, and the pooling process is performed by selecting the average or maximum value of the acquired element data to acquire the value. Then, the pooling processing unit 103 sets the element data p07 of the data 402 that outputs the acquired value. After that, the pooling processing unit 103 acquires the value by the pooling process while advancing the thick line frame 412 in the row direction by the element data until the thick line frame 412 reaches the end of the line of the data 401. Then, the pooling processing unit 103 sets the element data p08 to p13 of the data 402 to output the acquired values, respectively.

The pooling processing unit 103 repeats the acquisition of the value by the pooling processing by alternately using the pooling size while lowering the target line one by one. Then, the pooling processing unit 103 sets the acquired values as element data p14 to p48 of the data 402 for outputting each of them.

For example, when the thick line frame 412 is arranged at the position on the data 401 shown in FIG. 17, the pooling processing unit 103 acquires the element data i09, i10, i17 and i18. Then, the pooling processing unit 103 performs pooling processing using the element data i09, i10, i17 and i18, and acquires a value. After that, the pooling processing unit 103 sets the acquired value as the element data p08 of the data 402.

Further, when the thick line frame 413 is arranged at the position on the data 401 shown in FIG. 17, the pooling processing unit 103 acquires the element data i32, i33, i40 and i41. Then, the pooling processing unit 103 performs pooling processing using the element data i32, i33, i40 and i41, and acquires a value. After that, the pooling processing unit 103 sets the acquired value as the element data p28 of the data 402.

The pooling processing unit 103 acquires the element data p00 to p48 and completes the data 402. After that, the pooling processing unit 103 outputs the data 402 to the next arithmetic processing layer 10.

As described above, the arithmetic processing unit according to the present embodiment can perform the pooling processing by using the top data which is the arithmetic result of the forward convolution operation as it is. Therefore, even when the bottom data is converted according to the new filter definition and the forward convolution operation is performed, the pooling process can be performed without increasing the process, and the efficiency of the operation process can be improved for the entire network. ..

Next, Example 3 will be described. The arithmetic processing unit according to this embodiment performs padding to make the input data and the output data the same size when the bottom data is converted according to the new filter definition. The arithmetic processing unit according to this example is also represented by FIGS. 1 to 4. In the following, description of the functions of the same parts as in the first embodiment will be omitted.

FIG. 18 is a diagram for explaining a forward convolution operation by the convolution operation unit according to the third embodiment. Here, a case where the forward convolution operation is performed using the 8 × 8 bottom data 201 and the weight data 221 using the new filter definition 301 will be described.

The input data processing unit 111 receives the input of the bottom data 201. Then, the input data processing unit 111 converts the bottom data 201 according to the new filter definition 221 to obtain the bottom data 210.

Then, the input data processing unit 111 adds element data 213 having a value of 0 around the bottom data 210 as shown in FIG. 18 to generate the bottom data 214. The process of adding the element data 213 having a value of 0 around the bottom data 210 is 0 padding. As a result, the input data processing unit 111 matches the size of the top difference data 203 with the size of the bottom data 210. Then, the input data processing unit 111 outputs the bottom data 214 to the multiplication unit 112.

The multiplication unit 112 receives the input of the bottom data 214 from the input data processing unit 111. Then, the multiplication unit 112 executes a forward convolution operation on the bottom data 214 using the weight data 221. As a result, the multiplication unit 112 calculates the same number of element data t00 to t63 of the top data 209 as the element data b00 to nb63 of the bottom data 210.

Here, in the case of the bottom data 201 of the 8 × 8 matrix, 36 element data 213 are used to perform 0 padding. On the other hand, in the case of the bottom data 210, 34 element data 213 are used to perform 0 padding. That is, when the bottom data 210 is used, the number of element data 213 used for 0 padding is smaller than that of the bottom data 201 before conversion.

As described above, the arithmetic processing unit according to the present embodiment performs 0 padding on the converted bottom data according to the new filter definition and performs the forward convolution operation. In this case, it is sufficient to add a smaller number of element data than performing 0 padding on the bottom data before conversion, and the data capacity can be reduced and the calculation efficiency can be improved.

Next, Example 4 will be described. The arithmetic processing unit according to this embodiment performs a forward convolution operation and a backward convolution operation on three-dimensional data using a new filter definition. The arithmetic processing unit according to this example is also represented by FIGS. 1 to 4. Each part according to the present embodiment has a function of executing the same processing as each part of the first embodiment having the same reference numerals to the three-dimensional data.

FIG. 19 is a diagram for explaining an example of a forward convolution operation using a new filter definition by the convolution operation unit according to the fourth embodiment. The weight data storage unit 115 stores weight data 222 using the new three-dimensional filter definition. Here, the conventional filter definition corresponding to the weight data 222 is a cube in which element data are arranged in 3 × 3 × 3. The weight data 222 has the same data arrangement shape as the new filter definition 301 of the first embodiment in the front view in the x to z directions.

The input data processing unit 111 receives the input of the bottom data 201, which is an 8 × 8 × 8 cube. Then, the input data processing unit 111 averages the adjacent element data arranged in the y-axis direction and the z-axis direction corresponding to the coordinates of FIG. 19 of the bottom data 201. As a result, the input data processing unit 111 generates the bottom data 210 having the appearance of the bottom data 201 shifted by half of the element data by intervals in the y-axis direction and the z-axis direction. Then, the input data processing unit 111 outputs the generated bottom data 210 to the multiplication unit 112.

The multiplication unit 112 receives the input of the bottom data 210. Then, the multiplication unit 112 uses the weight data 222 using the new filter definition for the bottom data 210 to perform a forward convolution operation.

Further, the convolution calculation unit 106 receives the input of the top difference data 203 having the same arrangement shape as the arrangement shape of the top data 209 calculated by the forward convolution operation using the bottom data 210 and the weight data 222. Then, the convolution calculation unit 106 executes a backward convolution operation using the bottom data 210, the weight data 222, and the acquired top difference data 203.

Next, with reference to FIG. 20, another example of the forward convolution operation using the new filter definition by the convolution operation unit 106 according to the fourth embodiment will be described. FIG. 20 is a diagram for explaining another example of the forward convolution operation using the new filter definition by the convolution operation unit according to the fourth embodiment. The weight data storage unit 115 stores weight data 223 using the new three-dimensional filter definition. This weight data 223 is also a new filter definition corresponding to a cube in which element data are arranged in 3 × 3 × 3. The weight data 223 also has the same data arrangement shape as the new filter definition 301 of the first embodiment in the front view in the x to z directions.

The input data processing unit 111 generates the bottom data 210 as in the case of FIG. The multiplication unit 112 uses the weight data 223 using the new filter definition for the bottom data 210 to perform a forward convolution operation. Further, the convolution calculation unit 106 uses the back word tatami using the top difference data 203 having the same data arrangement shape as the top data 209 calculated by the forward convolution calculation using the bottom data 210 and the weight data 223. Performs inclusive operations.

As described above, the arithmetic processing unit according to the present embodiment performs forward convolution operation and backward convolution operation for three-dimensional data by using a new filter definition having less element data than before. .. Therefore, the arithmetic processing unit according to the present embodiment can improve the arithmetic efficiency while suppressing the capacity of the storage apparatus used in the deep learning arithmetic using the three-dimensional data.

(Program description example)
FIG. 21 is a diagram for explaining a description example of a program for forward convolution operation. As shown in FIG. 21, the forward convolution operation can be expressed by multiplication and addition in the operation using the bottom data 201 (bottom_y) and the top data 209 (top_x). The forward convolution operation is performed by designating the number of data Ci of the bottom data 201, the number of data Co of the top difference data 203, the number of batches mb, the number of strides W, and the number of pads as parameters for adjusting the top size. .. Here, adjusting the top size corresponds to inflating the top size.

FIG. 22 is a diagram for explaining a description example of a program for backward convolution weight difference calculation. As shown in FIG. 22, the backward convolution weight difference calculation can be expressed by multiplication and addition in the calculation using the bottom data 201 (bottom_y) and the top difference data 203 (top_x). In this case, the weight difference data (ew) is calculated. In the backward convolution weight difference calculation, the number of data Ci of the bottom data 201, the number of data Co of the top difference data 203, the number of batches mb, the number of strides W, and the number of pads as parameters for adjusting the top size are specified. Is done. Here, adjusting the top size corresponds to inflating the top size.

FIG. 23 is a diagram for explaining a description example of a program for backward convolution bottom difference calculation. As shown in FIG. 23, the backward convolution bottom difference operation can be expressed by multiplication and addition of the operation using the bottom data 201 (bottom_y) and the top difference data 203 (top_x). In this case, the bottom difference data 205 (bottom_ey) is calculated. In the backward convolution bottom difference calculation, the number of data Ci of the bottom data 201, the number of data Co of the top difference data 203, the number of batches mb, the number of strides W, and the number of pads as parameters for adjusting the top size are specified. Is done. Here, adjusting the top size corresponds to inflating the top size.

(Hardware configuration)
FIG. 25 is a hardware configuration diagram of the arithmetic processing unit. The arithmetic processing unit 1 includes a CPU (Central Processing Unit) 91, a memory 92, an accelerator 93, and a memory 94. The memory 92 is a memory dedicated to the CPU 91 and may be included in the CPU 91. Further, the memory 94 is a memory of the accelerator 93 and may be included in the accelerator 93.

The memory 92 stores various programs including an OS (Operating System) and a learning program used in each arithmetic processing layer 10. Further, the memory 92 stores the input data 2 and the expected value 207.

The CPU 91 executes the OS stored in the memory 92. Further, the CPU 91 outputs various programs including the learning program included in the memory 92, and various data including the input data 2, the weight data 202, and the expected value 207 to the accelerator 93. The weight data 202 includes weight data 221 and the like according to the new filter definition to be used. Then, the CPU 91 instructs the accelerator 93 to execute the deep learning process. After that, the CPU 91 acquires the trained weight data 202 from the accelerator 93 and updates the weight data 202 stored in the memory 92.

The accelerator 93 is, for example, a GPU, an FPGA (Field Programmable Gate Array), or the like. The accelerator 93 stores various programs including the learning program input from the CPU 91, and various data including the input data 2 and the expected value 207 in the memory 94. Then, the accelerator 93 executes the deep learning process using various programs including the learning program stored in the memory 94 and various data. As a result, the accelerator 93 includes the convolution calculation unit 101, the activation processing unit 102, the pooling processing unit 103, the pooling processing unit 104, the activation processing unit 105, and the convolution calculation unit 106 of the arithmetic processing layer 10 illustrated in FIG. Realize each function of. The accelerator 93 outputs weight data 202, which is a learning result in each arithmetic processing layer 10, to the CPU 91. The accelerator 93 executes processing in the same manner for all the arithmetic processing layers 10. Here, the accelerator 93 may acquire data from the CPU 91 for each processing of each arithmetic processing layer 10, or may collectively acquire data used for processing of each arithmetic processing layer 10.

1 Arithmetic processing device 2 Input data 3 Output data 10 to 14 Arithmetic processing layer 101, 106 Convolution calculation unit 102, 105 Activation processing unit 103, 104 Pooling processing unit 111 Input data processing unit 112 Multiplying unit 113 Addition unit 114 Output data Creation unit 115 Weight data storage unit 2011,210,211 Bottom data 202,221,222,223 Weight data 203 Top difference data 204 Weight difference data 205 Bottom difference data 207 Expected value 209 Top data

Claims (6)

A storage unit that stores the first data having the element data forming the matrix and the second data having the arrangement shape obtained by excluding a predetermined number of element data from the element data forming the matrix.
A conversion unit that converts the first data based on the arrangement shape of the second data, and
A calculation processing unit including a convolution calculation unit that performs a convolution calculation using the second data as a filter for the first data converted by the conversion unit. The arithmetic processing unit according to claim 1, wherein the second data has a symmetrical arrangement shape in the vertical, horizontal, and diagonal directions. The first data has the same number of element data in the row direction and the column direction.
In the second data, the element data including the row was removed one by one according to the distance from the middle row from the state where the same number of element data was arranged in the row direction and the column direction, and the element data was excluded. It has an arrangement shape in which the element data is arranged so that the position of half of the row and the position of half of the previous row excluding the element data match.
The arithmetic processing unit according to claim 1 or 2, wherein the conversion unit performs conversion for averaging adjacent element data in intervals of the first data. The operation according to any one of claims 1 to 3, further comprising a pooling processing unit that executes a pooling process by using the value of element data included in the calculation result by the convolution calculation unit as it is. Processing equipment. The convolution calculation unit adds element data having a value of 0 so as to surround the first data with a minimum number, and the convolution calculation unit adds the element data having a value of 0 to the first data. The arithmetic processing apparatus according to any one of claims 1 to 4, wherein the convolution operation is performed using the second data as a filter, and an arithmetic result having the same number of element data as the first data is acquired. .. A control method for an arithmetic processing unit that stores first data having element data forming a matrix and second data having an arrangement shape obtained by removing a predetermined number of element data from the element data forming a matrix.
The first data is converted based on the arrangement shape of the second data.
A control method of an arithmetic processing unit, characterized in that a convolution operation is performed on the converted first data by using the second data as a filter.

Images