KR102498066B1 - Deep Reinforcement Learning Accelerator - Google Patents

Abstract

The present invention relates to a deep reinforcement learning (DRL) technology, and a DRL accelerator is an actor that receives input data and outputs the result of processing through a DNN (Deep Neural Network) as an action (Action) It includes a learner circuit block that learns weight values of the DNN using an actor circuit block and a plurality of state data pairs.

Description

Deep Reinforcement Learning Accelerator

The present invention relates to Deep Reinforcement Learning (DRL) technology, in particular, to dynamically switch different data reuse methods of inference and training of reinforcement learning so that it can be used in a mobile environment, and for learning. It is about a deep learning reinforcement learning accelerator that compresses and stores data.

Recently, DRL has been spotlighted as one of the most powerful techniques applied in sequential decision-making problems for motion control and object tracking. Here, the artificial intelligence that performs DRL is referred to as a DRL agent, and an example is an intelligent robot. The DRL agent can learn optimal motions from input/output information, that is, experience, collected by interacting with the external environment on its own without human supervision, and can perform complex tasks almost at the same level as humans. Unlike reasoning-only AI, which lacks the ability to learn, DRL agents can dynamically adjust their behavior to target goals, even in constantly changing environmental conditions.

In the case of DRL, the learner uses the DNN for inference, but also needs to update (a kind of learning) at the same time, so it requires a larger maximum memory bandwidth than simple inference tasks. In addition, since a plurality of DNNs are required, DNN circuits implemented on a chip must be spatially and temporally mapped to a separate DNN, which may cause performance degradation.

The prior art documents below suggest technical means of introducing an accelerator that promotes parallel processing of data in DNN hardware that performs inference.

J. Lee, C. Kim, S. Kang, D. Shin, S. Kim, and H.-J. Yoo, "UNPU: A 50.6 TOPS/W unified deep neural network accelerator with 1b-to-16b fully-variable weight bit-precision," in 2018 IEEE International Solid-State Circuits Conference-(ISSCC), 2018: IEEE, pp. 218-220. K. Ueyoshi et al., "QUEST: A 7.49 TOPS multi-purpose log-quantized DNN inference engine stacked on 96MB 3D SRAM using inductive-coupling technology in 40nm CMOS," in 2018 IEEE International Solid-State Circuits Conference-(ISSCC) , 2018: IEEE, pp. 216-218.

The technical problem to be solved by the present invention is that memory access is not optimized for DRL applications that require adaptive data reuse even in the case of learning because conventional DNN hardware only performed inference or only a fixed data path existed in data parallelism. We want to overcome the technical limitations of doing it.

In order to solve the above technical problem, a deep reinforcement learning (DRL) accelerator according to an embodiment of the present invention receives input data and outputs the result of processing through a deep neural network (DNN) as an action. Actor circuit block; and a learner circuit block that learns weight values of the DNN using a plurality of data pairs including {current state, median value, action, reward, next state}.

In the DRL acceleration device according to an embodiment, in the actor circuit block, a MAC combining a multiplier and an accumulator forms a 2-dimensional PE array (Processing Element Array), and a plurality of PE arrays are inside. It integrates and receives the current state input from the outside, and can output the action and next state through DNN reasoning.

In addition, in the DRL accelerator device according to an embodiment, the actor circuit block, during a learner operation, a plurality of {current state, intermediate value, action, compensation, where the plurality of PE arrays were stored instead of external input .

In the DRL accelerator according to an embodiment, when mapping DNNs for actors and runners and other DNNs to the PE array, the usable PEs are spatially divided, and each of the divided PEs is divided into the Actor and DNNs for runners and other DNNs can be allocated and processed in parallel. In addition, it may further include a control logic (controller) that detects a change in the amount of computation or memory bandwidth required for DNN inference and changes the number of PEs allocated to each of the DNNs according to the detected result.

In the DRL accelerator according to an embodiment, when DNNs and other DNNs for actors and runners are mapped to the PE array, the DNNs and other DNNs for actors and runners can be temporally divided and used. PEs can be sequentially allocated and processed. In addition, in the DRL accelerator according to an embodiment, when DNNs for actors and runners and other DNNs are mapped to the PE array, PEs usable according to space-time division are mapped to a plurality of DNNs for parallel processing. However, it is possible to sequentially allocate and process the PEs assigned to the previously terminated DNN to other waiting DNNs. Furthermore, in the DRL accelerator according to an embodiment, when the operation of one DNN is completed, the current weight value, input value, and output value are stored in an external memory, and another DNN in standby is the PE assigned to the previously completed DNN. It may further include a control logic (controller) that reads weight values, input values, and output values required by the other waiting DNNs from an internal or external memory so that can be used.

The DRL accelerator according to an embodiment may further include a compressor that encodes or decodes a plurality of data pairs including the {current state, intermediate value, action, reward, next state}.

In the DRL accelerator device according to an embodiment, the PE array is configured such that input directions in actors and input directions in runners are different from each other, so that input features (IF) and weights (Input Feature, IF) and weights ( It is possible to form a transition PE (transposable processing element, tPE) structure that changes the data reuse of Weight, W).

In order to solve the above technical problem, a deep reinforcement learning (DRL) accelerator according to another embodiment of the present invention includes a plurality of DRL cores performing deep reinforcement learning (DRL); and an upper shared memory connected to the plurality of DRL cores through an on-chip network under the control of an upper controller, and receives input data and processes the result through a deep neural network (DNN) into an action. While the actor outputting to is running, the upper shared memory loads the weight values of the DNN into the DRL core, and multiple values including {current state, median value, action, reward, next state} Experience data shared by the plurality of DRL cores is stored in the upper shared memory while a learner that learns weight values of the DNN using data pairs of is executed.

In the DRL accelerator device according to another embodiment, the plurality of DRL cores include: a core controller controlling the DRL cores; a compressor for encoding or decoding a plurality of data pairs including the {current state, intermediate value, action, reward, next state}; and a PE array (Processing Element Array) that receives a current state and outputs an action and a next state through DNN inference.

In the DRL accelerator according to another embodiment, the plurality of DRL cores are connected to the compressor and receive weights (Weight, W) and input features (Input Feature, IF) broadcast memory (BMEM) and unicast memory (UMEM); and the broadcast memory and the unicast memory may provide input data to the PE array through a B-buffer and a U-buffer, respectively.

In the DRL acceleration device according to another embodiment, the core controller automatically fetches the weights and the input characteristics to the broadcast memory or the unicast memory according to the DNN network structure, and when an actor or runner operates The configuration of the PE array can be set to change the data reuse of weights and input features, respectively.

In the DRL accelerator device according to another embodiment, the plurality of DRL cores include: an activation unit processing a nonlinear function; and a 1-D SIMD unit that performs log functions, addition and multiplication for weight update and loss computation.

In the DRL accelerator according to another embodiment, while the PE array processes a DNN operation, the compressor scans an output buffer, and when the output buffer is full, the compressor generates a code word, an exponent ( exponent and mantissa are sequentially encoded, and when an input feature is transmitted to the DRL core, the compressor scans the code word and recombines the sequentially input input mantissa and exponent to stream the input feature. -Can stream-out.

In the DRL acceleration device according to another embodiment, the PE array includes a data pair including a plurality of {current state, median value, action, reward, next state} stored instead of external input during a runner operation. It is possible to generate a loss function by reading , and update the weight of the DNN by performing a back-propagation operation based on the generated loss function.

In the DRL accelerator according to another embodiment, the PE array is composed of a two-dimensional array of MACs, and each row shares a row buffer for receiving broadcast data and multiplies an input value through the MAC, but new data is broadcast every cycle to perform a parallel matrix operation, and if all unicast data provided from the U-buffer are reused, new unicast data can be input through the U-buffer.

In the DRL accelerator device according to another embodiment, the PE array is configured such that input directions in the actor and input directions in the runner are different from each other, so that the input feature (IF) and weight (Input Feature, IF) and weight ( It is possible to form a transition PE (transposable processing element, tPE) structure that changes the data reuse of Weight, W). In addition, in the PE array, the row length of the weight matrix (W Matrix) representing the number of output channels (Co) is greater than the column length of the input feature matrix (IF Matrix) representing the number of batch processing (Batch Size, BA). W broadcast (WBC) can be selected when it is relatively large, and IF broadcast (IFBC) can be selected when the row length of the weight matrix is relatively smaller than the column length of the input feature matrix.

Embodiments of the present invention have the advantage of significantly reducing external peak memory bandwidth and power consumption by using a compressor and a PE array structure that dynamically switches data reuse for both learning and inference.

1 is a diagram for explaining a processing process according to a subject performing deep learning reinforcement learning.
2 is a diagram for explaining an Actor-Critic algorithm that can be utilized in the technical field to which the present invention belongs.
FIG. 3 is a diagram for explaining data reuse in a matrix operation in a DRL agent.
4 is a diagram showing differences in data reuse methods of input features and weights.
5 is a diagram illustrating a dynamic change of a dynamic data reuse pattern during learning.
6 is a diagram illustrating a PE array that can be utilized in a DNN chip.
7 and 8 are diagrams illustrating a space division distribution method and a time division distribution method that can be utilized by embodiments of the present invention, respectively.
9 to 11 are diagrams illustrating space-time division and distribution methods and various combinations that can be utilized by embodiments of the present invention.
12 is a block diagram illustrating a DRL accelerator for dynamically switching data reuse according to an embodiment of the present invention.
13 is a diagram showing the structure of a DRL accelerator for dynamically switching data reuse according to another embodiment of the present invention.
14 is a diagram showing the structure of a compressor proposed by embodiments of the present invention.
15 is a diagram showing the structure of a transposable processing element (tPE) array proposed by embodiments of the present invention.
16 is a diagram for explaining a method of dynamically selecting W broadcast (WBC) and IF broadcast (IFBC) by utilizing a switched PE array structure.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, detailed descriptions of well-known functions or configurations that may obscure the gist of the present invention will be omitted in the following description and accompanying drawings. In addition, throughout the specification, 'including' a certain component means that other components may be further included, not excluding other components unless otherwise stated.

Terms used in the present invention are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the terms "comprise" or "comprise" are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, but that one or more other features or It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Unless specifically defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in this application, they are not interpreted in an ideal or excessively formal meaning. .

1 is a diagram for explaining a processing process according to a subject performing deep learning reinforcement learning. A DRL agent generally receives the environmental state (State, S _t ) observed by the DNN equipped inside through two components, Actor and Learner, which perform learning and inference, respectively, and optimizes itself. Outputs the action (Action, A _t ) of From the observed current state (S _t ), the DRL agent determines the optimal action (A _t ) using the DNN weight (Weight, θ) and policy (π, Policy). During DRL processing, the actor continuously interacts with the environment, and the runner trains the DNN regularly to maximize the reward (R _t ) from the environment. Strictly speaking, we learn in the direction of maximizing Return-Gt (=R _t +γR _t+1 +γ ² R _t+2 +...), the sum of the rewards that will be obtained when acting according to the current policy (γ is the Discounting Factor has a value of 0 < γ < 1. This plays a role of putting more weight on the reward that can be obtained directly from the current action, like the behavior of animals).

The actor observes the state of the environment (S _t ) and selects an action (A _t ) determined by the current DNN weight (Weight, θ) and its policy (π, Policy). After the DRL agent performs an action, the agent receives the next observed external environment state (S _t+1 ) and a scalar reward (R _t ) as an input. Each time, the experience stored in the experience memory is sampled and the runner trains the DNN using it. In order to utilize the "Experience Replay technique" widely used in DRL algorithms for learning stable DRL agents, the experience memory needs to store a large number of experiences (approximately 10,000 or more). The runner fetches a number of random experience samples in batch form and utilizes a loss function to maximize the reward for the action determined by the current DNN. Then, the runner updates the DNN weights using the gradient calculated from the loss function.

In general, DRL learning can be largely divided into On-Policy Learning and Off-Policy Learning. On-Policy Learning refers to a learning method in which the learning policy (Learner's Policy) and the acting policy (Actor's Policy) are the same. On the contrary, Off-Policy means that the policy DNN is trained by randomly sampling the accumulated experience after accumulating the DRL's Action Generation several times with a learning method in which the two policies are not necessarily the same. The biggest difference between the two is in the way they use their previous experience. In the case of using the On-Policy learning method, the best policy is found from the current experience every time, and a new experience is obtained with that policy to have a faster learning rate, but it is in a state where it is lost in Local-minima and cannot find a better policy anymore. Falling out happens. Complementing this, Off-Policy Learning reuses previously accumulated experience multiple times, enabling efficient data use for tasks where experience collection is difficult. In addition, it learns from randomly sampled experiences and has a slower learning speed than On-Policy Learning, but provides stable learning that does not fall into Local-minima through various trials and errors.

DQN (Deep-Q Network) is a representative Off-Policy Learning algorithm and Actor-Critic is a representative On-Policy Learning algorithm. ).

DQN includes 2 DNN networks consisting of 1 Main Q-network and 1 Target Network. DQN determines Action a _t from the current state s _t through the Main Q-network. Actor outputs are grouped into a set of {current state s _t , action a _t , reward R _{t +1} , next state s _{t+ 1} } and stored in a separate memory (replay memory), and then a data set is randomly selected from the replay memory. Call out to learn the target network. This target network is synchronized with the Main Q-network at regular intervals, and through this, noise generated in updates is reduced to provide stable DQN learning. Learn the target network through the equation below.

는 현재 학습 중인 Target network에서 구한다. (2)식의 앞의 첫 항은 다음 상태 s_t+1에서 현재 학습 중인 Target Network가 추론한 행동으로부터 얻을 수 있는 보상의 총합을 나타내며, 두 번째 항은 다음 상태 s_t+1에서 현재 Main Q-network가 추론한 행동으로부터 얻을 수 있는 보상의 총합을 나타낸다. 즉 다음 상태 s_t+1에서의 Target Network가 추론한 Q 값이 최대가 되는 액션 a와 현재 에이전트의 행동 정책으로부터 얻을 수 있는 Q 값의 차를 최소화함으로써 최대의 보상을 갖는 정책을 찾아낼 수 있다. 따라서 이를 손실 함수(Loss Function)로 사용하여 Target Network와 Main Q-network의 가중치(Weight) 값들을 갱신(Update)한다.Here, Q _m points to the Main Q-network, namely the Actor DNN, and Q _t points to the Target Network.

is obtained from the target network currently being trained. The first term in equation (2) represents the sum of rewards that can be obtained from the actions inferred by the target network currently learning in the next state s _t+1 , and the second term represents the current Main Q in the next state s _t+1. -Represents the total amount of rewards that can be obtained from actions inferred by the network. That is, the policy with the maximum reward can be found by minimizing the difference between the action a for which the Q value inferred by the target network in the next state s _t+1 is maximized and the Q value that can be obtained from the action policy of the current agent. . Therefore, this is used as a loss function to update the weight values of the target network and the main Q-network.

2 is a diagram for explaining an Actor-Critic algorithm that can be utilized in the technical field to which the present invention belongs. In the Actor-Critic algorithm, another DNN is created and entrusted with approximation of the Q function. That is, in addition to the existing Policy Network DNN (where the Policy Network encompasses the Main Q-network and Target Network of the DQN) that approximates the policy, a Value Network DNN that approximates the Q value is added. Each Policy Network and Value Network use a total of 4 DNNs, including the Main Q-network and Target Network. In this case, it can be divided into a case where the Policy Network and a Value Network are completely independent, and a case where the front several layers of one DNN are shared and only the last few layers are installed independently.

을 구할 수 있고 Value Network 출력으로부터 Time Difference Error,

를 구할 수가 있으며 이를 곱하여

를 최대화하도록 가중치(Weight) θ를 구하는 것으로 학습한다. Value Network의 학습은 출력

가 정확한 값을 갖도록 하여야 하는데 결국 실제 행동을 취해서 얻게 되는 행동가치

와 일치시키는 것으로 결국

으로 학습(learning)을 진행한다. Policy Network는 Cross-Entropy Error Function과 Time Difference Error의 곱으로 새로운 오류함수를 정의하고 이 오류함수로 Policy Network를 갱신(Update)한다.Both On-Policy Learning and Off-Policy Learning are possible for Actor-Critic Algorithm. Here, On-Policy Learning will be explained as an example. From the output of Policy Network, Cross-Entropy Error Function,

can be obtained, and from the Value Network output, Time Difference Error,

can be obtained and multiplied by

It learns by finding the weight θ to maximize . Value Network learning is an output

should have an accurate value, but in the end, the action value obtained by taking an actual action

Finally, by matching

proceed with learning. Policy Network defines a new error function as the product of Cross-Entropy Error Function and Time Difference Error, and updates Policy Network with this error function.

In the DQN algorithm described above, two DNNs are required, the Main DNN and the Target DNN of the actor, and the Actor-Critic algorithm requires multiple DNNs, such as Actor DNN and Critic DNN. In the present invention described below, a DRL CMOS chip in which a plurality of DNNs are supported and both the DQN algorithm and the Action-Critic algorithm operate in one chip is proposed.

Earlier, in the case of DRL, it was pointed out that the runner uses the DNN for inference, but also needs to update it at the same time, so it requires a larger maximum memory bandwidth than simple inference work. In addition, it is pointed out that since a plurality of DNNs are required, DNN circuits implemented on a chip must be spatially and temporally mapped to separate DNNs, which can cause performance degradation.

For convenience of explanation, let's assume a DRL agent in which a bipedal walking simulation robot learns to walk on its own. The above agent consists of 4 fully connected hidden layers with 512 neurons/layers and 212 input parameters. 3 is a diagram for explaining data reuse in the DRL Agent during FCL (Fully-Connected Layer) and RNN Matrix calculation. In the DRL Agent, the maximum memory bandwidth required by the Learner is the same. It shows that it is 10 times larger than the demand from the actors in the DRL agent. 4 is a diagram showing differences in data reuse methods of input features and weights. When analyzed based on memory bandwidth analysis, actors and runners have different data reuse methods of input features (Input Feature, IF or Activation) and weights (DNN Weight, W) as shown in FIG. In general, DNN operations can be simplified and expressed as the product of an IF matrix and a W matrix. If input IF and W data are continuously reused and operations are performed without the need to re-enter the same IF and W data, there is no need to fetch the same IF and W data, so the input/output bandwidth can be reduced.

Referring to FIGS. 3 and 4 , one IF (indicated by a circle) may be reused for Ws (indicated by a triangle) located in the same row in the W matrix. Therefore, the maximum reuse count of IF is the row length of the W matrix, equal to the number of Output Channels (Co) in one layer. Similarly, the maximum reuse count of one W data is equal to the column length of the IF matrix, which is equal to the batch size (B). Because actors typically take a single IF as input while interacting with the environment, IF reuse is more efficient. In the case of the runner, since the Co size is different for each layer, IF reuse or W reuse may be more efficient depending on the situation.

5 is a diagram illustrating a dynamic change of a dynamic data reuse pattern during learning, and shows a dynamic change of a data reuse pattern for each layer during runner processing. As can be seen in the figure, the number of memory accesses can be reduced to ~x10 through adaptive selection between IF reuse and W reuse.

As noted earlier, previous DNN hardware only performed inference, and their data parallels had only fixed data paths, so memory access was optimized only for DNN inference, but for DRL applications that require adaptive data reuse for training. It didn't work.

Therefore, the embodiments of the present invention presented below provide spatial division, temporal division, spatio-temporal division of the PE array present in the chip to the DNN in order to implement a plurality of DNNs required by various DRL algorithms with high performance. We propose a segmentation and composite segmentation mapping method. In addition, we propose an energy-efficient DRL accelerator that is optimized for learning various DNNs such as RNN (Recurrent Neural Network), FCL (Fully-Connected Layer), and CNN (Convolution Neural Network) by increasing the reconfigurability of PE. In particular, since the data reuse method of W and IF in DNNs suitable for learning is different from that of inference, we propose a new reconfigurable PE array, Transposable PE (tPE) array. Furthermore, we propose a compressor that can reduce data by compressing experience and intermediate layer data in the runner.

6 is a diagram illustrating a PE array that can be utilized in a DNN chip. As shown, a general DNN chip is composed of PE arrays, and the PE array is composed of several to thousands of PEs. This large number of PEs can be spatially partitioned and assigned to each DNN, which is called the spatial division distribution method.

7 is a diagram illustrating a spatial division and distribution method that can be utilized by embodiments of the present invention. A certain number of PEs are assigned to actors, a certain number of PEs are assigned to Critic, and a certain number of PEs are assigned to other DNNs required by the algorithm. These allocations can be changed dynamically in time, so that they can be changed again during operation according to the computational requirements of each DNN. Each DNN performs DNN operation independently using the assigned PEs, and parallel processing is possible at the same time because there is little dependency between DNNs. However, the PE arrays allocated to the DNN on which processing has been completed are placed in a standby state until the next processing request, which may reduce the utilization of PEs.

On the other hand, all of the PE arrays are used to process one DNN, and DNNs required by the algorithm, such as Actor DNN, Critic DNN, and other DNNs, alternate in time to use the entire PE array can be considered, and this is called the time-division distribution method.

8 is a diagram illustrating a time division distribution method that embodiments of the present invention can utilize. Parallel processing as in the spatial division distribution method is impossible, but since all PE arrays implement one DNN, there is an advantage that the processing speed of one DNN is fast. In particular, in the case of algorithms in which multiple DNNs share many layers, using time division makes it much simpler because other DNNs only need to receive the intermediate results of the first DNN operation and operate only on the last few DNN layers of their own.

However, since each DNN has a different required amount of computation and memory bandwidth, a space-time division-allocation method combining a space-division distribution method and a time-division distribution method considering this may also be considered. Referring to FIG. 9, all PE arrays are allocated to the Actor DNN in the time interval t ₁ -t ₂ , and in the time interval t ₂ -t ₃ , the Critic DNN and other DNNs divide and use all PEs by the spatial division distribution method. . In this case, in the period t ₃ -t ₄ , PE arrays allocated to other DNNs are in an idle state, and PE utilization may decrease. As a way to prevent this decrease in PE utilization, when the operation of other DNNs is completed, the PE arrays allocated to it can be allocated to the next stage, Actor DNN, so that Actor DNN operation can be started early.

In addition, referring to FIG. 10, by further developing the parallel processing by the spatial division distribution method, all PE arrays are distributed to Actor DNN, Critic DNN, and other DNNs to perform DNN operations, and PE arrays that have been performed are transferred to other DNNs. The utilization of the PE array can be improved by assigning it to a DNN. Furthermore, a composite method in which various methods described above are combined and used at once is also possible. As shown in FIG. 11, time division, space-time division, and space division may all be used together.

Since each DNN has a different neural network structure and weight values, in the case of time division, if the values for the previous time DNN are used later, they must be stored in memory and recalled later, and a separate controller to control them must be integrated on a chip.

Hereinafter, an outline of a DRL accelerator for dynamically switching data reuse proposed by embodiments of the present invention will be reviewed, and then a specific chip structure will be sequentially described.

12 is a block diagram illustrating a DRL acceleration device 300 that dynamically switches data reuse according to an embodiment of the present invention, and explains conceptual operations and processing methods centered on performers (actors and runners). . The DRL accelerator 300 may be implemented as a CMOS chip.

The actor circuit block 100 receives input data, processes it through a deep neural network (DNN), and outputs a result as an action. The learner circuit block 200 learns weight values of the DNN using a plurality of data pairs including {current state, median value, action, reward, next state}.

Here, in the actor circuit block 100, a MAC combining a multiplier and an accumulator forms a two-dimensional PE array (Processing Element Array), and a plurality of PE arrays are internally integrated to input from the outside. It receives the current state and can output the action and next state through DNN reasoning. In addition, the actor circuit block 100 includes a plurality of {current state, intermediate value, action, compensation, next state} stored in the plurality of PE arrays instead of external input during a runner operation. A loss function may be generated by reading a data pair, and weights of the DNN may be updated by performing a back-propagation operation based on the generated loss function. At this time, the DNN can apply various algorithms such as a recurrent neural network (RNN), a convolutional neural network (CNN), and a fully-connected layer (FCL).

Further, from an implementation point of view, it is preferable to further include a compressor that encodes or decodes a plurality of data pairs including {current state, intermediate value, action, reward, next state}. These compressors reduce the size of memory for storing the multiple data pairs described above and require separate hardware to compress and decompress these pairs in order to reduce the memory bandwidth required for fetch and store. It was created because

Furthermore, the PE array is configured such that input directions in the actor and input directions in the runner are different from each other, thereby reusing data of input features (IF) and weights (Weight, W) during the operation of the actor or runner, respectively. It is possible to form a transition PE (transposable processing element, tPE) structure that changes. As described above with reference to FIGS. 3 to 5, this was devised in consideration of the fact that the data reuse method of W and IF in a DNN suitable for learning is different from that of inference. A more specific structure and conversion method will be described in detail later with reference to FIG. 16 .

Meanwhile, as described above, the DRL accelerator according to the embodiments of the present invention may allocate a plurality of DNNs to the PE array in various ways.

First, in the case of mapping DNNs for actors and runners and other DNNs to the PE array, the available PEs are spatially partitioned, and the DNNs for actors and runners and other DNNs are divided into each divided PE. can be allocated and processed in parallel. From an implementation point of view, it is preferable to further include a controller that detects a change in the amount of computation required for DNN inference or memory bandwidth, and changes the number of PEs allocated to each of the DNNs according to the detected result. .

Second, when DNNs for actors and runners and other DNNs are mapped to the PE array, DNNs for actors and runners and other DNNs are temporally divided to sequentially allocate and process available PEs. can

Third, when DNNs for actors and runners and other DNNs are mapped to the PE array, PEs that can be used according to space-time division are mapped to a plurality of DNNs for parallel processing, but the PEs assigned to the previously terminated DNNs are mapped. can be processed by sequentially assigning to other waiting DNNs.

In the second and third cases from the implementation point of view, when the operation of one DNN is completed, the current weight value, input value, and output value are stored in external memory, and the other waiting DNN is the PE that was assigned to the DNN that was completed earlier. It is preferable to further include a control logic (controller) for reading weight values, input values, and output values required by the other waiting DNNs from an internal or external memory so that .

13 is a diagram showing the structure of a DRL accelerator for dynamically switching data reuse according to another embodiment of the present invention, and a specific chip architecture is presented.

A plurality of DRL cores (or t-Cores) 20 (eg, 4 top-level controllers of 64 KB) are provided in the DRL accelerator to perform deep reinforcement learning (DRL). The upper shared memory 10 is connected to the plurality of DRL cores 20 through an on-chip network under the control of an upper controller (Top Ctrlr). While an actor that receives input data and outputs the result of processing through a deep neural network (DNN) as an action is running, the upper shared memory 10 recalls the weight values of the DNN. A runner that loads into the DRL core 20 and learns the weight values of the DNN using a plurality of data pairs including {current state, median value, action, reward, next state} While (Learner) is being executed, experience data shared by the plurality of DRL cores 20 is stored in the upper shared memory 10.

The plurality of DRL cores 20 include a core controller 21 that controls the DRL cores, and a compressor that encodes or decodes a plurality of data pairs including {current state, intermediate value, action, compensation, next state}. ) 22 and a PE array (Processing Element Array) 23 that receives a current state and outputs an action and a next state through DNN inference.

In addition, the plurality of DRL cores 20 are connected to the compressor 22 and receive weights (Weight, W) and input features (Input Features, IF) in a broadcast memory (BMEM) 24 (64 KB). exemplified) and a unicast memory (UMEM) 26 (exemplified as 32 KB) may be further included. The broadcast memory 24 and the unicast memory 26 may provide input data to the PE array 23 through a B-buffer (not shown) and a U-buffer (not shown), respectively. , the output of the PE array 23 is input to an accumulation unit and added. Here, the core controller 21 automatically fetches the weight and the input feature to the broadcast memory 24 or the unicast memory 26 according to the DNN network structure, and operates an actor or runner. The configuration of the PE array 23 may be set to change the data reuse of the weight W and the input feature IF, respectively.

In addition, the plurality of DRL cores 20 may further include an activation unit for processing non-linear functions and a 1-D SIMD unit 28 for performing log functions, additions and multiplications for updating weights and calculating losses. .

14 is a diagram showing the structure of a compressor proposed by embodiments of the present invention. The compressor 22 that compresses, stores, and loads the data collected in real time may be composed of an encoder and a decoder. Since the distribution of the entire data is wide due to a small error and ΔW during learning, the input feature (IF) can be expressed as bfloat16. It is a distribution value of exponent data of input features stored as experience, which is highly concentrated in a narrow range. In the case of Learner's 20,480 Experience and Intermediate Layer data, this degree of concentration is even greater, with the average proportions of the three most frequent exponent values (top-3 indices) being 68 each. %, 85%. The 2b code word is data added for compression and indicates whether the currently displayed IF data is compressed. Through this code word, if the current node has an index included in the upper-3 index, the 8-bit exponent is skipped, and when the current node has an index not included in the upper-3 index, the exponent data is skipped. can't Code '00' indicates that indexing cannot be skipped because the current node does not fall within the top-3 index. The other codes '01', '10' and '11' indicate that the index of the current node is skipping index values in the top 1, top 2 and top 3 indices, respectively.

Referring to FIGS. 13 and 14 , while the PE array 23 processes a DNN operation, the compressor 22 scans the output buffer to check the three most frequently used index values. When the output buffer is full, the compressor 22 sequentially encodes a code word, an exponent, and a mantissa, and when an input feature (IF) is transmitted to the DRL core, The compressor 22 (Decompressor in FIG. 14) scans the code word and recombines the sequentially input input mantissa and exponent to stream-out the input feature. The following Table 1 is the result of measuring the compression rate with the DRL agent that can train the bipedal robot walker exemplified above to walk without falling. According to the measurement result, the average compression rate is 35%

15 is a diagram showing the structure of a transposable processing element (tPE) array 23 proposed by embodiments of the present invention, showing the architecture of a two-dimensional transition PE array for reinforcement learning processing that performs matrix multiplication. show B ₁₁ , B ₂₁ , B ₃₁ , and B ₄₁ data in the first row PE can be multiplied by A ₁ , and new data among them can be broadcast each cycle to perform parallel matrix operation. Integrates 16 4 4 4 Transposable PE arrays to share row buffers receiving broadcast data from BMEM. And each PE array is connected to a U-buffer 27 providing unicast data. The FP-FXP MAC performs multiplication of 16b bfloat type data by 16b integer, which is integrated in each PE. The results of four 4b multiplications and two 8b multiplications can be obtained simultaneously and in parallel at no additional cost so that the weight bit precision can be varied for different requirements of accuracy and performance. If all unicast data is reused, new unicast data is input to the PE array 23 through the U-buffer.

The PE array 23 generates a loss function by reading data pairs including a plurality of stored {current state, median value, action, reward, next state} instead of external input during a runner operation, , it is possible to update the weights of the DNN by performing a back-propagation operation based on the generated loss function. To this end, the PE array 23 is composed of a two-dimensional array of MACs, and each row shares a row buffer for receiving broadcast data and multiplies an input value through the MAC, and new data is generated every cycle. It is broadcast to perform a parallel matrix operation, and if all unicast data provided from the U-buffer are reused, new unicast data is input through the U-buffer.

16 is a diagram for explaining a method of dynamically selecting W broadcast (WBC) and IF broadcast (IFBC) by utilizing a switched PE array structure. As previously introduced, the PE array is configured so that the input direction in the actor and the input direction in the runner are different from each other, so that when the actor or runner operates, the input feature (IF) and weight (Weight, W) It forms a transposable processing element (tPE) structure that changes data reuse.

First, the reuse of IF is more efficient when the size of _CO is larger than that of BA. In this case, as shown in (A) of FIG. 16, W broadcast is adopted. PEs in the same row store IFs in the same input channels in different batches, and PEs in the same column store IFs in different input channels in the same batch. For example, at the start of DNN processing (T=0), PE ₀ stores the IF of a circle (BA=0), C _i =0) and PE ₁ stores the IF of a rectangle (BA=1, C _i =0). do. And PE ₄ stores the IF of the triangle (BA=0, C _i =1). The B-Buffer broadcasts 4 different Ws of 4 different input channels every cycle, and the results of PEs in the same column are added in an Accumulation Unit. After the W broadcast of the four input channels is complete, the PE array fetches a new IF from the U-buffer, and the PE array continues DNN processing. That is, in the PE array, the row length of the weight matrix (W Matrix) representing the number of output channels (Co) is relative to the column length of the input feature matrix (IF Matrix) representing the number of batch processing (Batch Size, BA). When , W broadcast (WBC) is preferably selected.

Second, when BA size is larger than _CO , reuse of W is more efficient than reuse of IF. In this case, as shown in (B) of FIG. 16, IF broadcast is adopted. PE ₀ and PE ₂ store W in different output channels, and PE ₀ and PE ₄ store W in the same output channel. That is, the PE array preferably selects IF broadcast (IFBC) when the row length of the weight matrix is relatively smaller than the column length of the input feature matrix.

When an actor operates, the batch size is larger than the size of the output channel, so W broadcast requires less external memory access than IF broadcast. In this case, W broadcast appears to be more efficient in terms of memory access, but many PEs cannot receive IF data that should be in idle (IDLE) state, so W broadcast has lower utilization of cores, so the frame rate is lower than IF broadcast. this lowers Therefore, IF broadcasts are more efficient for actors. Upon actor and runner operation, the configuration of the transition PE array can be automatically selected by the core controller.

Table 2 measures the proposed scheme with worker simulation of the DRL agent according to the embodiments of the present invention. If only IF broadcast (IFBC) is used, it consumes less power than W broadcast (WBC) because actors take up most of the DRL's processing time. However, the IFBC-only case requires higher memory bandwidth than the WBC-only case because WBC is more efficient in memory access than IFBC during runner processing. The IFBC and WBC are adaptively selected to reduce power consumption and peak memory bandwidth by utilizing the switched PE array. We also measure the power consumption and peak memory bandwidth of the switched PE array with Experience compression. Through the adaptive selection of IFBC and WBC together with experience compression, it was obtained as an experimental result that the average power consumption was reduced by 31% and the maximum memory bandwidth was reduced by 41%.

Embodiments of the present invention adopt a floating-point system and use a data compression scheme and an adaptive data path to handle large memory bandwidth and different data reuse in DRL processing. The result is precise DNN training for real-time, low-power DRL operation. Embodiments of the present invention propose a DRL accelerator equipped with a disposable PE array and an experience compressor to realize autonomous DRL operation in a dynamic environment. In addition, it was possible to reuse adaptive data for reasoning and training, and it was obtained as an experimental result that power and maximum memory bandwidth were reduced by 31% and 41%, respectively.

Meanwhile, the embodiments of the present invention can be implemented as computer readable codes in a computer readable recording medium. The computer-readable recording medium includes all types of recording devices in which data that can be read by a computer system is stored.

Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. In addition, the computer-readable recording medium may be distributed to computer systems connected through a network, so that computer-readable codes may be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the present invention can be easily inferred by programmers in the technical field to which the present invention belongs.

In the above, the present invention was examined focusing on various embodiments thereof. Those of ordinary skill in the art pertaining to the present invention will be able to understand that the present invention can be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from an illustrative rather than a limiting point of view. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the equivalent scope will be construed as being included in the present invention.

100: Actor circuit block
200: runner circuit block
300: DRL accelerator
10: upper shared memory
20: DRL core
21: core controller
22: Compressor
23: PE array
24: Broadcast memory (BMEM)
25: B-buffer
26: Unicast Memory (UMEM)
27: U-buffer
28: SIMD unit

Claims (22)

delete delete delete delete An actor circuit block that receives input data and outputs a result of processing through a deep neural network (DNN) as an action; and
DRL (including a Learner circuit block that learns weight values of the DNN using a plurality of data pairs including {current state, median value, action, reward, next state}) As a deep reinforcement learning) accelerator,
In the actor circuit block, a MAC combining a multiplier and an accumulator forms a two-dimensional PE array (Processing Element Array), and a plurality of PE arrays are internally integrated to receive a current state input from the outside, Output the action and next state through DNN inference,
When mapping DNNs for actors and runners and other DNNs to the PE array, the available PEs are spatially partitioned and the DNNs for actors and runners and other DNNs are allocated to each of the divided PEs. to process in parallel,
A control logic (Controller) that senses a change in the amount of computation or memory bandwidth required for DNN inference, and changes the number of PEs allocated to each of the DNNs according to the detected result; further comprising a DRL accelerator. An actor circuit block that receives input data and outputs a result of processing through a deep neural network (DNN) as an action; and
DRL (including a Learner circuit block that learns weight values of the DNN using a plurality of data pairs including {current state, median value, action, reward, next state}) As a deep reinforcement learning) accelerator,
In the actor circuit block, a MAC combining a multiplier and an accumulator forms a two-dimensional PE array (Processing Element Array), and a plurality of PE arrays are internally integrated to receive a current state input from the outside, Output the action and next state through DNN inference,
When DNNs for actors and runners and other DNNs are mapped to the PE array, a DRL that divides the DNNs for actors and runners and other DNNs in time and sequentially allocates and processes usable PEs. accelerator. An actor circuit block that receives input data and outputs a result of processing through a deep neural network (DNN) as an action; and
DRL (including a Learner circuit block that learns weight values of the DNN using a plurality of data pairs including {current state, median value, action, reward, next state}) As a deep reinforcement learning) accelerator,
In the actor circuit block, a MAC combining a multiplier and an accumulator forms a two-dimensional PE array (Processing Element Array), and a plurality of PE arrays are internally integrated to receive a current state input from the outside, Output the action and next state through DNN inference,
When mapping DNNs for actors and runners and other DNNs to the PE array, the available PEs are spatially partitioned and the DNNs for actors and runners and other DNNs are allocated to each of the divided PEs. to process in parallel,
When DNNs for actors and runners and other DNNs are mapped to the PE array, PEs available according to space-time division are mapped to a plurality of DNNs for parallel processing, but the PE assigned to the DNN that has been terminated first is waited for. A DRL accelerator that sequentially assigns and processes other DNNs in progress. According to any one of claims 6 and 7,
When the operation of one DNN is finished, the current weight values, input values, and output values are stored in external memory, and other waiting DNNs can use the PE allocated to the DNN that was completed earlier. A control logic (Controller) for reading weight values, input values, and output values from an internal or external memory; further comprising a DRL accelerator. delete An actor circuit block that receives input data and outputs a result of processing through a deep neural network (DNN) as an action; and
DRL (including a Learner circuit block that learns weight values of the DNN using a plurality of data pairs including {current state, median value, action, reward, next state}) As a deep reinforcement learning) accelerator,
In the actor circuit block, a MAC combining a multiplier and an accumulator forms a two-dimensional PE array (Processing Element Array), and a plurality of PE arrays are internally integrated to receive a current state input from the outside, Output the action and next state through DNN inference,
When mapping DNNs for actors and runners and other DNNs to the PE array, the available PEs are spatially partitioned and the DNNs for actors and runners and other DNNs are allocated to each of the divided PEs. parallel processing,
The PE array,
Conversion PE (transposable processing PE) that changes data reuse of input features (IF) and weights (W), respectively, when the input direction in the actor and the input direction in the runner are configured to be different from each other, when the actor or runner operates. Element, tPE) structure, DRL accelerator. A plurality of DRL cores that perform DRL (Deep Reinforcement Learning); and
An upper shared memory connected to the plurality of DRL cores through an on-chip network under the control of a higher controller;
While an actor that receives input data and outputs the result of processing through a deep neural network (DNN) as an action is running, the upper shared memory transfers the weight values of the DNN to the DRL core. load,
While a learner that learns the weight values of the DNN using multiple data pairs including {current state, median value, action, reward, next state} is running, the plurality of DRLs A deep reinforcement learning (DRL) accelerator in which experience data shared by the core is stored in the upper shared memory. According to claim 11,
The plurality of DRL cores,
a core controller that controls the DRL core;
a compressor for encoding or decoding a plurality of data pairs including the {current state, intermediate value, action, reward, next state}; and
A DRL accelerator including, respectively, a PE array (Processing Element Array) that receives a current state and outputs an action and a next state through DNN inference. According to claim 12,
The plurality of DRL cores,
A broadcast memory (BMEM) and a unicast memory (UMEM) connected to the compressor and receiving weights (Weight, W) and input features (Input Feature, IF); further comprising,
Wherein the broadcast memory and the unicast memory provide input data to the PE array through a B-Buffer and a U-Buffer, respectively. According to claim 13,
The core controller,
automatically fetching the weights and the input features to the broadcast memory or the unicast memory according to the DNN network structure;
A DRL accelerator for setting the configuration of the PE array to change data reuse of weights and input features, respectively, when an actor or runner operates. According to claim 12,
The plurality of DRL cores,
an activation unit that processes a nonlinear function; and
1-D SIMD unit performing log functions, addition and multiplication for weight update and loss calculation; DRL accelerator device further comprising. According to claim 12,
While the PE array processes DNN operations, the compressor scans the output buffer;
When the output buffer is full, the compressor sequentially encodes a code word, an exponent, and a mantissa;
When an input feature is transmitted to the DRL core, the compressor scans the code word and recombines the sequentially input input mantissa and exponent to stream-out the input feature. According to claim 12,
The PE array,
During the runner operation, a loss function is generated by reading data pairs including a number of stored {current state, median value, action, reward, next state} instead of external input, and based on the generated loss function A DRL accelerator that updates the weights of the DNN by performing a back-propagation operation. According to claim 12,
The PE array,
MAC consists of a two-dimensional array,
Each row shares a row buffer that receives broadcast data and multiplies an input value through MAC, and new data is broadcast every cycle to perform parallel matrix operation,
If all unicast data provided from the U-buffer is reused, new unicast data is input through the U-buffer. According to claim 12,
The PE array,
Conversion PE (transposable processing PE) that changes data reuse of input features (IF) and weights (W), respectively, when the input direction in the actor and the input direction in the runner are configured to be different from each other, when the actor or runner operates. Element, tPE) structure, DRL accelerator. According to claim 19,
The PE array,
When the row length of the weight matrix (W Matrix) representing the number of output channels (Co) is relatively larger than the column length of the input feature matrix (IF Matrix) representing the number of batches (Batch Size, BA) W broadcast (WBC),
A DRL accelerator that selects IF broadcast (IFBC) when the row length of the weight matrix is relatively smaller than the column length of the input feature matrix. The method of any one of claims 5, 6, 7, and 10,
The actor circuit block,
During a learner operation, the plurality of PE arrays generate a loss function by reading data pairs including a plurality of {current state, median value, action, reward, next state} stored instead of external input, A DRL accelerator that updates the weights of the DNN by performing a back-propagation operation based on the generated loss function. The method of any one of claims 5, 6, 7, and 10,
A compressor for encoding or decoding a plurality of data pairs including the {current state, intermediate value, action, compensation, next state}; further comprising a DRL accelerator.

Images