JP5229731B2 - Cache mechanism based on update frequency - Google Patents

Description

  The present invention relates to a cache mechanism based on update frequency. In particular, the present invention relates to a cache mechanism that can be used in a programming model that is an extension of the Map-Reduce programming model that can describe a highly parallel program.

  In recent years, a Map-Reduce programming model, which is a distributed programming model for processing data (files) arranged in a plurality of servers, has been used (Non-Patent Document 1). The Map-Reduce programming model is composed of a Map function that generates key / value pairs from arbitrary data and a Reduce function that integrates values having the same key from the intermediate data. Generally, hundreds to thousands of personal computers are used. -It is executed on a computer (Personal computer: PC) cluster. An example of the implementation of this Map-Reduce programming model is open source software called Hadoop (Non-patent Document 2).

  The Map-Reduce process according to the Map-Reduce programming model is performed on a large number of PCs. Among them, there are two servers, a “master” and a “worker”. The master manages the operation of the entire Map-Reduce process and allocates work to workers. The worker executes either the Map function or the Reduce function according to the request of the master. However, the worker can perform any process as needed, not one of the Map function and the Reduce function.

  Hereinafter, the execution of the Map function by the worker is referred to as Map processing, and the worker performing the Map processing is referred to as a Map worker (denoted as MapWorker in the drawing). Similarly, execution of a Reduce function on data by a worker is referred to as Reduce processing, and a worker performing the Reduce processing is referred to as a Reduce worker (denoted as ReduceWorker in the drawing).

  Specifically, in the Map process, a Map worker generates a map-type result (Map result) from local data using a Map function. In the Reduce process, a Reduce worker uses a Reduce function to generate one value (Reduce result) from all values having the same key value in all Map results. The outline of the Map-Reduce process will be described by taking a word count of a plurality of files as an example. FIG. 14 is a diagram illustrating an outline of Map-Reduce processing according to the related art.

  First, the Map worker generates the number of appearances of all the words included in the file each of which is in charge (Map processing). The Map worker 1 generates, as a Map result 10, the number of appearances of all words included in the two files file1 (cat, fox, dog, cat) and file2 (fox, fox, fox, rat). Similarly, Map worker 2 generates two files, file 3 and file 5, Map worker 3 generates one file of file 4, and the number of appearances of all words included in each file is generated as Map results 20 and 30, respectively. That is, as shown in FIG. 14, the Map result is generated from the Map worker as Map type data in which the word is “key” and the number of occurrences of the word is “value”.

  Next, the Reduce worker calculates and generates the number of appearances for each key of the Map result (Reduce process). In FIG. 14, the Reduce worker 1 performs the Reduce process for the key “cat” and the key “dog”, and the Reduce worker 2 performs the Reduce process for the key “fox” and the key “rat”.

  The Reduce worker 1 collects the Map result for the key “cat” from the Map results 10, 20, and 30, and generates the sum of the values included in the Map result as the Receive result 11. The Reduce worker 1 performs a Reduce process similar to that for the key “cat” with respect to the key “dog”, and generates a Receive result 11. Also, the Reduce worker 2 performs a Reduce process on each of the key “fox” and the key “rat”, and generates a Receive result 21. Note that the Map results 10 ′, 20 ′, and 30 ′ are the results of performing a work called “Shuffle” that is performed between the Map process and the Reduce process and aggregates the values of the same keys in the Map result. . In this way, by using the Map-Reduce programming model, it is possible to process a large number of files such as word aggregation in parallel. As a result, the entire process can be realized with low latency.

Jeffrey Dean and Sanjay Ghemawat, “MapReduce: Simplified Data Processing on Large Clusters” [online], June 3, 2008, [Search August 26, 2008], Internet, <URL: http: // labs. google.com/papers/mapreduce-osdi04.pdf> Hadoop “Welcome to Hadoop”, [online], [October 7, 2008 search], Internet, <URL: http://hadoop.apache.org/core/>

  In order to improve the efficiency of the Map-Reduce process, it is conceivable to extend the Map-Reduce programming model by adding a function for the Reduce worker to save the Reduce result in its own cache. In other words, the Reduce result can be stored in the cache of the server device that is the Reduce worker that generated the Reduce result. Thereby, when the Map-Reduce process is periodically executed on the same file, the Reduce result stored in the cache can be used.

  However, the Reduce result stored in the cache is consistent with the Reduce result obtained from the updated Map result when the Map-Reduce processing target file is updated and the Map result generated by the Map worker is updated. Since it is not available, it must be disabled. Therefore, there is a problem that even if the Reduce result is stored in the cache, it cannot be used effectively.

  Therefore, in view of the above-described problems, the present invention extends Map-Reduce programming to programming that can effectively use a cache, and builds a cache mechanism in processing of the extended Map-Reduce programming model based on the update frequency, and The purpose is to provide a system.

  In one aspect of the present invention, the following solution is provided.

  According to one aspect of the present invention, in a Map-Reduce processing system that executes Map processing and Reduce processing and performs distributed processing of a plurality of data, a method for constructing a cache mechanism for the Reduce processing, Method for constructing a cache mechanism in a Map-Reduce-ReduceMerge processing system, in which a ReduceMerge process that adds a ReduceMerge process that partially executes the Reduce process to a Map process result and processes the partially processed result in stages I will provide a. A method of constructing a cache mechanism includes a step of dividing a plurality of data to be processed into a plurality of groups based on the update frequency of each of the data, and a data update that is an update frequency of data constituting each of the plurality of groups A step of calculating a group update frequency, which is an update frequency of each of the plurality of groups, based on the frequency, and for each of the groups having a group update frequency equal to or lower than a preset group update frequency threshold among the plurality of groups Executing a Map process to generate a plurality of Map results; partially executing a Reduce process on the plurality of Map results to generate a plurality of Partial Reduce results; and a plurality of Partial Reduces Execute ReduceMerge processing on the results step by step to create a new partial R And generating a duce results, and the step of caching the generated partial Reduce results.

  According to this aspect, by adding the ReduceMerge process to the Map-Reduce process, the Reduce process is not performed at a time, but a partial Reduce result is generated and the Reduce process is performed step by step. In the Map-Reduce-ReduceMerge process expanded by adding the ReduceMerge process to the Map-Reduce process, the Reduce process for the group whose group update frequency is equal to or less than the threshold and the partial Reduce result in the ReduceMerge process stage are stored in the cache. That is, by storing the Reduce result for the data group that is not updated so much and the partial Reduce result of the ReduceMerge processing stage in the cache, it can be reused in the next Map-Reduce-ReduceMerge process.

  Here, the partial reduction result of the Reduce process and the Reduce-Merge process stage is a Reduce result for a plurality of group parts to be processed, and is generated in stages in the Map-Reduce process. For example, in the Map-Reduce-ReduceMerge process for groups A, B, C, and D, the Reduce results of groups A, B, C, and D, which are parts of groups A, B, C, and D, are partially reduced. It is a result. In addition, the Reduce results of groups C and D generated from the Reduce results (partial Reduce results) of the groups C and D are also partial Reduce results.

  Also, in this aspect, the Map-Reduce-ReduceMerge process combines groups having a group update frequency equal to or less than a threshold among a plurality of groups based on the group update frequency, and generates a partial Reduce result for each group. Based on the step and the group update frequency of the group corresponding to the partial reduce result, the step of calculating the partial update frequency that is the update frequency of the partial reduce result and the partial reduce result cannot be created. Up to generating partial new Reduce results by combining the partial Reduce results according to the partial update frequency.

  According to this aspect, it is possible to generate partial Reduce results in stages based on the partial update frequency, and by storing them in the cache, the cache is valid and can be reused. Can be built. By generating partial Reduce results stepwise based on the partial update frequency, when data is updated, a partial Reduce result cache that is not affected by the data update can be used as it is.

  Here, the partial update frequency is the update frequency of the partial Reduce result, that is, the update frequency of the Reduce result for a plurality of group parts to be processed. In addition, for example, in the Map-Reduce-ReduceMerge process of the groups A, B, and C, the partial Reduce result of the group A, the partial Reduce result of the group B, and the group C Generation of partial reduction results until the final reduction results of groups A, B, and C are obtained, such as partial reduction results of, and partial reduction results of groups A and B. . As a result, even when the group C is updated, the caches of the partial reduction result of the group A not including the group C, the partial reduction result of the group B, and the partial reduction result of the groups A and B remain unchanged. Can be used.

  The present invention can be combined with existing technologies such as the distributed file system GFS (Google (registered trademark) File System) and the large-scale distributed database BigTable, and such a combination is also within the technical scope of the present invention. included. The technique of the present invention also provides the steps of constructing a cache mechanism, including FPGA (field programmable gate array), ASIC (application specific integrated circuit), equivalent hardware logic elements, and programmable. The integrated circuit, or a combination thereof, may be provided in the form of a program that can be stored, that is, as a program product. Specifically, means for implementing a cache mechanism according to the present invention as a form of a custom LSI (Large Scale Integrated Circuit) having a data input / output, data bus, memory bus, system bus, etc., device, and embedded apparatus The form of the program product stored in the integrated circuit is also included in the technical scope of the present invention.

  According to the present invention, in the Map-Reduce-ReduceMerge process, which is an extension of the Map-Reduce programming so that the cache can be effectively used, by effectively using the cache, the Map-Reduce programming is compared with the conventional Map-Reduce process. -Reduce-ReduceMerge processing can be realized with low latency, that is, the CPU cost and communication cost can be reduced, and the processing efficiency can be improved. The cache mechanism applied to Map-Reduce processing based on the update frequency Construction methods and systems can be provided.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. These are merely examples, and the technical scope of the present invention is not limited to these.

  First, a case where a cache is used in conventional Map-Reduce processing will be described. A Reduce result caching method in the above-described Map-Rdeuce process of FIG. 14 will be described with reference to FIG.

  FIG. 15 is a diagram illustrating a cache example of the Reduce result in the Map-Reduce process according to the related art. The Reduce worker 1 performs a Reduce process on the key “dog” and the key “cat”, generates a Reduce result 11, and saves the Reduce result 11 in the cache 1 of the Reduce worker 1 (denoted as Cache in the drawing) 1. . Similarly, the Reduce worker 2 performs the Reduce processing of the key “fox” and the key “rat”, generates the Reduce result 21, and stores the Reduce result 21 in the cache 2 of the Reduce worker 2.

  In this way, the Reduce results for the second and subsequent keys “dog”, “cat”, “fox”, and “rat” for the same file can be acquired from the caches 1 and 2. That is, the second and subsequent Map processing and Reduce processing can be omitted, and the CPU cost and communication cost can be reduced.

  However, the Reduce result stored in the cache is consistent with the Reduce result obtained from the updated Map result when the Map-Reduce processing target file is updated and the Map result generated by the Map worker is updated. Since it is not available, it must be disabled. Therefore, there is a problem that even if the Reduce result is stored in the cache, it cannot be used effectively. In the Map-Reduce process in which the Reduce result shown in FIG. 15 is stored in the cache, a case where the Map result is updated will be described with reference to FIG.

  FIG. 16 is a diagram illustrating file update when a Reduce result stored in a cache is used according to the related art. Changes from FIG. 15 to FIG. 16 are shown in bold. When the fox of file1 shown in FIG. 16 is updated to dog, the map result 10 of the map worker 1 in charge of file1 is updated, and the reduced results 11, 21 of the updated words “fox” and “dog” are updated. The

  That is, since the updated words “fox” and “dog” have a different Reduce result from the values stored in the caches of the Reduce workers 1 and 2, the Reduce workers 1 and 2 again perform the Reduce process, and the Reduce process is performed. Values stored in the caches of workers 1 and 2 cannot be used. As described above, when the Map result, that is, when the file is updated, the Reduce result stored in the cache cannot be used and is invalidated.

  FIG. 17 is a diagram illustrating an actual Map-Reduce process according to the related art. 14 to 17, the outline of the Map-Reduce process has been described assuming that one Map worker processes one or two files. However, in the actual Map-Reduce process, as shown in FIG. Map workers process a large number of files. For this reason, the file is frequently updated, and the Reduce worker cache is frequently invalidated. As a result, in the Map-Reduce process, even if the Reduce result is cached, it cannot be used effectively.

  Next, an embodiment of the present invention that solves the problem that the cache cannot be effectively used in the conventional Map-Reduce process will be described. In one embodiment of the present invention, a cache mechanism that can effectively use a cache by using a Map-Reduce-ReduceMerge process, which is obtained by adding a ReduceMerge process to a Map-Reduce process and expanding the Map-Reduce process, is provided. To construct. The Map-Reduce-ReduceMerge process can realize substantially the same function as the Map-Reduce process.

  The Map-Reduce-ReduceMerge process includes a Map process, a Reduce process, and a ReduceMerge process. As described in the background art, the Map process applies a Map function to a plurality of files to generate a Map result, and the Reduce process applies a Reduce function to the Map result to reduce a result (partial Reduce result). Is generated.

  The ReduceMerge process generates a new Reduce result (partial Reduce result) by applying the ReduceMerge function to a plurality of Reduce results for the same key. The ReduceMerge process ends when all Map results are reflected in the Reduce result, that is, when Reduce results for all files to be processed are generated. The ReduceMerge process is performed by a worker, and the worker performing the ReduceMerge process is referred to as a ReduceMerge worker.

  The ReduceMerge function is a function that merges a plurality of Reduce results for the same key to generate one new Reduce result. The input of the ReduceMerge function is a Reduce result generated by the Reduce process or the ReduceMerge process.

  That is, by ReduceMerge processing, a plurality of Reduce results for the same key generated by Reduce processing or ReduceMerge processing are merged in stages, and finally one Reduce result for all files to be processed is generated for each key. The The ReduceMerge process will be specifically described with reference to FIG.

  FIG. 1 is a diagram illustrating an example of a Map-Reduce-ReduceMerge process according to an embodiment of the present invention. In this example, it is assumed that the Map worker does not process the file according to the update frequency. In the case of the Map-Reduce-ReduceMerge process, as in the conventional Map-Reduce process, first, Reduce results 200, 201, and 202 are obtained from a plurality of input files (ID = 1 to 30000) by Map workers and Reduce workers. Is generated.

  A ReduceMerge worker (denoted as ReduceMergeWorker in the drawing) 1 applies a ReduceMerge function to the Reduce results 200 and 201 to generate a new Reduce result 210. Further, the ReduceMerge worker 2 applies the ReduceMerge function to the Reduce results 202 and 210 to generate a new Reduce result 220.

  As described above, a ReduceMerge worker can generate a partial Reduce result for a file to be processed by merging the Reduce results in stages. By storing the partial Reduce result in the cache, a cache mechanism that can effectively use the cache can be constructed. That is, the Reduce process, which has been performed once in the conventional Map-Reduce process, is generated in the Map-Reduce-ReduceMerge process according to an embodiment of the present invention, and a partial Reduce result is generated in stages. By performing the processing, it is possible to construct a cache mechanism that can effectively use the cache, which was not possible with the conventional Reduce processing.

  A method for constructing a cache mechanism in Map-Reduce-ReduceMerge processing according to an embodiment of the present invention broadly includes three methods, and a method for generating a cache during the first Map-Reduce-ReduceMerge processing (Cache generation method), a method of invalidating the cache when the data is updated (cache invalidation method) in Map-Reduce-ReduceMerge processing in which the cache has already been generated, and Map- There is a method of performing Reduce-ReduceMerge processing (cache reuse method). Hereinafter, each method will be described.

  In one embodiment of the present invention, the cache is a storage means for storing data and programs in the server, a semiconductor medium such as RAM and ROM, a hard disk, and a magnetic medium / electric medium assigned by the OS by a device driver. Etc.

  FIG. 2 is a flowchart illustrating a cache generation method according to an embodiment of the present invention. By using the cache generated by the cache generation method shown in FIG. 2, it is possible to reduce part of the second and subsequent Map-Reduce-ReduceMerge processes.

  S10: The Map worker groups the file (data) in charge for each file update frequency (data update frequency) of the file. The process of step S10 will be specifically described by taking as an example a case where files with ID = 1 to ID = 8 are grouped for each file update frequency.

  FIG. 3 is a diagram showing an example of file grouping according to an embodiment of the present invention. In the example of FIG. 3, files of ID = 1 to ID = 8 handled by a Map worker are divided into two groups (High, Low), a group A with a high file update frequency and a group B with a low file update frequency It is a case where it classifies. The file update frequency of each file is represented by the number of updates in a fixed time. For example, when the file is updated once every 10 seconds, it is set to 1/10.

  In FIG. 3, the file contents, ID, and file update frequency are collectively shown for each file. Here, assuming that the file update frequency threshold of group A and group B is 1/150, files with file update frequency of 1/150 or more, ID = 1, 2, 5, 8 are divided into group A. On the other hand, files whose file update frequency is 1/150 or less and ID = 3,4,6,7 are divided into group B. Note that the threshold value of the file update frequency is arbitrary, and testing may be performed while changing the file update frequency to find an optimum value that can effectively use the cache.

Returning to FIG.
S20: The Map worker calculates the group update frequency for each group divided in step S10.

  For example, when calculating the group update frequency of the group A in FIG. 3 described above, the file update frequencies of ID = 1, 2, 5, and 8 constituting the group A are added. That is, 1/10 (ID = 1) +3/20 (ID = 2) +1/5 (ID = 5) +1/40 (ID = 8) = 19/40 is the group update frequency of group A.

  S30: The Map worker applies the Map function for each group, generates a Map result for each group, and sets the group update frequency of the corresponding group as the update frequency of the Map result. That is, the group update frequency for each group calculated in step S20 is the update frequency of the corresponding Map result. The process of step S30 will be specifically described with reference to FIG.

  FIG. 4 is a diagram illustrating a processing example of a Map worker according to an embodiment of the present invention. The example of FIG. 4 is a case where files with ID = 1 to ID = 30000 are divided into 6 groups A to F, and the Map function is applied to each group. In FIG. 4 and subsequent figures, a description for obtaining the Reduce result of the key “dog” is used to omit the description.

  Group A is a group of files whose update frequency is higher than a predetermined threshold among files with ID = 1 to ID = 10000, while group B is a group of files whose update frequency is lower than a predetermined threshold. . Similarly, the groups C and D are divided into files with ID = 10001 to ID = 20000, and the groups E and F are divided into files with ID = 20000 to ID = 30000 depending on whether the update frequency is higher or lower than a predetermined threshold. Group.

  The Map worker 1 applies a Map function to the group A and generates a Map result 100. When the group update frequency of group A is 1/8, the Map result 100 takes over the group update frequency of group A and becomes 1/8. Similarly, the Map worker 1 applies the Map function to the group B and generates a Map result 101. The Map result 101 takes over the group update frequency of the group B and becomes 1/1000. Similarly, for the groups C to F, a Map result is generated for each group, and the update frequency of the Map result is determined together.

  As described above, the update frequency of the Map result generated by the Map worker is determined based on the group update frequency of the corresponding group.

Returning to FIG.
S40: The Reduce worker applies a Reduce function to one or more Map results to generate a Reduce result. However, all Map results, that is, Reduce results from all groups, are generated by a plurality of Reduce workers.

  S50: The Reduce worker saves the Reduce result generated in Step S40 in the cache. It should be noted that the Reduce result generated by each Reduce worker is stored in the cache of each Reduce worker. Also, the Reduce worker stores the name of the group, which is the original data of the Reduce result, in the cache as the original data information. Furthermore, the Reduce worker can obtain the update frequency of the generated Reduce result as the sum of the group update frequencies of the group that is the original data, and stores the update frequency of the Reduce result in the cache. The process of step S40 will be specifically described with reference to FIG.

  FIG. 5 is a diagram illustrating an example of processing by a Reduce worker according to an embodiment of the present invention. The example of FIG. 5 is a case where Reduce results obtained from six groups A to F are stored in the cache of each Reduce worker.

  The Reduce worker 1 stores the Reduce result 110 obtained by using the Map result 100 in the cache 1. Also, the Reduce worker 1 stores the A group, which is the original data of the Reduce result 110, in the cache 1 as the original data information. This is because the Map result 100 that is input data of the Reduce result 110 is a value obtained from the A group. Also, the Reduce worker 1 stores the group update frequency 1/8 of the A group that is the original data of the Reduce result 110 in the cache 1 as the update frequency of the Reduce result 110. This is because the update frequency of the Reduce result 110 is the update frequency of the Map result 100 that is input data, and the update frequency of the Map result 101 is the group update frequency of the A group.

  The Reduce worker 2 stores the Reduce result 111 obtained using the Map results 101, 103, and 105 in the cache 2. The Reduce worker 2 stores the original data of the Reduce result 111 in the cache 2 as original data information indicating that the original data is the B, D, and F groups. This is because the Map results 101, 103, and 105, which are input data of the Reduce result 111, are values obtained from the B, D, and F groups, respectively.

  Also, the Reduce worker 2 stores the sum of the update frequencies of the B, D, and F groups, which are the original data of the Reduce result 111, 1/1000 + 1/2000 + 1/4000 = 7/4000 in the cache 2 as the update frequency of the Reduce result 111. To do. This is because the update frequency of the Reduce result 111 is the sum of the update frequencies of the Map results 101, 103, and 105 that are input data, and the Map results 101, 103, and 105 are update frequencies of the B, D, and F groups, respectively.

  Note that, when the Reduce worker applies the Reduce function to a plurality of Map results like the Reduce worker 2 described above, the Map results are selected so that the update frequency does not increase. This is to effectively use the cache of the Reduce worker. A method for selecting the Map result so that the update frequency does not increase will be described later with reference to FIGS.

Returning to FIG.
S60: The ReduceMerge worker applies a ReduceMerge function to the Reduce results generated by the plurality of Reduce workers, and generates a new Reduce result.

  S70: The ReduceMerge worker saves the Reduce result generated in Step S60 in the cache. Also, the ReduceMerge worker stores the name of the group, which is the original data of the Reduce result, in the cache as the original data information. Further, the ReduceMerge worker can obtain the update frequency of the generated Reduce result as the sum of the group update frequencies of the group that is the original data, and stores the update frequency of the Reduce result in the cache.

  S80: The master determines whether the Reduce results of all groups have been merged. If YES, the process ends. On the other hand, if NO, the process returns to step S60. Note that the ReduceMerge worker may determine whether the Reduce results of all the groups in Step S80 have been merged. The process from step S60 to S80 will be specifically described with reference to FIG.

  FIG. 6 is a diagram illustrating an example of processing by a ReduceMerge worker according to an embodiment of the present invention. It is a continuation of the processing example of the Reduce worker shown in FIG. The ReduceMerge worker applies the ReduceMerge function to a plurality of Reduce results, and finally generates a Reduce result including all the groups. The ReduceMerge worker 1 applies the ReduceMerge function to the Reduce result 110 generated by the Reduce worker 1 and the Reduce result 111 generated by the Reduce worker 2 to generate the Reduce result 120. The ReduceMerge worker 1 stores the Reduce result 120 in the cache 11.

  Further, the ReduceMerge worker 1 stores in the cache 11 that the original data of the Reduce result 120 is the A, B, D, and F groups. Since the Reduce result 120 is generated from the Reduce result 110 having the A group as the original data and the Reduce result 111 having the B, D, and F groups as the original data, the A, B, D, and F groups have the Reduce result 120. This is because the original data.

  Furthermore, the update frequency of the Reduce result 120 is stored in the cache 11. The update frequency of the Reduce result 120 is the sum of the update frequency 1/8 of the Reduce result 110 and the update frequency 7/4000 of the Reduce result 111, which is 1/8 + 7/4000 = 507/4000. The update frequency of the Reduce result 120 is the sum of the update frequencies of the A, B, D, and F groups that are the original data.

  Further, the ReduceMerge worker 2 applies a ReduceMerge function to the Reduce result 112 generated by the Reduce worker 3 and the Reduce result 120 generated by the ReduceMerge worker 1 to generate a Reduce result 130. Also, the ReduceMerge worker 2 stores the Reduce result 130 in the cache 21.

  In this way, the ReduceMerge worker can apply the Reduce function to the Reduce results generated by other ReduceMerge workers until a Reduce result including all groups is finally generated. Processing of the ReduceMerge worker continues. That is, a plurality of partial Reduce results of a plurality of groups are generated and finally combined to generate one Reduce result. In this way, the ReduceMerge worker generates Reduce results in stages and stores them in the cache.

  By storing each Reduce result generated in stages in the cache, a staged cache mechanism can be constructed. As a result, the cache can be effectively used in the Map-Reduce-ReduceMerge process. In other words, by adding the ReduceMerge process to the Map-Reduce process, it is possible to extend the Map-Reduce process that is easy to cache.

  Each means in the system for constructing the cache mechanism will be described using the cache generation processing flow shown in FIG. When the classifying unit receives the file to be processed, it performs the process of step S10 in FIG. The frequency calculation means receives the update frequency of each file, and performs the process of step S20 in FIG. The result generation means receives the group classified by the classification means, and performs the processing from step S30 to S40 in FIG. 2 for each group. The cache unit receives the Reduce result created by the result generation unit, and performs the process of step S50 in FIG.

  Further, the result generation means can receive the Reduce result created by the result generation means, and can generate a new Reduce result by the process of step S60 of FIG. Further, the cache unit receives the Reduce result newly created by the result generation unit, and performs the process of step S70 in FIG.

  Next, a cache invalidation method for partially invalidating the cache mechanism constructed by the above-described cache generation processing when a file is updated will be described. FIG. 7 is a flowchart showing cache invalidation processing according to an embodiment of the present invention. Note that the cache invalidation process may be performed by a map worker or a master. Hereinafter, a case where the master performs cache invalidation processing will be described.

  S100: The master detects a file update. Specifically, the master makes a program called a robot or a crawler resident and detects the update of the file.

  S110: The master specifies a group including the updated file detected in step S100.

  S120: The master invalidates all Reduce result caches generated using the group specified in step S110 as one of the original data. Specifically, the master searches the cache including the group specified in step S110 in the original data information, and invalidates the cache when detecting the cache including the group specified in step S110 in the original data information.

  FIG. 8 is a diagram illustrating an example of cache invalidation according to an embodiment of the present invention. The above-described cache invalidation method will be specifically described with reference to FIG. FIG. 8 is a diagram obtained by merging FIG. 5 and FIG.

  Assume that a file in group C has been updated. Update locations are shown in bold. First, the master detects that a file has been updated, and identifies a group that includes the updated file, in this case, group C. The master invalidates the cache 3 and the cache 21 including the group C in the original data information based on the fact that the group whose file is updated is the group C.

  Also, when the update frequency of the individual file becomes larger than the update frequency threshold of the group (High or Low) to which the file belongs, group reconfiguration and cache invalidation are performed according to the following procedure.

  FIG. 9 is a diagram illustrating an example of cache invalidation by group reconfiguration according to an embodiment of the present invention. The case where the data in B among groups A and B in FIG. 9 is frequently updated will be described.

  First, the master specifies a file that is frequently updated. Whether or not the file is frequently updated is determined based on whether or not the file update frequency exceeds a threshold when the files are grouped. The master acquires information on the group of the identified file and the destination group. In this example, since the file specified by the master is included in group B, the source group is group B. Since the group with a high file update frequency corresponding to the group B is the group A, the destination group is the A group.

  The master invalidates the cache based on the acquired information. In this example, the master invalidates the caches 1, 2, 11, and 21 including the groups A and B in the original data information. However, at this time, the cache 3 derived from a group other than A and B is held as it is.

  Next, the master moves the identified data from group B, which is the movement source group, to group A, which is the movement destination group.

  The same procedure can be used when transferring a file with a low update frequency from A to B. Since files are only moved within the same server, there is no change in the splitting status between servers. Also, in this example, it appears that many caches have been invalidated, but in reality there are more caches and invalidation is local.

  Next, a cache reuse method using the cache mechanism constructed by the cache generation method and the cache invalidation method described above will be described.

  FIG. 10 is a flowchart showing Map-Reduce-ReduceMerge processing using a cache mechanism according to an embodiment of the present invention.

  S200: The master searches all valid caches and selects a cache so that all groups of Reduce results are included. However, the group should not be fogged.

  S210: The master requests the Map worker in charge of the group not included in the selected one or more caches to perform the Map process.

  S220: The Reduce worker generates a Reduce result based on the Map result created by the Map worker requested to perform the Map process from the master in Step S210.

  S230: ReduceMerge processing is performed on the Reduce result of the cache selected by the master in Step S200 and the Reduce result generated by the Reduce worker in Step S220, and a final Reduce result is generated.

  FIG. 11 is a diagram showing an example of Map-Reduce-ReduceMerge processing using a cache mechanism according to an embodiment of the present invention. For example, when the cache is as shown in FIG. 11, the cache 11 of the ReduceMerge worker 1 includes the cache 1 of the Reduce worker 1 and the cache 2 of the Reduce worker 2. In this case, the caches of the ReduceMerge worker 1 or the caches 1 and 2 of the Reduce worker 1 and the Reduce worker 2 are selected as caches of the Reduce results of the groups A, B, D, and F, respectively. In FIG. 11, the cache 11 of the ReduceMerge worker 1 is selected.

  Next, the Map worker applies a Map function to a group not included in the selected cache, and generates a Map result. Since the Reduce results that do not include the groups C and E are stored in the cache 11 of the Reduce Merge worker 1, the Map workers 2 and 3 apply the Map function, and the Map results 12 and 12 of the groups C and E respectively. 14 is generated.

  Next, the Reduce worker generates a Reduce result from the Map result generated by the Map worker. In FIG. 11, the Reduce worker 3 generates a Reduce result 112 from the Map results 12 and 14.

  Finally, a ReduceMerge process is performed on the Reduce result of the selected cache and the Reduce result generated by the Reduce worker to generate a final Reduce result. In FIG. 11, the ReduceMerge worker 2 generates a Reduce result 130 from the Reduce result 120 stored in the cache 11 of the ReduceMerge worker 1 and the Reduce result 112 generated by the Reduce worker 3.

  In FIG. 11, a portion displayed by a dotted line indicates processing omitted due to the use of the cache. As can be seen from FIG. 11, many processes can be omitted by using a cache. As a result, the load on the entire system can be reduced, and the entire processing can be realized with low latency compared to the conventional Map-Reduce processing.

(Modification 1)
In the above embodiment, a map worker or a reduce worker may be any server, and even if a map result created by a map worker of a server is processed by a reduce worker of the same server, a reduce worker of another server It may be processed. However, it is more efficient for the Reduce worker of the same server to process the Map result created by a certain Map worker. This is because the Map result is not passed. Therefore, by optimizing the physical arrangement of the Map worker and the Reduce worker, more efficient Map-Reduce-ReduceMerge processing can be realized.

(Modification 2)
In the above embodiment, all Reduce results are stored in the cache regardless of the update frequency. However, since the number of times the Reduce result cache having a high update frequency can be used is smaller than that of the Reduce result cache having a low update frequency, the Reduce result having the high update frequency may be excluded from the cache target. As a result, processing for caching Reduce results with high update frequency and processing for invalidating the cache can be omitted, and more efficient Map-Reduce-ReduceMerge processing can be realized.

(Modification 3)
In the above embodiment, the Reduce result is stored in the cache. However, the Map result may be stored in the cache in the same manner as the Reduce result. The Map result is stored in the cache of the Map worker that generated it. Further, at this time, the Map result having a high update frequency may be excluded. As a result, even when the Reduce result cache cannot be used, the Map result cache can be used, so that more efficient Map-Reduce-ReduceMerge processing can be realized.

(Modification 4)
In the above embodiment, the uncached group is identified by checking all the cache groups. However, by placing an index for storing the cache position on one server, the uncached group is identified. It can be realized with low latency. As a result, more efficient Map-Reduce-ReduceMerge processing can be realized.

(Modification 5)
In the above embodiment, the grouping based on the file update frequency is divided into an arbitrary number other than the two groups of the file update frequency group (High) and the file update frequency group (Low). It may be possible. By finely grouping files for each file update frequency, it is possible to increase the number of effective caches, and to realize more efficient Map-Reduce-ReduceMerge processing.

(Modification 6)
In the above embodiment, the combination for reducing (ReduceMerge) is arbitrary, but in order to increase the number of valid caches, processing for optimizing the combination for reducing (ReduceMerge) may be added.

FIG. 12 is a flowchart of a process for optimizing a combination for Reduce (ReduceMerge) according to an embodiment of the present invention.
S300: The master acquires a threshold α of the update frequency of the Reduce result generated by the Reduce (ReduceMerge) process. The update frequency threshold may be arbitrary and may be determined by a human. However, the threshold should not be too close to 1. This is because when the threshold value is close to 1, the Reduce result is a result for many groups, so the probability that the Reduce result cache is invalidated increases and the cache cannot be used effectively.

  S310: The master selects one Reduce worker (ReduceMerge worker). At this time, a Reduce worker (ReduceMerge worker) in which a file to be processed has already been determined is excluded. Also, the master empties the group set G. The group set G is used to temporarily store the combination when creating a combination for Reduce (ReduceMerge).

  S320: The master selects one of the unselected groups with the lowest group update frequency (hereinafter referred to as group g).

  S330: When the master adds the group g selected in step S320 to the group set G and performs the Reduce process with the group set G as an input, the master determines whether the update frequency of the Reduce result exceeds α. If the determination result is YES, the process proceeds to step S350, and if the determination result is NO, the process proceeds to step S340.

  S340: If the master determines YES in step S330, it adds group g to group set G and returns to step S320. In this way, the group g is added to the group set G until the update frequency of the Reduce result obtained from the group set G exceeds the threshold value α.

  S350: When the master determines NO in step S330, the master determines whether the group set G is empty. If the determination result is YES, the process proceeds to step S360; if the determination result is NO, the process proceeds to step S370.

S360: The master adds group g to group set G. This is because, in one group, when the update frequency of the Reduce result obtained therefrom exceeds the threshold value α, the Reduce process is performed alone without being combined with other groups.
S360: The master uses the Reduce (ReduceMerge) worker selected in step S310 as a target of Reduce (ReduceMerge) for all groups in the group set G.

  S380: The master determines whether all groups have been selected, that is, whether all groups have been processed by a Reduce (ReduceMerge) worker. If the determination result is YES, the process ends. If the determination result is NO, the process returns to step S310.

  In this way, groups with low group update frequency are grouped together, and one Reduce result is generated from them and stored in the cache. By doing so, it is possible to optimize the combination of groups that are reduction targets, and it is possible to realize more efficient Map-Reduce-ReduceMerge processing.

FIG. 13 is a flowchart of the ReduceMerge process according to an embodiment of the present invention.
S400: The master acquires threshold values γ and Δ. Here, the master sets the update frequency α = γ of the Reduce result generated by the ReduceMerge process. Note that γ and Δ may be arbitrary and may be determined by a human.
S410: The master inputs the update frequency α and performs a process of optimizing the combination to be reduced shown in FIG.
S420: Each Reduce worker performs a Reduce process with the group that is the processing target determined in Step S410 as an input.

  S430: The master determines whether there is one Reduce result generated in step S420. If the determination result is NO, the process proceeds to step S440. On the other hand, if YES, the process ends. The end condition of the ReduceMerge process is that Reduce results corresponding to all groups have been generated. The fact that there is only one Reduce result generated in step S420 is that the result is a Reduce result corresponding to all groups and satisfies the termination condition of the ReduceMerge process.

S440: The master sets the update frequency α = α + Δ. By increasing the update frequency little by little, the number of Reduce results generated can be reduced in stages. That is, ReduceMerge processing can be performed step by step.
S450: The master inputs the update frequency α, and performs the process of optimizing the combination for ReduceMerge shown in FIG.
S460: The ReduceMerge worker that has made two or more groups subject to ReduceMerge processing in Step S450 performs ReduceMerge processing, and returns the processing to Step S430.

  In this way, by performing ReduceMerge processing in several stages according to the update frequency of the Reduce result, it is possible to construct a cache mechanism that can effectively use the cached Reduce result even if the update frequently occurs. it can. As a result, more efficient Map-Reduce-ReduceMerge processing can be realized.

  Although the present invention has been described based on the embodiment, the present invention is not limited to the above-described embodiment. The effects described in the embodiments of the present invention are only the most preferable effects resulting from the present invention, and the effects of the present invention are limited to those described in the embodiments or examples of the present invention. Is not to be done.

It is a figure which shows the Map-Reduce-ReduceMerge process example based on one Embodiment of this invention. 4 is a flowchart illustrating a cache generation method according to an embodiment of the present invention. It is a figure which shows the example of grouping of the file based on one Embodiment of this invention. It is a figure which shows the process example of the Map worker based on one Embodiment of this invention. It is a figure which shows the example of a process by the Reduce worker based on one Embodiment of this invention. It is a figure which shows the example of a process by the ReduceMerge worker based on one Embodiment of this invention. It is a flowchart which shows the cache invalidation process based on one Embodiment of this invention. It is a figure which shows the example of cache invalidation based on one Embodiment of this invention. It is a figure which shows the example of the cache invalidation by group reconfiguration | reconstruction based on one Embodiment of this invention. It is a flowchart of Map-Reduce-ReduceMerge processing using a cache mechanism according to an embodiment of the present invention. It is a figure which shows the Map-Reduce-ReduceMerge process example using a cache mechanism based on one Embodiment of this invention. It is a flowchart of the optimization process of the group which performs Reduce (ReduceMerge) according to an embodiment of the present invention. 6 is a flowchart of ReduceMerge processing according to an embodiment of the present invention. It is a figure which shows the Map-Reduce process outline | summary based on a prior art. It is a figure which shows the cache example of a Reduce result in the Map-Reduce process based on a prior art. It is a figure which shows file update at the time of utilization of the Reduce result preserve | saved at the cache based on a prior art. It is a figure which shows the actual Map-Reduce process based on a prior art.

Claims (13)

In a Map-Reduce processing system that executes Map processing and Reduce processing and distributes a plurality of data, a cache mechanism for the Reduce processing is constructed.
In the Map-Reduce-ReduceMerge processing system, in which the Reduce process is partially executed on the result of the Map process and a ReduceMerge process is added to process the partially processed result in stages.
Classifying the plurality of data into a plurality of groups based on a data update frequency that is an update frequency of each of the plurality of data;
Calculating a group update frequency, which is an update frequency of each of the plurality of groups, based on the data update frequency of data constituting each of the plurality of groups;
A step of generating the plurality of Map results by executing the Map processing for each of the plurality of groups having a group update frequency equal to or less than a preset threshold of the group update frequency;
Performing the Reduce process partially on the plurality of Map results to generate a plurality of partial Reduce results;
Performing the ReduceMerge process in a stepwise manner on the plurality of partial reduce results to generate a new partial reduce result;
Storing the partial Reduce result in a cache;
Including methods.   The method according to claim 1, further comprising storing information specifying a group corresponding to the partial reduction result in the cache when the partial reduction result is stored in the cache. And calculating a partial update frequency that is an update frequency of the partial reduce result based on the group update frequency of the group corresponding to the partial reduce result.
The method according to claim 1, wherein when the partial reduction result is stored in a cache, the partial update frequency is stored in the cache together with the partial update frequency. The step of generating the partial reduction result further comprises:
Combining the partial reduction results based on the partial update frequency, and newly generating the partial reduction results for each;
Repeating the step of newly generating the partial reduction result until no combination of the partial reduction results can be created;
The method of claim 3 comprising: The step of generating a new said partial Reduce result is:
The sum of the partial update frequencies of the partial reduction results to be combined is less than or equal to a preset threshold value of the partial update frequency,
The repeating step includes
The method according to claim 4, wherein the step of newly generating the partial reduce result is repeated by gradually increasing a threshold value of the partial update frequency. And identifying the group containing updated data in response to at least one of the plurality of data being updated;
Invalidating the partial Reduce result cache corresponding to the identified group;
The method of claim 1 comprising: Furthermore, the data update frequency is remarkably different in response to the fact that the data update frequency of at least one of the plurality of data has changed significantly compared to when the plurality of data is divided into the plurality of groups. Changing the group to which the changed data belongs,
Invalidating the cache of the partial reduce result corresponding to the group whose data update frequency has changed significantly before the change and the group included after the change;
The method of claim 1 comprising: Creating an index for the cache location of the partial reduce result;
Storing the created index in any one of the computers constituting the Map-Reduce processing system;
The method of claim 1 comprising:   The step of generating the Map result generates the plurality of Map results by executing the Map process for each of the plurality of groups in which the group update frequency exceeds the group update frequency threshold. The method of claim 1.   The method of claim 1, further comprising storing the plurality of Map results in a cache. In a Map-Reduce processing system that executes Map processing and Reduce processing and distributes a plurality of data, a cache mechanism for the Reduce processing is constructed.
In the Map-Reduce-ReduceMerge processing system, in which the Reduce process is partially executed on the result of the Map process and a ReduceMerge process is added to process the partially processed result in stages.
Dividing the plurality of data into a plurality of groups based on a data update frequency that is an update frequency of each of the plurality of data;
Calculating a group update frequency, which is an update frequency of each of the plurality of groups, based on the data update frequency of data constituting each of the plurality of groups;
A step of generating the plurality of Map results by executing the Map processing for each of the plurality of groups having a group update frequency equal to or less than a preset threshold of the group update frequency;
Performing the Reduce process partially on the plurality of Map results to generate a plurality of partial Reduce results;
Performing the ReduceMerge process in a stepwise manner on the plurality of partial reduce results to generate a new partial reduce result;
Calculating a partial update frequency, which is an update frequency of the partial Reduce result, based on the group update frequency of the group corresponding to the partial Reduce result;
Storing the partial reduce result, information identifying a group corresponding to the partial reduce result, and an update frequency of the partial reduce result in a cache;
Including
Furthermore,
When the partial reduction result, information identifying a group corresponding to the partial reduction result, and the update frequency of the partial reduction result are stored in a cache,
Identifying the group containing updated data in response to at least one of the plurality of data being updated;
Invalidating the partial Reduce result cache corresponding to the identified group;
The data update frequency has changed significantly in response to the fact that the data update frequency of at least one of the plurality of data has changed significantly compared to when the plurality of data is divided into the plurality of groups. Changing the group to which the data belongs;
Invalidating the partial reduce result cache corresponding to the groups whose data update frequency has changed significantly before and after the group included in the change. In a Map-Reduce processing system that executes Map processing and Reduce processing and distributes a plurality of data, a program for constructing a cache mechanism for the Reduce processing,
In the Map-Reduce-ReduceMerge processing system, in which the Reduce process is partially executed on the result of the Map process and a ReduceMerge process is added to process the partially processed result in stages.
In the computer constituting the Map-Reduce-ReduceMerge processing system,
Dividing the plurality of data into a plurality of groups based on a data update frequency that is an update frequency of each of the plurality of data;
Calculating a group update frequency, which is an update frequency of each of the plurality of groups, based on the data update frequency of data constituting each of the plurality of groups;
A step of generating the plurality of Map results by executing the Map processing for each of the plurality of groups having a group update frequency equal to or less than a preset threshold of the group update frequency;
Performing the Reduce process partially on the plurality of Map results to generate a plurality of partial Reduce results;
Performing the ReduceMerge process in a stepwise manner on the plurality of partial reduce results to generate a new partial reduce result;
Storing the partial Reduce result in a cache;
A program that executes In a Map-Reduce processing system that executes Map processing and Reduce processing and distributes a plurality of data, a system for constructing a cache mechanism for the Reduce processing,
In the Map-Reduce-ReduceMerge processing system, in which the Reduce process is partially executed on the result of the Map process and a ReduceMerge process is added to process the partially processed result in stages.
Classifying means for dividing the plurality of data into a plurality of groups based on a data update frequency that is an update frequency of each of the plurality of data;
A frequency calculating means for calculating a group update frequency, which is an update frequency of each of the plurality of groups, based on the data update frequency of data constituting each of the plurality of groups;
Map processing means for generating a plurality of Map results by executing the Map processing for each of the plurality of groups whose group update frequency is equal to or less than a preset threshold of the group update frequency;
Reduce processing means for partially executing the Reduce process on the plurality of Map results to generate a plurality of partial Reduce results;
ReduceMerge processing means for executing the ReduceMerge process stepwise on the plurality of partial Reduce results and generating a new partial Reduce result; and
Cache means for storing the partial Reduce result in a cache;
A system comprising:

Images