US10586129B2 - Generating artificial images for use in neural networks - Google Patents
Generating artificial images for use in neural networks Download PDFInfo
- Publication number
- US10586129B2 US10586129B2 US15/900,921 US201815900921A US10586129B2 US 10586129 B2 US10586129 B2 US 10586129B2 US 201815900921 A US201815900921 A US 201815900921A US 10586129 B2 US10586129 B2 US 10586129B2
- Authority
- US
- United States
- Prior art keywords
- image data
- matrix
- image
- filter
- resulting
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related, expires
Links
- 238000013528 artificial neural network Methods 0.000 title claims abstract description 36
- 239000011159 matrix material Substances 0.000 claims abstract description 145
- 238000000034 method Methods 0.000 claims abstract description 58
- 238000004590 computer program Methods 0.000 claims description 8
- 238000013213 extrapolation Methods 0.000 claims description 4
- 238000010586 diagram Methods 0.000 description 26
- 238000012545 processing Methods 0.000 description 20
- 230000006870 function Effects 0.000 description 18
- 230000008569 process Effects 0.000 description 13
- 210000002569 neuron Anatomy 0.000 description 8
- 238000013527 convolutional neural network Methods 0.000 description 7
- 238000012549 training Methods 0.000 description 6
- 230000009466 transformation Effects 0.000 description 6
- 230000005540 biological transmission Effects 0.000 description 5
- 238000007726 management method Methods 0.000 description 5
- 230000003190 augmentative effect Effects 0.000 description 4
- 239000003086 colorant Substances 0.000 description 4
- 238000004891 communication Methods 0.000 description 4
- 230000008520 organization Effects 0.000 description 4
- 210000000225 synapse Anatomy 0.000 description 4
- 238000000844 transformation Methods 0.000 description 4
- 238000007792 addition Methods 0.000 description 3
- 238000013459 approach Methods 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000007781 pre-processing Methods 0.000 description 3
- RYGMFSIKBFXOCR-UHFFFAOYSA-N Copper Chemical compound [Cu] RYGMFSIKBFXOCR-UHFFFAOYSA-N 0.000 description 2
- 239000000654 additive Substances 0.000 description 2
- 230000000996 additive effect Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 2
- 230000003416 augmentation Effects 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 229910052802 copper Inorganic materials 0.000 description 2
- 239000010949 copper Substances 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000004744 fabric Substances 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 239000000203 mixture Substances 0.000 description 2
- 230000006855 networking Effects 0.000 description 2
- 230000001902 propagating effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 238000012384 transportation and delivery Methods 0.000 description 2
- 238000012952 Resampling Methods 0.000 description 1
- 230000003466 anti-cipated effect Effects 0.000 description 1
- 238000013529 biological neural network Methods 0.000 description 1
- 210000004556 brain Anatomy 0.000 description 1
- 230000009172 bursting Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000007635 classification algorithm Methods 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000002790 cross-validation Methods 0.000 description 1
- 238000012517 data analytics Methods 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000012886 linear function Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000013439 planning Methods 0.000 description 1
- 229920001690 polydopamine Polymers 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 230000001242 postsynaptic effect Effects 0.000 description 1
- 230000000644 propagated effect Effects 0.000 description 1
- 238000013468 resource allocation Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G06K9/6255—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F18/2148—Generating training patterns; Bootstrap methods, e.g. bagging or boosting characterised by the process organisation or structure, e.g. boosting cascade
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/28—Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
-
- G06K9/42—
-
- G06K9/4619—
-
- G06K9/4652—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/20—Image enhancement or restoration by the use of local operators
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/56—Extraction of image or video features relating to colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20048—Transform domain processing
- G06T2207/20064—Wavelet transform [DWT]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Definitions
- the present invention relates to image processing using neural networks, and more specifically, to generating artificial images for use in neural networks.
- ConvNet use a variation of multilayer perceptrons designed to require minimal pre-processing. They are also known as shift invariant or space invariant artificial neural networks (MANN) based on their shared-weights architecture and translation invariance characteristics.
- MNN space invariant artificial neural networks
- ConvNet use relatively little pre-processing compared to other image classification algorithms. This means that the network learns the filters that in traditional algorithms were hand-engineered.
- Neural networks typically use a matrix vector linear product between layers with a common function applied element wise to the resulting vector function which may result in non-linear layers such as radial basis or threshold rectified layers.
- non-linear layers such as radial basis or threshold rectified layers.
- more complex non-linear layers requiring a per element function such as Fourier decompositions, polynomial or absolute value or multiple thresholds are difficult to implement and the back propagation through such layers is not always solvable.
- the result of layers replaces the input data and is not an addition to the input data.
- Multiple paths in neural networks have to be added if the previous input needs to be propagated such as with Residual Neural Networks.
- FIGS. 5A, 5B and 5C are photographic diagrams of an input image and concatenated images, in accordance with an embodiment of the present invention.
- FIG. 7 is a block diagram of an embodiment of a computer system or cloud server in which the present invention may be implemented.
- FIG. 9 is a diagram of abstraction model layers of a cloud computing environment in which the present invention may be implemented.
- the described method improves image recognition for neural networks by pre-computing image filters by augmenting images to generate artificial images.
- Artificial images are generated by augmenting an initial image by pre-computing image filters and obtaining new image data.
- the artificial images may be injected into the neural networks for better image recognition.
- the method may determine 110 if there is another color plane. If so, the method may iterate to input 105 a matrix of the next color plane and apply the filters to the next color plane.
- a flow diagram 200 shows an example embodiment of the aspect of FIG. 1 of determining 104 color planes or converting to a greyscale or panchromatic image.
- the (width, height) size of the new image becomes (width, height*(1+number of filters)).
- Other embodiments could select (width*(1+number of filters), height) or (width*(1+x), height (1+y)) where (1+x)*(1+y) ⁇ 1>number of filters.
- FIG. 5A shows a created artificial image 510 with the original image 511 and four filter images 512 , 513 , 514 , 515 .
- the computed filters result in matrices that are used as if they were image data.
- the reason for creating new images is that the process of usual neural networks for analyzing images is unchanged as it will take additional processed data as if it was an image data.
- FIG. 5B shows a created artificial image 520 with 11 filters added as a rectangular filling.
- FIG. 5C shows a created artificial image 530 illustrating further possible filters without resizing of Wavelet filters.
- the filters shown in FIGS. 5A, 5B and 5C are shown as greyscale photographs; however, they may include color images when processing color planes.
- FIG. 7 depicts a block diagram of components of the computing system 600 of FIG. 6 , in accordance with an embodiment of the present invention. It should be appreciated that FIG. 7 provides only an illustration of one implementation and does not imply any limitations with regard to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made.
- One or more operating systems 710 , and application programs 711 are stored on one or more of the computer readable storage media 708 for execution by one or more of the processors 702 via one or more of the respective RAMs 704 (which typically include cache memory).
- each of the computer readable storage media 708 can be a magnetic disk storage device of an internal hard drive, CD-ROM, DVD, memory stick, magnetic tape, magnetic disk, optical disk, a semiconductor storage device such as RAM, ROM, EPROM, flash memory, or any other computer readable storage media that can store a computer program and digital information, in accordance with embodiments of the invention.
- Computing system 600 can also include a network adapter or interface 716 , such as a TCP/IP adapter card or wireless communication adapter.
- Application programs 711 on computing system 600 can be downloaded to the computing device from an external computer or external storage device via a network (for example, the Internet, a local area network or other wide area networks or wireless networks) and network adapter or interface 716 . From the network adapter or interface 716 , the programs may be loaded into the computer readable storage media 708 .
- the network may comprise copper wires, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers and edge servers.
- the present invention may be a system, a method, and/or a computer program product at any possible technical detail level of integration
- the computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention
- PaaS Platform as a Service
- the consumer does not manage or control the underlying cloud infrastructure including networks, servers, operating systems, or storage, but has control over the deployed applications and possibly application hosting environment configurations.
- Nodes 10 may communicate with one another. They may be grouped (not shown) physically or virtually, in one or more networks, such as Private, Community, Public, or Hybrid clouds as described hereinabove, or a combination thereof.
- This allows cloud computing environment 50 to offer infrastructure, platforms and/or software as services for which a cloud consumer does not need to maintain resources on a local computing device.
- the types of computing devices 54 A-N shown in FIG. 8 are intended to be illustrative only and that computing nodes 10 and cloud computing environment 50 can communicate with any type of computerized device over any type of network and/or network addressable connection (e.g., using a web browser).
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Computation (AREA)
- Data Mining & Analysis (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Image Processing (AREA)
Abstract
Description
Claims (20)
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/900,921 US10586129B2 (en) | 2018-02-21 | 2018-02-21 | Generating artificial images for use in neural networks |
CN201910125584.8A CN110175968B (en) | 2018-02-21 | 2019-02-20 | Generating artificial images for use in neural networks |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/900,921 US10586129B2 (en) | 2018-02-21 | 2018-02-21 | Generating artificial images for use in neural networks |
Publications (2)
Publication Number | Publication Date |
---|---|
US20190258898A1 US20190258898A1 (en) | 2019-08-22 |
US10586129B2 true US10586129B2 (en) | 2020-03-10 |
Family
ID=67617361
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/900,921 Expired - Fee Related US10586129B2 (en) | 2018-02-21 | 2018-02-21 | Generating artificial images for use in neural networks |
Country Status (2)
Country | Link |
---|---|
US (1) | US10586129B2 (en) |
CN (1) | CN110175968B (en) |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111833263A (en) * | 2020-06-08 | 2020-10-27 | 北京嘀嘀无限科技发展有限公司 | Image processing method and device, readable storage medium and electronic equipment |
CN113344832A (en) * | 2021-05-28 | 2021-09-03 | 杭州睿胜软件有限公司 | Image processing method and device, electronic equipment and storage medium |
Citations (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5815608A (en) * | 1992-03-25 | 1998-09-29 | Mitsubishi Denki Kabushiki Kaisha | Apparatus and method for sensing and processing images |
US6792159B1 (en) * | 1999-12-29 | 2004-09-14 | Ge Medical Systems Global Technology Company, Llc | Correction of defective pixels in a detector using temporal gradients |
US20080212887A1 (en) * | 2005-05-12 | 2008-09-04 | Bracco Imaging S.P.A. | Method For Coding Pixels or Voxels of a Digital Image and a Method For Processing Digital Images |
WO2015065697A1 (en) | 2013-10-28 | 2015-05-07 | Molecular Devices, Llc | Method and system for classifying and identifying individual cells in a microscopy image |
CN104992177A (en) | 2015-06-12 | 2015-10-21 | 安徽大学 | Internet porn image detection method based on deep convolution nerve network |
US9542626B2 (en) | 2013-09-06 | 2017-01-10 | Toyota Jidosha Kabushiki Kaisha | Augmenting layer-based object detection with deep convolutional neural networks |
CN106780466A (en) | 2016-12-21 | 2017-05-31 | 广西师范大学 | A kind of cervical cell image-recognizing method based on convolutional neural networks |
US20170254932A1 (en) * | 2016-03-07 | 2017-09-07 | Microsoft Technology Licensing, Llc | Imaging through highly diffusive media with wavefront shaping |
US20180124330A1 (en) * | 2016-10-27 | 2018-05-03 | Leica Geosystems Ag | Method for visually representing scanning data |
Family Cites Families (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
DE19836071A1 (en) * | 1998-08-10 | 2000-02-17 | Schlafhorst & Co W | Process for the detection of thread residues on spinning head sleeves |
JP2001338288A (en) * | 2000-05-25 | 2001-12-07 | Nec Corp | Method and system for processing image, and image display controller |
US8837849B2 (en) * | 2007-06-26 | 2014-09-16 | Google Inc. | Method for noise-robust color changes in digital images |
KR101624210B1 (en) * | 2010-08-27 | 2016-05-31 | 한화테크윈 주식회사 | Method for reconstructing super-resolution image, and system for detecting illegally parked vehicles therewith |
CN103514583B (en) * | 2012-06-30 | 2016-08-24 | 华为技术有限公司 | Image sharpening method and equipment |
CN104680190B (en) * | 2013-11-29 | 2018-06-15 | 华为技术有限公司 | Object detection method and device |
US10198624B2 (en) * | 2016-02-18 | 2019-02-05 | Pinscreen, Inc. | Segmentation-guided real-time facial performance capture |
-
2018
- 2018-02-21 US US15/900,921 patent/US10586129B2/en not_active Expired - Fee Related
-
2019
- 2019-02-20 CN CN201910125584.8A patent/CN110175968B/en active Active
Patent Citations (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5815608A (en) * | 1992-03-25 | 1998-09-29 | Mitsubishi Denki Kabushiki Kaisha | Apparatus and method for sensing and processing images |
US6792159B1 (en) * | 1999-12-29 | 2004-09-14 | Ge Medical Systems Global Technology Company, Llc | Correction of defective pixels in a detector using temporal gradients |
US20080212887A1 (en) * | 2005-05-12 | 2008-09-04 | Bracco Imaging S.P.A. | Method For Coding Pixels or Voxels of a Digital Image and a Method For Processing Digital Images |
US9542626B2 (en) | 2013-09-06 | 2017-01-10 | Toyota Jidosha Kabushiki Kaisha | Augmenting layer-based object detection with deep convolutional neural networks |
WO2015065697A1 (en) | 2013-10-28 | 2015-05-07 | Molecular Devices, Llc | Method and system for classifying and identifying individual cells in a microscopy image |
CN104992177A (en) | 2015-06-12 | 2015-10-21 | 安徽大学 | Internet porn image detection method based on deep convolution nerve network |
US20170254932A1 (en) * | 2016-03-07 | 2017-09-07 | Microsoft Technology Licensing, Llc | Imaging through highly diffusive media with wavefront shaping |
US20180124330A1 (en) * | 2016-10-27 | 2018-05-03 | Leica Geosystems Ag | Method for visually representing scanning data |
CN106780466A (en) | 2016-12-21 | 2017-05-31 | 广西师范大学 | A kind of cervical cell image-recognizing method based on convolutional neural networks |
Non-Patent Citations (3)
Title |
---|
Alom et al., "Inception Recurrent Convolutional Neural Network for Object Recognition", arXiv:1704.07709v1 [cs.CV], Apr. 25, 2017, pp. 1-11. |
Mell et al., "The NIST Definition of Cloud Computing", National Institute of Standards and Technology, Special Publication 800-145, Sep. 2011, pp. 1-7. |
Pan et al., "Efficient Object Recognition Using Boundary Representation and Wavelet Neural Network", IEEE Transactions on Neural Networks, vol. 19, No. 12, Dec. 2008, pp. 2132-2149. |
Also Published As
Publication number | Publication date |
---|---|
US20190258898A1 (en) | 2019-08-22 |
CN110175968B (en) | 2023-05-09 |
CN110175968A (en) | 2019-08-27 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10373312B2 (en) | Automated skin lesion segmentation using deep side layers | |
US10229499B2 (en) | Skin lesion segmentation using deep convolution networks guided by local unsupervised learning | |
US11651206B2 (en) | Multiscale feature representations for object recognition and detection | |
US10936914B2 (en) | Convolutional neural network with augmentation features | |
US10699055B2 (en) | Generative adversarial networks for generating physical design layout patterns | |
US10990810B2 (en) | Automated facial recognition detection | |
US10706200B2 (en) | Generative adversarial networks for generating physical design layout patterns of integrated multi-layers | |
US11853395B2 (en) | Augmentation loss function for image classification | |
US20200410336A1 (en) | Dataset Dependent Low Rank Decomposition Of Neural Networks | |
US10586129B2 (en) | Generating artificial images for use in neural networks | |
US10671891B2 (en) | Reducing computational costs of deep reinforcement learning by gated convolutional neural network | |
US11315291B1 (en) | Shine visualization | |
US11620435B2 (en) | Domain specific model compression | |
US20230245481A1 (en) | Text detection algorithm for separating words detected as one text bounding box | |
US11593961B2 (en) | Difference-guided video analysis | |
US11688041B2 (en) | System and method of automatic image enhancement using system generated feedback mechanism | |
CN113837948B (en) | Image generation method, apparatus and storage medium | |
US11676363B2 (en) | Enhanced digital image loading | |
US11157554B2 (en) | Video response generation and modification | |
US11636345B2 (en) | Training generative adversarial networks | |
US20220164572A1 (en) | Optical character recognition segmentation | |
US20220036245A1 (en) | EXTRACTING SEQUENCES FROM d-DIMENSIONAL INPUT DATA FOR SEQUENTIAL PROCESSING WITH NEURAL NETWORKS | |
US20230162475A1 (en) | Radial summation preprocessing for image classification | |
US20230082448A1 (en) | Adaptive redundancy reduction for efficient video understanding |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: INTERNATIONAL BUSINESS MACHINES CORPORATION, NEW Y Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:FIAMMANTE, MARC;REEL/FRAME:044985/0045 Effective date: 20180221 Owner name: INTERNATIONAL BUSINESS MACHINES CORPORATION, NEW YORK Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:FIAMMANTE, MARC;REEL/FRAME:044985/0045 Effective date: 20180221 |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT RECEIVED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
FEPP | Fee payment procedure |
Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
LAPS | Lapse for failure to pay maintenance fees |
Free format text: PATENT EXPIRED FOR FAILURE TO PAY MAINTENANCE FEES (ORIGINAL EVENT CODE: EXP.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STCH | Information on status: patent discontinuation |
Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362 |
|
FP | Lapsed due to failure to pay maintenance fee |
Effective date: 20240310 |