;^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^Lance Simms, Stanford University 2009 ;pro Find_Leaky_Pixels ; ;PURPOSE: ; To try to find the pixels that appear to leak in flat frame images ; and 'get leaked into' in dark images. These pixels are usually at the ; center of a cross pattern and have a very large difference with respect ; to the pixels above, below, to the left, and to the right of them. ; ; The method of finding them will be to take the difference between ; a frame and the same frame shifted up, down, left, and right by 1 pixel ; and make a cut on the difference of all values. ; ; If the program is run in MODE=0, a simple plot of the pixel and its four nearest ; neighbors will be made. If MODE=1, the entire file will be searched. ; If MODE=2, the LeakyPix directory will be searched and a new leaky mask ; will be generated from the individual files. ; ;INPUTS: ; FITSFILENAMEORLIST: string ; MODE = 0 or MODE = 1 ; The path to the fits file that should be used to find the leaky pixels. ; This should be a slopefitted median flat field. ; MODE = 2 ; The path to the file that contains the list of fits files that will be used ; to generate the mask. ; ;OUTPUTS: ; FILE 1: ; KeyStrLeakyDir+'LeakyPix_'+KeyStr.RawObjectKey+'_'+KeyStr.Date+'.fits' ; - This file is a data cube in with the following slices ; 0 - Pixel values are ratio of center pixel to nearest neighbors ; 1 - Pixel values are the slopefit values of pixels; all pixels ; not flagged as leaky have 0 value. ; FILE 2 ; KeyStrLeakyDir+'LeakyPix_'+KeyStr.RawObjectKey+'_'+KeyStr.Date+'_RGB.fits' ; - This file will be an RGB data cube with the following color scheme ; Red - Pixels where the ratio is < 0.1 ; Green - Pixels where the ratio is > 0.1 and < 0.2 ; Blue - Pixels where the ratio is > 0.2 ; ;KEYWORDS: ; DATE: ; The Date during which they were taken. Also the subdirectory that ; the files are in. ; OFFSET: ; An offset for files that were written improperly. The readu ; function reads the values with this offset in place. ; NREADS: ; The number of reads (NAXIS3) that are present in the file. This ; is also a string in the filename that will be used as a regular ; expression in searching for matched files. ; THISFILTER: ; The filter through which the files were taken. ('I', 'G', or 'Y') ; OVERRIDE: ; Do the processing even if the output file already exists ; ALLFILES: ; 1 - Process all files matching the #Reads, Filter, and Object ; 0 - Only process one file (the first by default) ; MODE: ; 0 - Just plot a leaky pixel and its 4 nearest neighbors ; 1 - Search for all of the leaky pixels in the file ; ROW: int ; If MODE = 0, this will be the row of the leaky pixel ; COL: int ; If MODE = 0, this will be the column of the leaky pixel ; USEREF: int ; 0 - Don't subtract the reference pixels ; 1 - Subtract the reference pixels ; ADDBIAS: int ; 0 - Don't add the bias back to the data cube ; 1 - Add the bias back to the data cube if it's a median file ; UPDATELEAKYDB: int ; 0 - Don't write the results to LeakyPix database ; 1 - write the results to the LeakyPix database ; ;EXAMPLES: ; GENERATE THE LEAKY MASK FROM INDIVIDUAL FILES ; find_leaky_pixels_xrg, '/nfs/slac/g/ki/ki03/lances/H2RG-32-147/LeakyPix/LeakyFiles.txt', mode=2 ; find_leaky_pixels_xrg, '/nfs/slac/g/ki/ki03/lances/H1RG-022/LeakyPix/LeakyFiles.txt', mode=2 ;************************************************************************** pro Find_Leaky_Pixels_xRG, FitsFileNameOrList, $ Mode=Mode, Row=Row, Col=Col, $ Offset = Offset, ThisFilter = ThisFilter, $ OverRide = OverRide, $ UseRef = UseRef, TvFlag = TvFlag, $ UpdateLeakyDB = UpdateLeakyDB If N_Elements(Mode) eq 0 then Mode = 0 If N_Elements(Col) eq 0 then Col = 20 If N_Elements(Row) eq 0 then Row = 20 If N_Elements(AddBias) eq 0 then AddBias = 0 If N_Elements(UpdateLeakyDB) eq 0 then UpdateLeakyDB = 0 if Mode ne 2 then begin ;No lists are used in this mode FitsFileName = FitsFileNameOrList ;Include KeywordStruct.pro file with all the defaults for keywords @KeywordStruct_xRG.pro @PlotSettings_xRG.pro ;Get a Few Kewyords FileCheck = Return_File_Params_xRG(FitsFileName) ;Form the output filenames LeakyValsFileName = KeyStr.LeakyDir+'LeakyPix_'+KeyStr.RawObjectKey+'_'+$ KeyStr.Date+'.fits' LeakyRGBFileName = KeyStr.LeakyDir+'LeakyPix_'+KeyStr.RawObjectKey+'_'+$ KeyStr.Date+'RGB.fits' ;Indices with skipping of reference pixels and channel 0 PosThreshold = 5. LeakyThreshold = 0.7 NAxis1 = KeyStr.NAxis1 NAxis2 = KeyStr.NAxis2 Ref = KeyStr.Ref stop ;If H4RG is used, take first channel as obsolete If KeyStr.DetStr eq 'H4RG-10-007' then begin Ch0W = KeyStr.Ch0W EndIf Else Begin Ch0W = Ref/2 EndElse XSizeM = KeyStr.Naxis1-KeyStr.Ch0W-KeyStr.Ref-2 YSizeM = KeyStr.Naxis2-KeyStr.Ref-2 EndIf Case Mode of 0: Begin ;//////////////////////////////////////////////////////////////////// ;PLOT PIXELS //////////////////////////////////////////////////////// ;Read in the 3x3 area Fits_Read_DataCube, FitsFileName, Im, H, XStart=Col-1, XStop=Col+1, $ YStart = Row-1, YStop = Row+1 ;Add the bias back to the median if requested If AddBias eq 1 then begin For ReadNum = 1, KeyStr.NAxis3-1 do begin Im(*,*,ReadNum) = Im(*,*,ReadNum)+Im(*,*,0) EndFor EndIf Else Begin Im = Im(*,*,1:KeyStr.NAxis3-1) EndElse ;Get the min, max, and fraction w/respect to surrounding pixels MaxIm = Max(Im) MinIm = Min(Im) AvgSr = (Im(0,1,KeyStr.NAxis3-2)+Im(2,1,KeyStr.NAxis3-2)+$ Im(1,0,KeyStr.NAxis3-2)+Im(1,2,KeyStr.NAxis3-2))/4 Frac = Im(1,1,KeyStr.NAxis3-2)/float(AvgSr) ;Do the plotting plot, Im(1,1,*), yrange = [MinIm, MaxIm], $ ytitle = 'Counts', $ title='Open Pixel in : ' + KeyStr.RawObjectBase, /normal, $ color = fsc_color('black'), background = fsc_color('white') oplot, Im(0,1,*), color = fsc_color('Red') oplot, Im(2,1,*), color = fsc_color('Blue') oplot, Im(1,0,*), color = fsc_color('Green') oplot, Im(1,2,*), color = fsc_color('Orange') xyouts, 0.20, 0.9, 'Pixel Coordinates', color = fsc_color('Black'), /normal xyouts, 0.20, 0.85, 'X : '+strtrim(Col, 2), color = fsc_color('Black'), /normal xyouts, 0.35, 0.85, 'Y : '+strtrim(Row, 2), color = fsc_color('Black'), /normal xyouts, 0.20, 0.80, 'Fraction of Neighbors: ' + strtrim(Frac,2), color=fsc_color('black'),$ /normal xyouts, 0.25, 0.75, 'Upper', color = fsc_color('Orange') , /normal xyouts, 0.25, 0.70, 'Center', color = fsc_color('Black') , /normal xyouts, 0.20, 0.70, 'Left', color = fsc_color('Red') , /normal xyouts, 0.32, 0.70, 'Right', color = fsc_color('Blue') , /normal xyouts, 0.25, 0.65, 'Lower', color = fsc_color('Green') , /normal OrigCharSize = !p.charsize !p.charsize = .75 xyouts, 0.10, 0.000, KeyStr.RawObjectPath, color=fsc_color('black'), /normal !p.charsize = OrigCharSize If KeyStr.TvFlag eq 1 then begin Img = tvread(/tiff, FileName = KeyStr.LeakyDir+'Plots'+KeyStr.PathDelim+$ KeyStr.RawObjectKey+'_'+Strtrim(Col,2)+'_'+Strtrim(Row,2)+'.tif',$ /nodialog) EndIf stop End 1: Begin ;////////////////////////////////////////////////////////////////////////// ;*********************** FIND ALL PIXELS IN FILE ************************** ;Form an array that will keep track of how many times this pixel was leaky LeakyPixCenMask = bytarr(Naxis1, Naxis2) HotPixCenMask = bytarr(Naxis1, Naxis2) ;Form arrays to track nearest neighbors so that averaging can take place LeakyPixNNMask = bytarr(Naxis1, Naxis2) HotPixNNMask = bytarr(Naxis1, Naxis2) ;Form array for the rgb image LeakyPixRGB = bytarr(Naxis1, Naxis2,3) ;Form arrays to keep average values LeakyPixRatios = fltarr(Naxis1, Naxis2) LeakyPixVals = fltarr(Naxis1, Naxis2) HotPixRatios = fltarr(Naxis1, Naxis2) HotPixVals = fltarr(Naxis1, Naxis2) MkHdr, HotHeader, HotPixVals FxAddPar, HotHeader, 'DIFTHRESH', PosThresHold, $ 'Diff. Threshold between center and nearest neighbors' FxAddPar, HotHeader, KeyStr.Date+'_'+KeyStr.RawObjectBase, $ 'Name of Files used to draw map' FxAddPar, HotHeader, 'NUMIPC', 0, $ 'Number of pixels THRESHOLD above nearest neighbors' ;*************************************************************************** ;Begin the main loop for the files and read numbers print, 'Reading File: ' + FitsFileName ;Open up the raw image whether dark, flat, or object Fits_Read_DataCube, FitsFileName, Im, RawHeader Frame = Float(Im(*,*,0)) ;Form difference frames. Waste of memory, but ease of use with where function FrameUpOne = FltArr(XSizeM, YSizeM) FrameDownOne = FltArr(XSizeM, YSizeM) FrameRightOne = FltArr(XSizeM, YSizeM) FrameLeftOne = FltArr(XSizeM, YSizeM) ;Do the ratios with nearest neighbors FrameUpOne(*,*) = $ float(Frame(1+Ch0W:Naxis1-Ref/2-2, 1+Ref/2:Naxis2-Ref/2-2)) / $ float(Frame(1+Ch0W:Naxis1-Ref/2-2, 2+Ref/2:Naxis2-Ref/2-1)) FrameDownOne(*,*) = $ float(Frame(1+Ch0W:Naxis1-Ref/2-2, 1+Ref/2:Naxis2-Ref/2-2)) / $ float(Frame(1+Ch0W:Naxis1-Ref/2-2, 0+Ref/2:Naxis2-Ref/2-3)) FrameLeftOne(*,*) = $ float(Frame(Ch0W+1:Naxis1-Ref/2-2 , 1+Ref/2:Naxis2-Ref/2-2)) / $ float(Frame(Ch0W+2:Naxis1-Ref/2-1 , 1+Ref/2:Naxis2-Ref/2-2)) FrameRightOne(*,*) = $ float(Frame(Ch0W+1:Naxis1-Ref/2-2 , 1+Ref/2:Naxis2-Ref/2-2)) / $ float(Frame(Ch0W+0:Naxis1-Ref/2-3 , 1+Ref/2:Naxis2-Ref/2-2)) ;Form a mask showing the number of differences that are greater than threshold FrameBadMask = IntArr(XSizeM, YSizeM) ;Find pixels above neighbors FrameBadUpOne = where(FrameUpOne gt PosThreshold, NumPosUp) If FrameBadUpOne(0) ne -1 then FrameBadMask(FrameBadUpOne) += 1 FrameBadDownOne = where(FrameDownOne gt PosThreshold, NumPosDown) If FrameBadDownOne(0) ne -1 then FrameBadMask(FrameBadDownOne) += 1 FrameBadLeftOne = where(FrameLeftOne gt PosThreshold, NumPosLeft) If FrameBadLeftOne(0) ne -1 then FrameBadMask(FrameBadLeftOne) += 1 FrameBadRightOne = where(FrameRightOne gt PosThreshold, NumPosRight) If FrameBadRightOne(0) ne -1 then FrameBadMask(FrameBadRightOne) += 1 ;Find pixels below neighbors FrameBadUpOne = where(FrameUpOne lt LeakyThreshold, NumLeakyUp) If FrameBadUpOne(0) ne - 1 then FrameBadMask(FrameBadUpOne) -= 1 FrameBadDownOne = where(FrameDownOne lt LeakyThreshold, NumLeakyDown) If FrameBadDownOne(0) ne -1 then FrameBadMask(FrameBadDownOne) -= 1 FrameBadLeftOne = where(FrameLeftOne lt LeakyThreshold, NumLeakyLeft) If FrameBadLeftOne(0) ne -1 then FrameBadMask(FrameBadLeftOne) -= 1 FrameBadRightOne = where(FrameRightOne lt LeakyThreshold, NumLeakyRight) If FrameBadRightOne(0) ne - 1 then FrameBadMask(FrameBadRightOne) -= 1 ;Indices where center pixel was PosThreshold above nearest neighbors; reform ;indices for the full array PosCrPix = where(FrameBadMask eq 4) XPosCrPix = (PosCrPix mod XSizeM) + Ch0W + 1 YPosCrPix = (PosCrPix/XSizeM)+Ref/2 + 1 PosCrPix = XPosCrPix+long(YPosCrPix)*Naxis2 Print, 'Found : '+strtrim(N_Elements(PosCrPix),2) +' Hot Pixels' ;Now form all the Indices of the nearest neighbors PosCrPixNN = [PosCrPix+Naxis1-1, PosCrPix+Naxis1, PosCrPix+Naxis1+1,$ PosCrPix-1 , PosCrPix , PosCrPix+1,$ PosCrPix-Naxis1-1, PosCrPix-Naxis1, PosCrPix-Naxis1+1] ;Indices where center pixel was LeakyThreshold below nearest neighbors LeakyPix = where(FrameBadMask eq -4) XLeakyPix = (LeakyPix mod XSizeM) + Ch0W + 1 YLeakyPix = (LeakyPix/XSizeM)+Ref/2 + 1 LeakyPix = XLeakyPix+long(YLeakyPix)*Naxis2 Print, 'Found : '+strtrim(N_Elements(LeakyPix),2) +' Leaky Pixels' ;Form the individual arrays of indices for nearest neighbors LeakyPixUp = LeakyPix+NAxis1 LeakyPixDown = LeakyPix-NAxis1 LeakyPixRight = LeakyPix+1 LeakyPixLeft = LeakyPix-1 ;Take ratio of center to average of nearest neighbors LeakyPixRatios(LeakyPix) = Frame(LeakyPix)/$ ((Frame(LeakyPixUp)+Frame(LeakyPixDown)+$ Frame(LeakyPixRight) + Frame(LeakyPixLeft))/4) ;Now form all the Indices of the nearest neighbors LeakyPixNN = [LeakyPix+Naxis1, LeakyPix-1, LeakyPix+1, LeakyPix-Naxis1] ;form all indices of 9 pixels in box LeakyPixUpLeft = LeakyPix+NAxis-1 LeakyPixUpRight = LeakyPix+NAxis+1 LeakyPixDownLeft = LeakyPix-NAxis-1 LeakyPixDownRight = LeakyPix-NAxis+1 LeakyPix9N = [LeakyPix+Naxis1-1, LeakyPix+Naxis1, LeakyPix+Naxis1+1,$ LeakyPix-1 , LeakyPix , LeakyPix+1,$ LeakyPix-Naxis1-1, LeakyPix-Naxis1, LeakyPix-Naxis1+1] ;Get the slopes LeakyPixVals(LeakyPixNN) = Frame(LeakyPixNN) ;Make the color array LeakyPixRed = LeakyPix(where(LeakyPixRatios(LeakyPix) lt 0.1)) LeakyPixGreen = LeakyPix(where(LeakyPixRatios(LeakyPix) ge 0.1 and $ LeakyPixRatios(LeakyPix) lt 0.17)) LeakyPixBlue = LeakyPix(where(LeakyPixRatios(LeakyPix) ge 0.17 and $ LeakyPixRatios(LeakyPix) lt 1.0)) LeakyPixRGB(LeakyPixRed) = 255 LeakyPixRGB(LeakyPixGreen+long(Naxis1)*Naxis2) = 255 LeakyPixRGB(LeakyPixBlue+2*long(Naxis1)*Naxis2) = 255 ;Average the frames over the number of times hotpixels occured ;HotPixNZ = where(HotPixNNMask ne 0) ;HotPixVals(HotPixNZ) = HotPixVals(HotPixNZ)/HotPixNNMask(HotPixNZ) ;LeakyPixNZ = where(LeakyPixNNMask ne 0) ;Change the header slightly for the mask LeakyPixArr = Fltarr(Naxis1,Naxis2,2) LeakyPixArr(*,*,0) = LeakyPixRatios LeakyPixArr(*,*,1) = LeakyPixVals ;FileNames and headers for masks and value leaky files MkHdr, LeakyHeader, LeakyPixArr FxAddPar, LeakyHeader, 'DIFTHRESH', LeakyThreshold, $ 'Diff. Threshold between center and nearest neighbors' FxAddPar, LeakyHeader, KeyStr.Date+'_'+KeyStr.RawObjectBase, $ 'Name of Files used to draw map' FxAddPar, LeakyHeader, 'NUMLEAKY', 0, $ 'Number of pixels THRESHOLD below nearest neighbors' FxAddPar, LeakyHeader, 'BITPIX', 8 UH = Update_Header_xRG(LeakyHeader) Fits_Write, LeakyValsFileName, LeakyPixArr, LeakyHeader Fits_Write, LeakyRGBFileName, LeakyPixRGB ;Update the Leaky Pixel Database if requested If UpdateLeakyDB eq 1 then begin openmysql, MySQLlun, 'LeakyPix', MyQLErr TableName = KeyStr.DetStr+'_'+KeyStr.ElecStr strreplace, TableName, '-','_' strreplace, TableName, '-','_' For PixelNum = 0, N_Elements(LeakyPix)-1 do begin mysqlcmd, MySQLLun, 'INSERT INTO '+TableName+' VALUES('+$ strtrim(long(LeakyPix(PixelNum) mod KeyStr.NAxis1),2)+','+$ strtrim(long(LeakyPix(PixelNum) / KeyStr.NAxis1),2)+','+$ "'"+strtrim(KeyStr.ThisFilter,2)+"',"+$ "'"+strtrim(KeyStr.Date,2)+"',"+$ strtrim(Frame(LeakyPixDownLeft(PixelNum)),2)+','+$ strtrim(Frame(LeakyPixDown(PixelNum)),2)+','+$ strtrim(Frame(LeakyPixDownRight(PixelNum)),2)+','+$ strtrim(Frame(LeakyPixLeft(PixelNum)),2)+','+$ strtrim(Frame(LeakyPix(PixelNum)),2)+','+$ strtrim(Frame(LeakyPixRight(PixelNum)),2)+','+$ strtrim(Frame(LeakyPixUpLeft(PixelNum)),2)+','+$ strtrim(Frame(LeakyPixUp(PixelNum)),2)+','+$ strtrim(Frame(LeakyPixUpRight(PixelNum)),2)+','+$ strtrim(LeakyPixRatios(LeakyPix(PixelNum)),2)+');', $ Answer, NLines EndFor endif stop ;Write the number of hot pixels to the header HotHeader = HeadFits(HotValsFileName) FxAddPar, HotHeader, 'NUMIPC', N_Elements(PosCrPix) ModFits, HotValsFileName, 0, HotHeader ;Change the header slightly for the mask FxAddPar, HotHeader, 'BITPIX', 8 Fits_Write, HotMaskFileName, HotPixCenMask, HotHeader End 2: Begin ;//////////////////////////////////////////////////////////////////// ;Get the names of fits files that will be used readcol, FitsFileNameOrList, FitsFileNames, format='A' FitsFileName = FitsFileNames(0) DirName = File_DirName(FitsFileName) LeakyFileName = DirName + Path_Sep()+'LeakyMask.fits' fits_read, FitsFileName, D, H, /header_only NAxis1 = long(SxPar(H, 'NAXIS1')) NAxis2 = long(SxPar(H, 'NAXIS2')) LeakyMask = BytArr(NAxis1, NAxis2) ;FileNames and headers for masks and value leaky files MkHdr, LeakyHeader, LeakyMask for FileNum = 0, N_Elements(FitsFileNames)-1 do begin FxAddPar, LeakyHeader, 'FILENUM'+Strtrim(FileNum,2), $ File_BaseName(FitsFileNames(FileNum)), $ 'Files used to generate mask' endfor for FileNum = 0, N_Elements(FitsFileNames)-1 do begin Fits_Read, FitsFileNames(FileNum), LeakyMaskInd, LeakyHeadInd LeakyMask(where(LeakyMaskInd ne 0)) = 1 endfor ;Write the fits file Fits_Write, LeakyFileName, LeakyMask, LeakyHeader End EndCase stop end