NEW:



OBS PROCESSING MEETING NOTES:

Note: only covers “new” items since 7/26/2011 Conventional/In-situ meeting and since 7/12/2011 Satellite/Remote meeting

CONVENTIONAL/IN-SITU meeting on 9/20/2011 in BLACK - with later updates in GREEN:

HURRICANE:

- Barry Choy asking why we are not assimilating HDOB data

o NCO must write a decoder

▪ Geoff: [NCO_1] NCO should get started on this

o On DML but not yet prioritized

- Vijay will be running impact tests, denying additional off-time RAOBS available during Irene

o Yangrong identified these RAOBs, later Richard Pasch at NHC did the same

- DYNAMO:

o GTS test sonde message released from the Revelle ship 14 z 9/2/11

▪ It came into NCEP

▪ Is it on reject list?

o DYNAMO begins in Oct!

▪ Per John D. & Su: some data are now coming in. Nothing is on reject list. Looks ok so far. More data will start coming in this weekend. Need to but untested platforms on reject list. [DK_1] Dennis ping all applicable folks.

- Eric: Why do t-6 NDAS PREPBUFR files contain fewer drops than GDAS/GFS (valid at same time) for Irene? [DK_2] Dennis needs to find answer.

o Likely due to SYNDATA tossing drops in vicinity of storm. [GD_1] Geoff – revisit this.

- Shelley has updated storm names 2011-2016 (replacing names of significant storms with their new names) and has submitted RFC. All 2011 names are currently ok so this is not a crisis.

MARINE SURFACE (NON-SAT):

- GLERL adjusted obs (coastal surface) – [SL_1] Steve is working on this – [JW_1] will need Jeff and Dennis involved to add to PREPBUFR

- Buoys:

o Per Dick Crout NDBC test data coming in from a "Refresh" buoy that will eventually replace "Legacy" buoys – this is not coming into NCEP

▪ "SXPA01 KWNB", 10-min frequency (all refresh buoys will have this bulletin header)

▪ DF is now bringing in, Jeff A. is decoding in dev

▪ Dave Behringer is looking at subsurface data – looks ok, [NCO_2] Jeff A. will promote to prod IF WE ARE OK WITH MET DATA – [YL_4] Yangrong see if you can use Brad’s codes to evaluate these in PREPBUFR files you make from Dennis’ cron sfcshp dumps from /dcomdev

▪ [DS_1] Diane will check if these data will be included in quips

o EROA2 lat has been corrected in prod dictionary from 38.97 to 58.97N per Steve

- More sources of visibility (per John Kelly):

o IOOS moored buoys (GOMOOS) in Gulf of Maine (e.g., 44034) – in our tanks per Jeff W. (incl. visibility)

o Also manual obs from ferries – doesn’t appear we are getting these (per Jeff W.)

o A few NOAA/NOS/PORTS “mesonets”

▪ Would be in either SHEF, CREX or C-MAN format tanks (not sure which) – not sure if we are getting these visibility obs (likely not)…

• Per [SL_2] Steve: No, these are not in any of the SHEF tanks, we will not be getting them (and any visibility obs from them) in the near future due to resource limitations at NDBC.

o NH DOT RWIS – per Steve these are on MADIS but not coming in

▪ We do not get visibility for any mesonet data except SHEF/Coop (which we do not dump yet) – not in BUFR template

• Request in to [NCO_3] NCO/SIB to update mesonet decoder to add visibility and cloud-height data

- In Dennis’ cron GFS/GDAS dumps, added coast guard to “sfcshp” (not recognized in PREPBUFR), and dumps “bathy”, “tesac” and “trkob” for Xu Li and marine folks on 8/17/11

o This will go in with NRL ACQC package

o For now Kate will copy to global parallel dump archive

o Can assume pmsl=pstn for Coast Guard sites since they are close to ground per Steve

RADIOSONDE:

- We are getting the 3 ARM/CART radiosondes (Yangrong investigated)

o Southern Great Plains, North Slope of AK, Tropical W. Pacific

o John W. received an e-mail asking if we assimilate international ARM RAOBs: Niamey, Niger; Blackforest Germany; Shouxian, Anhui, China; and Graciosa Island, Azores, Portugal – [YL_1] Yangrong will check

- Kingston, Jamaica site (78397) has wind direction bias of 15 deg since 8/6/11

o Brad added winds to reject list

o Site notified of problem

- Working with [NCO_4] Becky to get 2 Antarctic RAOBs into dev decoder (for testing)

o These are used by MeteoFrance but we never got them in here

- Neil Jacobs TAMDAR presentation in Aug noted that some RAOBS in NCEP PREPBUFR files with drift appear to cut off prematurely – then they back off a bit from this – we need to check this out – [YL_2] Yangrong please check, ask them to send a sample

- Per Jeff A. Nancy France (07180), which had been reporting an illegal instrument type of 34, has now been corrected on-site so that any future reports will now contain the correct instrument type of 70 (this site reports sporadically) – [YL_3] Yangrong please check

MESONET:

- Mesonets with missing pressure (thus using U.S. Std Atmos p) and “x” in character 8 of id:

o Will no longer be assimilated in next NAM upgrade – only pressure will be tossed

o Still assimilated in RUC/RR - ?? [DK_3] Dennis ask Stan

- Puerto Rican mesonet (found by McQueen)

o Not in MADIS, [SL_3] Steve ping them about availability in realtime and reliability

o Reliable - ?

- Some HADS stations were not getting into MADIS

o Leon B. corrected this

- NEPP network very low, Steve indicates sites are reporting but they are not getting to MADIS - ?

o Is b255/xx101 coopmd tank (where NEPP comes in) populated from MADIS or GTS? MADIS is the answer and these are not part of integrated surface mesonet feed

o These sites may be coming in as something else since they were found on NWSTG feed

▪ C-MAN maybe - ?????

o b255/xx101 is currently not dumped so these data do not make it into msonet dumps or PREPBUFR files

- ESRL/MADIS reports corrupt lat/lon/elev in 18, 19, 20z files for 9/7/11.

- NJ Mesonet data: We may not be getting all of it since not all gets to MADIS

o e.g., New Brunswick and Jersey City – Rutgers sent email to Mike Barth on how to get these. Old item – [SL_4] Steve will email Leon.

- CRN in b255/xx030: there is more info than what we decode, NCO would have to write special decoder to get everything else

o Steve L. finds that the decoder is actually putting out the wrong info (e.g., dewpoint is actually RH)

▪ [NCO_5] NCO should shut this down and then write new decoder to handle existing info properly and add new info

▪ [SL_5] Steve – check RTMA, RUC and NAM mesonet reject list – hopefully provider CRN data other than temperature is not being used.

- Lightning:

o Vaisala Global Lightning Data (GLD) 360: Stan asking if we get this for RR

▪ No, but NCO is pulling for NAWIPS – [NCO_6] Stan asked NCO/DF to pull to CCS for RR – more to come …. ([DK_4] Dennis: add to DML)

- Clarus:

o Good QC documentation – could be applied to QC of other “conventional” obs types

o Briefing to NWS and OAR – Webinar, Doodle poll, more docs week of 9/12 - ?

▪ Steve and Shelley will “attend”

- NEON (National Ecological Observatory Network) funded by NSF to build 62 sites for observing change in ecosystems

o Research oriented activity and it is a long way off

o Focused on hydro/meteo/eco of terrestrial ecosystems (so lots of soil and veg parameters will be part of the measurements, in addition to standard met)

o Very good dataset for use of cal/val in model dev (e.g., land-surface and PBL physics in EMC's models) – eventually contact Mike Ek re: this

o Latency of the data will likely make it of limited utility in the NWP systems

o [DK_5] Dennis: Add to DML

METAR:

- Per Alan Overbay at FAA, some of the Gulf METAR stations have bad winds due to obstructions

o Height of anemometer and presser sensors – likely not 224 m which we have for them – Alan has more info for Steve’s metadata

o May be a feet vs. meters issue!

o Steve has correct wind sensor height and helipad height (= height for temp/pressure sensors?) [SL_6] Steve will work with NCO to update METAR dictionary to get correct heights in /dcomdev first, then if all is well into /dcom. We will then eventually be able to take them off reject list and they will be in adpsfc dumps and assimilated.

- Per Yangrong and Steve, obs coming in that are not in prod dictionary

o Steve has metadata, Jeff A. has added to dev dictionary

▪ [YL_5] Yangrong use Brad’s codes to evaluate in PREPBUFR files made from Dennis cron adpsfc dumps from /dcomdev. If all looks good ask Jeff A. to promote to prod.

- Steve requesting script to automatically (monthly) check for updates to NCDC’s ISH station lists

- RTMA needs Runway Visual Range (RVR) which is in BUFR template in dumps – 1st RVR

o Along with this, template has runway id, 2nd RVR, RVR tendency, replicates for different runways

o Geoff: Don’t add to PREPBUFR until we sample % of obs reporting it

▪ Can someone in the group do this when possible?

▪ LATE WORD FROM GEOFF: Stand down on this for now, only a few obs report this per Steve

AIRCRAFT:

- QC “feature” – all types except MDCRS get QM=14 on temp automatically if QM=14 on winds

o Comments say needed for (old) ACQC

o [DK_6] Dennis: ask Pat Pauley if this is needed for new QC

- AMDAR:

o Proposed changes to BUFR template – Dave Helms wants comments .. more to come … [DK_7] Dennis ping Dave for more info.

o South African AMDARS fixed (some frequently had incorrect locations) per Brad B. – removed from reject list 9/7/11

- AirDAT provided full TAMDAR feed 00z 8/26 through 23z 8/29(?) in support of NWP for Hurricane Irene

o RTDM showed spike in dump counts

o Brad B. is evaluating impact (plus Russ and Su?)

▪ T, q, u/v used by NAM/NDAS

▪ T, q used by RUC/RR

▪ nothing used by GFS/GDAS, but monitoring stats will be generated

▪ We only get Chautauqua, Mesaba, PenAir via MADIS, the belief is that these are only in AirDAT feed as well (??)

• Did other types come in that fell on the floor here at NCEP?

o e.g., Horizon, Aeromexico, ERA Alaska, Piedmont

o Should we get a few of these routinely even in limit 10% feed-?

▪ [SM_1] Shelley check: 1) did types other than the 3 usual ones come into MADIS and/or AirDAT tanks during Irene; 2) do types other than the 3 usual ones come into AirDAT tanks routinely; 3) is MADIS sending us types other than the 3 usual ones routinely, if so how does NCO’s decoder handle them since we have specific tanks for the 3 usual types

WIND PROFILERS/RASS:

- There is no bird migration alg. directly in PROFCQC code, but NPN profilers are QC’d for birds upstream at Hub

o Also PROFCQC code does check for vertical and temporal consistency in winds

o WFIP-profiles in CAP have some bird QC(??) but Jim Wilczak is seeing bird issues in some data

▪ PSD may install Gabor bird processing algorithm on the Glacial Ridge NM profiler

▪ [DK_8] Dennis: send Jim email on status.

- WFIP-relocated profilers decoded into MAP tanks – these are NOT restricted

o There is no change from the MADIS QC as originally thought (per Jim W.) – good news

o NCEP runs honor QM=14 form reject list (NAM, RUC/RR)

o Dennis’ cron PREPBUFR for RUC/RR now (since 7/27/11) remove QM=14 from dumps and allow original MADIS QM to remain in place (converted to PREPBUFR QM) – these are being used by ESRL RR runs (NCEP RR runs use /com PREPBUFR files which honor QM=14)

o Also found out that 4 SODARS are included here – these have been added to sdmedit reject list for operational NCEP runs and are also rejected in the ESRL RR runs – these are also not restricted

▪ Jeff A. updated MAP decoder in /dcomdev 8/29 to recognize SODARs vs. Profilers in BUFR entry A4ME (always missing right now)

• Stan validated that this looked ok

• Jeff A.’s RFC to fix this impl. 9/13

o WindLogics tower and Nacelle data is ready to be sent to NCEP per Jim W. – [DK_9] Dennis send Jim e-mail re: status.

▪ Direct transfer from ESRL to NCEP (like GPSIPW data) – ESRL is BUFRizing Nacelle data – not yet for tower data (will NCEP have to BUFRize this?)

▪ [NCO_7] NCO needs to set up a new HIGH-restriction classification for these data

- WFIP-relocated RASS decoded into RASS tanks (only RASS component for profilers, none for SODARs)

o We didn’t think of these – they were not added to sdmedit reject list!!

o Asked Brad to add them to reject list in sdmedit – he did so on 9/9

o Stan reports in general that the RUC does not seem to be using all of the RASS data (from all sources) – he does not know why …

- ESRL discovered ~ 8/24 that all CAP data (including WFIP-relocated profilers and SODARS), as well as RASS data have the wrong height assignment in NCEP BUFR tanks/dump/PREPBUFR

o Turned off in ESRL runs 8/24, restored CAP on 8/26 with temporary correction (RUC temporary correction was supposed to be implemented as ESRL the week of 8/29)

o NCO CAP/RASS decoder was assuming MADIS heights were AGL when in fact they were ASL (via the MADIS API NetCDF reader) (thus decoder ended up adding elevation twice to height)

▪ NPN profiler and RASS data are not affected

▪ This problem has occurred since the BUFR MAP feed was replaced by the MADIS MAP feed in ????

• [DK_10] DENNIS: NOTE THIS IN HISTORICAL DOC, get dates, apply patch in PREPBUFR processing

▪ Fixed in /dcomdev on 8/29, Stan verified it looked ok (RASS and CAP)

▪ Jeff A.’s RFC to fix this impl. 9/13

• ESRL removed their temporary patches

▪ These were never added to sdmedit reject list since it is hard to do so with current EDTBFR and these are not used in ops + have been this way all along and are about to be fixed

MISCELLANEOUS from SATELLITE/REMOTE meeting on 8/23/11 in BLACK:

updates from SATELLITE/REMOTE meeting on 9/6/11 in GREEN:

updates from CONVENTIONAL/IN-SITU meeting on 9/20/2011 in RED - with later updates in BLUE:

- Urgent Satellite/Remote Topics: none

- QC Topics – NRL ACQC !!! – 99% of Dennis’ work right now – steady progress, shaved another 30 sec off run time -- next contact Pat Pauley

- “RAP” will replace “RUC’ in all scripts and codes when RR is impl. (mid-November-?) - /com/ruc will become /com/rap

o Chris M. met with Geoff M. last week

▪ Dennis can submit 1 RFC for all scripts/codes w.r.t. name change

▪ Dennis made a list of all prod scripts and codes that either contain RUC or ruc, or have the files name ruc – many!!

o What about the extra dumps Dennis is making in his RR cron – do these need to be implemented?

▪ 1bamua, 1bamub, 1bhrs3, 1bhrs4, 1bmhs (widened from +/- 1 to +/- 3 hr time window)

▪ airsev.tm00.bufr_d (widened from +/- 2 to +/- 3 hr time window)

▪ radwnd (widened from +/- 0.75 to +/- 1 hr time window)

▪ ascatt/ascatw, esamua, esamub, eshrs3, esmhs, lghtng, lgycld, nexrad, wdsatr/wndsat (new)

▪ all dumps and prepbufr files for 00 and 12z ruc2a_early runs

▪ split PREP and ANAL jobs

▪ AND SET OF LATER DUMPS FOR RR’s CATCH-UP CYCLE - ??.

- METOP – outage for all products for 7 hours on 8/22/11, blind data

- Obs Workshop was held Sept 14-15 in at OFCM in SS

o Reps from FNMOC, NAVO, AFWA, NCEP, JCSDA, NESDIS, NBDC, NCDC, NODC, NGDC, 14’th Air Sq, TOC

o We all need a full listing of bulletin headers (from WMO?), TOC is of no help

▪ We are all missing data – need to compare notes

• Compare obs counts, etc. (are we all seeing spikes and gaps at the same time for the same sobs types)

o C-MANS are not platforms! They look more like short lighthouses on the coasts

- Introductory Fortran training was held last week

- DATA MINING LIST

o Dennis update

o Call meeting within EMC before involving NCO

▪ Automate via dB – talk to Shelley

▪ Google Docs?

- GAEA meeting on 7 Sep

o NOAA R&D machine at Oak Ridge (Zeus also coming to Fairmont to repl. Vapor) – transition of parts of EMC processing to this

o How does this affect obs processing codes? – restricted data!

- BUFR_DUPSAT implemented 8/9/11

o Removes “near-dups” in ATOVS 1b data

o Logic changed to not perform longitude correction what latitude is unreasonable

▪ No more possibility of long running dumps (e.g., 2 cases of AMSR-E dumps)

- BUFR_EDTBFR – RFC updated and resubmitted as a stand-alone (no longer part of Restricted Ship package)

o Met with Kelly, Brad B and Joey 8/29/11 to go over enhancements to SDMEDIT process, delivered write up for them

▪ Joey asked for one additional change: given YYYYMMDD and wildcard hour, all reports on YYYYMMDD would be rejected regardless of HH

• Updated RFC and resubmitted – NCO tested and Dennis validated

o NCO was going to implement 6/13 but it was withdrawn due to another code that needs to be updated as well by Kelly – EDTLST

▪ Late last week Dennis discovered that code is not working quite right for wildcard stn id’s – updating and will resubmit RFC later this week

• Will also put in check to prevent ALL reports for being flagged if SDM accidentally enters all wildcards!

- Fallout from change in bufrtab.012 to add MISST AMSR-E swath data:

o Several dump reprocessing codes failed in 20110803 00z dumps across new day and table change

▪ Submitted RFCs to make minor changes to W3MISCAN, PREPOBS_PREPSSMI, BUFR_DCODWINDSAT, WAVE_DCODQUIKSCAT, BUFR_SUPERTMI to correct this

• Not sure why they didn’t work right as was, but changes guarantee they will work in future cases such as this

o Langley cloud “lgycld” dump failed in BUFR_DUPSAT for 00z

▪ May be due to fact that these data are compressed

• When tanks are uncompressed, dumps work ok

• [NCO_8] Possible BUFRLIB issue uncovered for embedded tables -?? email sent to Jeff Ator

▪ Dennis has saved off everything related to this case

- Started up monthly OBSPROC roundtable meetings – 9/1/11

- BUFRLIB:

o Gentleman at UCAR seeing ~5 less reports pulled out of *.nr PREPBUFR files via UFBTAB w/ new BUFRLIB vs. previous version

▪ Asked Yangrong, Shelley, Jeff W. to investigate

▪ Jack says it occurs when the BUFR file being read is already opened

• Usually UFBTAB is called with the file closed

o Yangrong found 2 prod codes where UFBTAB is called after file is opened godas2_v1_getBufrJsn and godas_v1_getBufrJsn – [DS_2] these are Diane’s codes, they are losing some obs per Jeff W. - will she submit a RFCs?

▪ Jack has a fix, Jeff A. will update BUFRLIB software on web site as v10.0.1

• Will not get updated in CCS until next BUFRLIB version is implemented

- PREPBUFR:

o Shelley added sfc obs with missing wind direction but valid wind speed ( SOB

▪ She will expand to ALWAYS add SOB even if direction is not missing

• 88/99% of sfc obs with missing direction have speed < 3/5 m/s

▪ METARS with speed < 3 m/s already get this (Stan request for RUC, only done in RUC?)

▪ Calm winds get UOB, VOB = 0

▪ Will also update to ALWAYS add direction ( DDO even if speed is not missing

o Shelley calc. q for sfc reports where pressure is missing

▪ Did sensitivity study comparing diff in q for reported Pstn vs. U.S. Std. Atmos. Pstn from reported Pmsl vs. U.S. Std. Atmos. Pstn from no reported Pmsl

• Diffs are small in 99% of cases so likely do not have to use guess Pstn to re-define q in PREVENTS

• Outliers are due to obvious bad reported Pstn

o In many cases Pstn is rejected as bad but q QM is still 2 – [SM_2] Shelley and Dennis must fix this!

o These bad SYNOP stations should be reported back to the sites – Steve is doing this

▪ They are 75670 (Mexico), 70260 (AK), 71250 (Alberta, Canada), 91212 (South China Sea)

• 71250 was in Alberta and closed in 2000, opened again in 2010 in Newfoundland

• [NCO_9] NCO needs to be proactive not reactive with these type of changes w.r.t. dictionaries

• All SYNOPs in dictionary that have not reported in some time should be added to reject list in case a re-used id opens up again in a new location - [YL_6] Yangrong investigating

• Steve found 193 SYNOPs with improper location in dictionary (70 Canadian SYNOPs have likely moved)

▪ [NCO_10] Brad B.: add to reject list

▪ Coincident METARs in 2 cases are ok

o Need to add UAPART info so that Wan-Shu can immediately identify raobs with sig-levels and turn on raob level expander in GSI – Yangrong

▪ NO – Wan-Shu does not need this – she will use CAT that is already present with the reports

SATELLITE/REMOTE meeting on 8/23/2011 in BLACK:

updates from SATELLITE/REMOTE meeting on 9/6/11 in GREEN:

updates through 9/20/2011 in RED:

RADAR:

- Latest NWS News says 5 WSR-88D sites are now dual-pol

o Double check this on DML – [NCO_11] we want full decoding of these from NCO not just look-alike to current

▪ Full decoder represents 150% incr. over current

- Echo top in mosaic

- Washington State (Cliff Mass) has a L2 radar on a hilltop that returns negative scans – do want to add this to DML?

- Canadian: on hold

o NSSL getting these (merged mosaic, QPE’s)

▪ Canadians ok with their getting them (restricted to U.S. Gov’t)

• OU should not get it based on this restriction, do they - ????

▪ They do not rebuild the volume scan (we need this to use our L2 QC package on these)

o ROC does not get these

- Level 2.5 problems corrected: implemented 8/9/11

o Recognizes stn heights originally in feet and not meters (and now converts feet to meters)

o missing RPIDs

- Level 2: Parallel NAM-GSI now changes reports to above MSL (rather than above ground), will be impl. in upcoming bundle per John D.

SURFACE MARINE – SATELLITE:

- Vera: ENVISAT SSHA from NAVO – orbit change complete, again being tanked - YES!

- Remove: Starting 7/19 and lasting for 3 weeks the 13 meter "B" antenna at the Fairbanks CDA is being moved and will be unavailable - during this time, all POES and JASON-2 passes at Fairbanks will be taken on the 13 meter "A" antenna

o Result is some schedule conflicts between POES and JASON-2

o Certain JASON-2 passes originally scheduled at Fairbanks have been deleted to accommodate this conflict.

o The result will be roughly ten blind passes/week, rather than the usual 3-4, with additional latencies of approximately 2 hours after each blind pass

- MCSST:

o Bert submitted 2 RFC packages for MISST AMSR-E – being tested by PMB

▪ "INGEST-OF-AMSR-E-SWATH-DATA" (into b012/xx034 – dump mnemonic "amsrem")

• Implemented 8/2/11

• Connection to ftp. sometimes slow, Diane believes [DK_11] “wget” would work better than “ftp” …

o Ingest failure 9/9 due to connection timeout

▪ "ADD-AMSR-E-TO-RTGSSTHR-ANALYSIS-AND-SWITCH-FROM-NOAA-18-TO-NOAA-19" – scheduled for 10/11/11

• NAVO wants to turn off NOAA-18 MCSST, this is last NCEP processing to use it - TIN required

• Orig. Dennis’ update to BUFR_TRANPOESSST (to put correct BUFR dcty messages at top of b012/xx011-xx012 tanks) was going in with this, but Bert’s codes fail with this change

o Bert’s impl. will go ahead w/o Dennis’ change

o Later Dennis’ change to TRANPOESSST + update to Bert’s codes will go in – contacted Diane to validate

o NAVO is attempting to create BUFR POES sst files which would replace the binary format files we currently read in

▪ They have our BUFR table, a sample BUFR file and the BUFR translation code

▪ Same BUFR files generated with BUFR table that is non-standard and contains conflicts and doesn’t define parameters correctly – they are trying to add extra info. that was never in our BUFR files and that is where the problems lie

• Guy who is doing work is obviously not very familiar with BUFR!

• Sent them back to the drawing board!

• They want to work with us to set up proper BUFR template

o We likely would want all data in BUFR (more than what we get now)

o Referred them to Bert but Mike P. says NCO should do this

▪ NAVO wants to know what we need – only what we convert to BUFR now or everything in their raw data – this is an EMC question, right?

▪ We will ask for everything to be converted to BUFR, [JW_2] Jeff W. will work with NAVO contact to establish BUFR table containing everything in raw NAVO files

o NAVO wants to turn off NOAA-18

SATELLITE RADIANCES:

- METOP

o IASI outage for 12 hours 8/18/11, instrument anomaly

o IASI 3 hour outage 7/25, reason unknown

o Repeat data from McMurdo 24-h 9/15-16 per NESDIS

▪ Greg sees no evidence of incr. dup’s tossed in dump dup-checks over this time, was hoping to see some more tossed here – but also no incr. in dump counts (good)

- GOES:

o Per Becky C. – NESDIS will repl. GOES-11 with GOES-15 in December – what does that mean for NCEP?

▪ We can use winds right away

▪ We cannot use radiances until 2013 (spin up bias correction coeffs.)

▪ Most file names use W or E for GOES, so no need to RFC J-jobs – a few use sat # , these will have to be RFC’d

• [GK_1] Ask NESDIS to change all files to E and W instead of satellite number - Greg

▪ We would like sample files

- SEVIRI:

o GSI failed in some GBL retrospective runs trying to read dumps

▪ Looks like it is related to dumps Greg generated from /dcomdev tanks – dumps Dennis generated from /dcom tanks have not caused a problem in GSI

• Per Jeff A.: no difference between dev and prod decoder – the mystery deepens!

▪ Seems no failures since July 29 (using Dennis’ dump files)

• Run your message reader on all files since then to be sure, maybe set up cron to run in real time to ensure all is now well

▪ Problem was a result of crons on both machines scp’ing dump file to vapor – sometimes they ran at exactly the same time

• Greg added logic to scp only from ops machine unless it’s job either didn’t run or is running slow, then scp from dev machine

▪ [GK_2] Greg: Ask Russ if these corrupt files on vapor will ever cause a problem in the future (for retrospective runs, etc.)

- GOES eclipse season 8/13/11 – 10/31/11

- CIP exercise: 8/16/11 – no known problems – 10 hour outage

o No Aqua data (AIRS, AMSU-A, AMSR-E, MODIS)

o No JASON-2

o Original CIP test on 8/11 aborted after 2.5 hours due to connectivity issues at Wallops

- SSM/IS outage – 7 hours on 7/16-17, degraded data

- Collocated AIRS/MODIS:

o NESDIS is making real-time thinned files (started 7/21?)

▪ John wonders if their delivery is delayed vs. AIRS – [GK_3] Greg please check

▪ Greg checked and they look good, they are being processed faster than full res. files before

- NPP:

o Yi Song completed draft of BUFR table for OMPS Nadir profile, asking NCO (Jeff A.) to review

▪ She has supplied comments on Jeff A.’s review of the SST and AOT BUFR tables

- ATOVS:

o NOAA-18 outage 7 hours 8/2/11, ground equipment problems

o NOAA-18 reduced gyro test 27-h 9/20-21

▪ Slight degradation in some products

SATELLITE WINDS:

- Mixed satellite winds – issue with how to define satellites in BUFR

o Per Jeff A. can’t use Dennis’ idea of replicated SAID because it is a class 1 (coordinate) descriptor and any new instance of it wipes out the previous occurrence of it w/i the overall sequence

- AVHRR POES IR:

o Jim Jung says we can get winds directly from UW/CIMSS which would not have gaps from 0-7z each day like the NESDIS feed has - [GK_4] Greg check into this, lets try and get these

- Korean COMS – winds on DML, NESDIS may start making winds - ???, imager radiances – John Paquette: do [JD_1] we want this??? Derber-??

o COMS-1 launched in 2010

- METEOSAT outages

o 6 hours 9/9, reason unknown

o 4 hours 9/14, scheduled maintenance on server

- JMA

o MTSAT-1R replaced MTSAT-2 from 8/3/11 to 8/16/11 due to ground system failures

▪ MTSAT-1R was set to “look like” MTSAT-2

- GOES EE/AVHRR dumps that Dennis generates in cron

o Gap in files on Global dump archive 12/01/10 through 08/04/11 due to bug in archiving software

▪ Cannot regenerate w/o raw files from NESDIS as tanks are not archived

▪ Su not yet using and will not make retrospective runs when she starts looking at these this fall

▪ Unless objections will not pursue re-generation (NESDIS likely can’t provide raw files anyway)

▪ It is the user’s job to archive data such as this (e.g., user=GSI testers)

SATELLITE RETRIEVALS:

- Dennis’ RFC to stop attempting to ingest ERS data (scatterometer, RA, SAR) impl. 7/26

o Dennis needs to [DK_12] submit another RFC to remove bufr_traners2 and foreign BUFR table – both no longer needed

- ASCAT outage 3 hours 7/23-24 – reason unknown

- GPS-RO

o Providers will switch from Edition 3 to Edition 4 BUFR later in 2011

▪ This should not affect our decoding of the data – never-the-less, notified NCO so they are in the loop

o 12-hour outage 28 August, file configuration problem

- WindSat:

o New version: better science, less latency

▪ Should are now appearing in real time soon

• Greg is tanking (low resolution – we can later go to high res – Day 2)

o [GK_5] Greg sees some diffs vs. current version & has pinged Vince Tabor. Greg will make sure all is well with these.

• [GK_5] Next, Greg and Dennis will dump and reprocess (need to make sure QC is working correctly based on flags in new version)

• New BUFR files will be put on DDS by NCEP when we switch over in prod, MetOffice will then get just as they get current files

▪ Dennis updated Greg’s code to handle either old or new format

▪ FNMOC/NESDIS anxious to transition, but the current files will continue as long as FNMOC makes them, they know we are not ready

o 144 hour outage 7/28-8/2, network upgrade at Monterey & one of the switches was not put back in place. 

- TRMM:

o Switch to new machine upstream of NCEP on 7/18 doesn’t not appear to have affected us

- Langley GOES 1x1 cloud

o Outage (7 hours) on 7/13 – Langley disk filled up

▪ Another starting 12Z 7/19, not sure for how long or why – it was an upstream issue

o LaRC putting raw files on new server “integrity” as of 8/19 – NCO/DF submitted crisis RFC to account for this

▪ This server is open to me

▪ Will ask DF to continue to copy raw files to /dcom in case server access closes at some point in the future

• [DK_13] Dennis needs to RFC housekeeping logic for these files – they are growing!

▪ Found out later LaRC is still populating “typhoon”, but it is the backup server

o Outage weekend of 8/26-29

▪ LaRC shut down “integrity” due to Hurricane Irene, “typhoon” was still up but we were not notified (and likely could not have made change anyway due to Critical Wx)

• LaRC will notify us in future if “integrity” is going down

• [DK_14] We need a way to automate switch to “typhoon” if “integrity” is down)

o Wget might work

OZONE/AQ:

- SMOKE/DUST

o [JM_1] McQ: New dust (verification?) dataset for DML (use for CMAQ, HYSPLIT and NGAC) – more details please ( MAYBE NOT AVAIL. YET FROM NESDIS - ???

- ROSES PBL analysis project is ending

- Jeff Mc. inquiring about GOME-2 data from Netherlands

o Per Craig Long, total ozone very noisy

o Contains profiles? Jeff McQ. wants these!! (from all sources)

▪ Larry Flynn says they could provide a backup in event of loss of U.S. satellite ozone profiles

▪ Craig says ozone profiles are not available from either OMI or TES from either NASA or NESDIS

- NOAA-19 SBUV calibration starts August 1 (ramping in over several weeks) (jumps will not be as large as originally thought – John D. saw no problem with it anyway)

- NESDIS added a calibration update “jump” to SBUV on 7/21 – not the usual 30-day ramp up

o John D. says it should not affect us

- OMI outages:

o 24 hours on 7/16-17 – entered scheduled Zoom mode

o 24 hours on 8/18/19 – entered scheduled Zoom mode

- CALIPSO: L 1.5 Real time vertical profiles of aerosol backscatter and extinction coefficients (from LaRC)

o Sarah has completed all info needed for DML (POC’s , etc.)

o [NCO_12] Asked NCO/PMB/DF to subscribe to LaRC – LaRC will push L 1.5 aerosol files to PMB to server of their choice (RZDM)

▪ Becky will have this in place by end of Aug – she is training new hires using this exercise

o [NCO_13] Asked NCO/SIB to begin writing decoder for L 1.5 aerosol files (NetCDF ( BUFR) using sample data set

▪ Jeff McQ may have told them to wait until some final details are ironed out - ???

o L2 is more accurate than L1.5 but there is latency – if we go L2 route could pull data ourselves and not get NCO involved (except in decoding?)

o There is also CALIPSO cloud (layer and/or profile) information (L2) (from LaRC, NCEP POC is Brad F.)

▪ Would that need to go through NCO DF/SIB (native format is HDF or maybe NetCDF - ?)

▪ Separate feed is assumed ([NCO_14] NCO/PMB/DF should subscribe to these L2 cloud data as with L1.5 Calipso aerosol)

▪ [JM_2] There are choices in what cloud products to receive, Sarah has pinged Brad F. et. al. (subset or all cloud info, vertical resolution, e.g.)

o There is also now L1.5 AERONET aerosol data which we could get in real time via FTP push from GSFC (100 sites – surface product – around the world)

▪ Sarah put in request to receive this via ftp push – for verification (also format, product descr., machine it is on, etc.)

• NASA sent her sample data file w/ descr.

• Per NASA: Data are noisy vs. level 2 data

• [NCO_15] Sarah has contacted NCO/DF to provide sever for push (likely RZDM)

o No decoding needed, these are text files and can be copied directly into wtxtbul directory in /dcom by NCO/DF

Work assignments by individual (from this document only):

Dennis:

1. 09/20/11: Check notes from 6/13/11 Dynamo meeting. Contact Dynamo folks as needed. Ping Brad B. to add untested/unknown platforms to reject list. (refer back)

2. 09/20/11: Find out why do t-6 NDAS PREPBUFR files contain fewer drops than GDAS/GFS (valid at same time) for Irene. Likely due to SYNDATA. (refer back)

3. 09/20/11: Ask Stan if he is assimilating mesonets with “x” in character 8 (est. pressure) in either RUC or RR. (refer back)

4. 09/20/11: Add Vaisala Global Lightning Data (GLD) 360 to DML. (refer back)

5. 09/20/11: Add NEON (National Ecological Observatory Network) to DML. (refer back)

6. 09/20/11: When contacting Pat Pauley re: NRL ACQC issues, also ask her if there is any reason to keep old ACQC-based rule that for all aircraft except MDCRS, a QM=14 is set on temp automatically if QM=14 on winds. (refer back)

7. 09/20/11: Ping Dave Helms for more info on changes to BUFR template. (refer back)

8. 09/20/11: Ping Jim W. on status of bird algorithm implementation in WFIP-profilers. (refer back)

9. 09/20/11: Ping Jim W. on status of WindLogics tower and Nacelle data being sent to NCEP. (refer back)

10. 09/20/11: Note dates of incorrect CAP profiler/RASS height assignment in historical documentation web page, add patch to PREPBUFR for historical re-runs. (refer back)

11. 09/6/11: Add option to use “wget” in place of “ftp” for Katz ingest scripts (would allow 2nd try for MISST AMSR-E swath data in event of slow connection to NCDC’s misst server). (refer back)

12. 08/23/11: Submit RFC to remove bufr_traners2 and foreign BUFR table since both no longer needed due to termination of ERS. (refer back)

13. 08/23/11: RFC housekeeping logic to scrub raw LaRC cloud files from /dcom Langley_cloud directory (to be done within Katz ingest scripts). (refer back)

14. 09/20/11: Need a way to automate switch for LaRC cloud data from to their “integrity” server to their “typhoon” if the former goes down. “Wget” might work well here. (refer back)

Jeff W.:

1. 09/20/11: Help Steve append GLERL adjusted reports to PREPBUFR files. (refer back)

2. 09/20/11: Help NAVO convert their POES SST processing to BUFR. Develop NCEP-flavored BUFR table that attempts to convert all raw data into BUFR. (refer back)

Greg:

1. 09/20/11: Ask NESDIS to change all GOES file name qualifiers to “E” and “W” instead of satellite number. (refer back)

2. 09/20/11: Ask Russ if the corrupt sevcsr dump files on vapor will ever cause a problem in the future (for retrospective runs, etc.). (refer back)

3. 09/06/11: Check with NESDIS to see if collocated AIRS/MODIS data are delayed vs. AIRS-only we currently receive. (refer back)

4. 08/23/11: Check to see if we can get AVHRR POES winds directly from UW/CIMSS since these do not have gaps from 0-7z each day like the NESDIS feed has. (refer back)

5. 09/20/11: Ping NESDIS and make sure that new WindSat tanks are correct. Then start dumping and run Dennis’ dump reprocessing on them. (refer back)

Shelley:

1. 09/20/11: Check: 1) did types other than the 3 usual ones come into MADIS and/or AirDAT tanks during Irene; 2) do types other than the 3 usual ones come into AirDAT anks routinely; 3) is MADIS sending us types other than the 3 usual ones routinely, if so how does NCO’s decoder handle them since we have specific tanks for the 3 usual types. (refer back)

2. 09/06/11: Update PREPDATA such that QM is NOT 2 for q’s calculated from pressure with bad QM. (refer back)

Yangrong:

1. 09/20/11: Do we get/assimilate international ARM RAOBs: Niamey, Niger; Blackforest Germany; Shouxian, Anhui, China; and Graciosa Island, Azores, Portugal. (refer back)

2. 09/20/11: Ask Neil Jacobs/AirDAT to send samples of RAOB sites he sees in PREPBUFR file that terminate early due to drift issues. Check our own PREPBUFR files for any evidence of this. (refer back)

3. 09/20/11: Check if Nancy France (07180) has changed from illegal instrument type of 34 to correct instrument type of 70. (refer back)

4. 08/23/11: Run Brad’s PREPBUFR stats on “refresh” buoy in Dennis’ cron sfcshp dumps from /dcomdev. (refer back)

5. 09/20/11: Run Brad’s PREPBUFR stats on METARs with lat/lon/elev updates in dev (using Dennis’ cron adpsfc dumps from /dcomdev). (refer back)

6. 09/20/11: Research all SYNOPs and METARs in NCO dictionaries that have not reported in at least 1 year. Work with NCO to add these to reject list so that will not be used if they ever come in again since they might be in a different location now. Esp. for Canadian sites. Does NCO see a problem with adding so many sites to SDMEDIT reject list? (refer back)

Steve:

1. 09/20/11: GLERL adjusted obs (coastal surface). 2.5 deg land/sea mask for now. Move obs from sea/land to land/sea and adjust temperature, moisture, wind. Later append new (changed) reports to PREPBUFR file. (refer back)

2. 09/20/11: Revisit NOAA/NOS/PORTS mesonets – are these all coming into the mesonet tank b255/xx010 or would we expect to see some in SHEF tanks? (refer back)

3. 09/20/11: Ping MADIS re: Puerto Rican mesonet – esp. re: availability in realtime and reliability. (refer back)

4. 09/20/11: Ping Leon B. to see if MADIS is now getting all NJ Mesonet data. (refer back)

5. 09/20/11: Check RTMA, RUC and NAM mesonet reject list – hopefully provider CRN data other than temperature is not being used (since Td is really RH). (refer back)

6. 09/20/11: Work with NCO to get proper heights installed in METAR dictionary for Gulf FAA stations. First in /dcomdev then if all is well promote to /dcom. (refer back)

John D.

1. 09/20/11: Korean COMS winds will soon be produced by NESDIS, maybe also imager radiances. Do we want these? (refer back)

Geoff D:

1. 09/20/11: Time to think about removing SYNDATA in NAM/NDAS. Things have changed! (refer back)

Vera: - none

Diane:

1. 09/20/11: Check to see if “refresh” buoys are QC’d by quips. (refer back)

2. 09/20/11: RFC fix for godas2_v1_getBufrJsn and godas_v1_getBufrJsn since that are losing obs due to calling UFBTAB with BUFR file open (issue with new BUFRLIB). (refer back)

Shun: - none

Jeff McQ:

1. 09/20/11: Please provide details on new dust (verification?) dataset. Can then be added to DML (use for CMAQ, HYSPLIT and NGAC). Is this available yet from NESDIS? (refer back)

2. 09/20/11: There are choices in what CALIPSO L2 cloud products to receive. Sarah has pinged Brad F. et. al. (subset or all cloud info, vertical resolution, e.g.). (refer back)

Action items for NCO (from this document only):

1. 09/20/11: SIB: Write a decoder for the RECCO planes high-density aircraft obs (HDOBs) that come in an odd format. (refer back)

2. 08/23/11: DF/SIB/PMB: Do not promote refresh buoys in “SXPA01 KWNB” to prod until new EDTBFR in implemented and this bulletin header can be added to SDMEDIT reject list. EMC needs to evaluate MET data. (refer back)

3. 09/20/11: SIB: Update mesonet decoder to add visibility and cloud-height data. (refer back)

4. 09/20/11: DF: Get two Antarctic RAOBs (89625 and 89662) into dev decoder (for testing). (refer back)

5. 09/20/11: SIB: Shut down decoding of provider CRN in b255/xx030 tank as it is not giving right output (e.g., RH is stored in Td location). Then write new decoder and put these data in their own new (non-mesonet) tank. (refer back)

6. 09/20/11: DF: Pull Vaisala Global Lightning Data (GLD) 360 to CCS for Stan B. (already pulling for NAWIPS). (refer back)

7. 09/20/11: PMB: Set up new HIGH-restriction classification for Nacelle tower and wind data from ESRL that will soon come in. (refer back)

8. 09/06/11: Why did Langley cloud “lgycld” dump failed in BUFR_DUPSAT for 00z when bufrtab.012 was changed? May be due to fact that these data are compressed. Possible BUFRLIB issue uncovered for embedded tables. (refer back)

9. 09/20/11: Need to be proactive rather than reactive in updating dictionaries to reflect station lat/lon/elev changes (e.g., when Canadian SYNOPs move). (refer back)

10. 09/20/11: PMB: Add 75670 (Mexico), 70260 (AK), 71250 (Alberta, Canada), 91212 to SDMEDIT reject list. (refer back)

11. 09/20/11: SIB: Full decoding of dual-pol L2 RADARs is needed (new decoder). (refer back)

12. 08/23/11: DF: Subscribe to LaRC’s push of CALIPSO: L 1.5 real time vertical profiles of aerosol backscatter and extinction coefficients. They would go to RZDM. (refer back)

13. 08/23/11: SIB: Check with Jeff McQueen, then if given go ahead, begin writing decoder for CALIPSO L 1.5 Real time vertical profiles of aerosol backscatter and extinction coefficients (NetCDF ( BUFR). Sample data sets are now available. (refer back)

14. 09/20/11: DF: Subscribe to LaRC’s push of CALIPSO: L 2 cloud (layer and/or profile) information. They would go to RZDM. (refer back)

15. 09/20/11: DM: Subscribe to NASA/GSFC’s real time push of L 1.5 AERONET aerosol data. These would go to RZDM. No decoding needed, these are text files and DF can copy them directly into wtxtbul directory in /dcom. (refer back)

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download