Version 5.2.2 of SCC

A new SCC version (v5.2.2) has been released (04/05/2021) with the following changes:

• NPW downloaded from the new Cloudnet data portal (the one used in the previous SCC version is going to be switched off).
• Possibility to force a cloud-free scenario in case strong aerosol layers are miss-classified as clouds by the automatic procedure. Please note that the automatic cloud screening procedure should be always considered the standard and preferred one. Only in case it fails and you are sure there are actually no clouds, the cloud-free option should be used. Moreover, for traceability reasons, all the SCC products calculated using the cloud-free option will be labelled as manual cloud-screened. More information on how to use this option can be found in the official SCC documentation (https://docs.scc.imaa.cnr.it/en/latest/faq/faq.html#force-cloud-free-conditions).

Please report problems or suggestions on the SCC forum available at: https://earlinetforum.imaa.cnr.it/viewforum.php?f=21

Version 5.2.1 of SCC

After the release of the SCC version 5.2, several feedback have been collected from you reporting some issues. So, a new SCC version (v5.2.1) has been released (10/03/2021) in which the following issues have been addressed:

• proper handling of the height array in GDAS model data we get from Cloudnet (in Cloudnet model data the 'height' variable is defined as height above the ground but actually the values provided are above see level). This issue affected only products calculated by using GDAS model (for all the other models the height was reported correctly in the
• corresponding Cloudnet datafiles).
• proper handling of height in sounding file in case sounding station and lidar station altitudes are not the same. This bug affects only products calculated with SCC v5.2 and only those ones for which there is a difference between sounding station and lidar station altitudes.
• proper handing of incomplete daily products delivered by Cloudnet. In same cases it could happen that the Cloudnet API delivers not complete daily model data (model data not covering the whole day but only the first few hours). This shouldn't happen and Cloudnet people are working to fix this issue. In the meanwhile a new control has been added in the SCC to not ingest these incomplete data. This issue was responsible for the ELPP return code 254: "General error (most probably segmentation fault)".
• added new ELDA error codes in order to reduce the occurrences in which the code 22 "unknown runtime exception" is returned
• after having edited a measurement ID using "Admin" section a re-processing should start automatically. Actually the edited measurement ID was queued but never processed. This problem should be fixed now.
• under certain circumstances when a new SCC products is created an error was returned in your browser: "Page unavailable. Sorry, but the requested page is unavailable due to a server error. The administrator has been notified, so check back later."

Beside these fixed issues there is still one pending problem in the SCC v5.2.1 on which we are currently working on. This problem affects the CloudScreen module which sometime seems to misclassify strong dust events as cirrus cloud. Unfortunately this problem cannot be fixed quickly because we need to implement and test new criteria to be able to distinguish between these two cases. You will kept updated on that. In the meanwhile, if you noticed strong dust events which are not processed by the SCC because of this misclassification you could provide your manual cloudmask in the raw input file. Obviously this is not a solution but just a work around to be used until the new version of the CloudScreen will be available. Please, download the documentation on how to submit your manual cloudmask in your raw data file.    

Please report problems or suggestions on the SCC forum available at: https://earlinetforum.imaa.cnr.it/viewforum.php?f=21

Version 5.2 of SCC

The new SCC version (released on 10/02/2021) brings many important updates and enhancements which are summarized in the SCC version 5.2 release notes. 

Major modifications regards:

• filename format
• cloudscreen algorithm
• more advanced handling of processing queue
• locking all the parameters for approved configurations
• possibility to configure a delayed processing (with respect the submission)
• different interpretation of the raw data file variable Molecular_Calc (0 - now means automatic)

Please note that the new CloudScreen module is more on conservative side with respect to the previous one. So in general it is possible that we come up with less products with respect to the past but hopefully more reliable in terms of cloud  screening.

In making the update we tried to not break any existing configuration. So you should be able to log-in with your old username and password and find on the server all the configurations and measurements you have already uploaded before the upgrade.

As a consequence all the products on the servers are still the ones generated by you running the previous SCC version (v5.1). If you like, you can re-calculate those products using the new version of the SCC. Please keep in mind that in this case you are going to overwrite all your products! So if you like to keep your old products you should make a back-up of them before trying to re-run the new SCC version.

Please report problems or suggestions on the SCC forum available at: https://earlinetforum.imaa.cnr.it/viewforum.php?f=21

Version 5.1.1 of SCC

The new SCC version (released on 06/06/2019) brings many important updates and enhancements:

1) SCC product file format

The SCC v5.1 generates products which are:

• CF compliant (CF-1.7).
• optical products (ELDA outputs) are fully compatible with the format required by the new EARLINET database (which will be released quite soon).
• all SCC products are compatible with Panoply (https://www.giss.nasa.gov/tools/panoply/).
  This means that it is possible to plot the contents of any SCC product using Panoply without the need to write custom tools.

2) Quicklook web interface

A new color was adopted to distinguish between cloud-free regions (white regions) from no-data region (grey regions) in cloudmask quicklooks.

Please, keep in mind that the quicklook web interface (https://quicklooks.earlinet.org) will be password protected until the 31 of July 2019. After this date the password will be removed and it will become accessible to anybody.

IMPORTANT: We recommend to take advantage from this period for checking that everything is fine for your station on EARLINET quicklook web interface (https://quicklooks.earlinet.org).
In particular, you should check that:

• no testing or calibration lidar systems show up there by enabling for quicklook (on https://scc.imaa.cnr.it) only trusted lidar/configurations
• the name of the lidar systems that show up are somehow meaningful (avoiding for example names like "test system" or "my lidar" and so on)
• the quicklooks images are ok

3) Other enhancements

• All the stations are now identified by 3 digits code according to the list provided by Lucia. This is going to be the same code that identifies the stations of the new EARLINET database.
• It is now possible to use 15 digits to define the  measurement ID (the previous SCC version allowed to use only 12 digits). This has been done to allow the usage of the following format for the measurement ID (which we recommend!):

HHHHMMDDssshhmm

where 'sss' is the new 3 digits station code and HHHHMMDD hhmm is the start of the measurement.

• the stability of all the modules has been improved

In making the update we tried to not break any existing configuration.
So you should be able to log-in with your old username and password and find on the server all the configurations and measurements you have already uploaded before the upgrade.

As a consequence all the products on the servers are still the ones generated by you running the previous SCC version (v5.0). If you like you can re-calculate those products using the new version of the SCC (actually we recommend to do so). Please, keep in mind that in this case you are going to overwrite all your products! So, if you like to keep your old products you should make a back-up of them before trying to re-run the new SCC version.

Please, always remember that in order to generate the new products (HiRELPP, Cloumask, ELIC and ELQUICK), it is necessary to define for the configuration(s) an high resolution product, following some basic instructions on how to do that.
Please report problems or suggestions on the SCC forum available at: https://earlinetforum.imaa.cnr.it/viewforum.php?f=21

Version 5.0 of SCC

The new SCC version (released on 08/11/2018) brings many important updates and enhancements:

1) New SCC workflow

The SCC workflow has been modified by including 4 new calculus modules: HiRELPP, Cloudmask, ELIC and ELQUICK

• 1.a) HiRELPP module ingests raw input data and generate high resolution RCS timeseries (un-calibrated)
• 1.b) Cloudmask module ingests the output of HiRELPP module and generates cloudmask as output
• 1.c) ELPP module uses the generated cloudmask to skip (or label as cloud) in the time/vertical integration cloud contaminated signals
• 1.d) ELDA module takes into account cloud information in calculating optical products and provides the constants used to calibrate the optical products. Please note that calibration constants are calculated by ELDA only for certain product types (more information: https://earlinetforum.imaa.cnr.it/viewtopic.php?f=30&t=275)
• 1.e) ELIC module calibrates HiRELPP and ELPP products using the constants calculated by ELDA. It produces high and low calibrated high resolution products consisting in total attenuated backscatter timeseries
• 1.f) ELQUICK module generates quicklooks images from the high resolution ELIC products.The generated quicklook images will be available on a separated interface that will be available soon.

2) SCC product file format

All the SCC products are in NetCDF format and are CF compliant. To meet the CF compliance we had to modify the NetCDF structure of all the SCC products with respect to the old ones. This means that your reading software for the SCC products should be changed accordingly.In particular, the optical products generated by the SCC v5.0 are different from the EARLINET files we have used up to now. The new format is the same we have used for the publication of the data on CERA database. For the moment such files CANNOT be uploaded on the EARLINET database because it still requires the old format. Anyway we are going to release the new version of EARLINET database (presumably in January 2019) that will accept the new format only.

IMPORTANT: only the SCC product file format is changed. The raw input file format is still the same. This means no specific modifications are needed in your own converter to generate input file for the new SCC version.

3) Other enhancements

• 3.a) Web interface has been improved in terms of user-friendliness. Now it is possible to define different versions of a lidar system and to each lidar version it is possible to link several lidar configurations.
• 3.b) It is possible to calculate molecular atmosphere using model data (ECMWF, ICON_IGLO, GDAS). Model data are taken from CloudNET. To use this new option you should first check (using the SCC web inteface) that your station the field "Cloudnet Station ID" is set. If it is set the calculation of molecular atmosphere using model data can be activated by setting in the input file: Molecular_Calc=2.I would like to thanks Ewan O'Connor that offered us this very interesting option. The update does not break any existing configuration.All the configurations and measurements already uploaded before the updating should be found on the server. In order to no start the re-analysis of all the submitted measurements with the new SCC version automatically, the HiRELPP flag has been manually set to failed status for all the measurements. As a consequence, all the products on the servers are still the ones previously generated running the previous SCC version (v4.0). It is possible to re-calculate those products using the new version of the SCC (recommended). In this case, all the previous products will be overwritten! In order to keep the own old products a back-up of them is necessary before trying to re-run the new SCC version.

Please, note that in order to generate the new products (HiRELPP, Cloumask, ELIC and ELQUICK), it is necessary to define for the configuration(s) an high resolution product, following some basic instructions on how to do that.
Please report problems or suggestions on the SCC forum available at: https://earlinetforum.imaa.cnr.it/viewforum.php?f=21