The DERM Data Drill
DERM HISTORICAL METEOROLOGICAL DATA SURFACES
The Data Drill is a facility for extracting data from an archive of interpolated rainfall and climate surfaces maintained by the Queensland Department of Environmant and Resource Management. These surfaces were constructed by spatially interpolating observational data collected by the Australian Bureau of Meteorology. The Bureau maintains an archive of observational rainfall and climate records which dates back to the mid-late 1800's. Unfortunately, few stations recorded climate data prior to 1957 and much of the available data are not in digital format. For this reason, the interpolated climate surfaces commence in 1957, and the rainfall surfaces commence in 1890.
The number and location of data points used to construct the interpolated surfaces varies in time. The number of stations rer of stations reporting monthly rainfall data are shown in Figure 1, and the number reporting climate data are presented in Figure 2. As stations commence or cease reporting data, the location of available data points varies and a single figure indicating station locations is not appropriate. However the spatial distribution of stations is indicative of the location of stations used to construct the interpolated climate surfaces.
Figure 1. Number of stations reporting rainfall data, as at April 2000.
Figure 2. Number of stations reporting climate data, as at April 2000.
1. Interpolation Procedure
The interpolated surfaces were computed on a regular 0.05 degree grid extending from 10o S to 44o S, and 112o E to 154o E. All surfaces are available on a daily timestep, however monthly rainfall and long term mean surfaces for both rainfall and climate elements are available upon request. In the following sections, we provide details regarding the interpolation of the rainfall and climate variables.
1.1 Climate variables
All climate variables (except mean sea level pressure) were interpolated using a trivariate thin plate smoothing spline (Wahba and Wendelberger, 1990) with latitude, longitude and elevation as independent variables. Elevation was expressed in kilometres to minimise the validated root mean square interpolation error (Hutchinson, 1995). Latitude and longitude were in units of degrees. All surfaces were fitted by minimising the Generalised Cross Validation (GCV) error with the constraint of first order smoothness imposed.
The only exception to the above is mean sea level pressure (MSLP). The conversion from station pressure to MSLP explicitly removes the elevation component and can thus be omitted from the interpolation. Consequently MSLP was interpolated using rpolated using a bivariate spline with latitude and longitude as independent variables.
A two pass interpolation algorithm was used to detect and remove erroneous data. In the first pass, the observational data were interpolated and the residual associated with each data point was computed. If any given residual exceeded a fixed threshold, the corresponding datum was flagged as a possible outlier. The maximum number of data points that could be rejected was caed was capped at 5%. Those data points which were not flagged as outliers were reinterpolated in a second pass, to produce the final surface. The thresholds used for outlier detection are shown in Table 1.
Table 1. Threshold values used for identifying outliers.
Vapour Pressure 3.0 hPa Pressure 3.5 hPa Maximum Temperature 1.4 C Minimum Temperature 1.6 C % E.T. Radiation 16.0 % Evaporation 2.7 mm Relative Humidity 10 % Vapour Pressure Deficit 1.5 hPa
Examples of the interpolated surfaces are presented in the following maps: maximum and minimum temperature, pan evaporation, vapour pressure and radiation.
1.1.1 Pan Evaporation
Interpolated surfaces have been computed using data recorded from class A pans which measure potential evaporation. Observational data prior to 1970 have not been interpolated because various measuring devices were in use before 1970, resulting in inconsistent and unreliable data. For example, sunken tanks were used prior to 1966, and many of the early pans were not fitted with bird exclosures.
1.1.2 Solar Radiation
Solar radiation estimates have been computed been computed using data taken from actual radiation measurements, hours of sunshine duration and estimates of cloud oktas. It was necessary to utilise data from different sources since only about 25 stations measure radiation directly. Approximately 60 stations measure hours of sunshine duration and about 500 climate recording stations estimate cloud oktas.
In order to estimate actual solar radiation from observational estimates of cloud oktas, empirical tables were constructed using data from stations which simultaneously recorded both oktas and actual radiation. These tables were constructed by optimising the correlation between estimated cloud oktas and the measured radiation. However, since the theoretical extra-terrestrial radiation that is available for atmospheric transmission is dependent upon latitude and time of year, all measured radiation data were converted to percent extra-terrestrial (%E-T) radiation. This removes location and time of year effects and allows radiation data for all available dates and stations to be used in the construction of empirical tables. The correlation The correlation (r2) between percent extra-terrestrial radiation and 9 a.m. estimates of cloud oktas was found to be 0.50. This was improved if one used an average of 9 a.m. and 3 p.m. cloud oktas (r2 = 0.60). However, best results were obtained using a two-dimensional table, consisting of both 9 a.m. and 3 p.m. estimates of cloud oktas (r2 = 0.71). A further marginal improvement was obtained using 9 a.m., 12 p.m. and 3 p.m. oktas, however, the number of stations which report all three estimates is unacceptably low. The empirical array used to estimate percent extra-terrestrial radiation from cloud oktas is shown in Table 2. Given the 9 a.m. and 3 p.m. cloud okta estimates, the percent extra-terrestrial radiation can be read directly from the table.
Table 2. Average percent extra-terrestrial radiation as a function of 9 a.m. and 3 p.m. cloud cover.
0 1 2 3 4 5 6 7 8 0 74.0 73.6 72.9 72.4 71.2 71.0 70.1 67.0 57.5 1 72.9 71.9 71.0 70.0 69.0 67.9 66.2 62.3 56.4 2 71.1 69.8 69.1 67.8 66.5 65.5 63.9 58.6 53.4 3 69.3 68.2 67.1 65.7 65.2 63.5 61.6 56.2 49.7 4 68.0 66.9 65.0 64.2 63.3 61.5 59.3 54.3 47.2 5 66.5 65.4 63.6 61.7 61.5 60.3 57.6 52.0 45.3 6 63.9 63.0 60.7 59.8 58.5 57.6 55.1 49.8 42.2 7 59.6 58.1 55.8 54.6 53.4 52.2 49.8 43.6 34.8 8 50.2 49.1 46.9 46.3 43.4 44.5 40.1 33.2 24.5 To estimate solar radiation from hours of sunshine duration, an empirical equation was constructed using data from stations which simultaneously recorded both sunshine duration and actual radiation. A linear relationship was fitted using regression:
%E-T = 4.07 . sunshine hours + 25.50
resulting in a correlation of r2 = 0.85 with the measured (%E-T) radiation.
The three sources of radiation data are merged to preferentially include the highest quality data source. Radiation measurements are the most accurate data and are used if available. Sunshine duration measurements are more accurate than cloud okta estimates and are used if measured radiation is unavailable. If neither measured radiation or sunshine duration is available, cloud okta data are utilised.
Percent extra-terrestrial radiation is interpolated using a three-dimensional smoothing spline with latitude, longitude and elevation as independent variables. Elevation is incorporated as atmospheric transmittance of radiation is affected by cloud and optical air mass, both of which are influenced by elevation. The interpolated surface of %E-T radiation is then converted, pixel by pixel, to a gridded surface of incident radiation (MJm-2). Actual radiation is not interpolated as this quantity is parametrically dependent upon latitude, and thus incorporates the component of spatial variation due to solar angle. This component is explicitly removed in the transformation to %E-T radiation and is expected to result in reduced interpolation error.
An algorithm for estimating radiation using geostationary infrared satellite data have recently been implemented. This procedure is currently undergoing validation testing, but as yet has not been employed in the construction of the climate database.
1.2 Rainfall
Daily rainfall is intrinsically difficult to interpolate due its high variability, short range spatial correlation and the variety of mechanisms that can result in precipitation. However as the accumulation period increases, one can obtain improved interpolation accuracy as the day-to-day variability is overcome by topographic effects which influence long term rainfall patterns. This fact has led to the widespread use of normalisation techniques which attempt to remove the topographic component of rainfall (by subtracting the mean rainfall) and reducing the data variance (by standardising). The normalised variable can then be regarded as an anomaly, representing departures from the mean rainfall pattern due to broad scale synoptic features which can be reliably interpolated.
The distribution of rainfall is positively skewed for timesteps ranging from hourly to monthly. If the observational data are raised to an appropriate power, one can obtain a distribution function that is approximately normal. Maximum likelihood has been used to determine those parameters (power, mean and variance) which define a truncated normal distribution for which it is most likely that the observational data could have arisen from. A truncated distribution is used as small rainfall amounts are unreliably reported, and the computed distribution must be positive semi-definite with respect to rainfall. The truncation level is currently set to 0.7mm.
A maximum likelihood algorithm was used to compute the power, mean and variance required to normalise monthly rainfall data at each station. These parameters were only computed for those stations having at least 40 years of monthly rainfall data. The resulting values were then interpolated using a trivariate smoothing spline. Monthly rainfall data were interpolated as follows. Firstly, the observational data were transformed to a variable which is approximately normal by raising each data value to the power appropriate for the given location. The transformed variable was then normalised using the mean and variance appropriate to that datum's location. The resulting anomalting anomaly was interpolated using Ordinary Kriging with zero nugget and a variable range. The nugget was set to zero to enforce exact interpolation, and under these conditions the sill can be set arbitrarily. The range was computed locally and set to (1.5 times) the average distance to the neighbouring data points. Those data points which were within a 75 km radius of the target location were included in the interpolation, but this radius may have been increased to ensure at least 25 data points were utilised. After the transformed variable was interpolated, the normalisation and transformation were reversed to yield interpolated monthly rainfall.
Interpolated daily rainfall surfaces were derived from monthly surfaces by partitioning the interpolated monthly rainfall on to individual days. At each grid cell, the distribution of rainfall throughout the month was computed by interpolating the daily rainfall data directly. The monthly rainfall at each grid cell was then partitioned on to individual days according to the computed daily distribution of rainfall. The main advantage of this technique, as compared to interpolating the daily data directly, is (1) the magnitude (as opposed to the day-to-day distribution) of the interpolated estimates have been computed using monthly data, which are of higher quality than daily data, and (2) accumulated daily rainfall values could be utilised as they could be incorporated into the monthly total. If daily data were being interpolated directly, the accumated values could not have been used. (Naturally these values could not be used in the daily interpolations used to determine the daily distribution. However the interpolated daily values were only used for partitioning the interpolated monthly value, and were not used for computing the actual magnitude of the daily rainfall.)
With the exception of those days in the current month, all daily rainfall surfaces have been derived from monthly data using the algorithm described above. Daily rainfall surfaces for days within the current month are generated by Kriging the available daily data. These surfaces are continually reinterpolated throughout the month as the near real-time datasets are updated with additional and error-checked data. At the end of the month, or typically a few days thereafter, the accumulated monthly rainfall becomes available. The monthly rainfall is then spatially interpolated and used to derive daily rainfall surfaces which supersede those surfaces computed using the daily data.
1.3 Error Analysis
A comprehensive analysis of the accuracy of the interpolated surfaces has been undertaken on a temporal and spatial basis. These results, and a detailed discussion of the psychrometric equations used for computing climate variables such as vapour pressure, mean sea level pressure, relative humidity etc. are described in Jeffrey et al., 2001.
References
1. Jeffrey, S.J., Carter, J.O., Moodie, K.M and Beswick, A.R.. "Using spatial interpolation to construct a comprehensive archive of Australian climate data", Environmental Modelling and Software, Vol 16/4, pp 309-330, 2001.
2. Carter et al. (1996) Development of a National Drought Alert Strategic Information System: Vol III, "Development of data rasters for model inputs." Final Report on QPI 20 to Land and Water Resources Research and Development Corporation. 76 pp.
3. Hutchinson, M.F. (1995) "Interpolating mean rainfall using thin plate smoothing splines", International Journal of Geographical Information Systems, 9:385-403.4. Wahba, G. and Wendelberger, J. (1980) "Some new mathematical methods for variational objective analysis using splines and cross validation", Monthly Weather Review, 108:1122-1143.
Send mail to author with questions or comments about this web site.
© 1998 SILO. All rights reserved. Legal Notices.