记录一下 下载EBAS-Thredds数据的流程, 以全站点的ozone为例
1. 下载ebas-thredds-harvest的脚本
https://git.nilu.no/ebas/ebas-thredds-harvest
2. 获取所有的文件名列表 => EBAS.filelist.txt
从这里复制过来, 并掐头去尾, 保持每一行格式固定,都是文件名
3. 获取含ozone的文件名列表 => EBAS.filelist.ozone.txt
# in wsl
grep ozone EBAS.filelist.txt > EBAS.filelist.ozone.txt
4. 获取站点列表 => EBAS.sites.txt
# in wsl
awk -F. '{print $1}' EBAS.filelist.ozone.txt | sort | uniq > EBAS.sites.txt
5. python配置必要环境
netCDF4包, thredds包等等, 按需来,
6. 执行脚本获取所有站点对应的urls => urls.txt
执行下面的powershell脚本, 循环调用python脚本, 获取http下载链接, 并保存至urls.txt
这一步比较慢
# get_ozone_urls.ps1
$sites = "AM0001R", "AR0001R", "AR0002G", "AR0004R", "AR0005R", "AT0002R", "AT0003R", "AT0004R", "AT0005R", "AT0030R", "AT0032R", "AT0033R", "AT0034G", "AT0037R", "AT0038R", "AT0040R", "AT0041R", "AT0042R", "AT0043R", "AT0044R", "AT0045R", "AT0046R", "AT0047R", "AT0048R", "AT0049R", "AT0050R", "AU0002G", "BB0001R", "BE0001R", "BE0032R", "BE0035R", "BG0053R", "BM0001R", "BO0001R", "CA0002R", "CA0003R", "CA0006R", "CA0010R", "CA0011R", "CA0012R", "CA0023R", "CA0103R", "CA0420G", "CH0001G", "CH0002R", "CH0003R", "CH0004R", "CH0005R", "CH0010U", "CH0031R", "CH0053R", "CL0001R", "CV0001G", "CY0002R", "CZ0001R", "CZ0003R", "CZ0005R", "DE0001R", "DE0002R", "DE0003R", "DE0004R", "DE0005R", "DE0006R", "DE0007R", "DE0008R", "DE0009R", "DE0011R", "DE0012R", "DE0013R", "DE0014R", "DE0017R", "DE0018R", "DE0026R", "DE0031R", "DE0035R", "DE0038R", "DE0039R", "DE0042R", "DE0043G", "DE0045R", "DE0046R", "DE0047R", "DE0054R", "DE0057G", "DE0060G", "DK0005R", "DK0009R", "DK0010G", "DK0012R", "DK0025G", "DK0031R", "DK0032R", "DK0041R", "EE0009R", "EE0011R", "EG0001U", "EG0002U", "EG0004R", "ES0001R", "ES0002R", "ES0003R", "ES0004R", "ES0005R", "ES0006R", "ES0007R", "ES0008R", "ES0009R", "ES0010R", "ES0011R", "ES0012R", "ES0013R", "ES0014R", "ES0015R", "ES0016R", "ES0017R", "ES0018G", "FI0004R", "FI0009R", "FI0017R", "FI0018R", "FI0022R", "FI0037R", "FI0096G", "FR0008R", "FR0009R", "FR0010R", "FR0011R", "FR0012R", "FR0013R", "FR0014R", "FR0015R", "FR0016R", "FR0017R", "FR0018R", "FR0019R", "FR0020R", "FR0023R", "FR0024R", "FR0025R", "FR0026R", "FR0028R", "FR0030R", "FR0031R", "FR0032R", "FR0034G", "GB0002R", "GB0006R", "GB0013R", "GB0014R", "GB0015R", "GB0031R", "GB0033R", "GB0035R", "GB0036R", "GB0037R", "GB0038R", "GB0039R", "GB0041R", "GB0043R", "GB0044R", "GB0045R", "GB0048R", "GB0049R", "GB0050R", "GB0051R", "GB0052R", "GB0053R", "GB0059G", "GB1055R", "GR0001R", "GR0002R", "GR0003R", "GR0101R", "HU0002R", "HU0003R", "HU0017R", "ID1013R", "IE0001R", "IE0031R", "IS0091R", "IT0001R", "IT0004R", "IT0009R", "IT0014R", "IT0015U", "IT0016R", "IT0018R", "IT0019R", "JP0002G", "JP0003U", "JP1020R", "JP1028G", "JP1029R", "KE0001G", "KG0002U", "KR0100R", "KR0101R", "LT0015R", "LV0010R", "LV0016R", "LV2000U", "MK0007R", "MT0001R", "NL0007R", "NL0009R", "NL0010R", "NL0011R", "NL0091R", "NL0644R", "NO0001R", "NO0002R", "NO0015R", "NO0030R", "NO0039R", "NO0041R", "NO0042G", "NO0043R", "NO0044R", "NO0045R", "NO0047R", "NO0048R", "NO0052R", "NO0055R", "NO0056R", "NO0058G", "NO0059G", "NO0488R", "NO0489R", "NO0492R", "NO0762R", "NO0977R", "NO1007R", "NO1011R", "NO1083R", "NO1200R", "NP0001G", "NZ0001R", "NZ0002R", "NZ0003G", "PE0001R", "PL0002R", "PL0003R", "PL0004R", "PL0005R", "PT0004R", "RO0003R", "RO0008R", "RS0005R", "RU0001R", "RU0013R", "RU0016R", "RU0018R", "RU0100R", "SE0002R", "SE0003R", "SE0005R", "SE0011R", "SE0012R", "SE0013R", "SE0014R", "SE0018R", "SE0019R", "SE0020R", "SE0022R", "SE0032R", "SE0033R", "SE0034R", "SE0035R", "SE0039R", "SE0094R", "SI0008R", "SI0031R", "SI0032R", "SI0033R", "SK0002R", "SK0004R", "SK0005R", "SK0006R", "SK0007R", "US0008R", "US0039G", "US0142R", "US0204R", "US0602R", "US0901R", "US1200R", "US6001R", "US6004G", "US6005G", "US6006R", "VN0001R", "ZA0001G"
foreach ($s in $sites){
echo $s
python ebas-thredds-harvest/get_tds_data.py -s "ozone" -c "$s" -p "http" >> urls.txt
}
7. 下载这些链接
wget, curl之类的去下载urls.txt里的http链接即可
后续的处理逻辑记在EBAS观测数据处理
