如何在国内高速下载SRA
下载公共测序数据，一般是通过在NCBI上搜索到study的Run SRA号，然后使用NCBI提供的prefetch下载数据，但因NCBI国内访问较为缓慢导致下载速度巨慢，因此推荐使用EBI提供的enaBrowserTools+Aspera的方式从ENA下载SRA。该下载方式相对aspera更好，只需要提供accession号即可（如SRR号），特推荐使用该方法下载。
### 软件
- Aspera
High-performace transfer brower plugin，IBM旗下的高速下载插件。
# 下载
wget wget https://download.asperasoft.com/download/sw/connect/3.9.9/ibm-aspera-connect-3.9.9.177872-linux-g2.12-64.tar.gz
# 解压
tar -zxvf ibm-aspera-connect-3.9.9.177872-linux-g2.12-64.tar.gz
# 安装 软件安装路径是用户根目录的 .aspera/
./ibm-aspera-connect-3.9.9.177872-linux-g2.12-64.sh
- enaBrowserTools
enaDataGet/enaGroupGet是enaBrowserTools的一个python脚本，enaBrowserTools是基于python3的与ENA web services接口的套件，可以方便访问ENA。安装方式简单，直接上github下载源代码，然后解压即可。
enaDataGet: all data for a given sequence, assembly, read or analysis accession or WGS set.
enaGroupGet: all data for a particular group(sequence, WGS, assembly, read or analysis) for a given sample or study accession.
# 下载&解压
wget https://github.com/enasequence/enaBrowserTools/archive/v1.6.tar.gz
tar -zxvf v1.6.tar.gz# 配置别名 方便查看脚本位置信息
alias enaDataGet=/disk/share/toolkits/enaBrowserTools/python3-1.6/enaDataGet
alias enaGroupGet=/disk/share/toolkits/enaBrowserTools/python3-1.6/enaGroupGet
配置
- 配置aspera：
# 配置许可(这一步需要账户有root权限，普通用户无法设置，也可以不需要设置)
sudo cp ~/.aspera/connect/etc/aspera-license /usr/local/bin/
# 添加环境变量
echo 'export PATH=~/.aspera/connect/bin:$PATH' >> ~/.bashrc
source ~/.bashrc
# 配置秘钥
mkdir /home/zouhua/.aspera/config/
# 复制到配置目录
cp ~/.aspera/connect/etc/asperaweb_id_dsa.openssh /home/zouhua/.aspera/config/
- 配置enaDataGet
如果使用Aspera下载数据，则需要配置aspera_settings.ini文件
# step1 配置aspera_settings.ini
cd /disk/share/toolkits/enaBrowserTools-1.6 & vi aspera_settings.ini
配置前
[aspera]
ASPERA_BIN = /path/to/ascp
ASPERA_PRIVATE_KEY = /path/to/aspera_dsa.openssh
ASPERA_OPTIONS =
ASPERA_SPEED = 100M
配置后: 1. 指定ascp脚本；2.指定密钥；3.设置下载速度
[aspera]
ASPERA_BIN = /home/zouhua/.aspera/connect/bin/ascp
ASPERA_PRIVATE_KEY = /home/zouhua/.aspera/connect/etc/asperaweb_id_dsa.openssh
ASPERA_OPTIONS =
ASPERA_SPEED = 500M
# step2 配置aspera
export ENA_ASPERA_INIFILE="/disk/share/toolkits/enaBrowserTools-1.6/aspera_settings.ini"
查看帮助文档
- enaDataGet：重要参数 1. -f 指定数据类型；2. -d 指定本地下载目录；3. -a 指定是否使用aspera
usage: enaDataGet [-h] [-f {embl,fasta,submitted,fastq,sra}] [-d DEST] [-w][-m] [-i] [-a] [-as ASPERA_SETTINGS] [-v]accessionDownload data for a given accessionpositional arguments:accession Sequence, coding, assembly, run, experiment oranalysis accession or WGS prefix (LLLLVV) to downloadoptional arguments:-h, --help show this help message and exit-f {embl,fasta,submitted,fastq,sra}, --format {embl,fasta,submitted,fastq,sra}File format required. Format requested must bepermitted for data type selected. sequence, assemblyand wgs accessions: embl(default) and fasta formats.read group: submitted, fastq and sra formats. analysisgroup: submitted only.-d DEST, --dest DEST Destination directory (default is current runningdirectory)-w, --wgs Download WGS set for each assembly if available(default is false)-e, --extract-wgs Extract WGS scaffolds for each assembly if available(default is false)-exp, --expanded Expand CON scaffolds when downloading embl format(default is false)-m, --meta Download read or analysis XML in addition to datafiles (default is false)-i, --index Download CRAM index files with submitted CRAM files,if any (default is false). This flag is ignored forfastq and sra format options.-a, --aspera Use the aspera command line client to download,instead of FTP.-as ASPERA_SETTINGS, --aspera-settings ASPERA_SETTINGSUse the provided settings file, will otherwise checkfor environment variable or default settings filelocation.-v, --version show program's version number and exit############################################################################
usage: enaGroupGet [-h] [-g {sequence,wgs,assembly,read,analysis}][-f {embl,fasta,submitted,fastq,sra}] [-d DEST] [-w] [-m][-i] [-a] [-as ASPERA_SETTINGS] [-t] [-v]accessionDownload data for a given study or sample, or (for sequence and assembly) taxonpositional arguments:accession Study or sample accession or NCBI tax ID to fetch dataforoptional arguments:-h, --help show this help message and exit-g {sequence,wgs,assembly,read,analysis}, --group {sequence,wgs,assembly,read,analysis}Data group to be downloaded for thisstudy/sample/taxon (default is read)-f {embl,fasta,submitted,fastq,sra}, --format {embl,fasta,submitted,fastq,sra}File format required. Format requested must bepermitted for data group selected. sequence, assemblyand wgs groups: embl and fasta formats. read group:submitted, fastq and sra formats. analysis group:submitted only.-d DEST, --dest DEST Destination directory (default is current runningdirectory)-w, --wgs Download WGS set for each assembly if available(default is false)-e, --extract-wgs Extract WGS scaffolds for each assembly if available(default is false)-exp, --expanded Expand CON scaffolds when downloading embl format(default is false)-m, --meta Download read or analysis XML in addition to datafiles (default is false)-i, --index Download CRAM index files with submitted CRAM files,if any (default is false). This flag is ignored forfastq and sra format options.-a, --aspera Use the aspera command line client to download,instead of FTP.-as ASPERA_SETTINGS, --aspera-settings ASPERA_SETTINGSUse the provided settings file, will otherwise checkfor environment variable or default settings filelocation.-t, --subtree Include subordinate taxa (taxon subtree) when queryingwith NCBI tax ID (default is false)-v, --version show program's version number and exit
使用
enaDataGet和enaGroupGet可配置aspera使用，参数为*-a/–aspera*，添加此参数则调用aspera。
- step1 先搜索accession在EBI的api接口；
- step2 本地创建下载日志文件目录 logs；
- step3 使用ascp软件下载accession；
/disk/share/toolkits/enaBrowserTools-1.6/python3/enaDataGet -f sra -a SRR212430 -d ./sra
# /disk/share/toolkits/enaBrowserTools-1.6/python3/enaDataGet -f sra SRR212430 -d ./sra
下载成功
下载失败: 出现session stop即为失败
- 完全下载失败
- 部分下载失败
不使用aspera下载
问题
-
出现session stop信息，解决方案：
-
ASPERA_SPEED 设置在100 - 400 M之间；
-
root权限下开通 udp端口
iptables -I INPUT -p udp --dport 3301 -j ACCEPT iptables -I OUTPUT -p udp --dport 3301 -j ACCEPT
-
参考
- enaBrowserTools
