This is the original submitted sequence file. This is a sequence file in either fasta or fastq format. It may have been edited to change all end-of-line characters into UNIX format.

started Sun, 15 Sep 2019 06:46:57 GMT - completed Sun, 15 Sep 2019 06:50:36 GMT

Compute statistics for the sequence, determine coverage information and preserve it for later stages.

The script executed at this step is available here. It uses the following software:

drisee -v -t <format> -f <input>

jellyfish count -C -m <6|15> -c 12 -s 1G <input>

started Sun, 15 Sep 2019 06:47:27 GMT - completed Sun, 15 Sep 2019 06:50:14 GMT

Detection and removal of adapter sequences using a bit-masked k-difference matching algorithm

The script executed at this step is available here. It uses the following software:

skewer -x <adaptorfile> -k 5 -l 0 --quiet -t 4 -r .2 -m any <input> <outname>

started Sun, 15 Sep 2019 06:50:15 GMT - completed Sun, 15 Sep 2019 06:53:47 GMT

Depending on the options chosen, the preprocessing step filters sequences based on length, number of ambiguous bases and quality values if available. The FASTA formatted file 100.preprocess.passed.fna contains the sequences which were accepted and will be passed on to the next stage of the analysis pipeline. The FASTA formatted file 100.preprocess.removed.fna contains the sequences which were rejected and will not be passed on to the next stage of the analysis pipeline.

The script executed at this step is available here. It uses the following software:

fastq-mcf 'n/a' <infile> -S -l 50 -k 0 --max-ns <max_lqb> -q <min_qual> -w 10 -o <outfile>

started Sun, 15 Sep 2019 06:53:48 GMT - completed Sun, 15 Sep 2019 06:56:57 GMT

PCR artifacts require removal, sequences are artificially duplicated during the preparation for sequencing (see http://www.nature.com/ismej/journal/v3/n11/full/ismej200972a.html) for metagenomes and metatranscriptomes, the technique cannot be used for amplicon reads. The optional dereplication step removes redundant 'technical replicate' sequences from the metagenomic sample. Technical replicates are identified by binning reads with identical first 50 base-pairs. One copy of each 50-base-pair identical bin is retained. The FASTA formatted file 150.dereplication.passed.fna contains the sequences which were retained and will be passed on to the next stage of the analysis pipeline. The FASTA formatted file 150.dereplication.removed.fna contains the sequences which were rejected and will not be passed on to the next stage of the analysis pipeline.

The script executed at this step is available here.

started Sun, 15 Sep 2019 06:56:58 GMT - completed Sun, 15 Sep 2019 06:58:36 GMT

The FASTA formatted file 299.screen.passed.fna contains the sequences which were retained and will be passed on to the next stage of the analysis pipeline.

The script executed at this step is available here. It uses the following software:

bowtie2 -f --reorder --un <output> -x <index> -U <input>

started Sun, 15 Sep 2019 06:54:19 GMT - completed Sun, 15 Sep 2019 06:58:00 GMT

We search all sequences for potentially rRNA genes with a cut-off of 70% identity to ribosomal sequences from a reduced version of M5RNA. The FASTA formatted file 425.search.rna.fna contains the predicted ribosomal sequences.

The script executed at this step is available here. It uses the following software:

sortmerna -e 0.1 --blast '1 cigar qcov qstrand' --reads <input> --ref <m5rna_reduced>

started Sun, 15 Sep 2019 06:58:32 GMT - completed Sun, 15 Sep 2019 06:58:41 GMT

Sequences are clustered at 97% identity. Since 97% identity for ribosomal RNA genes is considered to be the same species, we cluster sequences from the same species together. Following the search, the original reads are loaded into MG-RAST for retrieval on-demand. The FASTA formatted file 440.cluster.rna97.fna contains sequence clusters that have at least 70% identity to ribosomal sequences and have sequences within 97% identity. The tab-delimited file 440.cluster.rna97.mapping identifies the sequence clusters and the sequences which describe them, each line describes a single cluster.

The script executed at this step is available here. It uses the following software:

cd-hit-est -n 9 -d 0 -c 0.97 -i <input>

started Sun, 15 Sep 2019 06:59:33 GMT - completed Sun, 15 Sep 2019 07:00:37 GMT

The similarity output is the file 450.rna.sims in BLAST m8 format. This includes the identifier for the query which is either the FASTA id or the cluster ID, and the internal identifier (md5sum) for the sequence that it hits.

The script executed at this step is available here. It uses the following software:

blat -out=blast8 -t=dna -q=dna -fastMap <m5rna> <input>

started Sun, 15 Sep 2019 06:59:33 GMT - completed Sun, 15 Sep 2019 07:05:25 GMT

Coding regions within the sequences are predicted using FragGeneScan, an ab-initio prokaryotic gene calling algorithm. Using a hidden Markov model for coding regions and non-coding regions, this step identifies the most likely reading frame and translates nucleotide sequences into amino acids sequences. The predicted genes, possibly more than one per fragment, are called features. The nucleotide sequence FASTA formatted file 350.genecalling.coding.fna contains the predicted coding regions.

The script executed at this step is available here. It uses the following software:

run_FragGeneScan.pl -complete 0 -train <type> -genome <input>

started Sun, 15 Sep 2019 07:05:26 GMT - completed Sun, 15 Sep 2019 07:06:45 GMT

Potentially protein coding features are masked if they overlap with a ribosomal RNA feature.

The script executed at this step is available here.

started Sun, 15 Sep 2019 07:07:16 GMT - completed Sun, 15 Sep 2019 07:07:58 GMT

Predicted protein coding sequences are clustered at 90% identity. We reduce the amount of sequences that are being searched in the similarity search step, reducing the computational cost. Following the search, the original reads are loaded into MG-RAST for retrieval on-demand. The tab-delimited file 550.cluster.aa90.mapping identifies the sequence clusters and the sequences which describe them, each line describes a single cluster. The amino acid sequence FASTA formatted file 550.cluster.aa90.faa contains the translations of one sequence from each cluster (by cluster ids starting with aa90_) and all the unclustered (singleton) sequences with the original sequence ID.

The script executed at this step is available here. It uses the following software:

cd-hit -n 5 -d 0 -c 0.90 -i <input>

started Sun, 15 Sep 2019 07:08:29 GMT - completed Sun, 15 Sep 2019 09:33:24 GMT

The similarity output from BLAT against the M5NR protein database is the file 650.superblat.sims in BLAST m8 format. This includes the identifier for the query which is either the FASTA id or the cluster ID, and the internal identifier (md5sum) for the sequence that it hits.

The script executed at this step is available here. It uses the following software:

blat -prot -fastMap -out=blast8 <m5nr> <input>

started Sun, 15 Sep 2019 09:33:25 GMT - completed Sun, 15 Sep 2019 09:37:23 GMT

Expand protein similarities into various technical namespaces.

The script executed at this step is available here.

started Sun, 15 Sep 2019 07:00:37 GMT - completed Sun, 15 Sep 2019 07:02:49 GMT

Expand rRNA similarities into various technical namespaces.

The script executed at this step is available here.

started Sun, 15 Sep 2019 09:37:24 GMT - completed Sun, 15 Sep 2019 09:38:02 GMT

Map the cluster annotations back for rRNA and protein annotations to the original sequences and create an index for fast access to individual sequences and similarities.

The script executed at this step is available here.

Expand similarities into various technical namespaces and index them for fast access.

The script executed at this step is available here.

started Sun, 15 Sep 2019 09:38:02 GMT - completed Sun, 15 Sep 2019 09:41:00 GMT

Compute the observed abundance per feature (M5NR hit) from the expanded similarities. We pivot the data structure from one line per similarity to one line per M5NR hit.

The script executed at this step is available here.

started Sun, 15 Sep 2019 09:37:24 GMT - completed Sun, 15 Sep 2019 09:37:58 GMT

Compute the observed abundance per LCA from the expanded similarities. We pivot the data structure from one line per similarity to one line per lowest common ancestor.

The script executed at this step is available here.

started Sun, 15 Sep 2019 09:37:24 GMT - completed Sun, 15 Sep 2019 09:37:40 GMT

Compute the observed abundance per data source from the expanded similarities. We pivot the data structure from one line per similarity to one line entry of each data source e.g. RefSeq, Subsystems, …

The script executed at this step is available here.

started Sun, 15 Sep 2019 09:33:25 GMT - completed Sun, 15 Sep 2019 09:34:33 GMT

Create fasta file of darkmatter, features that are predicted by FragGeneScan but have no similarity hit with the M5NR.

The script executed at this step is available here.

started Mon, 16 Sep 2019 16:13:04 GMT - completed Tue, 17 Sep 2019 00:20:00 GMT

Load the computed profile data into the Cassandra database.

The script executed at this step is available here.

Compute the abundace profiles from the expanded similarities. We pivot the data structure from one line per similarity to one line per M5NR hit. Load the computed profile data into the database.

The script executed at this step is available here.

started Tue, 17 Sep 2019 00:20:00 GMT - completed Tue, 17 Sep 2019 00:37:57 GMT

Finalize the job and compute summary statistics for the overview page.

The script executed at this step is available here.

started Tue, 17 Sep 2019 00:37:57 GMT - completed Tue, 17 Sep 2019 00:38:02 GMT

Send email to the user.

The script executed at this step is available here.